Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-04 19:28:46 +01:00
Code Issues Packages Projects Releases Wiki Activity

Preserve reviewed chat catalog eval source without production candidates

Browse Source 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed 2026-10-28 20:31:12 -05:00
1 parent 0098da56f4
commit 24e236c0f5
73 files changed
+3466 -364
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/runner-e2e/CATALOG-PRESERVATION.md
+33
View File
@@ -0,0 +1,33 @@
# Preserved chat catalog source
This branch consolidates reviewed eval changes for all **244** frozen scenarios:
242 lazy catalog entries and the two existing standalone chat flows. It preserves
source implementations, controls and evidence contracts. It does not establish
that all scenarios are ready to run or that this composed head has passed a live
campaign. The machine-readable inventory is `chat-catalog-preservation.json`.
The production baseline remains `39e811c9c5226979fce42e052a27cfb28301018a`.
No production, workflow, lockfile or SDK source change is included. The real
new-company, formal-hire and routine-history definitions retain prerequisites
18103, 18105 and 18033 respectively. Those focused fixes are excluded here; their
candidate outcomes remain separate from frozen-baseline qualification.
Composition preserves the per-scenario descriptor union, both independent SDK
privacy and new-company security sections, the final formal-hire clock module,
and the stronger shared Notes provider used by status cards and routine history.
The accepted original-deadline and multipart controls are included, as is the
new-company successor's original phase deadline and remaining-time wizard clamp.
Frozen queries, contexts, outcomes, turn/human counts and overall caps are retained.
Readiness is separate from importability. Declared unsupported fresh contexts,
public capability/profile gaps and external resource/authorization requirements
remain explicit. The preserved plugin configure/disable fixture still has a known
secret-ref schema mismatch: it sends an object where its public string schema
requires a secret UUID. Its spent failures are not repaired, erased or rerun by
this consolidation. Other measured failures retain their actual interpretation.
Every historical result remains attached to its original source revision, full
fingerprint, provider/model/build/database evidence, ordinal, usage, cost, privacy
and cleanup. The integrated source changes broad fingerprint closures; none of
the older qualified passes is transferred to this head. No attempt budget is reset,
and no live launch or publication authority comes from this source preservation.
tests/runner-e2e/SECURITY.md
+40
View File
@@ -588,3 +588,43 @@ evidence secret finding withholds all screenshots for that attempt. The launcher
retains a closed withholding reason and removes public markers while preserving
original private PNG bytes and hashes. Later cleanup labels and empty sanitized
export scans cannot clear this decision; the publisher and gallery enforce it.
### SDK canary compiled fingerprint admission
The source-local `plugins.use-plugin-tool` canary verifies the existing startup
build and private retention before any live fixture can create an agent or run
its hello. The gate binds the authenticated 500 Git inputs and restored lock,
compiler/helper identities, both target certificates and complete 1611-file
compiled manifests. It preserves every privacy scan and raw observation.
The normal case fingerprint is deferred until those genuine compiled bytes are
available in that same real scenario startup. A complete non-null fingerprint
and source closure must be retained privately before fixtures. The later normal
definition must match it. Missing or changed proof fails before fixture/model
work. A failed framework start still consumes that attempt; this gate adds no
retry, diagnostic operation, allowance or grading exception. The other five SDK
cases remain subject to their separate admission holds. Local declaration-only
metadata carries null compiled digests and makes no runtime qualification claim.
Preview-tool fixture discovery uses the installed manifest key for the board
registry filter and verifies the returned database UUID and exact namespaced
`preview` name. Enabled setup must expose exactly one tool. Disable verification
requires the same selected record to be disabled and its actual key-filtered
inventory to be empty; an enabled positive precedes that removal check. Agent
discovery remains under its existing UUID-filtered authorized catalog. This
identity correction does not release or rerun the exhausted SDK canary, change
SDK/compiler/scanner behavior, or authorize the other five cases.
The isolated `onboarding.new-company` native1017 candidate keeps its original
source-company chat and ordinary human company creation. Only its credential
setup runs in the existing standalone private child. It uses the real saved
company's public `agents/new` route, same first-agent setup ID, personal
OpenRouter account grant, exact local environment, native hello and public hire.
No draft/localStorage write, legacy CLI fallback or source-company account
substitution is allowed. Partial created connection IDs are retained privately
for public cleanup before the new company is removed. Native probe usage stays
unknown outside heartbeat accounting, never zero. This candidate includes only
the focused local native-probe prerequisite from core PR18103; all candidate
outcomes are separate from frozen39e qualification. Source acceptance is not
permission to run the unused initial.
tests/runner-e2e/chat-catalog-contract.ts
+14
View File
@@ -26,6 +26,20 @@ export interface ChatCatalogContext {
/** These exact selected OpenCode/local journeys use the original flow clock for
* native observation phases. Other cases/lanes keep their authored phase caps. */
const originalClockExecutionIds = new Set([
"chat-features.runner-opencode.local.companies.export-company",
"chat-features.runner-opencode.local.projects.project-defaults",
"chat-features.runner-opencode.local.questions-and-approvals.ambiguous-yes",
"chat-features.runner-opencode.local.questions-and-approvals.review-plan",
"chat-features.runner-opencode.local.questions-and-approvals.necessary-question",
"chat-features.runner-opencode.local.task-coordination.block-on-design",
"chat-features.runner-opencode.local.gateways-profiles.rotate-token",
"chat-features.runner-opencode.local.gateways-profiles.create-gateway",
"chat-features.runner-opencode.local.workspaces.preview-service",
"chat-features.runner-opencode.local.assistant-connections.consent-read-only",
"chat-features.runner-opencode.local.navigation-preferences.quick-search",
"chat-features.runner-opencode.local.navigation-preferences.edit-profile",
"chat-features.runner-opencode.local.navigation-preferences.broken-link",
"chat-features.runner-opencode.local.projects.share-project",
"chat-features.runner-opencode.local.access.cli-authorization",
"chat-features.runner-opencode.local.access.other-company",
"chat-features.runner-opencode.local.access.private-resource",
tests/runner-e2e/chat-catalog-descriptors.json
+30 -22
View File
@@ -183,9 +183,10 @@
"chat-catalog-domains/internal-definition.ts",
"chat-catalog-domains/internal-support.ts",
"chat-catalog-domains/internal-activity.ts",
"chat-scenarios/activity.yaml"
"chat-scenarios/activity.yaml",
"chat-catalog-domains/recorded-work-claims.ts"
],
"oracleVersion": "paperclip.chat-catalog.internal.v1",
"oracleVersion": "paperclip.chat-catalog.activity-overview.grounded.v2",
"coverageGaps": [],
"expectedRunCount": 1,
"turnTimeoutMs": 120000,
@@ -1400,12 +1401,13 @@
"scenarioId": "cases.related-work",
"sourceFiles": [
"chat-catalog-domains/integrator-case-related.ts",
"chat-catalog-domains/recorded-work-claims.ts",
"chat-catalog-domains/integrator-support.ts",
"chat-catalog-domains/internal-support.ts",
"chat-catalog-domains/internal-definition.ts",
"chat-scenarios/cases.yaml"
],
"oracleVersion": "paperclip.chat-catalog.integrator.v1",
"oracleVersion": "paperclip.chat-catalog.case-related.grounded.v2",
"expectedRunCount": 2,
"expectedHumanMessageCount": 1,
"turnTimeoutMs": 120000,
@@ -3136,6 +3138,7 @@
{
"scenarioId": "gateways-profiles.inspect-gateway-error",
"sourceFiles": [
"chat-catalog-domains/gateway-public-receipt.ts",
"chat-catalog-domains/external-gateway-error.ts",
"chat-catalog-domains/external-provider.ts",
"chat-catalog-domains/external-support.ts",
@@ -3150,7 +3153,7 @@
"../../ui/src/App.tsx",
"chat-scenarios/gateways-profiles.yaml"
],
"oracleVersion": "paperclip.chat-catalog.external.v1",
"oracleVersion": "paperclip.chat-catalog.gateway-public-receipt.v2",
"expectedRunCount": 3,
"expectedHumanMessageCount": 1,
"minimumRunCount": 3,
@@ -3688,11 +3691,13 @@
},
{
"requiredCredentials": [
"OPENAI_API_KEY"
"OPENROUTER_API_KEY"
],
"scenarioId": "onboarding.new-company",
"sourceFiles": [
"chat-catalog-domains/identity-new-company.ts",
"chat-catalog-domains/identity-new-company-native.ts",
"chat-catalog-domains/identity-new-company-native-proof.ts",
"chat-catalog-domains/identity-wizard.ts",
"chat-catalog-domains/identity-private-browser.ts",
"chat-catalog-domains/identity-private-browser-worker.ts",
@@ -3767,6 +3772,7 @@
"scenarioId": "plugins.configure-plugin",
"sourceFiles": [
"chat-catalog-domains/external-plugins.ts",
"chat-catalog-domains/external-plugin-tool-identity.ts",
"chat-catalog-domains/external-plugin-fixture.ts",
"chat-catalog-domains/external-support.ts",
"chat-catalog-domains/internal-support.ts",
@@ -3789,6 +3795,7 @@
"scenarioId": "plugins.disable-plugin",
"sourceFiles": [
"chat-catalog-domains/external-plugins.ts",
"chat-catalog-domains/external-plugin-tool-identity.ts",
"chat-catalog-domains/external-plugin-fixture.ts",
"chat-catalog-domains/external-support.ts",
"chat-catalog-domains/internal-support.ts",
@@ -3833,6 +3840,7 @@
"scenarioId": "plugins.use-plugin-tool",
"sourceFiles": [
"chat-catalog-domains/external-plugin-tool.ts",
"chat-catalog-domains/external-plugin-tool-identity.ts",
"chat-catalog-domains/external-plugin-fixture.ts",
"chat-catalog-domains/external-support.ts",
"chat-catalog-domains/internal-support.ts",
@@ -4169,11 +4177,12 @@
"scenarioId": "questions-and-approvals.formal-hire-revision",
"sourceFiles": [
"chat-catalog-domains/identity-hire-revision.ts",
"chat-catalog-domains/identity-hire-model-choice.ts",
"chat-catalog-domains/identity-support.ts",
"chat-catalog-domains/internal-support.ts",
"chat-scenarios/questions-and-approvals.yaml"
],
"oracleVersion": "paperclip.chat-catalog.identity.v1",
"oracleVersion": "paperclip.chat-catalog.hire-revision.native1017.v2",
"expectedRunCount": 1,
"turnTimeoutMs": 120000,
"attemptTimeoutMs": 300000,
@@ -4631,13 +4640,15 @@
"scenarioId": "routines.read-history",
"sourceFiles": [
"chat-catalog-domains/integrator-routine-history.ts",
"chat-catalog-domains/runtime-native-failure.ts",
"chat-catalog-domains/routine-history-work-failure.ts",
"chat-catalog-domains/external-provider.ts",
"chat-catalog-domains/external-support.ts",
"chat-catalog-domains/internal-definition.ts",
"chat-catalog-domains/internal-support.ts",
"chat-catalog-domains/integrator-support.ts",
"chat-scenarios/routines.yaml"
],
"oracleVersion": "paperclip.chat-catalog.integrator.v1",
"oracleVersion": "paperclip.chat-catalog.integrator.routine-work-failure.v2",
"expectedRunCount": 3,
"expectedHumanMessageCount": 1,
"turnTimeoutMs": 120000,
@@ -4645,12 +4656,12 @@
"runPolicy": {
"kind": "terminal-counts",
"terminalStatusCounts": {
"failed": 1,
"succeeded": 2
"succeeded": 3
}
},
"modulePath": "./chat-catalog-domains/integrator-routine-history.js",
"exportName": "integratorRoutineHistoryCases"
"exportName": "integratorRoutineHistoryCases",
"authentication": "human-session"
},
{
"scenarioId": "routines.run-now",
@@ -4725,9 +4736,10 @@
"sourceFiles": [
"chat-catalog-domains/runtime-support.ts",
"chat-catalog-domains/runtime-runs-adapters.ts",
"chat-scenarios/runs-adapters.yaml"
"chat-scenarios/runs-adapters.yaml",
"chat-catalog-domains/model-option-catalog.ts"
],
"oracleVersion": "paperclip.chat-catalog.runtime.v2",
"oracleVersion": "paperclip.chat-catalog.model-options.scoped-catalog.v3",
"coverageGaps": [],
"expectedRunCount": 1,
"turnTimeoutMs": 120000,
@@ -5057,11 +5069,7 @@
"../../server/src/services/company-skills.ts",
"../../packages/shared/src/validators/skill-policy.ts"
],
"oracleVersion": "paperclip.chat-catalog.integrator.v1",
"coverageGaps": [
"Source-qualified prerequisite limitation: the supported membership permission route currently rejects agent principals, so agents:configure cannot be assumed from CEO/legacy hire authority. Actual grant/configuration HTTP probes must establish the runtime blocker or succeed before this scenario can run.",
"Unsupported effective-use context: reserved setup probes the actual unsupported use action plus every real lifecycle action under a denied policy and the actual still-permitted board assignment, restoring policy/assignment in finally. Source runtime version selection has no use-policy principal/action. These receipts establish a prerequisite limitation, never the required model refusal: effective-use-denial outcome remains unqualifiable until a supported actual use-denial contract exists."
],
"oracleVersion": "paperclip.chat-catalog.skill-enable.standard-config.v2",
"expectedRunCount": 2,
"expectedHumanMessageCount": 1,
"turnTimeoutMs": 120000,
@@ -5267,7 +5275,8 @@
"scenarioId": "status-cards.refresh-card",
"sourceFiles": [
"chat-catalog-domains/integrator-status-cards.ts",
"chat-catalog-domains/runtime-native-failure.ts",
"chat-catalog-domains/status-card-work-failure.ts",
"chat-catalog-domains/external-provider.ts",
"chat-catalog-domains/runtime-support.ts",
"chat-catalog-domains/identity-support.ts",
"chat-catalog-domains/internal-definition.ts",
@@ -5275,7 +5284,7 @@
"chat-catalog-domains/integrator-support.ts",
"chat-scenarios/status-cards.yaml"
],
"oracleVersion": "paperclip.chat-catalog.integrator.v1",
"oracleVersion": "paperclip.chat-catalog.status-card.required-work-failure.v2",
"expectedRunCount": 7,
"expectedHumanMessageCount": 4,
"turnTimeoutMs": 120000,
@@ -5283,8 +5292,7 @@
"runPolicy": {
"kind": "terminal-counts",
"terminalStatusCounts": {
"failed": 1,
"succeeded": 6
"succeeded": 7
}
},
"modulePath": "./chat-catalog-domains/integrator-status-cards.js",
tests/runner-e2e/chat-catalog-domains/external-assistant-consent.ts
+2 -1
View File
@@ -1,3 +1,4 @@
import { chatCatalogOriginalPhaseDeadline } from "../chat-catalog-contract.js";
import { randomBytes, randomUUID } from "node:crypto";
import { expect } from "@playwright/test";
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
@@ -47,7 +48,7 @@ export const assistantConsentExternalCases = [definition("assistant-connections.
authentication: "human-session",
sourceFiles: ["chat-catalog-domains/external-assistant-consent.ts", "chat-catalog-domains/external-assistant-connections.ts", "chat-catalog-domains/identity-auth.ts", "chat-catalog-domains/external-support.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/identity-support.ts", "public-mcp-client.ts", "chat-scenarios/assistant-connections.yaml"],
async act(ctx, state) {
await waitForReply(ctx); const viewer = people.get(ctx); if (!viewer || !ctx.input.privateSecrets) throw new Error("Viewer consent fixture missing");
await waitForReply(ctx, chatCatalogOriginalPhaseDeadline(ctx)); const viewer = people.get(ctx); if (!viewer || !ctx.input.privateSecrets) throw new Error("Viewer consent fixture missing");
const page = await viewer.context.newPage();
try {
async function signedInPerson() { const response = await page.context().request.get(`${ctx.input.api.baseURL}/api/auth/get-session`); if (!response.ok()) throw new Error("Viewer browser session check failed"); return (await response.json() as { user: { id: string } }).user.id; }
tests/runner-e2e/chat-catalog-domains/external-gateway-error.ts
+10 -3
View File
@@ -25,6 +25,7 @@ import { collectRunEvents } from "../run-observations.js";
import { isValidNativePrpEnvelope } from "../native-event-envelope.js";
import { privateScanJson } from "./external-secret-scan.js";
import { observedRedactedPublicReceipt } from "./gateway-public-receipt.js";
const providers = new WeakMap<ChatCatalogContext, NotesProvider>();
type Row = Record<string, unknown>;
const object = (value: unknown): Row =>
@@ -355,7 +356,9 @@ export const gatewayErrorExternalCases = [
minimumRunCount: 3,
expectedHumanMessageCount: 1,
attemptTimeoutMs: 660000,
oracleVersion: "paperclip.chat-catalog.gateway-public-receipt.v2",
sourceFiles: [
"chat-catalog-domains/gateway-public-receipt.ts",
"chat-catalog-domains/external-gateway-error.ts",
"chat-catalog-domains/external-provider.ts",
"chat-catalog-domains/external-support.ts",
@@ -422,10 +425,12 @@ export const gatewayErrorExternalCases = [
),
),
);
let ascendingMorganComments: Entity[] = [];
for (const prefix of ["morgan", "own"]) {
const comments = await ctx.input.api.get<Entity[]>(
`/api/issues/${state[`${prefix}TaskId`]}/comments?order=asc`,
);
if (prefix === "morgan") ascendingMorganComments = comments;
state[`${prefix}ReceiptNativeBound`] = comments.some(
(comment) =>
comment.createdByRunId === state[`${prefix}RunId`] &&
@@ -456,7 +461,7 @@ export const gatewayErrorExternalCases = [
}
});
const reports = await ctx.input.api.get<Entity[]>(
`/api/issues/${state.morganTaskId}/comments?order=asc`,
`/api/issues/${state.morganTaskId}/comments`,
),
report = reports.find(
(row) =>
@@ -475,8 +480,10 @@ export const gatewayErrorExternalCases = [
`/api/heartbeat-runs/${run.id}/events?afterSeq=${afterSeq}&limit=1000`,
),
);
if (observedMorganReceiptRead(events, report))
state.canonicalReadMorganReceipt = true;
if (observedMorganReceiptRead(events, report) || observedRedactedPublicReceipt(events, object(report), reports.map(object), {
companyId: o.companyId, leadId: o.leadId, canonicalRunId: run.id, workerId: String(state.morganId),
workerTaskId: String(state.morganTaskId), workerRunId: String(state.morganRunId),
}, [ascendingMorganComments.map(object)])) state.canonicalReadMorganReceipt = true;
}
return { ...o, state };
},
tests/runner-e2e/chat-catalog-domains/external-gateway-rotation.ts
+2 -1
View File
@@ -1,3 +1,4 @@
import { chatCatalogOriginalPhaseDeadline } from "../chat-catalog-contract.js";
import { publicRotationMessages, rotationQuestionHandoff, type RotationPublicMessage, type RotationTarget } from "./external-gateway-handoff.js";
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
import type { ToolMcpGatewayWithTokens, ToolMcpGatewayTokenCreated, ToolProfileWithDetails } from "../../../packages/shared/src/types/tool-access.js";
@@ -67,7 +68,7 @@ export const gatewayRotationExternalCases = [definition("gateways-profiles.rotat
}, gradeGatewayRotation, {
authentication: "human-session", sourceFiles: ["chat-catalog-domains/external-gateway-rotation.ts", "chat-catalog-domains/external-gateway-handoff.ts", "../../ui/src/components/AppsSidebar.tsx", "../../ui/src/components/IssueThreadInteractionCard.tsx", "../../ui/src/pages/apps/gateways/GatewaysList.tsx", "../../ui/src/pages/apps/gateways/GatewayDetail.tsx", "chat-catalog-domains/external-provider.ts", "chat-catalog-domains/external-secret-scan.ts", "chat-catalog-domains/external-support.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/external-private-browser.ts", "chat-catalog-domains/external-private-browser-worker.ts", "run-observations.ts", "chat-scenarios/gateways-profiles.yaml"],
async act(ctx, state) {
await waitForReply(ctx); const fixture = fixtures.get(ctx)!, gateway = state.gateway as ToolMcpGatewayWithTokens;
await waitForReply(ctx, chatCatalogOriginalPhaseDeadline(ctx)); const fixture = fixtures.get(ctx)!, gateway = state.gateway as ToolMcpGatewayWithTokens;
const before = (await ctx.input.api.get<{ gateways: ToolMcpGatewayWithTokens[] }>(`${root(ctx)}/tools/gateways`)).gateways.find(row => row.id === gateway.id)!;
state.noPrematureMutation = before.tokens.length === 2 && before.tokens.every(row => !row.revokedAt);
const observation = await observeExternal(ctx, state);
tests/runner-e2e/chat-catalog-domains/external-gateways-profiles.ts
+2 -1
View File
@@ -1,3 +1,4 @@
import { chatCatalogOriginalPhaseDeadline } from "../chat-catalog-contract.js";
import { expect } from "@playwright/test";
import { definition, externalObservation, observeExternal, root, checks } from "./external-support.js";
import { probeAgentPermission, waitForReply } from "./identity-support.js";
@@ -23,7 +24,7 @@ export const gatewayExternalCases = [definition("gateways-profiles.create-gatewa
}, gradeGatewayCreate, {
sourceFiles: ["chat-catalog-domains/external-gateways-profiles.ts", "chat-catalog-domains/external-support.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/identity-support.ts", "chat-scenarios/gateways-profiles.yaml"],
async act(ctx, state) {
await waitForReply(ctx);
await waitForReply(ctx, chatCatalogOriginalPhaseDeadline(ctx));
const before = await ctx.input.api.get<{ gateways: Entity[] }>(`${root(ctx)}/tools/gateways`); state.gatewayCountBeforeHuman = before.gateways.length;
if (before.gateways.length !== 0) throw new Error("Agent created a management endpoint before the human action");
const source = ctx.input.page.url();
tests/runner-e2e/chat-catalog-domains/external-plugin-tool-identity.test.ts
+78
View File
@@ -0,0 +1,78 @@
import { expect, it, vi } from "vitest";
import type { PluginRecord } from "../../../packages/shared/src/types/plugin.js";
import { readPreviewToolInventory, requirePreviewToolInventory, PreviewToolRemovalPendingError } from "./external-plugin-tool-identity.js";
const mocks = vi.hoisted(() => ({ databaseFactory: vi.fn(() => { throw new Error("No database allowed in tool inventory calibration"); }) }));
vi.mock("../../../server/src/services/plugin-registry.js", () => ({ pluginRegistryService: mocks.databaseFactory }));
vi.mock("../../../server/src/middleware/logger.js", () => { const logger = { info: vi.fn(), debug: vi.fn(), warn: vi.fn(), error: vi.fn(), child: () => logger }; return { logger }; });
import { createPluginToolDispatcher } from "../../../server/src/services/plugin-tool-dispatcher.js";
const id = "11111111-1111-4111-8111-111111111111", otherId = "22222222-2222-4222-8222-222222222222", key = "paperclip.eval-preview-calibration";
const selection = { id, key, count: 1 as const };
const record = (status: "ready" | "disabled" = "ready") => ({ id, pluginKey: key, status });
function fixture(pluginKey = key, pluginId = id) {
const dispatcher = createPluginToolDispatcher();
dispatcher.registerPluginTools(pluginKey, { id: pluginKey, tools: [{ name: "preview", displayName: "Preview", description: "Synthetic preview", parametersSchema: { type: "object", properties: {} } }] } as never, pluginId);
const routes: string[] = [];
const api = { async get<T>(route: string): Promise<T> { routes.push(route); const query = new URL(route, "http://synthetic.invalid").searchParams.get("pluginId"); return dispatcher.listToolsForAgent(query ? { pluginId: query } : undefined) as T; } };
return { dispatcher, api, routes };
}
it("reads the supported board stable-key filter and binds the actual UUID and exact preview namespace", async () => {
const { dispatcher, api, routes } = fixture();
expect(dispatcher.listToolsForAgent({ pluginId: id })).toEqual([]); // The original vacuous UUID lookup.
expect(await readPreviewToolInventory(api, record(), selection)).toEqual([expect.objectContaining({ pluginId: id, name: `${key}:preview` })]);
expect(routes).toEqual([`/api/plugins/tools?pluginId=${key}`]); expect(mocks.databaseFactory).not.toHaveBeenCalled();
});
it("proves actual dispatcher removal, after an enabled positive, before accepting disabled zero", async () => {
const { dispatcher, api } = fixture();
expect(await readPreviewToolInventory(api, record(), selection)).toHaveLength(1);
dispatcher.unregisterPluginTools(key);
expect(await readPreviewToolInventory(api, record("disabled"), { ...selection, count: 0 })).toEqual([]);
expect(mocks.databaseFactory).not.toHaveBeenCalled();
});
it("rejects a disabled label while the real tool remains registered", async () => {
const { api } = fixture();
await expect(readPreviewToolInventory(api, record("disabled"), { ...selection, count: 0 })).rejects.toThrow("inventory");
});
it("only the same exact still-registered tool can be pending within the original disable deadline", () => {
const same = [{ name: `${key}:preview`, pluginId: id }];
expect(() => requirePreviewToolInventory(record("disabled"), { ...selection, count: 0 }, same)).toThrow(PreviewToolRemovalPendingError);
for (const rows of [[{ name: `${key}:preview`, pluginId: otherId }], [{ name: "foreign:preview", pluginId: id }], [...same, ...same]]) {
let observed: unknown;
try { requirePreviewToolInventory(record("disabled"), { ...selection, count: 0 }, rows); } catch (error) { observed = error; }
expect(observed).toBeInstanceOf(Error); expect(observed).not.toBeInstanceOf(PreviewToolRemovalPendingError); expect(String(observed)).toContain("inventory");
}
});
it("rejects absent registration for an enabled plugin", async () => {
const { dispatcher, api } = fixture(); dispatcher.unregisterPluginTools(key);
await expect(readPreviewToolInventory(api, record(), selection)).rejects.toThrow("inventory");
});
it("rejects the correct key namespace with a foreign returned database UUID", async () => {
const { api } = fixture(key, otherId);
await expect(readPreviewToolInventory(api, record(), selection)).rejects.toThrow("inventory");
});
it.each([
{ selected: { ...selection, id: otherId }, plugin: record() },
{ selected: { ...selection, key: "paperclip.foreign" }, plugin: record() },
{ selected: { ...selection, key: id }, plugin: record() },
{ selected: selection, plugin: { ...record(), id: otherId } },
{ selected: selection, plugin: { ...record(), pluginKey: "paperclip.foreign" } },
{ selected: selection, plugin: record("disabled") },
{ selected: { ...selection, count: 0 as const }, plugin: record() },
])("rejects wrong selected identity/lifecycle before an empty query could hide it: %j", async ({ selected, plugin }) => {
const { api, routes } = fixture(); await expect(readPreviewToolInventory(api, plugin, selected)).rejects.toThrow("identity or lifecycle"); expect(routes).toEqual([]);
});
it.each([
undefined,
{ tools: [{ pluginId: id, name: `${key}:preview` }] },
[],
[{ pluginId: id, name: `${key}:other` }],
[{ pluginId: otherId, name: `${key}:preview` }],
[{ name: `${key}:preview` }],
[{ pluginId: id, name: `${key}:preview` }, { pluginId: id, name: `${key}:preview` }],
[{ pluginId: id, name: `${key}:preview` }, { pluginId: otherId, name: "paperclip.foreign:preview" }],
])("does not accept a wrong shape, count or foreign contribution: %j", value => {
expect(() => requirePreviewToolInventory(record(), selection, value)).toThrow("inventory");
});
it("permits zero only for the same installed disabled identity", () => {
expect(requirePreviewToolInventory(record("disabled"), { ...selection, count: 0 }, [])).toEqual([]);
expect(() => requirePreviewToolInventory({ ...record("disabled"), pluginKey: "paperclip.foreign" }, { ...selection, count: 0 }, [])).toThrow("identity or lifecycle");
});
tests/runner-e2e/chat-catalog-domains/external-plugin-tool-identity.ts
+29
View File
@@ -0,0 +1,29 @@
import type { PluginRecord } from "../../../packages/shared/src/types/plugin.js";
export interface PreviewToolDescriptor { name: string; pluginId: string }
export interface PreviewPluginSelection { id: string; key: string; count: 0 | 1 }
interface PublicToolReader { get<T>(route: string): Promise<T> }
export class PreviewToolRemovalPendingError extends Error {
constructor() { super("Actual selected preview tool inventory is still registered during disable"); }
}
function requireSelectedPlugin(plugin: Pick<PluginRecord, "id" | "pluginKey" | "status">, selected: PreviewPluginSelection) {
if (!/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/i.test(selected.id)
|| !/^[a-z0-9][a-z0-9._-]+$/.test(selected.key) || plugin.id !== selected.id || plugin.pluginKey !== selected.key
|| plugin.status !== (selected.count === 1 ? "ready" : "disabled"))
throw new Error("Selected preview plugin identity or lifecycle does not match its actual installed record");
}
/** Board discovery filters by manifest key; descriptors identify the database UUID.
* Bind both instead of treating an empty query under the other identity as removal. */
export function requirePreviewToolInventory(plugin: Pick<PluginRecord, "id" | "pluginKey" | "status">, selected: PreviewPluginSelection, value: unknown): PreviewToolDescriptor[] {
requireSelectedPlugin(plugin, selected);
if (!Array.isArray(value) || value.some(row => !row || typeof row !== "object"
|| row.name !== `${selected.key}:preview` || row.pluginId !== selected.id))
throw new Error("Actual selected preview tool inventory does not match its exact identity and count");
if (selected.count === 0 && value.length === 1) throw new PreviewToolRemovalPendingError();
if (value.length !== selected.count) throw new Error("Actual selected preview tool inventory does not match its exact identity and count");
return value as PreviewToolDescriptor[];
}
export async function readPreviewToolInventory(api: PublicToolReader, plugin: Pick<PluginRecord, "id" | "pluginKey" | "status">, selected: PreviewPluginSelection): Promise<PreviewToolDescriptor[]> {
requireSelectedPlugin(plugin, selected);
const value = await api.get<unknown>(`/api/plugins/tools?pluginId=${encodeURIComponent(plugin.pluginKey)}`);
return requirePreviewToolInventory(plugin, selected, value);
}
tests/runner-e2e/chat-catalog-domains/external-plugin-tool.ts
+4 -3
View File
@@ -1,3 +1,4 @@
import { readPreviewToolInventory } from "./external-plugin-tool-identity.js";
import { createHash } from "node:crypto";
import { mkdir,writeFile,readFile,rename } from "node:fs/promises";
import path from "node:path";
@@ -49,15 +50,15 @@ export const pluginToolExternalCases=[definition("plugins.use-plugin-tool",async
const packageFile=path.join(scaffold.packageRoot,"package.json"),pkg=JSON.parse(await readFile(packageFile,"utf8"));pkg.paperclipPlugin.ui="./dist/ui";await writeFile(packageFile,JSON.stringify(pkg));
await ctx.input.evidence("chat-catalog-vetted-preview-tool-package.json",{synthetic:true,manifestSha256:hash(JSON.stringify(manifest)),workerSha256:hash(worker),uiSha256:hash(ui),artifactSha256:hash(html)});
const installed=await ctx.input.api.post<PluginRecord>("/api/plugins/install",{packageName:scaffold.packageRoot,isLocalPath:true});installedId=installed.id;const plugin=await pollUntil({label:"real contributed preview tool",deadlineAt:Date.now()+45000,load:()=>ctx.input.api.get<PluginRecord>(`/api/plugins/${installed.id}`),accept:row=>row.status==="ready",reject:row=>row.status==="error"?"Preview fixture loader failed; details withheld":undefined});
const tools=await ctx.input.api.get<Array<{name:string;pluginId?:string}>>(`/api/plugins/tools?pluginId=${plugin.id}`);if(tools.length!==1)throw new Error("Setup blocker: actual plugin loader must register exactly one preview tool");
const tools=await readPreviewToolInventory(ctx.input.api,plugin,{id:installed.id,key:scaffold.key,count:1});
const profile=await ctx.input.api.post<{id:string}>(`${root(ctx)}/tools/profiles`,{profileKey:"owned-website-preview",name:"Owned website preview",defaultAction:"deny",entries:[{selectorType:"tool_name",effect:"include",toolName:tools[0]!.name}]});profileId=profile.id;await ctx.input.api.post(`${root(ctx)}/tools/profiles/${profile.id}/bind`,{targetType:"agent",targetId:ctx.input.fixtures.agent.id});
const visible=await readAgentResource(ctx,`/api/plugins/tools?pluginId=${plugin.id}`),toolVisibleToAgent=visible.status===200&&Array.isArray(visible.data)&&visible.data.some((row:{name?:string})=>row.name===tools[0]!.name);if(!toolVisibleToAgent)throw new Error("Setup blocker: installed plugin tool is not actually admitted to the selected agent");
const visible=await readAgentResource(ctx,`/api/plugins/tools?pluginId=${plugin.id}`),toolVisibleToAgent=visible.status===200&&Array.isArray(visible.data)&&visible.data.length===1&&visible.data.some((row:{name?:string;pluginId?:string})=>row.name===tools[0]!.name&&row.pluginId===plugin.id);if(!toolVisibleToAgent)throw new Error("Setup blocker: installed plugin tool is not actually admitted to the selected agent");
await ctx.input.api.post(`${root(ctx)}/chats/${ctx.input.fixtures.agent.id}`);await ctx.refreshIssue();await ctx.input.api.patch(`/api/issues/${ctx.issue()!.id}`,{projectId:project.id,description:`The selected active plugin is Website Preview Eval (${plugin.id}). Its owned synthetic website artifact is website.html and this task belongs to the Synthetic website preview project. Use its admitted preview contribution to show the website.`});
return {pluginId:plugin.id,pluginReady:true,toolVisibleToAgent,artifactExistsBefore:true,artifactSha256:hash(html),projectId:project.id,profileId:profile.id,companyPrefix:ctx.input.fixtures.company.issuePrefix,route};
}catch(error){fixtures.delete(ctx);try{if(profileId)await ctx.input.api.post(`${root(ctx)}/tools/profiles/${profileId}/unbind`,{targetType:"agent",targetId:ctx.input.fixtures.agent.id});if(installedId)await ctx.input.api.delete(`/api/plugins/${installedId}`,{allowNotFound:true});}finally{await scaffold.close();}throw error;}
},gradePluginTool,{
expectedRunCount:2,expectedHumanMessageCount:2,attemptTimeoutMs:480000,
sourceFiles:["chat-catalog-domains/external-plugin-tool.ts","chat-catalog-domains/external-plugin-fixture.ts","chat-catalog-domains/external-support.ts","chat-catalog-domains/internal-support.ts","chat-catalog-domains/identity-support.ts","chat-flow.ts","../../cli/node_modules/tsx/dist/loader.mjs","../../cli/src/index.ts","../../packages/plugins/create-paperclip-plugin/src/index.ts","../../packages/plugins/sdk/src/index.ts","../../packages/plugins/sdk/dist/index.js","chat-scenarios/plugins.yaml"],
sourceFiles:["chat-catalog-domains/external-plugin-tool.ts","chat-catalog-domains/external-plugin-tool-identity.ts","chat-catalog-domains/external-plugin-fixture.ts","chat-catalog-domains/external-support.ts","chat-catalog-domains/internal-support.ts","chat-catalog-domains/identity-support.ts","chat-flow.ts","../../cli/node_modules/tsx/dist/loader.mjs","../../cli/src/index.ts","../../packages/plugins/create-paperclip-plugin/src/index.ts","../../packages/plugins/sdk/src/index.ts","../../packages/plugins/sdk/dist/index.js","chat-scenarios/plugins.yaml"],
async act(ctx,state){
await waitForReply(ctx);const first=await observeExternal(ctx,state),reply=first.phaseReplies.at(-1);if(!reply)throw new Error("Initial source preview reply missing");state.initialRunId=reply.runId;
const href=previewLink(first.links,first.origin,state.companyPrefix,state.route);state.initialPreviewLinkMatched=!!href;if(!href)throw new Error("Actual plugin preview link missing from model reply");const source=ctx.input.page.url();await ctx.input.page.goto(new URL(href,first.origin).href,{waitUntil:"domcontentloaded"});await expect(ctx.input.page.getByRole("heading",{name:"Synthetic website preview",exact:true})).toBeVisible();await expect(ctx.input.page.frameLocator('iframe[title="Website preview"]').getByRole("heading",{name:"Synthetic website",exact:true})).toBeVisible();state.initialPreviewVisible=true;await ctx.input.page.goto(source,{waitUntil:"domcontentloaded"});
tests/runner-e2e/chat-catalog-domains/external-plugins.ts
+24 -5
View File
@@ -5,6 +5,7 @@ import type { PluginRecord } from "../../../packages/shared/src/types/plugin.js"
import { pollUntil } from "../api.js";
import { definition, externalObservation, observeExternal, root, checks, verifyRenderedReply } from "./external-support.js";
import { createPreviewFixture } from "./external-plugin-fixture.js";
import { readPreviewToolInventory, PreviewToolRemovalPendingError } from "./external-plugin-tool-identity.js";
import { waitForReply, probeAgentPermission } from "./identity-support.js";
type Fixture = Awaited<ReturnType<typeof createPreviewFixture>>;
@@ -39,12 +40,16 @@ export function gradePluginManagement(value: unknown) {
}
export const pluginExternalCases = ["configure-plugin", "disable-plugin"].map(id => definition(`plugins.${id}`, async ctx => {
const fixture = await createPreviewFixture(ctx); fixtures.set(ctx, fixture);
let installedId: string | undefined, secretId: string | undefined;
try {
const installed = await ctx.input.api.post<PluginRecord>("/api/plugins/install", { packageName: fixture.packageRoot, isLocalPath: true });
installedId = installed.id;
const plugin = await pollUntil({ label: "disposable preview plugin activation", deadlineAt: Date.now() + 45_000,
load: () => ctx.input.api.get<PluginRecord>(`/api/plugins/${installed.id}`), accept: row => row.status === "ready", reject: row => row.status === "error" ? "Synthetic plugin activation failed; details withheld" : undefined });
const toolsBefore = await readPreviewToolInventory(ctx.input.api, plugin, { id: installed.id, key: fixture.key, count: 1 });
const value = `synthetic-preview-${randomBytes(24).toString("hex")}`; ctx.input.privateSecrets?.push(value);
const secret = await ctx.input.api.postSensitive<{ id: string }>(`${root(ctx)}/secrets`, { name: "Preview fixture credential", key: "PREVIEW_FIXTURE_API_KEY", provider: "local_encrypted", value });
secretId = secret.id;
if (id === "disable-plugin") await ctx.input.api.post(`/api/plugins/${plugin.id}/config`, { companyId: ctx.input.fixtures.company.id, configJson: { apiKey: { type: "secret_ref", secretId: secret.id, version: "latest" } } });
const configBefore = await ctx.input.api.get<ConfigRecord | null>(`/api/plugins/${plugin.id}/config?companyId=${ctx.input.fixtures.company.id}`);
const jobs = await ctx.input.api.get<Array<{ id: string }>>(`/api/plugins/${plugin.id}/jobs`);
@@ -54,11 +59,16 @@ export const pluginExternalCases = ["configure-plugin", "disable-plugin"].map(id
await ctx.input.api.post(`${root(ctx)}/chats/${ctx.input.fixtures.agent.id}`); await ctx.refreshIssue();
await ctx.input.api.patch(`/api/issues/${ctx.issue()!.id}`, { description: `Selected preview plugin: Website Preview Eval, key ${fixture.key}. ${id === "configure-plugin" ? "Its required preview credential has not been bound. The existing organization secret is named Preview fixture credential." : "It is active and contributes a scoped preview route and an annual managed refresh job."}` });
return { pluginId: plugin.id, pluginKey: fixture.key, syntheticPlugin: true, pluginReadyBeforeQuery: plugin.status === "ready", secretId: secret.id,
configBefore, jobId: jobs[0]!.id, jobCountBefore: jobs.length, agentMutationStatus,
toolsBeforeCount: toolsBefore.length, previewToolName: toolsBefore[0]!.name, configBefore, jobId: jobs[0]!.id, jobCountBefore: jobs.length, agentMutationStatus,
resultPath: `/${ctx.input.fixtures.company.issuePrefix}/company/settings/instance/plugins/${plugin.id}`, resultLabel: "Website Preview Eval" };
} catch (error) { fixtures.delete(ctx); await fixture.close(); throw error; }
} catch (error) {
fixtures.delete(ctx);
try { if (installedId) await ctx.input.api.delete(`/api/plugins/${installedId}`, { allowNotFound: true }); }
finally { try { if (secretId) await ctx.input.api.delete(`/api/secrets/${secretId}`, { allowNotFound: true }); } finally { await fixture.close(); } }
throw error;
}
}, gradePluginManagement, {
sourceFiles: ["chat-catalog-domains/external-plugins.ts", "chat-catalog-domains/external-plugin-fixture.ts", "chat-catalog-domains/external-support.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/identity-support.ts", "../../cli/node_modules/tsx/dist/loader.mjs", "../../cli/src/index.ts", "../../packages/plugins/create-paperclip-plugin/src/index.ts", "../../packages/plugins/sdk/src/index.ts", "../../packages/plugins/sdk/dist/index.js", "chat-scenarios/plugins.yaml"],
sourceFiles: ["chat-catalog-domains/external-plugins.ts", "chat-catalog-domains/external-plugin-tool-identity.ts", "chat-catalog-domains/external-plugin-fixture.ts", "chat-catalog-domains/external-support.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/identity-support.ts", "../../cli/node_modules/tsx/dist/loader.mjs", "../../cli/src/index.ts", "../../packages/plugins/create-paperclip-plugin/src/index.ts", "../../packages/plugins/sdk/src/index.ts", "../../packages/plugins/sdk/dist/index.js", "chat-scenarios/plugins.yaml"],
async act(ctx, state) {
await waitForReply(ctx);
const before = await ctx.input.api.get<PluginRecord>(`/api/plugins/${state.pluginId}`);
@@ -80,7 +90,14 @@ export const pluginExternalCases = ["configure-plugin", "disable-plugin"].map(id
} else {
const row = ctx.input.page.locator("li").filter({ hasText: "Website Preview Eval" });
await row.getByRole("button", { name: "Disable", exact: true }).click();
await pollUntil({ label: "human-disabled preview plugin", deadlineAt: Date.now() + 20_000, load: () => ctx.input.api.get<PluginRecord>(`/api/plugins/${state.pluginId}`), accept: row => row.status === "disabled" }); state.humanDisabled = true;
const fixture = fixtures.get(ctx); if (!fixture || typeof state.pluginId !== "string") throw new Error("Owned preview plugin identity is missing");
await pollUntil({ label: "human-disabled preview plugin and removed tool", deadlineAt: Date.now() + 20_000,
async load() {
const plugin = await ctx.input.api.get<PluginRecord>(`/api/plugins/${state.pluginId}`);
if (plugin.status !== "disabled") return { plugin, toolsRemoved: false };
try { await readPreviewToolInventory(ctx.input.api, plugin, { id: String(state.pluginId), key: fixture.key, count: 0 }); return { plugin, toolsRemoved: true }; }
catch (error) { if (error instanceof PreviewToolRemovalPendingError) return { plugin, toolsRemoved: false }; throw error; }
}, accept: row => row.plugin.status === "disabled" && row.toolsRemoved }); state.humanDisabled = true;
}
await ctx.input.page.goto(source, { waitUntil: "domcontentloaded" });
},
@@ -91,7 +108,9 @@ export const pluginExternalCases = ["configure-plugin", "disable-plugin"].map(id
if (id === "configure-plugin") state.configuredReadback = response.ok() && (await response.json()).configured === true;
else {
state.previewStatusAfter = response.status();
state.toolsAfterCount = (await ctx.input.api.get<unknown[]>(`/api/plugins/tools?pluginId=${state.pluginId}`)).length;
const fixture = fixtures.get(ctx); if (!fixture || typeof state.pluginId !== "string") throw new Error("Owned preview plugin identity is missing");
const tools = await readPreviewToolInventory(ctx.input.api, state.pluginAfter as PluginRecord, { id: state.pluginId, key: fixture.key, count: 0 });
state.toolsAfterCount = tools.length;
const triggered = await ctx.input.api.request.post(`/api/plugins/${state.pluginId}/jobs/${state.jobId}/trigger`); state.disabledJobTriggerStatus = triggered.status();
state.retainedConfigUnchanged = JSON.stringify((state.configAfter as ConfigRecord | null)?.configJson ?? null) === JSON.stringify((state.configBefore as ConfigRecord | null)?.configJson ?? null);
}
tests/runner-e2e/chat-catalog-domains/external-provider.ts
+13 -7
View File
@@ -1,14 +1,18 @@
import { createServer } from "node:http";
import { randomBytes } from "node:crypto";
import { createHash, randomBytes } from "node:crypto";
import { listenOnFetchAllowedPort } from "../../e2e/fetch-allowed-port.js";
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
import { root } from "./external-support.js";
export interface Tool { name: string; title: string; description: string; readOnly: boolean }
export interface ProviderCall { method: string; name?: string; authorized: boolean; arguments?: Record<string, unknown>; sequence: number }
export async function startNotesProvider(marker: string, options: { nonDestructiveWrites?: boolean } = {}) {
export interface ProviderCall { method: string; name?: string; authorized: boolean; arguments?: Record<string, unknown>; sequence: number; requestIdSha256?: string; responseHttpStatus?: number; toolError?: boolean; receivedAt?: number; respondedAt?: number; responseSha256?: string }
export function notesReadResponse(marker: string, unavailable: boolean) {
return unavailable ? { isError: true, content: [{ type: "text", text: "Synthetic current checklist is unavailable; no current source verification was returned" }] }
: { content: [{ type: "text", text: marker }] };
}
export async function startNotesProvider(marker: string, options: { nonDestructiveWrites?: boolean; readContent?: () => string } = {}) {
const token = randomBytes(32).toString("hex");
const calls: ProviderCall[] = []; let writeResult: string | undefined; let failWrites = false;
const calls: ProviderCall[] = []; let writeResult: string | undefined; let failWrites = false; let failReads = false;
const tools: Tool[] = [
{ name: "launch_read", title: "Read launch page", description: "Read the synthetic launch page.", readOnly: true },
{ name: "launch_update", title: "Update launch page", description: "Write the synthetic launch update supplied by the user.", readOnly: false },
@@ -19,8 +23,8 @@ export async function startNotesProvider(marker: string, options: { nonDestructi
for await (const part of req) { length += part.length; if (length > 65_536) throw new Error("Request too large"); parts.push(part); }
const message = JSON.parse(Buffer.concat(parts).toString("utf8") || "{}");
const authorized = req.headers.authorization === `Bearer ${token}`;
calls.push({ method: message.method ?? "unknown", name: message.params?.name, authorized,
arguments: message.params?.arguments, sequence: calls.length });
const call: ProviderCall = { method: message.method ?? "unknown", name: message.params?.name, authorized,
arguments: message.params?.arguments, sequence: calls.length, requestIdSha256: createHash("sha256").update(String(message.id)).digest("hex"), receivedAt: Date.now() }; calls.push(call);
if (!authorized) { res.writeHead(401); res.end(); return; }
let result: unknown = {};
if (message.method === "initialize") result = { protocolVersion: "2024-11-05", capabilities: { tools: {} }, serverInfo: { name: "paperclip-eval-synthetic-notes", version: "1.0.0" } };
@@ -29,16 +33,18 @@ export async function startNotesProvider(marker: string, options: { nonDestructi
if (message.method === "tools/call") {
const tool = tools.find(tool => tool.name === message.params?.name);
if (!tool) result = { isError: true, content: [{ type: "text", text: "Unknown action" }] };
else if (tool.readOnly) result = notesReadResponse(options.readContent?.() ?? marker, failReads);
else if (!tool.readOnly && failWrites) result = { isError: true, content: [{ type: "text", text: "Synthetic write failed; page unchanged" }] };
else { if (!tool.readOnly) writeResult = message.params?.arguments?.text;
result = { content: [{ type: "text", text: tool.readOnly ? marker : `Saved synthetic launch update: ${writeResult ?? ""}` }] }; }
}
call.responseSha256 = createHash("sha256").update(JSON.stringify(result)).digest("hex"); call.responseHttpStatus = 200; call.toolError = !!(result as { isError?: boolean }).isError; call.respondedAt = Date.now();
res.writeHead(200, { "content-type": "application/json" }); res.end(JSON.stringify({ jsonrpc: "2.0", id: message.id ?? null, result }));
} catch { res.writeHead(400); res.end(); }
});
const port = await listenOnFetchAllowedPort(server);
return { url: `http://127.0.0.1:${port}/`, token, tools, calls,
written: () => writeResult, failWrites: () => { failWrites = true; }, allowWrites: () => { failWrites = false; },
written: () => writeResult, failReads: () => { failReads = true; }, failWrites: () => { failWrites = true; }, allowWrites: () => { failWrites = false; },
close: () => new Promise<void>(resolve => server.close(() => resolve())) };
}
export type NotesProvider = Awaited<ReturnType<typeof startNotesProvider>>;
tests/runner-e2e/chat-catalog-domains/gateway-public-receipt.test.ts
+43
View File
@@ -0,0 +1,43 @@
import { expect, it } from "vitest";
import { executeRunnerApi } from "../../../server/src/services/native-runtime/runner-api-client.js";
import { redactRunnerApiPayloads } from "../../../server/src/services/native-runtime/runner-api-log-redaction.js";
import { nativeSha256 } from "../../../server/src/services/native-runtime/canonical.js";
import { observedRedactedPublicReceipt } from "./gateway-public-receipt.js";
const binding = { companyId: "company", leadId: "ceo", canonicalRunId: "canonical", workerId: "morgan", workerTaskId: "worker-task", workerRunId: "worker-run" };
async function valid() {
const report = { id: "public-comment", companyId: "company", issueId: "worker-task", authorAgentId: "morgan", createdByRunId: "worker-run", body: "The required notes write returned a provider error and the page remained unchanged.", createdAt: "2026-10-28T12:00:00.000Z" };
const request = { operationId: "GET /api/issues/{id}/comments", pathParams: { id: "worker-task" } };
let requests = 0;
const response = await executeRunnerApi(request, { companyId: "company", issueId: "canonical-task", runId: "canonical", workMode: "standard" }, {
apiUrl: "https://synthetic.invalid", token: "synthetic-test-token", fetch: async (url, init) => { requests++; expect(new URL(String(url)).pathname).toBe("/api/issues/worker-task/comments"); expect(init?.method).toBe("GET"); return new Response(JSON.stringify([report]), { status: 200, headers: { "content-type": "application/json" } }); },
readFile: async () => { throw Error("unexpected file read"); }, saveResponse: async () => { throw Error("unexpected artifact write"); },
});
expect(requests).toBe(1);
const { operationId, ...rest } = response;
const wire = { ...rest, apiOperationId: operationId };
const event = (seq: number, eventType: string, payload: unknown) => ({ companyId: "company", agentId: "ceo", runId: "canonical", seq, eventType, createdAt: `2026-10-28T12:00:0${seq}.000Z`, protocolSchemaVersion: 2, payload: { prpEvent: { schema: "paperclip.prp.event.v2", schemaVersion: 2, sourceKind: "runner", payload } } });
const events = [event(1, "item.started", { kind: "dynamicToolCall", item: { id: "call", type: "tool_use", name: "call_api", input: redactRunnerApiPayloads(request) } }), event(2, "item.completed", { kind: "dynamicToolCall", item: { id: "call", tool_use_id: "call", type: "tool_result", result: redactRunnerApiPayloads(wire) } })];
return { report, comments: [report], events, binding: { ...binding } };
}
const observe = (o: Awaited<ReturnType<typeof valid>>) => observedRedactedPublicReceipt(o.events, o.report, o.comments, o.binding);
const item = (o: Awaited<ReturnType<typeof valid>>, index = 1): any => o.events[index]!.payload.prpEvent.payload;
it("binds the actual production API envelope/redaction to an independent public comment without a transport-ID assumption", async () => { const o = await valid(); expect(observe(o)).toBe(true); expect((item(o).item.result as any).apiPayloadSha256).toBe(nativeSha256({ ok: true, status: 200, contentType: "application/json", retryAfter: null, data: o.comments, apiOperationId: "GET /api/issues/{id}/comments" })); });
it.each(["companyId", "agentId", "runId"])("rejects a foreign event %s", async field => { const o = await valid(); (o.events[1] as any)[field] = "foreign"; expect(observe(o)).toBe(false); });
it.each(["companyId", "leadId", "canonicalRunId", "workerId", "workerTaskId", "workerRunId"])("rejects a foreign expected binding %s", async field => { const o = await valid(); (o.binding as any)[field] = "foreign"; expect(observe(o)).toBe(false); });
it.each(["body", "id", "authorAgentId", "createdByRunId", "companyId", "issueId"])("rejects mutated independently saved receipt %s", async field => { const o = await valid(); (o.report as any)[field] = "wrong"; expect(observe(o)).toBe(false); });
it.each(["apiPayloadSha256", "apiOperationId", "status", "ok", "contentType", "data", "apiPayloadRedacted"])("rejects a wrong or missing typed result %s", async field => { const o = await valid(); delete item(o).item.result[field]; expect(observe(o)).toBe(false); });
it("rejects reordered/incomplete/comment-contaminated public DTO collections", async () => { for (const mode of ["extra", "empty", "foreign"]){ const o = await valid(); if(mode==="empty") o.comments=[]; else o.comments.push({...o.report,id:"another",...(mode==="foreign"?{companyId:"foreign"}:{})}); expect(observe(o)).toBe(false); } });
it("rejects wrong/missing/duplicate call starts and results, errors and bad chronology", async () => { for(const mutate of [(o:any)=>o.events.shift(),(o:any)=>o.events.push(structuredClone(o.events[0])),(o:any)=>o.events.push(structuredClone(o.events[1])),(o:any)=>item(o,0).item.name="read_task",(o:any)=>item(o,0).item.input.operationId="GET /api/companies/{companyId}/activity",(o:any)=>item(o).item.tool_use_id="other-call",(o:any)=>item(o).item.is_error=true,(o:any)=>o.events[0].seq=3,(o:any)=>o.events[1].createdAt="2026-10-28T11:59:59.000Z",(o:any)=>o.events[0].createdAt="2026-10-28T12:00:03.000Z",(o:any)=>o.events[1].protocolSchemaVersion=1,(o:any)=>o.events[1].payload.prpEvent.sourceKind="provider",(o:any)=>item(o).item.result.unproven="extra"]){ const o=await valid(); mutate(o);expect(observe(o)).toBe(false); } });
it("never replaces Morgan's public receipt with the CEO's own real receipt", async () => { const o=await valid(); o.comments=[{...o.report, id:"own-comment",authorAgentId:"ceo",createdByRunId:"own-run",issueId:"own-task"}]; expect(observe(o)).toBe(false); });
it("requires both actual response digest and the public DTO; a hash label alone cannot prove a read",async()=>{const o=await valid();item(o).item.result.apiPayloadSha256="a".repeat(64);expect(observe(o)).toBe(false)});
it("matches supported actual public response orders without synthesizing or sorting private response bytes",async()=>{
const o=await valid(),prior={...o.report,id:"prior-comment",body:"Checking the required source.",createdAt:"2026-10-28T11:59:00Z"};
const ascending=[prior,o.report],descending=[o.report,prior];
item(o).item.result=redactRunnerApiPayloads({ok:true,status:200,contentType:"application/json",retryAfter:null,data:descending,apiOperationId:"GET /api/issues/{id}/comments"});
expect(observedRedactedPublicReceipt(o.events,o.report,ascending,o.binding)).toBe(false);
expect(observedRedactedPublicReceipt(o.events,o.report,ascending,o.binding,[descending])).toBe(true);
expect(observedRedactedPublicReceipt(o.events,o.report,ascending,o.binding,[[{...o.report,body:"changed"},prior]])).toBe(false);
item(o).item.result=redactRunnerApiPayloads({ok:true,status:200,contentType:"application/json",retryAfter:null,data:ascending,apiOperationId:"GET /api/issues/{id}/comments"});
expect(observedRedactedPublicReceipt(o.events,o.report,descending,o.binding,[ascending])).toBe(true);
});
tests/runner-e2e/chat-catalog-domains/gateway-public-receipt.ts
+40
View File
@@ -0,0 +1,40 @@
import { nativeSha256 } from "../../../server/src/services/native-runtime/canonical.js";
import { REDACTED_EVENT_VALUE } from "../../../server/src/redaction.js";
import { isValidNativePrpEnvelope } from "../native-event-envelope.js";
type Row = Record<string, unknown>;
const object = (value: unknown): Row => value && typeof value === "object" && !Array.isArray(value) ? value as Row : {};
const time = (value: unknown) => typeof value === "string" || value instanceof Date ? new Date(value).getTime() : NaN;
const operation = "GET /api/issues/{id}/comments";
export interface PublicReceiptBinding { companyId: string; leadId: string; canonicalRunId: string; workerId: string; workerTaskId: string; workerRunId: string }
function matchesPublicReceiptResponse(events: Row[], report: Row, comments: Row[], binding: PublicReceiptBinding) {
if (!binding.companyId || !binding.leadId || !binding.canonicalRunId || !binding.workerId || !binding.workerTaskId || !binding.workerRunId
|| binding.workerId === binding.leadId || binding.workerRunId === binding.canonicalRunId || !report.id || typeof report.body !== "string"
|| report.companyId !== binding.companyId || report.issueId !== binding.workerTaskId || report.authorAgentId !== binding.workerId || report.createdByRunId !== binding.workerRunId
|| !Array.isArray(comments) || !comments.length || !comments.every(row => row.companyId === binding.companyId && row.issueId === binding.workerTaskId)
|| !comments.some(row => nativeSha256(row) === nativeSha256(report))) return false;
const rows = events.flatMap(event => {
const envelope = object(object(event.payload).prpEvent), payload = object(envelope.payload);
return event.companyId === binding.companyId && event.agentId === binding.leadId && event.runId === binding.canonicalRunId
&& Number.isSafeInteger(Number(event.seq)) && Number(event.seq) > 0 && isValidNativePrpEnvelope(envelope, event.protocolSchemaVersion) && envelope.sourceKind === "runner"
? [{ event, payload }] : [];
});
return rows.some(({event, payload}) => {
const item = object(payload.item), result = object(item.result);
if (event.eventType !== "item.completed" || item.type !== "tool_result" || typeof item.id !== "string" || item.tool_use_id !== item.id || item.is_error === true
|| !Number.isFinite(time(report.createdAt)) || !Number.isFinite(time(event.createdAt)) || time(event.createdAt) < time(report.createdAt)
|| result.apiOperationId !== operation || result.ok !== true || result.status !== 200 || typeof result.contentType !== "string" || !result.contentType.includes("json")
|| result.data !== REDACTED_EVENT_VALUE || result.apiPayloadRedacted !== true || typeof result.apiPayloadSha256 !== "string" || !/^[a-f0-9]{64}$/.test(result.apiPayloadSha256)
|| Object.keys(result).some(key => !["ok", "status", "contentType", "apiOperationId", "data", "apiPayloadRedacted", "apiPayloadSha256"].includes(key))) return false;
const starts = rows.filter(row => row.event.eventType === "item.started" && object(row.payload.item).id === item.id && object(row.payload.item).type === "tool_use" && Number(row.event.seq) < Number(event.seq));
if (starts.length !== 1 || rows.filter(row => row.event.eventType === "item.completed" && object(row.payload.item).type === "tool_result" && object(row.payload.item).id === item.id && object(object(row.payload.item).result).apiOperationId === operation).length !== 1) return false;
const start = starts[0]!, request = object(object(start.payload.item).input);
if (!Number.isFinite(time(start.event.createdAt)) || time(start.event.createdAt) > time(event.createdAt)
|| object(start.payload.item).type !== "tool_use" || object(start.payload.item).name !== "call_api" || request.operationId !== operation
|| request.apiPayloadRedacted !== true || typeof request.apiPayloadSha256 !== "string" || !/^[a-f0-9]{64}$/.test(request.apiPayloadSha256)) return false;
return nativeSha256({ ok: true, status: 200, contentType: result.contentType, retryAfter: null, data: comments, apiOperationId: operation }) === result.apiPayloadSha256;
});
}
export function observedRedactedPublicReceipt(events: Row[], report: Row, comments: Row[], binding: PublicReceiptBinding, additionalPublicResponses: Row[][] = []) {
return [comments, ...additionalPublicResponses].some(response => matchesPublicReceiptResponse(events, report, response, binding));
}
tests/runner-e2e/chat-catalog-domains/identity-hire-model-choice.test.ts
+72
View File
@@ -0,0 +1,72 @@
import { describe, expect, it, vi } from "vitest";
import { CHAT_OPENCODE_MODEL } from "../chat-runtime-profile.js";
import { HIRE_MODEL_METADATA_URL, HIRE_ORIGINAL_MODEL, proveHireModelChoice, readHireModelChoice, validHireModelChoice } from "./identity-hire-model-choice.js";
function fixture() {
const original = { id: HIRE_ORIGINAL_MODEL.replace(/^openrouter\//, ""), pricing: { prompt: "0.00000066", completion: "0.00000198", input_cache_read: "0.000000022" } as Record<string, unknown> };
const target = { id: CHAT_OPENCODE_MODEL.replace(/^openrouter\//, ""), pricing: { prompt: "0.00000015", completion: "0.0000006", input_cache_read: "0.000000003",
overrides: [{ utc_days: ["monday"], utc_start: 100, utc_end: 400, prompt: "0.0000003", completion: "0.0000012", input_cache_read: "0.000000006" }] } as Record<string, unknown> };
return { body: { data: [original, target] }, catalog: [HIRE_ORIGINAL_MODEL, CHAT_OPENCODE_MODEL].map(id => ({ id })), original, target };
}
describe("real catalog identity and every published price variant", () => {
it("proves exact requested1017 using worst target and best original rates", () => {
const f = fixture(), proof = proveHireModelChoice(f.body, f.catalog);
expect(proof.originalModel).toBe(HIRE_ORIGINAL_MODEL); expect(proof.cheaperModel).toBe(CHAT_OPENCODE_MODEL);
expect(proof.targetMaximumRates.prompt).toBe("0.0000003"); expect(proof.targetRateVariants).toBe(2);
expect(proof.units).toBe("USD/token"); expect(proof).not.toHaveProperty("costUsd");
});
it.each([
["missing target catalog", (f: ReturnType<typeof fixture>) => { f.catalog.pop(); }],
["duplicate catalog identity", (f: ReturnType<typeof fixture>) => { f.catalog.push(f.catalog[0]!); }],
["missing official identity", (f: ReturnType<typeof fixture>) => { f.body.data.pop(); }],
["ambiguous official identity", (f: ReturnType<typeof fixture>) => { f.body.data.push(f.target); }],
["alias instead of fixed model", (f: ReturnType<typeof fixture>) => { f.target.id = "deepseek/deepseek-v4.1-flash"; }],
["missing amount", (f: ReturnType<typeof fixture>) => { delete f.target.pricing.prompt; }],
["unknown amount", (f: ReturnType<typeof fixture>) => { f.target.pricing.prompt = null; }],
["numeric amount", (f: ReturnType<typeof fixture>) => { f.target.pricing.prompt = 0; }],
["negative amount", (f: ReturnType<typeof fixture>) => { f.target.pricing.prompt = "-0.1"; }],
["nonfinite amount", (f: ReturnType<typeof fixture>) => { f.target.pricing.prompt = "NaN"; }],
["incompatible ancillary fee", (f: ReturnType<typeof fixture>) => { f.original.pricing.request = "0.1"; }],
["unknown new fee even zero", (f: ReturnType<typeof fixture>) => { f.target.pricing.request = "0"; }],
["missing ancillary price", (f: ReturnType<typeof fixture>) => { delete f.original.pricing.input_cache_read; }],
["unknown override set", (f: ReturnType<typeof fixture>) => { f.target.pricing.overrides = null; }],
["missing override price", (f: ReturnType<typeof fixture>) => { delete (f.target.pricing.overrides as Array<Record<string, unknown>>)[0]!.completion; }],
["unrecognized override fee", (f: ReturnType<typeof fixture>) => { (f.target.pricing.overrides as Array<Record<string, unknown>>)[0]!.request = "0.1"; }],
["base-only false cheap comparison", (f: ReturnType<typeof fixture>) => { (f.target.pricing.overrides as Array<Record<string, unknown>>)[0]!.prompt = "1"; }],
["expensive model sometimes cheaper", (f: ReturnType<typeof fixture>) => { f.original.pricing.overrides = [{ prompt: "0.00000001", completion: "0.00000001", input_cache_read: "0.000000001" }]; }],
["lower input but higher output", (f: ReturnType<typeof fixture>) => { f.target.pricing.completion = "1"; }],
["higher cached input fee", (f: ReturnType<typeof fixture>) => { f.target.pricing.input_cache_read = "1"; }],
])("rejects %s", (_label, mutate) => { const f = fixture(); mutate(f); expect(() => proveHireModelChoice(f.body, f.catalog)).toThrow(); });
it("does not manufacture zero for unknown but preserves an explicitly published zero", () => {
const f = fixture(); f.target.pricing.input_cache_read = "0";
(f.target.pricing.overrides as Array<Record<string, unknown>>)[0]!.input_cache_read = "0";
expect(proveHireModelChoice(f.body, f.catalog).targetMaximumRates.input_cache_read).toBe("0");
});
it("uses exact decimals beyond floating point precision", () => {
const f = fixture(); delete f.target.pricing.overrides;
for (const field of ["prompt", "completion", "input_cache_read"]) { f.original.pricing[field] = "0.100000000000000002"; f.target.pricing[field] = "0.100000000000000001"; }
expect(proveHireModelChoice(f.body, f.catalog).targetMaximumRates.prompt).toBe("0.100000000000000001");
f.original.pricing.prompt = f.target.pricing.prompt; f.original.pricing.completion = f.target.pricing.completion;
expect(() => proveHireModelChoice(f.body, f.catalog)).toThrow("strictly cheaper");
});
});
describe("bounded public metadata read and replay custody", () => {
it("uses one credential-free GET and recomputable selected pricing", async () => {
const f = fixture(), fetcher = vi.fn<typeof fetch>().mockResolvedValue(Response.json(f.body));
const proof = await readHireModelChoice(f.catalog, fetcher);
expect(fetcher).toHaveBeenCalledExactlyOnceWith(HIRE_MODEL_METADATA_URL, { method: "GET", headers: { Accept: "application/json" }, signal: expect.any(AbortSignal) });
expect(validHireModelChoice(proof)).toBe(true);
expect(validHireModelChoice({ ...proof, cheaperModel: "other" })).toBe(false);
expect(validHireModelChoice({ ...proof, metadataSha256: "" })).toBe(false);
expect(validHireModelChoice({ ...proof, retrievedAt: "unknown" })).toBe(false);
expect(validHireModelChoice({ ...proof, targetMaximumRates: { ...proof.targetMaximumRates, prompt: "0" } })).toBe(false);
});
it("rejects an oversized metadata response without any model call", async () => {
const fetcher = vi.fn<typeof fetch>().mockResolvedValue(new Response("x".repeat(512 * 1024 + 1)));
await expect(readHireModelChoice(fixture().catalog, fetcher)).rejects.toThrow("read limit"); expect(fetcher).toHaveBeenCalledTimes(1);
});
it("rejects an unavailable public catalog without retry or credential fallback", async () => {
const fetcher = vi.fn<typeof fetch>().mockResolvedValue(new Response("unavailable", { status: 503 }));
await expect(readHireModelChoice(fixture().catalog, fetcher)).rejects.toThrow("could not be read"); expect(fetcher).toHaveBeenCalledTimes(1);
});
});
tests/runner-e2e/chat-catalog-domains/identity-hire-model-choice.ts
+88
View File
@@ -0,0 +1,88 @@
import { createHash } from "node:crypto";
import { CHAT_OPENCODE_MODEL } from "../chat-runtime-profile.js";
export const HIRE_ORIGINAL_MODEL = "openrouter/deepseek/deepseek-v4-pro-0813";
export const HIRE_MODEL_METADATA_URL = "https://openrouter.ai/api/v1/models?q=deepseek-v4&output_modalities=text";
const PRICE_FIELDS = ["prompt", "completion", "input_cache_read"] as const;
type PriceField = typeof PRICE_FIELDS[number];
type Rates = Record<PriceField, string>;
interface ModelRecord { id: string; pricing: Record<string, unknown> }
export interface HireModelChoice {
originalModel: string;
cheaperModel: typeof CHAT_OPENCODE_MODEL;
eligibleModels: string[];
sourceRecords: ModelRecord[];
targetMaximumRates: Rates;
originalMinimumRates: Rates;
targetRateVariants: number;
originalRateVariants: number;
units: "USD/token";
metadataSha256?: string;
retrievedAt?: string;
}
const record = (value: unknown): Record<string, unknown> => value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
const digest = (value: string) => createHash("sha256").update(value).digest("hex");
function amount(value: unknown): bigint {
if (typeof value !== "string" || value.length > 48 || !/^\d{1,20}(?:\.\d{1,18})?$/.test(value)) throw new Error("Published model price is missing or unsupported");
const [whole, fraction = ""] = value.split(".");
return BigInt(whole!) * 10n ** 18n + BigInt(fraction.padEnd(18, "0"));
}
function variants(model: ModelRecord): Rates[] {
const pricing = record(model.pricing);
if (Object.keys(pricing).filter(key => key !== "overrides").sort().join(",") !== [...PRICE_FIELDS].sort().join(",")) throw new Error("Published ancillary price fields are incompatible");
const overrides = Object.hasOwn(pricing, "overrides") ? pricing.overrides : [];
if (!Array.isArray(overrides) || overrides.length > 32) throw new Error("Published rate variants are unsupported");
const rows = [pricing, ...overrides.map(record)];
for (const row of rows) {
if (row !== pricing && Object.keys(row).some(key => ![...PRICE_FIELDS, "utc_days", "utc_start", "utc_end"].includes(key))) throw new Error("Published rate variant fields are unsupported");
for (const field of PRICE_FIELDS) amount(row[field]);
}
return rows.map(row => Object.fromEntries(PRICE_FIELDS.map(field => [field, row[field]])) as Rates);
}
export function proveHireModelChoice(body: unknown, catalog: readonly { id: string }[]): HireModelChoice {
const data = record(body).data;
if (!Array.isArray(data) || data.length > 64) throw new Error("Official model metadata is invalid");
const models = [HIRE_ORIGINAL_MODEL, CHAT_OPENCODE_MODEL].map(nativeId => {
if (catalog.filter(row => row.id === nativeId).length !== 1) throw new Error("Both proposed models must exist in the actual public OpenRouter catalog");
const rows = data.filter(row => record(row).id === nativeId.replace(/^openrouter\//, ""));
if (rows.length !== 1) throw new Error("Official model identity is missing or ambiguous");
const row = record(rows[0]);
return { id: String(row.id), pricing: record(row.pricing) };
});
const original = variants(models[0]!), target = variants(models[1]!);
const targetMaximumRates = {} as Rates, originalMinimumRates = {} as Rates;
for (const field of PRICE_FIELDS) {
targetMaximumRates[field] = target.map(row => row[field]).reduce((a, b) => amount(a) >= amount(b) ? a : b);
originalMinimumRates[field] = original.map(row => row[field]).reduce((a, b) => amount(a) <= amount(b) ? a : b);
if (amount(targetMaximumRates[field]) > amount(originalMinimumRates[field])) throw new Error("The selected model does not dominate every published rate variant");
}
if (!["prompt", "completion"].some(field => amount(targetMaximumRates[field as PriceField]) < amount(originalMinimumRates[field as PriceField]))) throw new Error("No strictly cheaper published text rate is proved");
return { originalModel: HIRE_ORIGINAL_MODEL, cheaperModel: CHAT_OPENCODE_MODEL, eligibleModels: [HIRE_ORIGINAL_MODEL, CHAT_OPENCODE_MODEL], sourceRecords: models,
targetMaximumRates, originalMinimumRates, targetRateVariants: target.length, originalRateVariants: original.length, units: "USD/token" };
}
export function validHireModelChoice(value: unknown): value is HireModelChoice {
const choice = record(value);
try {
const recomputed = proveHireModelChoice({ data: choice.sourceRecords }, (choice.eligibleModels as string[]).map(id => ({ id })));
return ["originalModel", "cheaperModel", "eligibleModels", "targetMaximumRates", "originalMinimumRates", "targetRateVariants", "originalRateVariants", "units"].every(key => JSON.stringify(choice[key]) === JSON.stringify(record(recomputed)[key]))
&& typeof choice.metadataSha256 === "string" && /^[a-f0-9]{64}$/.test(choice.metadataSha256)
&& typeof choice.retrievedAt === "string" && Number.isFinite(Date.parse(choice.retrievedAt));
} catch { return false; }
}
export async function readHireModelChoice(catalog: readonly { id: string }[], fetcher: typeof fetch = fetch): Promise<HireModelChoice> {
const response = await fetcher(HIRE_MODEL_METADATA_URL, { method: "GET", headers: { Accept: "application/json" }, signal: AbortSignal.timeout(10_000) });
if (!response.ok || !response.body) throw new Error("Official public model metadata could not be read");
const reader = response.body.getReader(), chunks: Uint8Array[] = [];
let bytes = 0;
try {
for (;;) {
const part = await reader.read();
if (part.done) break;
bytes += part.value.byteLength;
if (bytes > 512 * 1024) throw new Error("Official model metadata exceeded its read limit");
chunks.push(part.value);
}
} finally { await reader.cancel().catch(() => {}); reader.releaseLock(); }
const text = Buffer.concat(chunks).toString("utf8");
return { ...proveHireModelChoice(JSON.parse(text), catalog), metadataSha256: digest(text), retrievedAt: new Date().toISOString() };
}
tests/runner-e2e/chat-catalog-domains/identity-hire-revision-clock.test.ts
+21
View File
@@ -0,0 +1,21 @@
import { afterEach, expect, it, vi } from "vitest";
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
const mocks = vi.hoisted(() => ({ wait: vi.fn() }));
vi.mock("./identity-support.js", async importOriginal => ({ ...await importOriginal<typeof import("./identity-support.js")>(), waitForReply: mocks.wait }));
import { hireRevisionIdentityCases } from "./identity-hire-revision.js";
afterEach(() => { vi.useRealTimers(); mocks.wait.mockReset(); });
it("passes the original remaining observation deadline after the incidental150s window without adding time", async () => {
vi.useFakeTimers(); vi.setSystemTime(190_000);
const ctx = { observationDeadlineAt: 270_000 } as ChatCatalogContext;
const stop = new Error("Controlled wait boundary; no UI or API action"); mocks.wait.mockRejectedValue(stop);
await expect(hireRevisionIdentityCases[0]!.act!(ctx, {})).rejects.toBe(stop);
expect(mocks.wait).toHaveBeenCalledExactlyOnceWith(ctx, 270_000);
expect(hireRevisionIdentityCases[0]!.attemptTimeoutMs).toBe(300_000);
});
it.each([undefined, Number.NaN, 190_000])("rejects missing or exhausted original clock before waiting: %s", async deadline => {
vi.useFakeTimers(); vi.setSystemTime(190_000);
const ctx = { observationDeadlineAt: deadline } as ChatCatalogContext;
mocks.wait.mockRejectedValue(new Error("Must not reach wait"));
await expect(hireRevisionIdentityCases[0]!.act!(ctx, {})).rejects.toThrow("Original identity observation budget exhausted or missing");
expect(mocks.wait).not.toHaveBeenCalled();
});
tests/runner-e2e/chat-catalog-domains/identity-hire-revision.test.ts
+20 -4
View File
@@ -1,16 +1,32 @@
import { describe, expect, it } from "vitest";
import { proveHireModelChoice } from "./identity-hire-model-choice.js";
import { gradeHireRevision } from "./identity-hire-revision.js";
import type { IdentityObservation } from "./identity-support.js";
function valid(): IdentityObservation {
const models = ["openrouter/deepseek/deepseek-v4-pro-0813", "openrouter/deepseek/deepseek-v4.1-flash-1017"];
const modelChoice = { ...proveHireModelChoice({ data: models.map((id, index) => ({ id: id.replace(/^openrouter\//, ""), pricing: { prompt: index ? "0.1" : "1", completion: index ? "0.2" : "2", input_cache_read: index ? "0.01" : "0.02" } })) }, models.map(id => ({ id }))), metadataSha256: "a".repeat(64), retrievedAt: "2026-10-28T00:00:00Z" };
return { scenarioId: "questions-and-approvals.formal-hire-revision", origin: "http://fixture.invalid", companyId: "acorn", leadId: "ceo", company: { id: "acorn" }, companies: [], agents: [{ id: "ceo" }], tasks: [], projects: [], goals: [], interactions: [], comments: [], documents: [], revisions: [], links: [], runs: [], humanQueries: [], reply: "The proposal is back for board approval.",
activity: [{ id: "revision", action: "approval.revision_requested", entityId: "approval", actorType: "user" }, { id: "resubmit", action: "approval.resubmitted", entityId: "approval", actorType: "agent", agentId: "ceo" }],
state: { originalApproval: { id: "approval", payload: { adapterConfig: { model: "gpt-6.1-sol" } } }, currentApproval: { id: "approval", companyId: "acorn", type: "hire_agent", requestedByAgentId: "ceo", status: "pending", decidedByUserId: null, decidedAt: null, payload: { adapterConfig: { model: "gpt-6-luna" } } },
originalModel: "gpt-6.1-sol", cheaperModel: "gpt-6-luna", eligibleModels: ["gpt-6.1-sol", "gpt-6-luna"], originalOtherFieldsPreserved: true, humanId: "alex", originalHistoryCommentId: "history", beforeAgentCount: 1, pendingProposalVisible: true, revisedModelVisible: true,
approvalComments: [{ id: "history", authorUserId: "alex", body: "Original gpt-6.1-sol, board selected gpt-6-luna" }, { id: "revision-comment", authorAgentId: "ceo", body: "Revised to gpt-6-luna and resubmitted for approval." }] } };
state: { modelChoice, originalApproval: { id: "approval", payload: { adapterType: "paperclip_runner", adapterConfig: { provider: "opencode", model: "openrouter/deepseek/deepseek-v4-pro-0813" } } }, currentApproval: { id: "approval", companyId: "acorn", type: "hire_agent", requestedByAgentId: "ceo", status: "pending", decidedByUserId: null, decidedAt: null, payload: { adapterType: "paperclip_runner", adapterConfig: { provider: "opencode", model: "openrouter/deepseek/deepseek-v4.1-flash-1017" } } },
originalModel: "openrouter/deepseek/deepseek-v4-pro-0813", cheaperModel: "openrouter/deepseek/deepseek-v4.1-flash-1017", eligibleModels: ["openrouter/deepseek/deepseek-v4-pro-0813", "openrouter/deepseek/deepseek-v4.1-flash-1017"], originalOtherFieldsPreserved: true, humanId: "alex", originalHistoryCommentId: "history", beforeAgentCount: 1, pendingProposalVisible: true, revisedModelVisible: true,
approvalComments: [{ id: "history", authorUserId: "alex", body: "Original openrouter/deepseek/deepseek-v4-pro-0813, board selected openrouter/deepseek/deepseek-v4.1-flash-1017" }, { id: "revision-comment", authorAgentId: "ceo", body: "Revised to openrouter/deepseek/deepseek-v4.1-flash-1017 and resubmitted for approval." }] } };
}
describe("formal hire remains a proposal after cost revision", () => {
it("requires actual model choice, attributed history and a still pending proposal", () => { expect(gradeHireRevision(valid()).passed).toBe(true); expect(gradeHireRevision(undefined).passed).toBe(false); });
it("rejects treating requested revision as approval or activating the hire", () => { const o = valid(); (o.state.currentApproval as { status: string }).status = "approved"; expect(gradeHireRevision(o).passed).toBe(false); const wrong = valid(); wrong.agents.push({ id: "new-hire" }); expect(gradeHireRevision(wrong).passed).toBe(false); });
it("rejects changing only prose while retaining the expensive config", () => { const o = valid(); (o.state.currentApproval as { payload: { adapterConfig: { model: string } } }).payload.adapterConfig.model = "gpt-6.1-sol"; expect(gradeHireRevision(o).passed).toBe(false); });
it("rejects changing only prose while retaining the expensive config", () => { const o = valid(); (o.state.currentApproval as { payload: { adapterType: "paperclip_runner", adapterConfig: { provider: "opencode", model: string } } }).payload.adapterConfig.model = "openrouter/deepseek/deepseek-v4-pro-0813"; expect(gradeHireRevision(o).passed).toBe(false); });
it("rejects an unsupported substituted model and missing historical record", () => { const o = valid(); o.state.eligibleModels = []; expect(gradeHireRevision(o).passed).toBe(false); const wrong = valid(); wrong.state.approvalComments = []; expect(gradeHireRevision(wrong).passed).toBe(false); });
it.each([
["missing price proof", (o: IdentityObservation) => { delete o.state.modelChoice; }],
["unrelated original model", (o: IdentityObservation) => { (o.state.originalApproval as { payload: { adapterConfig: { model: string } } }).payload.adapterConfig.model = "openrouter/other/model"; }],
["wrong proposed runtime", (o: IdentityObservation) => { (o.state.currentApproval as { payload: { adapterConfig: { provider: string } } }).payload.adapterConfig.provider = "codex"; }],
["wrong proposal", (o: IdentityObservation) => { (o.state.currentApproval as { id: string }).id = "other"; }],
["wrong company", (o: IdentityObservation) => { (o.state.currentApproval as { companyId: string }).companyId = "other"; }],
["wrong requesting agent", (o: IdentityObservation) => { (o.state.currentApproval as { requestedByAgentId: string }).requestedByAgentId = "other"; }],
["human decision already saved", (o: IdentityObservation) => { (o.state.currentApproval as { decidedByUserId: string | null }).decidedByUserId = "alex"; }],
["changed unrelated configuration", (o: IdentityObservation) => { o.state.originalOtherFieldsPreserved = false; }],
["duplicate agent resubmission", (o: IdentityObservation) => { o.activity.push({ ...o.activity[1]!, id: "duplicate" }); }],
["approval side effect", (o: IdentityObservation) => { o.activity.push({ id: "approved", action: "approval.approved", entityId: "approval" }); }],
["invisible pending proposal", (o: IdentityObservation) => { o.state.pendingProposalVisible = false; }],
])("rejects %s", (_label, mutate) => { const o = valid(); mutate(o); expect(gradeHireRevision(o).passed).toBe(false); });
});
tests/runner-e2e/chat-catalog-domains/identity-hire-revision.ts
+19 -11
View File
@@ -2,19 +2,24 @@ import { expect } from "@playwright/test";
import type { Approval, ApprovalComment } from "../../../packages/shared/src/types/approval.js";
import type { ChatCatalogCase } from "../chat-catalog-contract.js";
import { checks, entity } from "./internal-support.js";
import { identityDefinition, identityObservation, observeIdentity, openStandingChat, waitForReply } from "./identity-support.js";
import { identityDefinition, identityObservation, observeIdentity, openStandingChat, remainingIdentityObservationBudget, waitForReply } from "./identity-support.js";
import { CHAT_OPENCODE_MODEL } from "../chat-runtime-profile.js";
import { readHireModelChoice, validHireModelChoice } from "./identity-hire-model-choice.js";
export function gradeHireRevision(value: unknown) {
const c = checks(); const o = identityObservation(value); c.add("complete-formal-revision", !!o, "Actual governance proposal, catalog choice, saved history and attributed resubmission are required."); if (!o) return c.result();
const approval = o.state.currentApproval as Approval | undefined; const original = o.state.originalApproval as Approval | undefined;
const adapter = approval?.payload.adapterConfig as { model?: string } | undefined;
const adapter = approval?.payload.adapterConfig as { model?: string; provider?: string } | undefined;
c.add("same-unapproved-governed-hire", !!approval && approval.id === original?.id && approval.companyId === o.companyId && approval.type === "hire_agent"
&& approval.requestedByAgentId === o.leadId && approval.status === "pending" && approval.decidedByUserId === null && approval.decidedAt === null,
"Revision returns the same actual hiring proposal to pending board review; it is never promoted to approved or a ready employee.");
c.add("actual-board-selected-cheaper-choice", typeof o.state.cheaperModel === "string" && adapter?.model === o.state.cheaperModel
&& (original?.payload.adapterConfig as { model?: string } | undefined)?.model !== adapter?.model
&& Array.isArray(o.state.eligibleModels) && o.state.eligibleModels.includes(adapter.model) && o.state.originalOtherFieldsPreserved === true,
"The real public adapter catalog contains the explicit board-selected lower-cost alternative, and only that proposed configuration changes; no rate is invented.");
&& Array.isArray(o.state.eligibleModels) && o.state.eligibleModels.includes(adapter.model) && o.state.originalOtherFieldsPreserved === true
&& approval?.payload.adapterType === "paperclip_runner" && adapter.provider === "opencode"
&& validHireModelChoice(o.state.modelChoice) && o.state.modelChoice.cheaperModel === adapter.model && o.state.modelChoice.originalModel === o.state.originalModel
&& (original?.payload.adapterConfig as { model?: string } | undefined)?.model === o.state.modelChoice.originalModel,
"The actual public catalog and official published text/cache rates prove the selected1017 configuration costs less across all published rate variants. Only the pending proposed model changes; no inference or realized savings is claimed for the original configuration.");
const comments = o.state.approvalComments as ApprovalComment[] | undefined;
c.add("inspectable-original-and-revised-history", !!comments?.some(comment => comment.id === o.state.originalHistoryCommentId && comment.authorUserId === o.state.humanId
&& comment.body.includes(String(o.state.originalModel)) && comment.body.includes(String(o.state.cheaperModel)))
@@ -30,10 +35,12 @@ export function gradeHireRevision(value: unknown) {
export const hireRevisionIdentityCases: readonly ChatCatalogCase[] = [identityDefinition("questions-and-approvals.formal-hire-revision", async ctx => {
const source = await openStandingChat(ctx); const { fixtures, execution, workspacePath, nonce } = ctx.input;
const configuration = execution.profile.buildAgent({ environmentId: fixtures.environment.id, environmentFixtureId: execution.environment.id, workspacePath, secretRefs: fixtures.secretRefs, executionId: nonce });
const config = configuration.adapterConfig as Record<string, unknown>; const cheaperModel = "gpt-6-luna";
const originalModel = typeof config.model === "string" && config.model !== cheaperModel ? config.model : "gpt-6.1-sol";
const models = await ctx.input.api.get<Array<{ id: string }>>(`/api/companies/${fixtures.company.id}/adapters/${configuration.adapterType}/models?provider=codex&environmentId=${fixtures.environment.id}`);
if (![originalModel, cheaperModel].every(model => models.some(row => row.id === model))) throw new Error("Setup blocker: both the proposed and board-selected cheaper models must exist in the actual target adapter catalog");
const config = configuration.adapterConfig as Record<string, unknown>;
if (configuration.adapterType !== "paperclip_runner" || config.provider !== "opencode" || config.model !== CHAT_OPENCODE_MODEL) throw new Error("This hire-revision fixture requires the requested native OpenCode1017 profile");
const models = await ctx.input.api.get<Array<{ id: string }>>(`/api/companies/${fixtures.company.id}/adapters/opencode_local/models?provider=openrouter&environmentId=${fixtures.environment.id}`);
const modelChoice = await readHireModelChoice(models);
const { originalModel, cheaperModel } = modelChoice;
await ctx.input.evidence("identity-hire-model-choice.json", { ...modelChoice, originalConfigurationExecuted: false, publishedRateComparisonOnly: true });
const session = await ctx.input.api.get<{ user: { id: string } }>("/api/auth/get-session");
const originalApproval = await ctx.input.api.post<Approval>(`/api/companies/${fixtures.company.id}/approvals`, { type: "hire_agent", requestedByAgentId: fixtures.agent.id, issueIds: [source.id],
payload: { ...configuration, name: `Launch researcher ${nonce}`, role: "researcher", reportsTo: fixtures.agent.id, adapterConfig: { ...config, model: originalModel } } });
@@ -43,11 +50,12 @@ export const hireRevisionIdentityCases: readonly ChatCatalogCase[] = [identityDe
await expect.poll(async () => (await ctx.input.api.get<Approval>(`/api/approvals/${originalApproval.id}`)).status).toBe("revision_requested");
await ctx.input.page.goto(url, { waitUntil: "domcontentloaded" });
return { originalApproval, originalHistoryCommentId: history.id, originalModel, cheaperModel, humanId: session.user.id,
eligibleModels: models.map(row => row.id), beforeAgentCount: (await ctx.input.api.get<unknown[]>(`/api/companies/${fixtures.company.id}/agents`)).length };
eligibleModels: models.map(row => row.id), modelChoice, beforeAgentCount: (await ctx.input.api.get<unknown[]>(`/api/companies/${fixtures.company.id}/agents`)).length };
}, gradeHireRevision, {
authentication: "human-session", sourceFiles: ["chat-catalog-domains/identity-hire-revision.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", "chat-scenarios/questions-and-approvals.yaml"],
authentication: "human-session", oracleVersion: "paperclip.chat-catalog.hire-revision.native1017.v2", sourceFiles: ["chat-catalog-domains/identity-hire-revision.ts", "chat-catalog-domains/identity-hire-model-choice.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", "chat-scenarios/questions-and-approvals.yaml"],
async act(ctx, state) {
await waitForReply(ctx); const id = entity(state, "originalApproval").id; const source = ctx.input.page.url();
const originalDeadlineAt = ctx.observationDeadlineAt; remainingIdentityObservationBudget(ctx);
await waitForReply(ctx, originalDeadlineAt); const id = entity(state, "originalApproval").id; const source = ctx.input.page.url();
const links = await ctx.input.page.locator('[id^="comment-"] a[href]').evaluateAll(rows => rows.map(row => row.getAttribute("href")!));
const link = links.find(href => new URL(href, ctx.input.api.baseURL).pathname.endsWith(`/approvals/${id}`));
if (!link) throw new Error("Actual pending revised proposal review link is missing");
tests/runner-e2e/chat-catalog-domains/identity-navigation-boundaries.ts
+2 -1
View File
@@ -1,3 +1,4 @@
import { chatCatalogOriginalPhaseDeadline } from "../chat-catalog-contract.js";
import { expect } from "@playwright/test";
import type { ChatCatalogCase, ChatCatalogContext } from "../chat-catalog-contract.js";
import { RunnerApiHttpError } from "../api.js";
@@ -71,7 +72,7 @@ export const navigationBoundaryIdentityCases: readonly ChatCatalogCase[] = [
}, gradeNavigationBoundary, {
sourceFiles: sources, authentication: "human-session",
async act(ctx, state) {
await waitForReply(ctx); const source = ctx.input.page.url(); const path = String(state.stalePath);
await waitForReply(ctx, chatCatalogOriginalPhaseDeadline(ctx)); const source = ctx.input.page.url(); const path = String(state.stalePath);
await ctx.input.page.goto(path, { waitUntil: "domcontentloaded" });
await expect(ctx.input.page.getByText("Issue not found", { exact: true })).toBeVisible(); state.missingVisible = true;
const actual = new URL(ctx.input.page.url()); state.stalePathPreserved = actual.pathname + actual.search === path;
tests/runner-e2e/chat-catalog-domains/identity-navigation-preferences.ts
+2 -1
View File
@@ -1,3 +1,4 @@
import { chatCatalogOriginalPhaseDeadline } from "../chat-catalog-contract.js";
import { expect } from "@playwright/test";
import { checks, entity, hasLink, project, task, verifyInternalResult } from "./internal-support.js";
import { identityDefinition, identityObservation, waitForReply, probeAgentPermission } from "./identity-support.js";
@@ -73,7 +74,7 @@ export const navigationIdentityCases = [
authentication: "human-session",
sourceFiles: ["chat-catalog-domains/identity-navigation-preferences.ts", "chat-catalog-domains/identity-auth.ts", "chat-catalog-domains/identity-private-browser.ts", "chat-catalog-domains/identity-private-browser-worker.ts", "chat-catalog-domains/identity-wizard.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", "chat-scenarios/navigation-preferences.yaml"],
async act(ctx, state) {
await waitForReply(ctx); const source = ctx.input.page.url();
await waitForReply(ctx, chatCatalogOriginalPhaseDeadline(ctx)); const source = ctx.input.page.url();
await ctx.input.page.keyboard.press("ControlOrMeta+k");
await ctx.input.page.getByPlaceholder("Search tasks, agents, projects...").fill("Launch status:backlog");
await expect(ctx.input.page.getByRole("dialog").getByText(entity(state, "task").title!, { exact: true })).toBeVisible();
tests/runner-e2e/chat-catalog-domains/identity-new-company-clock.test.ts
+44
View File
@@ -0,0 +1,44 @@
import { afterEach, expect, it, vi } from "vitest";
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
const mocks = vi.hoisted(() => ({ wait: vi.fn(), wizard: vi.fn(), get: vi.fn() }));
vi.mock("./identity-support.js", async original => ({ ...await original<typeof import("./identity-support.js")>(), waitForReply: mocks.wait }));
vi.mock("./identity-new-company-native.js", () => ({ finishNativeNewCompany: mocks.wizard }));
import { newCompanyIdentityCases } from "./identity-new-company.js";
const stop = new Error("Controlled wizard boundary; no browser, account, server or model");
afterEach(() => { vi.useRealTimers(); mocks.wait.mockReset(); mocks.wizard.mockReset(); mocks.get.mockReset(); });
function context(deadline: number | undefined = 570_000) {
mocks.wizard.mockRejectedValue(stop); mocks.get.mockResolvedValue([{ id: "source-company" }]);
return { observationDeadlineAt: deadline, input: { nonce: "synthetic-clock", privateInputs: { credentials: { OPENROUTER_API_KEY: "synthetic-never-dispatched-key" } }, privateSecrets: [],
api: { baseURL: "http://127.0.0.1:1", get: mocks.get }, page: { url: () => "http://127.0.0.1:1/SRC/chat" } } } as unknown as ChatCatalogContext;
}
it("uses the SAME original phase beyond150s and preserves the existing360s human setup maximum", async () => {
vi.useFakeTimers(); vi.setSystemTime(200_000); const ctx = context(); mocks.wait.mockResolvedValue(undefined); mocks.wizard.mockRejectedValue(stop);
await expect(newCompanyIdentityCases[0]!.act!(ctx, {})).rejects.toBe(stop);
expect(mocks.wait).toHaveBeenCalledExactlyOnceWith(ctx, 570_000);
expect(mocks.wizard).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ deadlineAt: 560_000 }));
expect(newCompanyIdentityCases[0]!.expectedRunCount).toBe(1); expect(newCompanyIdentityCases[0]!.expectedHumanMessageCount ?? 1).toBe(1);
expect(newCompanyIdentityCases[0]!.attemptTimeoutMs).toBe(600_000);
});
it("caps late human setup at the remaining original phase instead of giving a fresh360s", async () => {
vi.useFakeTimers(); vi.setSystemTime(200_000); const ctx = context(); mocks.wait.mockImplementation(async () => { vi.setSystemTime(500_000); }); mocks.wizard.mockRejectedValue(stop);
await expect(newCompanyIdentityCases[0]!.act!(ctx, {})).rejects.toBe(stop);
expect(mocks.wait).toHaveBeenCalledExactlyOnceWith(ctx, 570_000);
expect(mocks.wizard).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ deadlineAt: 570_000 }));
});
it.each([undefined, Number.NaN, Number.POSITIVE_INFINITY, 200_000])("rejects missing/nonfinite/exhausted phase before waiting: %s", async deadline => {
vi.useFakeTimers(); vi.setSystemTime(200_000); const ctx = context(); (ctx as { observationDeadlineAt?: number }).observationDeadlineAt = deadline;
await expect(newCompanyIdentityCases[0]!.act!(ctx, {})).rejects.toThrow("Original identity observation budget exhausted or missing");
expect(mocks.wait).not.toHaveBeenCalled(); expect(mocks.wizard).not.toHaveBeenCalled(); expect(ctx.input.api.get).not.toHaveBeenCalled();
});
it("rechecks the original phase after waiting before any company or human setup action", async () => {
vi.useFakeTimers(); vi.setSystemTime(200_000); const ctx = context(); mocks.wait.mockImplementation(async () => { vi.setSystemTime(570_000); });
await expect(newCompanyIdentityCases[0]!.act!(ctx, {})).rejects.toThrow("Original identity observation budget exhausted or missing");
expect(mocks.wizard).not.toHaveBeenCalled(); expect(ctx.input.api.get).not.toHaveBeenCalled();
});
it("does not start human setup if the authenticated company read consumes the remaining phase", async () => {
vi.useFakeTimers(); vi.setSystemTime(500_000); const ctx = context(); mocks.wait.mockResolvedValue(undefined);
mocks.get.mockImplementationOnce(async () => { vi.setSystemTime(570_000); return [{ id: "source-company" }]; });
await expect(newCompanyIdentityCases[0]!.act!(ctx, {})).rejects.toThrow("Original identity observation budget exhausted or missing");
expect(mocks.wizard).not.toHaveBeenCalled();
});
tests/runner-e2e/chat-catalog-domains/identity-new-company-native-proof.ts
+78
View File
@@ -0,0 +1,78 @@
import type { AiConnectionBinding, AiConnectionList } from "../../../packages/shared/src/ai-connections.js";
import type { ConnectionGrantsResponse, ToolConnectionInstall } from "../../../packages/shared/src/types/tool-access.js";
export const NEW_COMPANY_NATIVE_MODEL = "openrouter/deepseek/deepseek-v4.1-flash-1017";
export const NEW_COMPANY_PROVIDER_MODEL = "deepseek/deepseek-v4.1-flash-1017";
export interface NativeCompany { id: string; name: string; issuePrefix: string; defaultResponsibleUserId?: string | null }
export interface NativeFirstAgent {
id: string; companyId: string; name: string; role: string; adapterType: string;
adapterConfig: Record<string, unknown>; runtimeConfig: { aiConnection?: AiConnectionBinding; heartbeat?: { enabled?: boolean } };
defaultEnvironmentId?: string | null; lifecycleState: string; status: string;
}
export interface NativeSetupRequest {
companyId: string; agentId: string; environmentId: string; adapterType: string; provider: string; model: string; sequence: number;
binding?: AiConnectionBinding; containsCredential: boolean;
}
export interface NativeSetupProbe extends NativeSetupRequest {
httpStatus: number; resultAdapterType: string; status: string; checks: Array<{ code: string; level: string }>;
}
export interface NewCompanyNativeReceipt {
companyId: string; companyPrefix: string; firstAgentId: string; companyName: string; agentName: string;
requestedHarness: "opencode_local"; actualHarness: "opencode_local"; model: string;
environmentId: string; environmentDriver: "local"; actualProbeEnvironmentId: string;
beforeCompanyIds: string[]; afterCompanyIds: string[]; beforeReadyAgents: number; afterAgentIds: string[];
noAgentBeforeRouteDeparture: true; publicSetupRoute: string; savedSetupAgentId: string;
hello: NativeSetupProbe; helloCount: number; hiredThroughPublicSetup: true; hireCount: number;
firstAgentReady: true; chatComposerVisible: true; heartbeatRunIds: string[]; readyConversationPath: string;
account: { id: string; grantId: string; provider: "openrouter"; method: "api_key"; status: "connected"; ownership: "personal"; ownerUserId: string; currentUserId: string; responsibleUserId: string; bindingMode: "delegated" };
exactPublicGrantAndAgentInstall: true; draftContainsCredential: boolean; publicAgentContainsCredential: boolean;
verificationKind: "public-native-hello-probe"; firstWorkExecuted: false; probeUsage: "unavailable-outside-heartbeat-accounting";
}
export function nativeSetupRequest(input: { companyId: string; adapterType: string; body: unknown; sequence: number }, credential: string): NativeSetupRequest {
const body = input.body as Record<string, unknown>; const config = body?.adapterConfig as Record<string, unknown> | undefined;
return { companyId: input.companyId, adapterType: input.adapterType, sequence: input.sequence,
agentId: typeof body?.agentId === "string" ? body.agentId : typeof body?.setupAgentId === "string" ? body.setupAgentId : "",
environmentId: typeof body?.environmentId === "string" ? body.environmentId : typeof body?.defaultEnvironmentId === "string" ? body.defaultEnvironmentId : "",
provider: typeof config?.provider === "string" ? config.provider : "", model: typeof config?.model === "string" ? config.model : "",
binding: (body?.aiConnection ?? (body?.runtimeConfig as { aiConnection?: AiConnectionBinding } | undefined)?.aiConnection) as AiConnectionBinding | undefined,
containsCredential: Boolean(credential && JSON.stringify(input.body).includes(credential)) };
}
export function requireNativeFirstAgentProof(input: {
company: NativeCompany; agent: NativeFirstAgent; humanId: string; environment: { id: string; driver: string };
savedSetupAgentId: string; probes: NativeSetupProbe[]; hires: Array<NativeSetupRequest & { httpStatus: number; resultAgentId: string }>;
connection: { connectionId: string; grantId: string }; accounts: AiConnectionList;
grants: ConnectionGrantsResponse; installs: ToolConnectionInstall[];
}): void {
const { company, agent, humanId, environment, savedSetupAgentId, probes, hires, connection, accounts, grants, installs } = input;
const binding = agent.runtimeConfig.aiConnection;
const matches = (request: NativeSetupRequest) => request.companyId === company.id && request.agentId === savedSetupAgentId
&& request.environmentId === environment.id && request.adapterType === "paperclip_runner" && request.provider === "opencode"
&& request.model === NEW_COMPANY_NATIVE_MODEL && request.containsCredential === false
&& request.binding?.provider === "openrouter" && request.binding.method === "api_key" && request.binding.mode === "delegated"
&& request.binding.connectionId === connection.connectionId && request.binding.grantId === connection.grantId;
if (environment.driver !== "local" || !environment.id || probes.length !== 1 || hires.length !== 1
|| !matches(probes[0]!) || probes[0]!.httpStatus !== 200 || probes[0]!.resultAdapterType !== "paperclip_runner" || probes[0]!.status !== "pass"
|| probes[0]!.checks.some(check => check.level === "error") || !probes[0]!.checks.some(check => check.code === "opencode_hello_probe_passed")
|| !Number.isInteger(probes[0]!.sequence) || probes[0]!.sequence < 1 || !Number.isInteger(hires[0]!.sequence) || probes[0]!.sequence >= hires[0]!.sequence
|| !matches(hires[0]!) || hires[0]!.httpStatus < 200 || hires[0]!.httpStatus >= 300 || hires[0]!.resultAgentId !== savedSetupAgentId)
throw new Error("New company requires one current same-agent native OpenCode account/model/local-target test and hire");
if (agent.id !== savedSetupAgentId || agent.companyId !== company.id || agent.role !== "ceo" || agent.adapterType !== "paperclip_runner"
|| agent.adapterConfig.provider !== "opencode" || agent.adapterConfig.model !== NEW_COMPANY_NATIVE_MODEL
|| agent.defaultEnvironmentId !== environment.id || agent.lifecycleState !== "ready" || !["idle", "active"].includes(agent.status)
|| agent.runtimeConfig.heartbeat?.enabled !== false
|| binding?.provider !== "openrouter" || binding.method !== "api_key" || binding.mode !== "delegated"
|| binding.connectionId !== connection.connectionId || binding.grantId !== connection.grantId
|| company.defaultResponsibleUserId !== humanId || accounts.currentUserId !== humanId)
throw new Error("New company public first-agent identity/configuration differs from the tested setup");
const account = accounts.connections.find(row => row.id === connection.connectionId && row.grantId === connection.grantId);
const grant = grants.grants.find(row => row.id === connection.grantId);
if (!account || account.companyId !== company.id || account.provider !== "openrouter" || account.method !== "api_key"
|| account.status !== "connected" || account.ownership !== "personal" || account.ownerUserId !== humanId
|| account.routing?.kind !== "openrouter" || account.routing.protocol !== "chat" || account.routing.auth !== "bearer"
|| account.routing.models.length !== 1 || account.routing.models[0]?.id !== NEW_COMPANY_PROVIDER_MODEL
|| grants.connection.id !== connection.connectionId || grants.currentUserId !== humanId || grant?.connectionId !== connection.connectionId
|| grant.companyId !== company.id || grant.kind !== "user" || grant.subjectUserId !== humanId || grant.status !== "active"
|| installs.length !== 1 || installs[0]?.companyId !== company.id || installs[0]?.connectionId !== connection.connectionId
|| installs[0]?.targetType !== "agent" || installs[0]?.targetId !== agent.id)
throw new Error("New company account requires its actual personal grant and exact first-agent public installation");
}
tests/runner-e2e/chat-catalog-domains/identity-new-company-native.test.ts
+102
View File
@@ -0,0 +1,102 @@
import { describe, expect, it } from "vitest";
import { readFileSync } from "node:fs";
import { NEW_COMPANY_NATIVE_MODEL, NEW_COMPANY_PROVIDER_MODEL, nativeSetupRequest, requireNativeFirstAgentProof } from "./identity-new-company-native-proof.js";
import { aiRoutingModel } from "../../../packages/shared/src/ai-provider-routing.js";
import { createAiConnectionSchema } from "../../../packages/shared/src/ai-connections.js";
import { resolveRouteOnboardingOptions } from "../../../ui/src/lib/onboarding-route.js";
type Proof = Parameters<typeof requireNativeFirstAgentProof>[0];
const companyId = "11111111-1111-4111-8111-111111111111", agentId = "22222222-2222-4222-8222-222222222222";
function proof(): Proof {
const binding = { provider: "openrouter", method: "api_key", mode: "delegated", connectionId: "connection", grantId: "grant" };
const request = { companyId, agentId, environmentId: "local-target", adapterType: "paperclip_runner", provider: "opencode", model: NEW_COMPANY_NATIVE_MODEL, binding, containsCredential: false };
// Synthetic projections of the actual public DTOs. No API, grant or agent is created.
return {
company: { id: companyId, name: "Website business", issuePrefix: "WEB", defaultResponsibleUserId: "human" },
agent: { id: agentId, companyId, name: "CEO", role: "ceo", adapterType: "paperclip_runner", adapterConfig: { provider: "opencode", model: NEW_COMPANY_NATIVE_MODEL }, runtimeConfig: { aiConnection: binding, heartbeat: { enabled: false } }, defaultEnvironmentId: "local-target", lifecycleState: "ready", status: "idle" },
humanId: "human", environment: { id: "local-target", driver: "local" }, savedSetupAgentId: agentId,
probes: [{ ...request, sequence: 1, httpStatus: 200, resultAdapterType: "paperclip_runner", status: "pass", checks: [{ code: "opencode_hello_probe_passed", level: "info" }] }],
hires: [{ ...request, sequence: 2, httpStatus: 201, resultAgentId: agentId }], connection: { connectionId: "connection", grantId: "grant" },
accounts: { currentUserId: "human", canManageConnections: true, connections: [{ id: "connection", grantId: "grant", companyId, provider: "openrouter", method: "api_key", name: "My OpenRouter", status: "connected", ownership: "personal", ownerUserId: "human", isDefault: false,
routing: { kind: "openrouter", protocol: "chat", auth: "bearer", models: [{ id: NEW_COMPANY_PROVIDER_MODEL }] } }] },
grants: { currentUserId: "human", connection: { id: "connection", uid: "connection" }, grants: [{ id: "grant", connectionId: "connection", companyId, kind: "user", subjectUserId: "human", status: "active" }] },
installs: [{ companyId, connectionId: "connection", targetType: "agent", targetId: agentId }],
} as unknown as Proof;
}
describe("native first-agent public setup custody", () => {
it("accepts the same saved CEO/account/local model test and hire", () => expect(() => requireNativeFirstAgentProof(proof())).not.toThrow());
for (const [name, mutate] of Object.entries<(p: Proof) => void>({
"remote target": p => { p.environment.driver = "remote"; },
"wrong selected target": p => { p.probes[0]!.environmentId = "other"; },
"wrong company": p => { p.probes[0]!.companyId = "other"; },
"foreign setup agent": p => { p.probes[0]!.agentId = "other"; },
"legacy adapter": p => { p.probes[0]!.adapterType = "opencode_local"; },
"legacy result": p => { p.probes[0]!.resultAdapterType = "opencode_local"; },
"wrong runtime": p => { p.probes[0]!.provider = "codex"; },
"wrong model": p => { p.probes[0]!.model = "openrouter/other"; },
"key validation only": p => { p.probes[0]!.checks = [{ code: "ai_connection_api_key_reverified", level: "info" }]; },
"wrong hello": p => { p.probes[0]!.checks = [{ code: "codex_hello_probe_passed", level: "info" }]; },
"probe denied": p => { p.probes[0]!.httpStatus = 403; },
"warning probe": p => { p.probes[0]!.status = "warn"; },
"probe error": p => { p.probes[0]!.checks.push({ code: "adapter_auth_missing", level: "error" }); },
"missing probe": p => { p.probes = []; },
"second probe": p => { p.probes.push(p.probes[0]!); },
"late probe": p => { p.probes[0]!.sequence = 3; },
"missing chronology": p => { p.probes[0]!.sequence = Number.NaN; },
"foreign account tested": p => { p.probes[0]!.binding = { ...p.probes[0]!.binding!, mode: "delegated", connectionId: "other", grantId: "grant" }; },
"default-account substitution": p => { p.probes[0]!.binding = { provider: "openrouter", method: "api_key", mode: "responsible_user" }; },
"raw credential in test": p => { p.probes[0]!.containsCredential = true; },
"duplicate hire": p => { p.hires.push(p.hires[0]!); },
"different hire model": p => { p.hires[0]!.model = "other"; },
"hire replaces setup": p => { p.hires[0]!.resultAgentId = "other"; },
"hire denied": p => { p.hires[0]!.httpStatus = 403; },
"final different company": p => { p.agent.companyId = "other"; },
"renamed general profile": p => { p.agent.role = "general"; },
"not ready": p => { p.agent.lifecycleState = "setup"; },
"wrong saved model": p => { p.agent.adapterConfig.model = "other"; },
"foreign saved environment": p => { p.agent.defaultEnvironmentId = "other"; },
"enabled first work": p => { p.agent.runtimeConfig.heartbeat = { enabled: true }; },
"foreign saved grant": p => { p.agent.runtimeConfig.aiConnection = { provider: "openrouter", method: "api_key", mode: "delegated", connectionId: "connection", grantId: "other" }; },
"other responsible person": p => { p.company.defaultResponsibleUserId = "other"; },
"other signed-in person": p => { p.accounts.currentUserId = "other"; },
"shared account": p => { p.accounts.connections[0]!.ownership = "shared"; },
"unavailable account": p => { p.accounts.connections[0]!.status = "needs_attention"; },
"foreign account owner": p => { p.accounts.connections[0]!.ownerUserId = "other"; },
"wrong route protocol": p => { p.accounts.connections[0]!.routing!.protocol = "responses"; },
"wrong catalog model": p => { p.accounts.connections[0]!.routing!.models[0]!.id = "other"; },
"no public grant": p => { p.grants.grants = []; },
"foreign grant": p => { p.grants.grants[0]!.companyId = "other"; },
"other grant owner": p => { p.grants.grants[0]!.subjectUserId = "other"; },
"revoked grant": p => { p.grants.grants[0]!.status = "revoked"; },
"no installation": p => { p.installs = []; },
"company-wide access": p => { p.installs[0]!.targetType = "company"; p.installs[0]!.targetId = companyId; },
"other agent access": p => { p.installs[0]!.targetId = "other"; },
})) it(`rejects ${name}`, () => { const p = proof(); mutate(p); expect(() => requireNativeFirstAgentProof(p)).toThrow(); });
it("normalizes the existing public test and hire shapes without exposing raw credentials", () => {
const p = proof(); const config = { provider: "opencode", model: NEW_COMPANY_NATIVE_MODEL };
expect(nativeSetupRequest({ companyId, adapterType: "paperclip_runner", sequence: 1, body: { agentId, environmentId: "local-target", adapterConfig: config, aiConnection: p.agent.runtimeConfig.aiConnection } }, "synthetic-private-key"))
.toMatchObject({ agentId, environmentId: "local-target", containsCredential: false, binding: p.agent.runtimeConfig.aiConnection });
const result = nativeSetupRequest({ companyId, adapterType: "paperclip_runner", sequence: 2, body: { setupAgentId: agentId, defaultEnvironmentId: "local-target", adapterConfig: { ...config, env: { OPENROUTER_API_KEY: "synthetic-private-key" } }, runtimeConfig: p.agent.runtimeConfig } }, "synthetic-private-key");
expect(result).toMatchObject({ agentId, environmentId: "local-target", containsCredential: true }); expect(JSON.stringify(result)).not.toContain("synthetic-private-key");
});
});
describe("actual supported route and provider contracts", () => {
it("ordinary saved-company agents/new does not mount onboarding", () => {
expect(resolveRouteOnboardingOptions({ pathname: "/WEB/agents/new", companyPrefix: "WEB", companies: [{ id: companyId, issuePrefix: "WEB" }] })).toBeNull();
expect(resolveRouteOnboardingOptions({ pathname: "/WEB/onboarding", companyPrefix: "WEB", companies: [{ id: companyId, issuePrefix: "WEB" }] })).not.toBeNull();
});
it("keeps public OpenRouter catalog ID and native model normalization consistent", () => {
const routing = proof().accounts.connections[0]!.routing!;
expect(aiRoutingModel(routing, "opencode_local", NEW_COMPANY_PROVIDER_MODEL)).toBe(NEW_COMPANY_NATIVE_MODEL);
expect(aiRoutingModel(routing, "opencode_local", NEW_COMPANY_NATIVE_MODEL)).toBe(NEW_COMPANY_NATIVE_MODEL);
expect(createAiConnectionSchema.parse({ provider: "openrouter", method: "api_key", name: "My OpenRouter", ownership: "personal", allAgents: false, agentIds: [agentId], routing, apiKey: "synthetic-private-key" })).toMatchObject({ ownership: "personal", agentIds: [agentId], allAgents: false });
});
it("binds the original production UI draft gate and public first-agent hold/reuse", () => {
const source = (file: string) => readFileSync(new URL(`../../../${file}`, import.meta.url), "utf8");
expect(source("ui/src/components/OnboardingWizard.tsx")).toContain('const setupSnapshot = step === 4');
expect(source("ui/src/context/DialogContext.tsx")).toContain('const [onboardingOpen, setOnboardingOpen] = useState(false)');
expect(source("ui/src/components/new-agent/AgentBasicsDialog.tsx")).toContain('role: existing.length ? "general" : "ceo"');
expect(source("ui/src/components/new-agent/AgentBasicsDialog.tsx")).toContain('buildNewAgentRuntimeConfig({ heartbeatEnabled: false })');
expect(source("ui/src/components/new-agent/NewAgentSetup.tsx")).toContain('setupAgentId: created.id');
expect(source("ui/src/lib/test-agent-setup.ts")).toContain('input.adapterType,');
});
});
tests/runner-e2e/chat-catalog-domains/identity-new-company-native.ts
+150
View File
@@ -0,0 +1,150 @@
import { expect, test, type Page, type Response } from "@playwright/test";
import { privateIdentityBrowser } from "./identity-private-browser.js";
import { pollUntil } from "../api.js";
import type { AiConnectionList } from "../../../packages/shared/src/ai-connections.js";
import type { ConnectionGrantsResponse, ToolConnectionInstall } from "../../../packages/shared/src/types/tool-access.js";
import { NEW_COMPANY_NATIVE_MODEL, NEW_COMPANY_PROVIDER_MODEL, nativeSetupRequest, requireNativeFirstAgentProof,
type NativeCompany, type NativeFirstAgent, type NativeSetupProbe, type NativeSetupRequest, type NewCompanyNativeReceipt } from "./identity-new-company-native-proof.js";
export type { NewCompanyNativeReceipt } from "./identity-new-company-native-proof.js";
interface Input {
page: Page; origin: string; companyName: string; agentName: string; credential: string; deadlineAt: number;
onCompany(id: string): void; onConnection(companyId: string, connectionId: string, grantId: string): void;
evidence(name: string, value: unknown): Promise<void>;
}
export async function finishNativeNewCompany(input: Input): Promise<NewCompanyNativeReceipt> {
const receipt = await privateIdentityBrowser({ kind: "new-company-native", origin: input.origin, startPath: input.page.url(),
cookies: await input.page.context().cookies(input.origin), companyName: input.companyName, agentName: input.agentName,
credential: input.credential, deadlineAt: input.deadlineAt }, { company: input.onCompany, connection: input.onConnection });
if (!receipt.newCompany) throw new Error("Private new-company setup returned no closed result");
await input.page.goto(new URL(receipt.newCompany.readyConversationPath, input.origin).href, { waitUntil: "domcontentloaded" });
await input.evidence("identity-public-new-company-native-result.json", receipt.newCompany); return receipt.newCompany;
}
/** Only the existing standalone private credential worker may call this. */
export async function runNativeNewCompanyInPrivateBrowser(input: Input & {
registerCompany(company: NativeCompany): Promise<void>;
registerConnection(companyId: string, connectionId: string, grantId: string): Promise<void>;
}): Promise<NewCompanyNativeReceipt> {
let attached = false; try { test.info(); attached = true; } catch { /* standalone only */ }
if (attached) throw new Error("Credential-bearing company setup requires the private child");
const { page, origin } = input;
const remaining = () => Math.max(1, Math.min(180_000, input.deadlineAt - Date.now()));
const get = async <T>(endpoint: string): Promise<T> => {
const response = await page.context().request.get(origin + endpoint, { timeout: Math.min(30_000, remaining()) });
if (!response.ok()) throw new Error(`Public company setup read returned HTTP ${response.status()}`); return response.json() as Promise<T>;
};
const session = await get<{ user: { id: string } }>("/api/auth/get-session");
const before = await get<NativeCompany[]>("/api/companies");
await page.getByRole("button", { name: /^Open .+ (?:organization|workspace|company) switcher$/i }).click();
await page.getByRole("menuitem", { name: "Create organization", exact: true }).click();
await expect(page.getByRole("heading", { name: "What is the name of your organization?", exact: true })).toBeVisible();
await page.locator("#onboarding-company-name").fill(input.companyName);
await page.getByRole("button", { name: "Continue", exact: true }).click();
await expect(page.locator("#onboarding-agent-name")).toBeVisible();
const company = await pollUntil({ label: "actual saved new company", deadlineAt: input.deadlineAt,
load: () => get<NativeCompany[]>("/api/companies"), accept: rows => rows.filter(row => !before.some(old => old.id === row.id) && row.name === input.companyName).length === 1 })
.then(rows => rows.find(row => !before.some(old => old.id === row.id) && row.name === input.companyName)!);
await input.registerCompany(company); input.onCompany(company.id);
if ((await get<NativeFirstAgent[]>(`/api/companies/${company.id}/agents`)).length) throw new Error("The wizard created an agent before the public route departure");
// Ordinary address-bar navigation to a supported public route. No hidden
// dialog action, draft edit or forced localStorage state is used.
const publicSetupRoute = `/${company.issuePrefix}/agents/new`;
await page.goto(new URL(publicSetupRoute, origin).href, { waitUntil: "domcontentloaded" });
await expect(page.getByTestId("onboarding-wizard")).not.toBeVisible();
await expect(page.getByRole("heading", { name: "Meet your next agent", exact: true })).toBeVisible();
await page.getByLabel("Agent name", { exact: true }).fill(input.agentName);
await page.getByRole("button", { name: "Choose adapter", exact: true }).click();
await expect(page.getByRole("heading", { name: "Choose an adapter", exact: true })).toBeVisible();
const setupAgents = await get<NativeFirstAgent[]>(`/api/companies/${company.id}/agents`);
if (setupAgents.length !== 1 || setupAgents[0]?.name !== input.agentName || setupAgents[0]?.role !== "ceo" || setupAgents[0]?.lifecycleState === "ready")
throw new Error("Public first-agent dialog did not save one setup-held CEO");
const provisional = setupAgents[0]!;
await page.getByRole("dialog").locator("summary").filter({ hasText: /^Advanced$/ }).click();
await page.getByRole("button", { name: "Experimental harness", exact: true }).click();
await page.getByRole("option", { name: "OpenCode (Paperclip Runner)", exact: true }).click();
await page.getByRole("button", { name: "Configure agent", exact: true }).click();
await expect(page.getByRole("heading", { name: "Configure your agent", exact: true })).toBeVisible();
if (new URL(page.url()).searchParams.get("setupAgentId") !== provisional.id) throw new Error("Public setup route replaced the saved first-agent identity");
const environments = await get<Array<{ id: string; driver: string }>>(`/api/companies/${company.id}/environments?driver=local`);
if (environments.length !== 1 || environments[0]?.driver !== "local") throw new Error("Company setup requires the actual available local target");
const environment = environments[0]!;
await page.getByRole("combobox", { name: "Environment", exact: true }).selectOption(environment.id);
await page.getByRole("combobox", { name: "Model", exact: true }).selectOption({ label: "Enter custom model…" });
await page.getByRole("textbox", { name: "Model ID", exact: true }).fill(NEW_COMPANY_NATIVE_MODEL);
let connection: { connectionId: string; grantId: string } | undefined; let responseSequence = 0;
const probes: NativeSetupProbe[] = [], hires: Array<NativeSetupRequest & { httpStatus: number; resultAgentId: string }> = [];
const reads = new Set<Promise<void>>(), errors: unknown[] = [];
const onResponse = (response: Response) => {
const url = new URL(response.url()); if (url.origin !== origin || response.request().method() !== "POST") return;
const account = url.pathname === `/api/companies/${company.id}/ai-connections`;
const probe = url.pathname.match(new RegExp(`^/api/companies/${company.id}/adapters/([^/]+)/test-environment$`));
const hire = url.pathname === `/api/companies/${company.id}/agent-hires`;
if (!account && !probe && !hire) return;
const sequence = ++responseSequence;
const pending = (async () => {
const body = response.request().postDataJSON() as Record<string, unknown>;
const result = await response.json() as Record<string, unknown>;
if (account) {
if (response.status() !== 201 || typeof result.connectionId !== "string" || typeof result.grantId !== "string") throw new Error("Public account creation receipt missing");
await input.registerConnection(company.id, result.connectionId, result.grantId);
if (connection || body.provider !== "openrouter" || body.method !== "api_key" || body.ownership !== "personal"
|| body.allAgents !== false || !Array.isArray(body.agentIds) || body.agentIds.length !== 1 || body.agentIds[0] !== provisional.id)
throw new Error("Public account creation did not preserve exact personal first-agent access");
connection = { connectionId: result.connectionId, grantId: result.grantId };
return;
}
const request = nativeSetupRequest({ companyId: company.id, adapterType: probe ? decodeURIComponent(probe[1]!) : String(body.adapterType ?? ""), body, sequence }, input.credential);
if (probe) probes.push({ ...request, httpStatus: response.status(), resultAdapterType: String(result.adapterType ?? ""), status: String(result.status ?? ""),
checks: Array.isArray(result.checks) ? result.checks.map(row => ({ code: String(row.code ?? ""), level: String(row.level ?? "") })) : [] });
else hires.push({ ...request, httpStatus: response.status(), resultAgentId: String((result.agent as { id?: string } | undefined)?.id ?? "") });
})(); reads.add(pending); void pending.catch(error => errors.push(error)).finally(() => reads.delete(pending));
};
page.on("response", onResponse);
const settleReads = async () => { await Promise.all([...reads].map(read => read.catch(() => {}))); if (errors.length) throw new Error("Public company setup response identity check failed"); };
try {
await page.getByRole("combobox", { name: "Connection", exact: true }).click();
await page.getByRole("option", { name: "Connect an account…", exact: true }).click();
const accountDialog = page.getByRole("dialog").filter({ has: page.getByRole("heading", { name: "Connect account", exact: true }) });
await accountDialog.locator("summary").filter({ hasText: /^Advanced providers$/ }).click();
await accountDialog.getByRole("button", { name: "OpenRouter", exact: true }).click();
await accountDialog.getByRole("button", { name: "Change", exact: true }).click();
await accountDialog.getByRole("radio", { name: "Just me", exact: true }).click();
await accountDialog.getByRole("radio", { name: "Just agents I pick", exact: true }).click();
await accountDialog.getByRole("button", { name: "Select agents", exact: true }).click();
await page.getByRole("checkbox", { name: `Allow ${input.agentName}`, exact: true }).setChecked(true);
await page.getByPlaceholder("Filter agents", { exact: true }).press("Escape");
await accountDialog.getByRole("textbox", { name: "Model IDs", exact: true }).fill(NEW_COMPANY_PROVIDER_MODEL);
await accountDialog.getByLabel("API key", { exact: true }).fill(input.credential);
await accountDialog.getByRole("button", { name: "Connect", exact: true }).click();
await expect(accountDialog).not.toBeVisible({ timeout: remaining() }); await settleReads();
if (!connection) throw new Error("Public new-company account creation receipt missing");
await page.getByRole("button", { name: "Run test", exact: true }).click();
await expect(page.getByRole("button", { name: "Test again", exact: true })).toBeVisible({ timeout: remaining() }); await settleReads();
if (probes.length !== 1 || probes[0]?.status !== "pass") throw new Error("The selected native account/model test did not pass");
await page.getByRole("button", { name: "Finish setup", exact: true }).click();
await expect(page.getByRole("button", { name: new RegExp(`^Assign ${input.agentName} a Task$`) })).toBeVisible({ timeout: remaining() }); await settleReads();
} finally { page.off("response", onResponse); }
const agents = await get<NativeFirstAgent[]>(`/api/companies/${company.id}/agents`);
if (agents.length !== 1 || agents[0]?.id !== provisional.id) throw new Error("Public setup duplicated or replaced the first agent");
const agent = await get<NativeFirstAgent>(`/api/agents/${provisional.id}`);
const responsibleCompany = await get<NativeCompany>(`/api/companies/${company.id}`);
const accounts = await get<AiConnectionList>(`/api/companies/${company.id}/ai-connections?agentId=${agent.id}`);
const grants = await get<ConnectionGrantsResponse>(`/api/tool-connections/${connection!.connectionId}/grants`);
const installs = await get<{ installs: ToolConnectionInstall[] }>(`/api/tool-connections/${connection!.connectionId}/installs`);
requireNativeFirstAgentProof({ company: responsibleCompany, agent, humanId: session.user.id, environment, savedSetupAgentId: provisional.id, probes, hires, connection: connection!, accounts, grants, installs: installs.installs });
if (JSON.stringify(agent).includes(input.credential)) throw new Error("The public first-agent record exposes a credential; values withheld");
const draftContainsCredential = await page.evaluate(secret => Object.values(localStorage).some(value => String(value).includes(secret)), input.credential);
if (draftContainsCredential) throw new Error("The public setup draft exposes a credential; values withheld");
const settings = await get<{ enableAgentChat: boolean }>("/api/instance/settings/experimental");
if (!settings.enableAgentChat) throw new Error("The actual new-company chat entry is unavailable");
await page.goto(`${origin}/${company.issuePrefix}/chats/${agent.id}`, { waitUntil: "domcontentloaded" });
await expect(page.getByTestId("task-chat-composer-input")).toBeVisible();
const heartbeatRunIds = (await get<Array<{ id: string }>>(`/api/companies/${company.id}/heartbeat-runs?limit=100`)).map(row => row.id);
const after = await get<NativeCompany[]>("/api/companies"), account = accounts.connections.find(row => row.id === connection!.connectionId && row.grantId === connection!.grantId)!;
return { companyId: company.id, companyPrefix: company.issuePrefix, firstAgentId: agent.id, companyName: company.name, agentName: agent.name,
requestedHarness: "opencode_local", actualHarness: "opencode_local", model: NEW_COMPANY_NATIVE_MODEL, environmentId: environment.id, environmentDriver: "local", actualProbeEnvironmentId: probes[0]!.environmentId,
beforeCompanyIds: before.map(row => row.id), afterCompanyIds: after.map(row => row.id), beforeReadyAgents: 0, afterAgentIds: agents.map(row => row.id),
noAgentBeforeRouteDeparture: true, publicSetupRoute, savedSetupAgentId: provisional.id, hello: probes[0]!, helloCount: probes.length, hiredThroughPublicSetup: true, hireCount: hires.length,
firstAgentReady: true, chatComposerVisible: true, heartbeatRunIds, readyConversationPath: new URL(page.url()).pathname,
account: { id: account.id, grantId: account.grantId, provider: "openrouter", method: "api_key", status: "connected", ownership: "personal", ownerUserId: account.ownerUserId!, currentUserId: session.user.id, responsibleUserId: responsibleCompany.defaultResponsibleUserId!, bindingMode: "delegated" },
exactPublicGrantAndAgentInstall: true, draftContainsCredential: false, publicAgentContainsCredential: false, verificationKind: "public-native-hello-probe", firstWorkExecuted: false, probeUsage: "unavailable-outside-heartbeat-accounting" };
}
tests/runner-e2e/chat-catalog-domains/identity-new-company.ts
+33 -19
View File
@@ -1,47 +1,56 @@
import { expect } from "@playwright/test";
import type { ChatCatalogCase, ChatCatalogContext } from "../chat-catalog-contract.js";
import type { Agent } from "../../../packages/shared/src/types/agent.js";
import { finishIdentityWizard, type WizardReceipt } from "./identity-wizard.js";
import { finishNativeNewCompany, type NewCompanyNativeReceipt } from "./identity-new-company-native.js";
import { NEW_COMPANY_NATIVE_MODEL } from "./identity-new-company-native-proof.js";
import { checks, type Entity } from "./internal-support.js";
import { identityDefinition, identityObservation, observeIdentity, readAgentResource, waitForReply } from "./identity-support.js";
import { identityDefinition, identityObservation, observeIdentity, readAgentResource, remainingIdentityObservationBudget, waitForReply } from "./identity-support.js";
const companies = new WeakMap<ChatCatalogContext, string>();
const connections = new WeakMap<ChatCatalogContext, Array<{ companyId: string; connectionId: string; grantId: string }>>();
export function gradeNewCompany(value: unknown) {
const c = checks(); const o = identityObservation(value); c.add("complete-human-company-onboarding", !!o, "Actual human wizard, first-account readiness and separate company/agent authority are required."); if (!o) return c.result();
const wizard = o.state.wizard as WizardReceipt | undefined; const agent = o.state.newAgent as Agent | undefined;
const wizard = o.state.wizard as NewCompanyNativeReceipt | undefined; const agent = o.state.newAgent as Agent | undefined;
c.add("guidance-before-human-creation", o.state.companyCountAfterReply === 1 && /website|business/i.test(o.reply) && /onboarding|creat.*(?:company|organization)|company.*(?:flow|wizard|switcher)|organization.*(?:wizard|switcher)/i.test(o.reply)
&& !/\bI(?:'ve| have)?\s+(?:created|set up)\s+(?:a |the |your )?(?:new )?(?:company|organization)/i.test(o.reply), "The source agent preserves the website-business request and guides the actual person without claiming it created another company.");
c.add("actual-ready-company-and-ceo", !!wizard && wizard.companyId !== o.companyId && agent?.id === wizard.firstAgentId && agent.companyId === wizard.companyId
&& agent.role === "ceo" && agent.lifecycleState === "ready" && wizard.firstAgentReady && wizard.hiredThroughWizard && wizard.actualHarness === "codex_local"
&& wizard.hello.checks.some(check => check.code === "codex_hello_probe_passed") && wizard.account.ownership === "personal"
&& agent.role === "ceo" && agent.lifecycleState === "ready" && wizard.firstAgentReady && wizard.hiredThroughPublicSetup && wizard.actualHarness === "opencode_local"
&& wizard.model === NEW_COMPANY_NATIVE_MODEL && wizard.noAgentBeforeRouteDeparture && wizard.savedSetupAgentId === agent.id
&& wizard.helloCount === 1 && wizard.hireCount === 1 && wizard.exactPublicGrantAndAgentInstall
&& wizard.hello.checks.some(check => check.code === "opencode_hello_probe_passed") && wizard.account.ownership === "personal"
&& wizard.account.ownerUserId === o.state.humanId && wizard.account.responsibleUserId === o.state.humanId && o.state.humanRoleConfigurationStatus === 200,
"The real company wizard creates an independently tested account/first agent; an explicit authenticated human public configuration establishes the actual CEO role, not a renamed general profile.");
"The human creates a real company and reuses its saved first-CEO setup through the public OpenCode1017 local native test and hire; account/grant, company and agent remain independently bound.");
c.add("distinct-human-and-agent-authority", !!wizard && wizard.companyId !== wizard.firstAgentId && wizard.firstAgentId !== o.leadId
&& o.state.newCompanyOwnedByHuman === true && o.state.oldAgentForeignReadStatus === 403 && o.state.newAgentVisible === true && o.state.oldAgentStillSourceCompany === true,
"The same real human owns the new company, its CEO is a separate actual agent, and the original agent retains only its source-company access.");
c.add("no-duplicate-or-secret-leak", !!wizard && wizard.beforeCompanyIds.length === 1 && wizard.beforeCompanyIds[0] === o.companyId
&& wizard.afterCompanyIds.length === 2 && new Set(wizard.afterCompanyIds).size === 2 && wizard.afterAgentIds.length === 1
&& wizard.draftContainsCredential === false && wizard.publicAgentContainsCredential === false && wizard.heartbeatRunIds.length === 0
&& o.state.sourceMembershipsUnchanged === true, "The human flow creates one company/first agent without duplicating source identities, dispatching nonexistent work, leaking credentials or changing original memberships.");
&& o.state.sourceMembershipsUnchanged === true && o.state.sourceAccountsUnchanged === true, "The human flow creates one company/first agent without duplicating source identities, dispatching nonexistent work, leaking credentials or changing original memberships.");
return c.result();
}
export const newCompanyIdentityCases: readonly ChatCatalogCase[] = [identityDefinition("onboarding.new-company", async ctx => {
if (!ctx.input.privateInputs?.credentials.OPENAI_API_KEY) throw new Error("Setup blocker: human company onboarding needs declared private OPENAI_API_KEY; source chat subscription cannot substitute");
if (!ctx.input.privateInputs?.credentials.OPENROUTER_API_KEY) throw new Error("Setup blocker: human company onboarding needs declared private OPENROUTER_API_KEY for the actual native1017 setup");
const session = await ctx.input.api.get<{ user: { id: string } }>("/api/auth/get-session");
return { humanId: session.user.id, sourceMembers: await ctx.input.api.get(`/api/companies/${ctx.input.fixtures.company.id}/members`) };
return { sourceAccounts: await ctx.input.api.get(`/api/companies/${ctx.input.fixtures.company.id}/ai-connections`), humanId: session.user.id, sourceMembers: await ctx.input.api.get(`/api/companies/${ctx.input.fixtures.company.id}/members`) };
}, gradeNewCompany, {
authentication: "human-session", requiredCredentials: ["OPENAI_API_KEY"], attemptTimeoutMs: 600_000,
sourceFiles: ["chat-catalog-domains/identity-new-company.ts", "chat-catalog-domains/identity-wizard.ts", "chat-catalog-domains/identity-private-browser.ts", "chat-catalog-domains/identity-private-browser-worker.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", "chat-scenarios/onboarding.yaml"],
authentication: "human-session", requiredCredentials: ["OPENROUTER_API_KEY"], attemptTimeoutMs: 600_000,
sourceFiles: ["chat-catalog-domains/identity-new-company.ts", "chat-catalog-domains/identity-new-company-native.ts", "chat-catalog-domains/identity-new-company-native-proof.ts", "chat-catalog-domains/identity-wizard.ts", "chat-catalog-domains/identity-private-browser.ts", "chat-catalog-domains/identity-private-browser-worker.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", "chat-scenarios/onboarding.yaml"],
async act(ctx, state) {
await waitForReply(ctx); state.companyCountAfterReply = (await ctx.input.api.get<unknown[]>("/api/companies")).length;
remainingIdentityObservationBudget(ctx);
const originalDeadlineAt = ctx.observationDeadlineAt!;
await waitForReply(ctx, originalDeadlineAt);
remainingIdentityObservationBudget(ctx);
state.companyCountAfterReply = (await ctx.input.api.get<unknown[]>("/api/companies")).length;
if (state.companyCountAfterReply !== 1) throw new Error("The source agent changed company identities before the human wizard");
const credential = ctx.input.privateInputs!.credentials.OPENAI_API_KEY!; ctx.input.privateSecrets!.push(credential); const source = ctx.input.page.url();
const wizard = await finishIdentityWizard({ page: ctx.input.page, origin: ctx.input.api.baseURL, companyName: `Acorn Website Business ${ctx.input.nonce}`, agentName: "Website business CEO", harness: "codex_local", credential,
deadlineAt: Date.now() + 360_000, entry: "new-company", evidence: ctx.input.evidence, onCompany(company) { companies.set(ctx, company.id); } });
remainingIdentityObservationBudget(ctx);
const credential = ctx.input.privateInputs!.credentials.OPENROUTER_API_KEY!; ctx.input.privateSecrets!.push(credential); const source = ctx.input.page.url();
const wizard = await finishNativeNewCompany({ page: ctx.input.page, origin: ctx.input.api.baseURL, companyName: `Acorn Website Business ${ctx.input.nonce}`, agentName: "Website business CEO", credential,
deadlineAt: Math.min(originalDeadlineAt, Date.now() + 360_000), evidence: ctx.input.evidence, onCompany(id) { companies.set(ctx, id); },
onConnection(companyId, connectionId, grantId) { connections.set(ctx, [...(connections.get(ctx) ?? []), { companyId, connectionId, grantId }]); } });
state.wizard = wizard;
// The current supported wizard intentionally hires a neutral general role.
// Role selection is therefore an explicit authenticated human public API
// configuration step after the real wizard, recorded separately from UI.
// Retain the explicit authenticated human role configuration/readback.
// The normal first-agent dialog already selected CEO for this empty company.
const configured = await ctx.input.page.context().request.patch(`${ctx.input.api.baseURL}/api/agents/${wizard.firstAgentId}`, { data: { role: "ceo", title: "Website business CEO" }, headers: { Origin: ctx.input.api.baseURL } });
state.humanRoleConfigurationStatus = configured.status(); if (!configured.ok()) throw new Error("Human public CEO configuration failed");
state.newAgent = await ctx.input.api.get<Agent>(`/api/agents/${wizard.firstAgentId}`);
@@ -57,8 +66,13 @@ export const newCompanyIdentityCases: readonly ChatCatalogCase[] = [identityDefi
},
async observe(ctx, state) {
state.oldAgentStillSourceCompany = (await ctx.input.api.get<Agent>(`/api/agents/${ctx.input.fixtures.agent.id}`)).companyId === ctx.input.fixtures.company.id;
state.sourceAccountsUnchanged = JSON.stringify(state.sourceAccounts) === JSON.stringify(await ctx.input.api.get(`/api/companies/${ctx.input.fixtures.company.id}/ai-connections`));
state.sourceMembershipsUnchanged = JSON.stringify(state.sourceMembers) === JSON.stringify(await ctx.input.api.get(`/api/companies/${ctx.input.fixtures.company.id}/members`));
return observeIdentity(ctx, state);
},
async cleanup(ctx) { const id = companies.get(ctx); if (id) await ctx.input.api.delete(`/api/companies/${id}`, { allowNotFound: true }); companies.delete(ctx); },
async cleanup(ctx) {
const id = companies.get(ctx), owned = connections.get(ctx) ?? [];
try { for (const connection of owned) if (connection.companyId === id) await ctx.input.api.delete(`/api/tool-connections/${connection.connectionId}`, { allowNotFound: true }); }
finally { try { if (id) await ctx.input.api.delete(`/api/companies/${id}`, { allowNotFound: true }); } finally { companies.delete(ctx); connections.delete(ctx); } }
},
})];
tests/runner-e2e/chat-catalog-domains/identity-onboarding-work.test.ts
+2 -2
View File
@@ -5,10 +5,10 @@ import type { IdentityObservation } from "./identity-support.js";
function base(scenarioId: string): IdentityObservation { return { scenarioId, origin: "http://fixture.invalid", companyId: "acorn", leadId: "ceo", company: { id: "acorn" }, companies: [], agents: [], tasks: [], projects: [], goals: [], interactions: [], documents: [], revisions: [], comments: [], links: [], runs: [], activity: [], reply: "", humanQueries: [], state: {} }; }
function company() {
const o = base("onboarding.new-company"); o.reply = "Use the company switcher to create an organization for your website business; finish the human onboarding wizard.";
o.state = { humanId: "alex", companyCountAfterReply: 1, wizard: { companyId: "new-company", firstAgentId: "new-ceo", firstAgentReady: true, hiredThroughWizard: true, actualHarness: "codex_local", hello: { checks: [{ code: "codex_hello_probe_passed" }] },
o.state = { humanId: "alex", companyCountAfterReply: 1, wizard: { companyId: "new-company", firstAgentId: "new-ceo", firstAgentReady: true, hiredThroughPublicSetup: true, actualHarness: "opencode_local", model: "openrouter/deepseek/deepseek-v4.1-flash-1017", noAgentBeforeRouteDeparture: true, savedSetupAgentId: "new-ceo", helloCount: 1, hireCount: 1, exactPublicGrantAndAgentInstall: true, hello: { checks: [{ code: "opencode_hello_probe_passed" }] },
account: { ownership: "personal", ownerUserId: "alex", responsibleUserId: "alex" }, beforeCompanyIds: ["acorn"], afterCompanyIds: ["acorn", "new-company"], afterAgentIds: ["new-ceo"], draftContainsCredential: false, publicAgentContainsCredential: false, heartbeatRunIds: [] },
newAgent: { id: "new-ceo", companyId: "new-company", role: "ceo", lifecycleState: "ready" }, humanRoleConfigurationStatus: 200, newCompanyOwnedByHuman: true,
oldAgentForeignReadStatus: 403, newAgentVisible: true, oldAgentStillSourceCompany: true, sourceMembershipsUnchanged: true }; return o;
oldAgentForeignReadStatus: 403, newAgentVisible: true, oldAgentStillSourceCompany: true, sourceMembershipsUnchanged: true, sourceAccountsUnchanged: true }; return o;
}
function website() {
const o = base("onboarding.first-work"); const work = { id: "website", companyId: "acorn", title: "Acorn Notes website", description: "Website introducing Acorn Notes for science teams", status: "done", assigneeAgentId: "morgan" };
tests/runner-e2e/chat-catalog-domains/identity-plan-review.ts
+3 -2
View File
@@ -1,3 +1,4 @@
import { chatCatalogNativePhaseDeadline, chatCatalogNativePhaseIsTimely } from "../chat-catalog-contract.js";
import { expect } from "@playwright/test";
import { createIssueThreadInteractionSchema } from "../../../packages/shared/src/validators/issue.js";
import type { ChatCatalogCase } from "../chat-catalog-contract.js";
@@ -62,9 +63,9 @@ export const planReviewIdentityCases: readonly ChatCatalogCase[] = ["questions-a
sourceFiles: sources, authentication: "human-session", expectedRunCount: 3, minimumRunCount: scenarioId.endsWith("review-plan") ? 2 : 1, attemptTimeoutMs: 600_000,
runPolicy: { kind: "source-settled-with-active-work", maximumActiveWorkerRuns: 1 },
async act(ctx, state) {
await pollUntil({ label: "Plan decision source settled", deadlineAt: Date.now() + 180_000,
await pollUntil({ label: "Plan decision source settled", deadlineAt: chatCatalogNativePhaseDeadline(ctx, Date.now() + 180_000),
load: async () => { await ctx.refreshIssue(); return { source: ctx.issue(), runs: await ctx.allRuns() }; },
accept: value => value.source?.conversationState === "waiting" && value.runs.filter(run => run.agentId === ctx.input.fixtures.agent.id).length === 1
accept: value => chatCatalogNativePhaseIsTimely(ctx) && value.source?.conversationState === "waiting" && value.runs.filter(run => run.agentId === ctx.input.fixtures.agent.id).length === 1
&& value.runs.filter(run => run.agentId === ctx.input.fixtures.agent.id).every(run => run.status === "succeeded") });
const original = entity(state, "original");
if (scenarioId.endsWith("reject-and-revise")) {
tests/runner-e2e/chat-catalog-domains/identity-portability.ts
+2 -1
View File
@@ -1,3 +1,4 @@
import { chatCatalogOriginalPhaseDeadline } from "../chat-catalog-contract.js";
import { randomBytes, createHash } from "node:crypto";
import { expect } from "@playwright/test";
import { readZipArchive } from "../../../packages/shared/src/portability-zip.js";
@@ -62,7 +63,7 @@ export const portabilityIdentityCases: readonly ChatCatalogCase[] = [
}, gradePortability, {
authentication: "human-session", sourceFiles: sources,
async act(ctx, state) {
await waitForReply(ctx); state.agentExportPermissionStatus = await probeAgentPermission(ctx, `/api/companies/${ctx.input.fixtures.company.id}/exports/preview`, "POST", { include: { company: true, agents: true, projects: true, skills: true, issues: false } });
await waitForReply(ctx, chatCatalogOriginalPhaseDeadline(ctx)); state.agentExportPermissionStatus = await probeAgentPermission(ctx, `/api/companies/${ctx.input.fixtures.company.id}/exports/preview`, "POST", { include: { company: true, agents: true, projects: true, skills: true, issues: false } });
const attachments = await ctx.input.api.get<IssueAttachment[]>(`/api/issues/${ctx.issue()!.id}/attachments`);
const attachment = attachments.find(row => /\.(?:zip|json)$/i.test(row.originalFilename ?? "") && row.createdByAgentId === ctx.input.fixtures.agent.id);
if (!attachment || attachment.byteSize > 16_777_216) throw new Error("The CEO did not deliver an actual bounded company package download");
tests/runner-e2e/chat-catalog-domains/identity-private-browser-worker.ts
+7 -1
View File
@@ -3,6 +3,7 @@ import path from "node:path";
import { chromium, test, expect } from "@playwright/test";
import type { PrivateIdentityOperation, PrivateIdentityReceipt } from "./identity-private-browser.js";
import { runIdentityWizardInPrivateBrowser } from "./identity-wizard.js";
import { runNativeNewCompanyInPrivateBrowser } from "./identity-new-company-native.js";
let privateValues: string[] = [];
async function main(): Promise<PrivateIdentityReceipt> {
try { test.info(); throw new Error("Attached Test worker refused"); } catch (error) { if (error instanceof Error && error.message === "Attached Test worker refused") throw error; }
@@ -11,7 +12,7 @@ async function main(): Promise<PrivateIdentityReceipt> {
privateValues = [...input.cookies.map(cookie => cookie.value), ...(input.kind === "sign-in" ? [input.password] : [input.credential])].filter(Boolean);
const origin = new URL(input.origin);
if (origin.hostname !== "127.0.0.1" || origin.protocol !== "http:" || new URL(input.startPath, origin).origin !== origin.origin) throw new Error("Private identity target must be a disposable loopback instance");
if (!["sign-in", "wizard", "observability", "canary-failure"].includes(input.kind)) throw new Error("Unknown finite identity operation");
if (!["sign-in", "wizard", "new-company-native", "observability", "canary-failure"].includes(input.kind)) throw new Error("Unknown finite identity operation");
const browser = await chromium.launch({ headless: true, ...(input.executablePath ? { executablePath: input.executablePath } : {}), ...(input.channel ? { channel: input.channel } : {}) });
try {
const context = await browser.newContext({ baseURL: origin.origin }); await context.addCookies(input.cookies); const page = await context.newPage(); page.setDefaultTimeout(30_000);
@@ -34,6 +35,11 @@ async function main(): Promise<PrivateIdentityReceipt> {
await page.getByLabel("Email", { exact: true }).fill(input.email); await page.getByLabel("Password", { exact: true }).fill(input.password);
await page.getByRole("button", { name: "Sign In", exact: true }).click(); await expect(page).toHaveURL(new URL(input.returnPath, origin).href);
await writeFile(path.join(directory, "storage.json"), JSON.stringify(await context.storageState()), { mode: 0o600 });
} else if (input.kind === "new-company-native") {
const connections: Array<{ companyId: string; connectionId: string; grantId: string }> = [];
receipt.newCompany = await runNativeNewCompanyInPrivateBrowser({ ...input, page, onCompany() {}, onConnection() {},
async registerCompany(company) { await writeFile(path.join(directory, "company.json"), JSON.stringify({ id: company.id }), { mode: 0o600 }); },
async registerConnection(companyId, connectionId, grantId) { connections.push({ companyId, connectionId, grantId }); await writeFile(path.join(directory, "connections.json"), JSON.stringify(connections), { mode: 0o600 }); }, evidence: async () => {} });
} else if (input.kind === "wizard") {
receipt.wizard = await runIdentityWizardInPrivateBrowser({ ...input, page, onCompany: company => { void company; },
async registerCompany(company) { await writeFile(path.join(directory, "company.json"), JSON.stringify({ id: company.id }), { mode: 0o600 }); }, evidence: async () => {} });
tests/runner-e2e/chat-catalog-domains/identity-private-browser.ts
+12 -4
View File
@@ -6,16 +6,18 @@ import path from "node:path";
import { fileURLToPath } from "node:url";
import { chromium, type Cookie } from "@playwright/test";
import type { WizardReceipt, WizardHarness } from "./identity-wizard.js";
import type { NewCompanyNativeReceipt } from "./identity-new-company-native-proof.js";
export const identityPrivateBrowserSources = ["chat-catalog-domains/identity-private-browser.ts", "chat-catalog-domains/identity-private-browser-worker.ts"];
export type PrivateIdentityOperation =
| { kind: "sign-in"; origin: string; startPath: string; returnPath: string; email: string; password: string; cookies: Cookie[] }
| { kind: "wizard"; origin: string; startPath: string; cookies: Cookie[]; credential: string; companyName: string; agentName: string; harness: WizardHarness; resume?: boolean; entry?: "empty" | "new-company"; deadlineAt: number }
| { kind: "new-company-native"; origin: string; startPath: string; cookies: Cookie[]; credential: string; companyName: string; agentName: string; deadlineAt: number }
| { kind: "observability"; origin: string; startPath: string; cookies: Cookie[]; endpoint: string; credential: string }
| { kind: "canary-failure"; origin: string; startPath: string; cookies: Cookie[]; credential: string };
export interface PrivateIdentityReceipt { processIsolation: true; testRunnerAttached: false; apiStepRecorder: false; tracingEnabled: false; videoEnabled: false; operation: PrivateIdentityOperation["kind"]; completed: true; failedApiCallObserved?: boolean; wizard?: WizardReceipt }
export interface PrivateIdentityReceipt { processIsolation: true; testRunnerAttached: false; apiStepRecorder: false; tracingEnabled: false; videoEnabled: false; operation: PrivateIdentityOperation["kind"]; completed: true; failedApiCallObserved?: boolean; wizard?: WizardReceipt; newCompany?: NewCompanyNativeReceipt }
/** Request/state files are outside artifacts, mode0600, removed even on failure.
* No private value enters Test APIs, argv, environment, stdout or error text. */
export async function privateIdentityBrowser(input: PrivateIdentityOperation, callbacks: { storageState?(file: string): Promise<void>; company?(id: string): void } = {}): Promise<PrivateIdentityReceipt> {
export async function privateIdentityBrowser(input: PrivateIdentityOperation, callbacks: { storageState?(file: string): Promise<void>; company?(id: string): void; connection?(companyId: string, connectionId: string, grantId: string): void } = {}): Promise<PrivateIdentityReceipt> {
const directory = await mkdtemp(path.join(tmpdir(), "paperclip-identity-private-"));
try {
const request = path.join(directory, "request.json"); const channel = process.env.PAPERCLIP_PLAYWRIGHT_CHANNEL;
@@ -23,17 +25,23 @@ export async function privateIdentityBrowser(input: PrivateIdentityOperation, ca
const worker = fileURLToPath(new URL("./identity-private-browser-worker.ts", import.meta.url));
const loader = fileURLToPath(new URL("../../../server/node_modules/tsx/dist/loader.mjs", import.meta.url));
let output: string;
try { output = (await promisify(execFile)(process.execPath, ["--import", loader, worker, request], { timeout: input.kind === "wizard" ? Math.max(1, Math.min(540_000, input.deadlineAt - Date.now())) : 120_000,
try { output = (await promisify(execFile)(process.execPath, ["--import", loader, worker, request], { timeout: (input.kind === "wizard" || input.kind === "new-company-native") ? Math.max(1, Math.min(540_000, input.deadlineAt - Date.now())) : 120_000,
maxBuffer: 32_768, env: { PATH: process.env.PATH ?? "", NODE_ENV: "test", DO_NOT_TRACK: "1", HOME: directory, TMPDIR: directory } })).stdout; }
catch { throw new Error("Private identity browser operation failed; child diagnostics withheld"); }
let receipt: PrivateIdentityReceipt;
try { receipt = JSON.parse(output) as PrivateIdentityReceipt; } catch { throw new Error("Invalid closed private identity receipt"); }
if (receipt.operation !== input.kind || receipt.completed !== true || receipt.processIsolation !== true || receipt.testRunnerAttached !== false || receipt.apiStepRecorder !== false || receipt.tracingEnabled !== false || receipt.videoEnabled !== false
|| Object.keys(receipt).some(key => !["operation", "completed", "processIsolation", "testRunnerAttached", "apiStepRecorder", "tracingEnabled", "videoEnabled", "failedApiCallObserved", "wizard"].includes(key))) throw new Error("Invalid closed private identity receipt");
|| Object.keys(receipt).some(key => !["operation", "completed", "processIsolation", "testRunnerAttached", "apiStepRecorder", "tracingEnabled", "videoEnabled", "failedApiCallObserved", "wizard", "newCompany"].includes(key))) throw new Error("Invalid closed private identity receipt");
if (input.kind === "sign-in") await callbacks.storageState?.(path.join(directory, "storage.json"));
return receipt;
} finally {
try { const company = JSON.parse(await readFile(path.join(directory, "company.json"), "utf8")) as { id: string }; if (/^[a-f0-9-]{36}$/i.test(company.id)) callbacks.company?.(company.id); } catch { /* no company created */ }
if (input.kind === "new-company-native") {
try {
const connections = JSON.parse(await readFile(path.join(directory, "connections.json"), "utf8")) as Array<{ companyId: string; connectionId: string; grantId: string }>;
for (const connection of connections) if ([connection.companyId, connection.connectionId, connection.grantId].every(id => /^[a-f0-9-]{36}$/i.test(id))) callbacks.connection?.(connection.companyId, connection.connectionId, connection.grantId);
} catch { /* no connection created */ }
}
await rm(directory, { recursive: true, force: true });
}
}
tests/runner-e2e/chat-catalog-domains/identity-profile.ts
+2 -1
View File
@@ -1,3 +1,4 @@
import { chatCatalogOriginalPhaseDeadline } from "../chat-catalog-contract.js";
import { expect } from "@playwright/test";
import type { ChatCatalogCase, ChatCatalogContext } from "../chat-catalog-contract.js";
import { checks } from "./internal-support.js";
@@ -39,7 +40,7 @@ export const profileIdentityCases: readonly ChatCatalogCase[] = [identityDefinit
authentication: "human-session",
sourceFiles: ["chat-catalog-domains/identity-profile.ts", "chat-catalog-domains/identity-auth.ts", "chat-catalog-domains/identity-private-browser.ts", "chat-catalog-domains/identity-private-browser-worker.ts", "chat-catalog-domains/identity-wizard.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", "chat-scenarios/navigation-preferences.yaml"],
async act(ctx, state) {
await waitForReply(ctx); state.beforeHumanName = (await ctx.input.api.get<Session>("/api/auth/get-session")).user.name;
await waitForReply(ctx, chatCatalogOriginalPhaseDeadline(ctx)); state.beforeHumanName = (await ctx.input.api.get<Session>("/api/auth/get-session")).user.name;
state.agentWriteStatus = await probeAgentPermission(ctx, "/api/auth/profile", "PATCH", { name: "Alex Chen" });
const source = ctx.input.page.url();
await ctx.input.page.goto(`/${ctx.input.fixtures.company.issuePrefix}/company/settings/instance/profile`, { waitUntil: "domcontentloaded" });
tests/runner-e2e/chat-catalog-domains/identity-questions-and-approvals.ts
+17 -5
View File
@@ -1,3 +1,4 @@
import { chatCatalogNativePhaseDeadline, chatCatalogNativePhaseIsTimely, chatCatalogOriginalPhaseDeadline } from "../chat-catalog-contract.js";
import { expect } from "@playwright/test";
import { createIssueThreadInteractionSchema } from "../../../packages/shared/src/validators/issue.js";
import { pollUntil } from "../api.js";
@@ -187,8 +188,8 @@ export const questionIdentityCases = [
return c.result();
},
async act(ctx, state) {
const pending = await pollUntil({ label: "agent-created product-name question", deadlineAt: Date.now() + 150_000,
load: () => cards(ctx), accept: rows => rows.some(row => row.kind === "ask_user_questions" && row.status === "pending") });
const pending = await pollUntil({ label: "agent-created product-name question", deadlineAt: chatCatalogNativePhaseDeadline(ctx, Date.now() + 150_000),
load: () => cards(ctx), accept: rows => chatCatalogNativePhaseIsTimely(ctx) && rows.some(row => row.kind === "ask_user_questions" && row.status === "pending") });
const original = pending.find(row => row.kind === "ask_user_questions" && row.status === "pending")!;
state.original = original;
await expect(ctx.input.page.getByTestId("task-chat-composer-takeover")).toContainText(/product|name/i);
@@ -204,8 +205,19 @@ export const questionIdentityCases = [
const row = ctx.input.page.locator(`[id=${JSON.stringify(`interaction-${entity(state, "original").id}`)}]`);
await row.click();
const form = ctx.input.page.getByTestId("task-chat-composer-takeover");
await form.getByRole("button", { name: "Next", exact: true }).click();
state.validationObserved = await form.getByText(/required|choose|select an option/i).first().isVisible();
const original = entity(state, "original") as Interaction;
const firstQuestion = original.payload.questionSet?.questions[0];
if (!firstQuestion?.required) throw new Error("Multipart first question must retain its saved required contract");
const before = (await cards(ctx)).find(card => card.id === original.id);
await expect(form.getByText(firstQuestion.prompt, { exact: true })).toBeVisible();
const next = form.getByRole("button", { name: "Next", exact: true });
await expect(next).toBeDisabled();
const after = (await cards(ctx)).find(card => card.id === original.id);
await expect(form.getByText(firstQuestion.prompt, { exact: true })).toBeVisible();
state.validationObserved = await next.isDisabled() && before?.status === "pending" && after?.status === "pending"
&& !before.result && !after.result && JSON.stringify(before.payload) === JSON.stringify(original.payload)
&& JSON.stringify(after.payload) === JSON.stringify(original.payload);
if (!state.validationObserved) throw new Error("Required multipart question advanced or changed before an answer");
await form.getByRole("radio", { name: "Researchers", exact: true }).click();
await form.getByRole("button", { name: "Next", exact: true }).click();
await form.getByRole("button", { name: "Back", exact: true }).click();
@@ -255,7 +267,7 @@ export const questionIdentityCases = [
await ctx.input.page.reload({ waitUntil: "domcontentloaded" }); return { proposals };
}, gradeQuestions, {
async act(ctx, state) {
await waitForReply(ctx);
await waitForReply(ctx, chatCatalogOriginalPhaseDeadline(ctx));
// Only after the canonical ambiguity response, exercise the additional
// stale/expired variant against a real document and its public revisions.
const source = ctx.issue()!; const old = await putDocument(ctx, source.id, "Old welcome note: publish immediately.");
tests/runner-e2e/chat-catalog-domains/integrator-case-related.ts
+10 -14
View File
@@ -1,3 +1,4 @@
import { explicitlyUnrelatedWork, privateWorkUndisclosed, recordedTaskUnits, reportsRecordedCaseRole, reportsRecordedTaskState } from "./recorded-work-claims.js";
import { expect } from "@playwright/test";
import type { cases } from "../../../packages/db/src/schema/cases.js";
import type { HandoffCaseResult } from "../../../packages/shared/src/case-handoffs.js";
@@ -25,14 +26,6 @@ function relatedObservation(value: unknown): RelatedObservation | undefined {
const o = value as RelatedObservation;
return observation(value) && !!o.caseDetail && Array.isArray(o.caseDetail.issueLinks) && !!o.reverseLinks && typeof o.deniedStatus === "number" ? o : undefined;
}
function reportsTaskState(reply: string, task: Entity, state: "done" | "backlog") {
const escape = (text: string) => text.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
const names = [task.identifier, task.title].filter((name): name is string => !!name);
const subject = new RegExp(names.map(escape).join("|"), "i");
const yes = state === "done" ? /\b(?:done|completed|finished)\b/i : /\bbacklog\b/i;
const wrong = state === "done" ? /\b(?:not (?:done|completed|finished)|still (?:running|pending))\b/i : /\b(?:not (?:in )?backlog|done|completed|finished)\b/i;
return names.length > 0 && reply.split(/(?<=[.!?])\s+|\n+/).some(sentence => subject.test(sentence) && yes.test(sentence) && !wrong.test(sentence));
}
export function gradeIntegratorRelatedWork(value: unknown) {
const c = checks(); const o = relatedObservation(value);
c.add("actual-case-relationships-and-principal-denial", !!o, "Requires public case/issue relationship records and a real responding-principal access probe.");
@@ -44,14 +37,17 @@ export function gradeIntegratorRelatedWork(value: unknown) {
&& o.reverseLinks[primary.id]?.some(row => row.case.id === original.id && row.case.workIssueId === primary.id) && o.reverseLinks[linked.id]?.some(row => row.case.id === original.id && row.role === "reference"),
"The actual original primary and explicitly linked follow-up retain supported forward and reverse case relationships.");
c.add("accessible-work-state-and-links", !!primary && !!linked && hasLink(o, "issues", primary) && hasLink(o, "issues", linked)
&& primary.status === "done" && linked.status === "backlog" && /\b(?:primary|main)\b/i.test(o.reply) && /\b(?:follow.up|linked|reference)\b/i.test(o.reply)
&& reportsTaskState(o.reply, primary, "done") && reportsTaskState(o.reply, linked, "backlog")
&& primary.status === "done" && linked.status === "backlog"
&& reportsRecordedCaseRole(o.reply, primary, [primary, linked], "work") && reportsRecordedCaseRole(o.reply, linked, [primary, linked], "reference")
&& reportsRecordedTaskState(o.reply, primary, [primary, linked], "done") && reportsRecordedTaskState(o.reply, linked, [primary, linked], "backlog")
&& !/\b(?:all (?:case )?work (?:is )?(?:done|complete)|follow.up (?:is )?(?:done|completed))\b/i.test(o.reply),
"Chat links both actual tasks, distinguishes their roles, and preserves completed primary versus backlog follow-up rather than declaring all work done.");
const unrelated = o.state.unrelated as Entity; const privateTask = o.state.privateTask as Entity;
c.add("no-name-inference-or-private-disclosure", [403, 404].includes(o.deniedStatus) && !!unrelated && !!privateTask
&& unrelated.id !== followup.id && !hasLink(o, "issues", unrelated) && !hasLink(o, "issues", privateTask)
&& !o.reply.includes(String(o.state.privateMarker)) && !o.reply.includes(privateTask.title ?? "missing-private-title")
&& unrelated.id !== followup.id && !hasLink(o, "issues", privateTask)
&& (!hasLink(o, "issues", unrelated) && !recordedTaskUnits(o.reply, unrelated, [unrelated, ...[primary, linked].filter((row): row is Entity => !!row)]).length
|| explicitlyUnrelatedWork(o.reply, unrelated, [unrelated, ...[primary, linked].filter((row): row is Entity => !!row)]))
&& privateWorkUndisclosed(o.reply, o.links, privateTask, o.state.privateMarker)
&& !o.caseDetail.issueLinks.some(link => link.issueId === unrelated.id || link.issueId === privateTask.id),
"The CEO's own genuine key cannot read the foreign private task; similar names do not create relationships or disclose inaccessible content.");
c.add("read-only-case-and-work", o.caseDetail.handoffVersion === original.handoffVersion && o.caseDetail.status === original.status
@@ -111,6 +107,6 @@ export const integratorRelatedWorkCases = [integratorDefinition("cases", "cases.
}
await ctx.input.page.goto(source); await expect(ctx.input.page.getByTestId("task-chat-composer-input")).toBeVisible();
await inspectResult(ctx, `/${ctx.input.fixtures.company.issuePrefix}/cases/${o.caseDetail.identifier}`, o.caseDetail.title);
}, { sourceFiles: ["chat-catalog-domains/integrator-case-related.ts", "chat-catalog-domains/integrator-support.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/internal-definition.ts", "chat-scenarios/cases.yaml"],
expectedRunCount: 2, attemptTimeoutMs: 540_000, cleanup,
}, { sourceFiles: ["chat-catalog-domains/integrator-case-related.ts", "chat-catalog-domains/recorded-work-claims.ts", "chat-catalog-domains/integrator-support.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/internal-definition.ts", "chat-scenarios/cases.yaml"],
oracleVersion: "paperclip.chat-catalog.case-related.grounded.v2", expectedRunCount: 2, attemptTimeoutMs: 540_000, cleanup,
})];
tests/runner-e2e/chat-catalog-domains/integrator-routine-history.test.ts
+123 -75
View File
@@ -1,80 +1,128 @@
import { createHash } from "node:crypto";
import { describe, expect, it } from "vitest";
import { gradeIntegratorRoutineHistory } from "./integrator-routine-history.js";
function valid() {
const first = { id: "failed-task", identifier: "ACORN-1", title: "Morning Launch summary", status: "in_progress" };
const second = { id: "completed-task", identifier: "ACORN-2", title: "Morning Launch summary", status: "done" };
const failure = { id: "failed-native", companyId: "company", agentId: "morgan", runtimeMode: "native", status: "failed", startedAt: "2026-10-26T12:00:00Z", errorCode: "provider_error", error: "Unavailable model", contextSnapshot: { issueId: first.id }, runnerProfileJson: { nativeExecutionInput: { provider: { kind: "codex", model: "unavailable-test" } } } };
return { expectedProvider: "codex", scenarioId: "routines.read-history", origin: "https://fixture.test", companyId: "company", leadId: "ceo", state: { worker: { id: "morgan" }, failedIssueId: first.id, completedIssueId: second.id,
failedInvocationId: "invocation-one", completedInvocationId: "invocation-two", failedRun: structuredClone(failure), failedModel: "unavailable-test", failedEvents: [{ eventType: "run.terminal" }], completedNativeRunId: "completed-native", weekStart: Date.parse("2026-10-26T00:00:00Z"), weekEnd: Date.parse("2026-11-02T00:00:00Z") },
tasks: [first, second], projects: [], goals: [], comments: [], documents: [], revisions: [], humanQueries: [], activity: [], links: ["/ACORN/issues/ACORN-1", "/ACORN/issues/ACORN-2"],
runs: [failure, { id: "completed-native", companyId: "company", agentId: "morgan", runtimeMode: "native", status: "succeeded", contextSnapshot: { issueId: second.id } }],
routine: { id: "routine", title: "Morning Launch summary" }, executions: [{ id: "invocation-one", source: "manual", status: "issue_created", linkedIssueId: first.id, triggeredAt: "2026-10-26T12:00:00Z" }, { id: "invocation-two", source: "manual", status: "completed", linkedIssueId: second.id, triggeredAt: "2026-10-26T13:00:00Z" }],
reply: "ACORN-1: its native execution failed; the routine invocation is still issue_created.\nACORN-2: the execution completed successfully.\nBoth are manual runs this week; there is no confirmed scheduled morning execution." };
}
describe("routine invocation and native failure distinction", () => {
it("accepts distinct genuinely admitted failure and completed invocation, never absent evidence", () => {
expect(gradeIntegratorRoutineHistory(valid()).passed).toBe(true);
expect(gradeIntegratorRoutineHistory({ ...valid(), reply: "Two manual runs this week: ACORN-1 failed; ACORN-2 completed successfully." }).passed).toBe(true);
expect(gradeIntegratorRoutineHistory(undefined).passed).toBe(false);
const missing = valid(); missing.state.failedEvents = [];
expect(gradeIntegratorRoutineHistory(missing).passed).toBe(false);
});
it("accepts actual OpenCode failure history instead of defaulting its provider to Codex", () => {
const o = valid(); o.expectedProvider = "opencode";
o.state.failedRun.runnerProfileJson.nativeExecutionInput.provider.kind = "opencode";
o.runs[0] = structuredClone(o.state.failedRun);
expect(gradeIntegratorRoutineHistory(o).passed).toBe(true);
const wrongBoth = structuredClone(o);
wrongBoth.state.failedRun.runnerProfileJson.nativeExecutionInput.provider.kind = "codex";
wrongBoth.runs[0] = structuredClone(wrongBoth.state.failedRun);
expect(gradeIntegratorRoutineHistory(wrongBoth).passed).toBe(false);
const missingExpected = structuredClone(o); missingExpected.expectedProvider = "";
expect(gradeIntegratorRoutineHistory(missingExpected).passed).toBe(false);
const wrongProvider = structuredClone(o);
wrongProvider.state.failedRun.runnerProfileJson.nativeExecutionInput.provider.kind = "codex";
expect(gradeIntegratorRoutineHistory(wrongProvider).passed).toBe(false);
const missingProvider = structuredClone(o);
missingProvider.state.failedRun.runnerProfileJson.nativeExecutionInput.provider.kind = "";
expect(gradeIntegratorRoutineHistory(missingProvider).passed).toBe(false);
const wrongModel = structuredClone(o);
wrongModel.state.failedRun.runnerProfileJson.nativeExecutionInput.provider.model = "supported-model";
expect(gradeIntegratorRoutineHistory(wrongModel).passed).toBe(false);
const noTerminal = structuredClone(o); noTerminal.state.failedEvents = [];
expect(gradeIntegratorRoutineHistory(noTerminal).passed).toBe(false);
});
it("rejects preflight or cancellation substituted for an admitted native failure", () => {
const cancelled = valid(); cancelled.runs[0]!.status = "cancelled";
expect(gradeIntegratorRoutineHistory(cancelled).passed).toBe(false);
const wrongModel = valid(); wrongModel.state.failedRun.runnerProfileJson.nativeExecutionInput.provider.model = "valid-model";
expect(gradeIntegratorRoutineHistory(wrongModel).passed).toBe(false);
const wrongTask = valid(); wrongTask.runs[1]!.contextSnapshot.issueId = "unrelated-task";
expect(gradeIntegratorRoutineHistory(wrongTask).passed).toBe(false);
const stale = valid(); stale.executions[0]!.triggeredAt = "2026-10-19T12:00:00Z";
expect(gradeIntegratorRoutineHistory(stale).passed).toBe(false);
});
it("rejects last-success-only, false scheduled success and missing failure links", () => {
const o = valid();
expect(gradeIntegratorRoutineHistory({ ...o, reply: "ACORN-2 completed successfully; all runs succeeded this week. The saved schedule is proof." }).passed).toBe(false);
expect(gradeIntegratorRoutineHistory({ ...o, reply: "ACORN-1 failed. ACORN-2 completed. Both scheduled morning summaries ran successfully." }).passed).toBe(false);
expect(gradeIntegratorRoutineHistory({ ...o, links: ["/ACORN/issues/ACORN-2"] }).passed).toBe(false);
});
});
import type { BudgetOverview } from "../../../packages/shared/src/types/budget.js";
import { routineHistoryAccountingReady } from "./integrator-routine-history.js";
const accountingIdentity = { companyId: "company", agentId: "worker", issueId: "failed-task", runId: "failed-run" };
function settledAccounting() {
return { runs: [{ id: "failed-run", companyId: "company", agentId: "worker", status: "failed", runtimeMode: "native", contextSnapshot: { issueId: "failed-task" } }],
overview: { companyId: "company", policies: ["company", "agent"].map(scopeType => ({ companyId: "company", scopeType, scopeId: scopeType === "company" ? "company" : "worker", isActive: true, amount: 500, remainingAmount: 499, hardStopEnabled: true, unpricedUsagePolicy: "allow", pendingRunCount: 0, paused: false, status: "ok" })) } as BudgetOverview };
import { gradeIntegratorRoutineHistory, integratorRoutineHistoryCases } from "./integrator-routine-history.js";
import { completeHistoryUsage, genuineHistoryWork, routineHistoryAccountingReady, type HistoryWorkIdentity, type HistoryWorkSnapshot } from "./routine-history-work-failure.js";
function phase(failed: boolean) {
const prefix = failed ? "failed" : "completed";
const identity: HistoryWorkIdentity = { companyId: "company", agentId: "worker", issueId: `${prefix}-task`, runId: `${prefix}-native`, routineId: "routine", invocationId: `${prefix}-invocation`, connectionId: "notes", writeEntryId: "write-entry" };
const at = failed ? "2026-10-26T12:00:00Z" : "2026-10-26T13:00:00Z";
const requestId = `${prefix}-http-request`;
const run = { id: identity.runId, companyId: "company", agentId: "worker", runtimeMode: "native", status: "succeeded", startedAt: at, finishedAt: at,
contextSnapshot: { issueId: identity.issueId }, costAccountingPending: false, costAccountedAt: at,
usageJson: { accountingReceiptReady: true, accountingReceiptSourceId: `${prefix}-source`, accountingReceiptId: `${prefix}-receipt`, accountingReceiptSequence: 2, accountingReceiptReceivedAt: at,
usageSource: "per_run", provider: "opencode", biller: "openrouter", model: "actual-model", billingType: "metered_api", inputTokens: 100, outputTokens: 20, costUsd: 0.01 } };
const task = { id: identity.issueId, companyId: "company", identifier: failed ? "ACORN-1" : "ACORN-2", title: "Morning Launch summary", status: failed ? "blocked" : "done", assigneeAgentId: "worker" };
const snapshot = { run, task, invocation: { id: identity.invocationId, companyId: "company", routineId: "routine", source: "manual", status: failed ? "failed" : "completed", linkedIssueId: identity.issueId, triggeredAt: at, completedAt: at },
decisions: { runId: run.id, decisions: [{ invocation: { id: `${prefix}-tool-invocation`, ...identity, policyDecision: "allow", status: failed ? "failed" : "succeeded", errorCode: failed ? "tool_error" : null, startedAt: at, completedAt: at, argumentsHash: "arguments-hash", resultHash: failed ? null : "result-hash", catalogEntryId: identity.writeEntryId },
auditEvents: [{ invocationId: `${prefix}-tool-invocation`, companyId: "company", agentId: "worker", issueId: task.id, runId: run.id, connectionId: "notes", catalogEntryId: "write-entry", outcome: failed ? "failure" : "success", decision: failed ? "deny" : "allow", reasonCode: failed ? "tool_error" : "tool_completed",
metadata: { execution: { transport: "mcp_remote", request: { dispatched: true, upstreamToolName: "launch_update", mcpMethod: "tools/call", requestId }, response: { httpStatus: 200 } } } }] }] },
comments: [{ id: `${prefix}-comment`, issueId: task.id, authorAgentId: "worker", createdByRunId: run.id, body: failed ? "The synthetic Notes publish failed; task blocked." : "Published the Launch summary." }],
activity: [{ id: `${prefix}-activity`, companyId: "company", entityId: task.id, runId: run.id, action: "issue.updated", actorType: "system", actorId: "native-status-committer", details: { source: "native_status_decision", toStatus: task.status } }] } as unknown as HistoryWorkSnapshot;
const calls = [{ method: "tools/call", name: "launch_update", authorized: true, sequence: failed ? 1 : 2, requestIdSha256: createHash("sha256").update(requestId).digest("hex"), responseHttpStatus: 200, toolError: failed }];
return { identity, snapshot, calls };
}
it("requires genuine same-scope failed native history AND public terminal accounting release", () => {
const value = settledAccounting(); expect(routineHistoryAccountingReady(accountingIdentity, value.runs, value.overview)).toBe(true);
for (const scope of [0, 1]) { const pending = settledAccounting(); pending.overview.policies[scope]!.pendingRunCount = 1; expect(routineHistoryAccountingReady(accountingIdentity, pending.runs, pending.overview)).toBe(false); }
function valid() {
const f = phase(true), s = phase(false);
return { scenarioId: "routines.read-history", origin: "https://fixture.test", companyId: "company", leadId: "ceo", expectedProvider: "opencode",
state: { worker: { id: "worker" }, failedIssueId: f.identity.issueId, completedIssueId: s.identity.issueId, failedInvocationId: f.identity.invocationId, completedInvocationId: s.identity.invocationId,
failedIdentity: f.identity, completedIdentity: s.identity, failedWork: f.snapshot, completedWork: s.snapshot, failedCalls: f.calls, completedCalls: s.calls,
failedWritten: null, completedWritten: "Launch payment approval remains pending.", completedNativeRunId: s.identity.runId, weekStart: Date.parse("2026-10-26T00:00:00Z"), weekEnd: Date.parse("2026-11-02T00:00:00Z") },
routine: { id: "routine", title: "Morning Launch summary" }, executions: [f.snapshot.invocation!, s.snapshot.invocation!], tasks: [f.snapshot.task, s.snapshot.task], runs: [f.snapshot.run!, s.snapshot.run!],
projects: [], goals: [], comments: [], documents: [], revisions: [], activity: [], humanQueries: [], links: ["/ACORN/issues/ACORN-1", "/ACORN/issues/ACORN-2"], reply: "ACORN-1 failed. ACORN-2 completed successfully. These are manual runs this week." };
}
function overview() {
return { companyId: "company", policies: ["company", "agent"].map(scopeType => ({ companyId: "company", scopeType, scopeId: scopeType === "company" ? "company" : "worker", isActive: true, amount: 500, remainingAmount: 499, hardStopEnabled: true, unpricedUsagePolicy: "allow", pendingRunCount: 0, paused: false, status: "ok" })) } as BudgetOverview;
}
describe("real routine work failure with successful native turns", () => {
it("accepts authoritative provider failure, blocked task and failed invocation, then independent success", () => {
expect(genuineHistoryWork(phase(true).identity, phase(true).snapshot, phase(true).calls, true)).toBe(true);
expect(genuineHistoryWork(phase(false).identity, phase(false).snapshot, phase(false).calls, false)).toBe(true);
expect(gradeIntegratorRoutineHistory(valid()).passed).toBe(true);
});
const mutations: Array<[string, (p: ReturnType<typeof phase>) => void]> = [
["foreign company", p => { p.snapshot.run!.companyId = "foreign"; }],
["foreign worker", p => { p.snapshot.run!.agentId = "other"; }],
["foreign task", p => { p.snapshot.run!.contextSnapshot!.issueId = "other"; }],
["failed native provider", p => { p.snapshot.run!.status = "failed"; }],
["cancelled native", p => { p.snapshot.run!.status = "cancelled"; }],
["still running native", p => { p.snapshot.run!.finishedAt = null; }],
["manufactured failure narration", p => { p.snapshot.decisions.decisions = []; }],
["unchanged task status", p => { p.snapshot.task.status = "in_progress"; }],
["wrong task owner", p => { p.snapshot.task.assigneeAgentId = "other"; }],
["stale public invocation", p => { p.snapshot.invocation!.status = "issue_created"; }],
["foreign routine", p => { p.snapshot.invocation!.routineId = "other"; }],
["foreign invocation task", p => { p.snapshot.invocation!.linkedIssueId = "other"; }],
["scheduled substitute", p => { p.snapshot.invocation!.source = "schedule" as never; }],
["policy denial", p => { p.snapshot.decisions.decisions[0]!.invocation.policyDecision = "deny"; }],
["denied error", p => { p.snapshot.decisions.decisions[0]!.invocation.errorCode = "policy_denied"; }],
["foreign catalog", p => { p.snapshot.decisions.decisions[0]!.invocation.catalogEntryId = "other"; }],
["foreign connection", p => { p.snapshot.decisions.decisions[0]!.invocation.connectionId = "other"; }],
["foreign audit run", p => { p.snapshot.decisions.decisions[0]!.auditEvents[0]!.runId = "other"; }],
["missing audit", p => { p.snapshot.decisions.decisions[0]!.auditEvents = []; }],
["HTTP authorization rejection", p => { p.calls[0]!.responseHttpStatus = 401; }],
["unauthorized provider call", p => { p.calls[0]!.authorized = false; }],
["successful work claimed failure", p => { p.calls[0]!.toolError = false; }],
["unrelated HTTP receipt", p => { p.calls[0]!.requestIdSha256 = "other"; }],
["no provider call", p => { p.calls = []; }],
["human-authored block", p => { p.snapshot.activity[0]!.actorId = "board"; }],
["foreign block run", p => { p.snapshot.activity[0]!.runId = "other"; }],
["no worker failure report", p => { p.snapshot.comments = []; }],
["forged worker report", p => { p.snapshot.comments[0]!.createdByRunId = "other"; }],
];
it.each(mutations)("rejects %s", (_name, mutate) => { const p = phase(true); mutate(p); expect(genuineHistoryWork(p.identity, p.snapshot, p.calls, true)).toBe(false); });
it("retains week, separate task, failure/success, link and schedule-grounding controls", () => {
for (const mutate of [
(v: ReturnType<typeof valid>) => { v.executions[0]!.triggeredAt = "2026-10-19T00:00:00Z" as never; },
(v: ReturnType<typeof valid>) => { v.executions.push(v.executions[0]!); },
(v: ReturnType<typeof valid>) => { v.links.shift(); },
(v: ReturnType<typeof valid>) => { v.reply = "All runs succeeded this week."; },
(v: ReturnType<typeof valid>) => { v.reply += " Both scheduled morning summaries ran successfully."; },
(v: ReturnType<typeof valid>) => { v.state.failedWritten = "changed" as never; },
(v: ReturnType<typeof valid>) => { v.state.completedWritten = ""; },
]) { const v = valid(); mutate(v); expect(gradeIntegratorRoutineHistory(v).passed).toBe(false); }
expect(gradeIntegratorRoutineHistory(undefined).passed).toBe(false);
});
it("declares exactly three normally succeeded turns and preserves original clocks", () => {
const c = integratorRoutineHistoryCases[0]!;
expect(c.runPolicy).toEqual({ kind: "terminal-counts", terminalStatusCounts: { succeeded: 3 } });
expect([c.expectedRunCount, c.expectedHumanMessageCount, c.turnTimeoutMs, c.attemptTimeoutMs]).toEqual([3, 1, 120000, 720000]);
});
});
it("does not force ready for foreign runs, fake failure, absent policy or blocked scope", () => {
for (const key of ["id", "companyId", "agentId"] as const) { const v = settledAccounting(); v.runs[0]![key] = "foreign"; expect(routineHistoryAccountingReady(accountingIdentity, v.runs, v.overview)).toBe(false); }
const wrongTask = settledAccounting(); wrongTask.runs[0]!.contextSnapshot.issueId = "other-task"; expect(routineHistoryAccountingReady(accountingIdentity, wrongTask.runs, wrongTask.overview)).toBe(false);
const cancelled = settledAccounting(); cancelled.runs[0]!.status = "cancelled"; expect(routineHistoryAccountingReady(accountingIdentity, cancelled.runs, cancelled.overview)).toBe(false);
const absent = settledAccounting(); absent.overview.policies.pop(); expect(routineHistoryAccountingReady(accountingIdentity, absent.runs, absent.overview)).toBe(false);
const blocked = settledAccounting(); blocked.overview.policies[0]!.paused = true; expect(routineHistoryAccountingReady(accountingIdentity, blocked.runs, blocked.overview)).toBe(false);
describe("complete normal accounting stays mandatory", () => {
it("accepts actual settled per-run receipt with positive recorded usage", () => { const p = phase(true); expect(completeHistoryUsage(p.snapshot.run)).toBe(true); expect(routineHistoryAccountingReady(p.identity, [p.snapshot.run!], overview())).toBe(true); });
it.each([
["pending settlement", "costAccountingPending", true], ["no settlement", "costAccountedAt", null],
])("rejects %s", (_n, key, value) => { const p = phase(true); (p.snapshot.run as unknown as Record<string, unknown>)[String(key)] = value; expect(completeHistoryUsage(p.snapshot.run)).toBe(false); });
it.each([
["partial receipt", "accountingReceiptReady", false], ["capture failed", "accountingCaptureFailed", true], ["bootstrap no-work", "accountingProviderWorkStarted", false],
["missing source", "accountingReceiptSourceId", null], ["missing id", "accountingReceiptId", null], ["missing sequence", "accountingReceiptSequence", 0],
["missing receipt time", "accountingReceiptReceivedAt", null], ["cumulative usage", "usageSource", "session_cumulative"], ["missing provider", "provider", "unknown"],
["no input usage", "inputTokens", null], ["string output usage", "outputTokens", "20"],
])("rejects %s", (_n, key, value) => { const p = phase(true); p.snapshot.run!.usageJson![String(key)] = value; expect(completeHistoryUsage(p.snapshot.run)).toBe(false); });
it("accepts complete settled unpriced usage under explicit allow without inventing an amount", () => {
const p = phase(true), u = p.snapshot.run!.usageJson!;
u.costStatus = "unpriced"; u.costUsd = null; u.costUsdExact = null; u.accountingCostIncomplete = true;
expect(completeHistoryUsage(p.snapshot.run)).toBe(true);
expect(routineHistoryAccountingReady(p.identity, [p.snapshot.run!], overview())).toBe(true);
expect([u.costUsd, u.costUsdExact]).toEqual([null, null]);
});
it("keeps unpriced-deny, stale/incomplete receipts, foreign runs and invented-zero readiness blocked", () => {
const p = phase(true), u = p.snapshot.run!.usageJson!; u.costStatus = "unpriced"; u.costUsd = null; u.costUsdExact = null;
const denied = overview(); denied.policies[0]!.unpricedUsagePolicy = "block";
expect(routineHistoryAccountingReady(p.identity, [p.snapshot.run!], denied)).toBe(false);
u.accountingReceiptReady = false; u.costUsd = 0;
expect(routineHistoryAccountingReady(p.identity, [p.snapshot.run!], overview())).toBe(false);
u.accountingReceiptReady = true; p.snapshot.run!.costAccountingPending = true;
expect(routineHistoryAccountingReady(p.identity, [p.snapshot.run!], overview())).toBe(false);
p.snapshot.run!.costAccountingPending = false; p.snapshot.run!.companyId = "foreign";
expect(routineHistoryAccountingReady(p.identity, [p.snapshot.run!], overview())).toBe(false);
});
it("does not infer free work, bypass scoped pending/hard-stop policy or accept foreign accounting", () => {
const p = phase(true); p.snapshot.run!.usageJson!.inputTokens = 0; p.snapshot.run!.usageJson!.outputTokens = 0; expect(completeHistoryUsage(p.snapshot.run)).toBe(false);
for (const index of [0, 1]) { const o = overview(); o.policies[index]!.pendingRunCount = 1; expect(routineHistoryAccountingReady(phase(true).identity, [phase(true).snapshot.run!], o)).toBe(false); }
const o = overview(); o.policies[0]!.paused = true; expect(routineHistoryAccountingReady(phase(true).identity, [phase(true).snapshot.run!], o)).toBe(false);
const foreign = phase(true); foreign.identity.runId = "other"; expect(routineHistoryAccountingReady(foreign.identity, [foreign.snapshot.run!], overview())).toBe(false);
});
});
tests/runner-e2e/chat-catalog-domains/integrator-routine-history.ts
+51 -47
View File
@@ -1,16 +1,17 @@
import type { BudgetOverview } from "../../../packages/shared/src/types/budget.js";
import type { ChatRun } from "../chat-flow.js";
import type { RoutineDetail, RoutineRunSummary } from "../../../packages/shared/src/types/routine.js";
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
import { pollUntil } from "../api.js";
import { checks, hasLink, observation, project, type Entity, type InternalObservation } from "./internal-support.js";
import { fixtureWorker } from "./internal-definition.js";
import { unavailableNativeModelFailure, admittedModelFailure } from "./runtime-native-failure.js";
import type { RuntimeAgent } from "./runtime-support.js";
import type { BudgetOverview } from "../../../packages/shared/src/types/budget.js";
import { prepareHistoryNotes, cleanupHistoryNotes, historyNotesPhase, historyNotesCalls, waitHistoryWork, readHistoryWork, genuineHistoryWork, routineHistoryAccountingReady, completeHistoryUsage, type HistoryWorkIdentity, type HistoryWorkSnapshot, type WorkCall } from "./routine-history-work-failure.js";
export { routineHistoryAccountingReady } from "./routine-history-work-failure.js";
import { groundedStatement, integratorDefinition, inspectResult, observeInternal } from "./integrator-support.js";
interface HistoryObservation extends InternalObservation { expectedProvider: string; routine: RoutineDetail; executions: RoutineRunSummary[] }
const root = (ctx: ChatCatalogContext) => `/api/companies/${ctx.input.fixtures.company.id}`;
async function observeHistory(ctx: ChatCatalogContext, state: Record<string, unknown>): Promise<HistoryObservation> {
state.failedWork = await readHistoryWork(ctx, state.failedIdentity as HistoryWorkIdentity);
state.completedWork = await readHistoryWork(ctx, state.completedIdentity as HistoryWorkIdentity);
const base = await observeInternal(ctx, state); const routine = state.routine as RoutineDetail;
return { ...base, expectedProvider: ctx.input.execution.profile.provider, routine: await ctx.input.api.get<RoutineDetail>(`/api/routines/${routine.id}`), executions: await ctx.input.api.get<RoutineRunSummary[]>(`/api/routines/${routine.id}/runs`) };
}
@@ -21,70 +22,73 @@ function historyObservation(value: unknown): HistoryObservation | undefined {
const escaped = (value: string) => value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
export function gradeIntegratorRoutineHistory(value: unknown) {
const c = checks(); const o = historyObservation(value);
c.add("actual-routine-and-native-execution-history", !!o, "Requires real public invocation history plus distinct actual native failure and successful execution receipts.");
c.add("actual-routine-and-native-execution-history", !!o, "Requires real public invocation history plus distinct actual failed work and successful execution receipts.");
if (!o) return c.result();
const failedTask = o.tasks.find(task => task.id === o.state.failedIssueId); const completedTask = o.tasks.find(task => task.id === o.state.completedIssueId);
const failedInvocation = o.executions.find(run => run.id === o.state.failedInvocationId); const completedInvocation = o.executions.find(run => run.id === o.state.completedInvocationId);
const failed = o.runs.find(run => run.id === (o.state.failedRun as { id?: string } | undefined)?.id);
const failureEvidence = o.state.failedRun as Parameters<typeof admittedModelFailure>[0]["run"];
const provider = o.expectedProvider;
const observedProvider = (failed as typeof failureEvidence)?.runnerProfileJson?.nativeExecutionInput?.provider?.kind;
c.add("genuine-task-bound-native-failure", !!failedTask && failed?.status === "failed" && failed.runtimeMode === "native" && !!failedInvocation && failedInvocation.linkedIssueId === failedTask.id
&& (provider === "codex" || provider === "opencode") && observedProvider === provider && admittedModelFailure({ provider, run: failureEvidence, events: o.state.failedEvents as Parameters<typeof admittedModelFailure>[0]["events"] ?? [], companyId: o.companyId, agentId: (o.state.worker as Entity | undefined)?.id ?? "", issueId: failedTask.id, model: String(o.state.failedModel) }), "The failure is a genuinely admitted task-bound unavailable-model native execution, never a rejected HTTP request or a patched/cancelled status.");
const failure = o.state.failedWork as HistoryWorkSnapshot | undefined;
const success = o.state.completedWork as HistoryWorkSnapshot | undefined;
const failedIdentity = o.state.failedIdentity as HistoryWorkIdentity;
const completedIdentity = o.state.completedIdentity as HistoryWorkIdentity;
c.add("genuine-task-bound-failed-work", !!failedTask && genuineHistoryWork(failedIdentity, failure, o.state.failedCalls as WorkCall[] ?? [], true)
&& failedIdentity.companyId === o.companyId && failedIdentity.agentId === (o.state.worker as Entity | undefined)?.id
&& failedIdentity.issueId === failedTask.id && failedIdentity.invocationId === failedInvocation?.id
&& failedTask.status === "blocked" && failedInvocation?.status === "failed" && completeHistoryUsage(failure?.run)
&& o.runs.some(run => run.id === failedIdentity.runId && run.status === "succeeded" && run.companyId === o.companyId && run.agentId === failedIdentity.agentId && run.contextSnapshot?.issueId === failedTask.id)
&& o.state.failedWritten === null, "Actual authorized synthetic MCP work returned HTTP 200/tool_error, with company/worker/task/native-run audit, a worker-blocked task, failed public invocation and complete normal usage. Narration or patched state alone is insufficient.");
c.add("genuine-recovered-work-and-accounting", genuineHistoryWork(completedIdentity, success, o.state.completedCalls as WorkCall[] ?? [], false)
&& completedIdentity.companyId === o.companyId && completedIdentity.agentId === (o.state.worker as Entity | undefined)?.id
&& completedIdentity.issueId === completedTask?.id && completedIdentity.invocationId === completedInvocation?.id
&& completeHistoryUsage(success?.run) && typeof o.state.completedWritten === "string" && !!o.state.completedWritten,
"The restored synthetic provider received a separate authorized successful write under the second task/native run, with completed public history and normal usage.");
c.add("separate-real-successful-invocation", !!completedTask && completedTask.status === "done" && completedInvocation?.status === "completed" && completedInvocation.linkedIssueId === completedTask.id
&& completedTask.id !== failedTask?.id && o.runs.some(run => run.id === o.state.completedNativeRunId && run.status === "succeeded" && run.runtimeMode === "native" && run.contextSnapshot?.issueId === completedTask.id), "A second independently invoked routine actually completed its normal task under a successful native run.");
const start = Number(o.state.weekStart), end = Number(o.state.weekEnd);
c.add("requested-week-and-exact-history-retained", [failedInvocation, completedInvocation].every(run => !!run && Date.parse(String(run.triggeredAt)) >= start && Date.parse(String(run.triggeredAt)) < end)
&& o.executions.length === 2 && failedInvocation?.source === "manual" && completedInvocation?.source === "manual", "Both actual manual invocations fall inside the measured current week, with no hidden or duplicate history rows.");
c.add("actual-failure-and-completion-linked", !!failedTask && !!completedTask && hasLink(o, "issues", failedTask) && hasLink(o, "issues", completedTask), "The answer links both real execution tasks.");
c.add("failed-native-not-inferred-routine-completion", !!failedTask?.identifier && !!completedTask?.identifier
c.add("failed-work-not-inferred-routine-completion", !!failedTask?.identifier && !!completedTask?.identifier
&& groundedStatement(o.reply, new RegExp(escaped(failedTask.identifier), "i"), /\b(?:failed|failure|unsuccessful)\b/i, new RegExp(`${escaped(failedTask.identifier)}[^\n.!?]{0,60}\\b(?:succeeded|successfully completed)\\b`, "i"))
&& groundedStatement(o.reply, new RegExp(escaped(completedTask.identifier), "i"), /\b(?:completed|succeeded|successful)\b/i, /\ball (?:runs|executions) (?:succeeded|completed successfully)\b/i)
&& !o.reply.split(/(?<=[.!?])\s+|\n+/).some(sentence => /\b(?:scheduled|scheduler|automatically|automatic|on schedule)\b/i.test(sentence)
&& /\b(?:ran|executed|succeeded|completed|delivered|successful)\b/i.test(sentence)
&& !/\b(?:no|not|cannot|can't|unconfirmed|unverified|unknown|missing)\b/i.test(sentence)), "The response preserves which actual native execution failed and which invocation completed without inferring unsupported scheduled success. A truthful concise account of the actual manual runs needs no unprompted schedule-denial sentence.");
&& !/\b(?:no|not|cannot|can't|unconfirmed|unverified|unknown|missing)\b/i.test(sentence)), "The response preserves which actual routine work failed and which invocation completed without inferring unsupported scheduled success. A truthful concise account of the actual manual runs needs no unprompted schedule-denial sentence.");
return c.result();
}
/** Terminal native failure alone does not release budget accounting admission. */
export function routineHistoryAccountingReady(expected: { companyId: string; agentId: string; issueId: string; runId: string }, runs: ChatRun[], overview: BudgetOverview) {
const run = runs[0];
if (runs.length !== 1 || !run || run.id !== expected.runId || run.companyId !== expected.companyId || run.agentId !== expected.agentId
|| run.contextSnapshot?.issueId !== expected.issueId || run.status !== "failed" || run.runtimeMode !== "native" || overview.companyId !== expected.companyId) return false;
const policies = overview.policies.filter(p => p.isActive && (p.scopeType === "company" && p.scopeId === expected.companyId || p.scopeType === "agent" && p.scopeId === expected.agentId));
return ["company", "agent"].every(scope => policies.some(p => p.scopeType === scope)) && policies.every(p => p.companyId === expected.companyId
&& p.hardStopEnabled && p.amount > 0 && p.unpricedUsagePolicy === "allow" && p.pendingRunCount === 0 && !p.paused && p.status !== "hard_stop" && p.remainingAmount > 0);
}
export const integratorRoutineHistoryCases = [integratorDefinition("routines", "routines.read-history", async ctx => {
const p = await project(ctx, "Launch", { description: "Synthetic Launch project: payment approval remains pending; no other active work." }); const worker = await fixtureWorker(ctx);
for (const scope of [{ scopeType: "company", scopeId: ctx.input.fixtures.company.id }, { scopeType: "agent", scopeId: worker.id }]) await ctx.input.api.post(`${root(ctx)}/budgets/policies`, { ...scope, amount: 500, hardStopEnabled: true, isActive: true, unpricedUsagePolicy: "allow" });
const routine = await ctx.input.api.post<RoutineDetail>(`${root(ctx)}/routines`, { title: "Morning Launch summary", projectId: p.id, assigneeAgentId: worker.id, concurrencyPolicy: "always_enqueue", description: "Summarize the actual Launch project in a short saved task comment, link the project, and mark this summary task done. Do not edit project work or create follow-up tasks." });
let failedInvocation: RoutineRunSummary | undefined;
const failure = await unavailableNativeModelFailure(ctx, worker as RuntimeAgent, async () => {
failedInvocation = await ctx.input.api.post<RoutineRunSummary>(`/api/routines/${routine.id}/run`, { source: "manual", idempotencyKey: `failed-history-${ctx.input.nonce}` });
if (!failedInvocation.linkedIssueId) throw new Error("Routine failure prerequisite did not admit an actual execution task");
return { issueId: failedInvocation.linkedIssueId };
});
const accountingIdentity = { companyId: ctx.input.fixtures.company.id, agentId: worker.id, issueId: failure.failedIssueId, runId: failure.failedRun.id };
const accounting = await pollUntil({ label: "genuine failed routine prerequisite final accounting before separate success invocation", deadlineAt: Date.now() + 60_000, intervalMs: 1000,
load: async () => ({ runs: await ctx.allRuns(), overview: await ctx.input.api.get<BudgetOverview>(`${root(ctx)}/budgets/overview`) }),
accept: value => routineHistoryAccountingReady(accountingIdentity, value.runs, value.overview) });
await ctx.input.evidence("chat-routine-history-prerequisite-accounting.json", { expected: accountingIdentity, ...accounting });
const completedInvocation = await ctx.input.api.post<RoutineRunSummary>(`/api/routines/${routine.id}/run`, { source: "manual", idempotencyKey: `success-history-${ctx.input.nonce}` });
if (!completedInvocation.linkedIssueId) throw new Error("Successful routine prerequisite did not create a real execution task");
const success = await pollUntil({ label: "actual successful routine history", deadlineAt: Date.now() + 180_000, intervalMs: 1000,
load: async () => ({ runs: await ctx.allRuns(), history: await ctx.input.api.get<RoutineRunSummary[]>(`/api/routines/${routine.id}/runs`) }),
accept: value => value.runs.length === 2 && value.runs.some(run => run.status === "succeeded" && run.contextSnapshot?.issueId === completedInvocation.linkedIssueId)
&& value.history.some(run => run.id === completedInvocation.id && run.status === "completed"),
reject: value => value.runs.length > 2 ? "Routine history fixture exceeded its two declared real native invocations" : undefined });
const connection = await prepareHistoryNotes(ctx, worker.id);
const routine = await ctx.input.api.post<RoutineDetail>(`${root(ctx)}/routines`, { title: "Morning Launch summary", projectId: p.id, assigneeAgentId: worker.id, concurrencyPolicy: "always_enqueue",
description: "Summarize the actual Launch project in a short saved task comment, link the project, and publish that summary once using the connected Synthetic Launch Notes launch_update action. This is a controlled synthetic dependency, not an official vendor service. If the actual publish returns an error, report the actual failure in this task's comment and block the task truthfully; do not mark failed work done. If the publish succeeds, save the result and mark this summary task done. Do not retry, repair the provider, edit project work or create follow-up tasks." });
const invoke = async (kind: string) => {
const invocation = await ctx.input.api.post<RoutineRunSummary>(`/api/routines/${routine.id}/run`, { source: "manual", idempotencyKey: `${kind}-history-${ctx.input.nonce}` });
if (!invocation.linkedIssueId) throw new Error("Routine prerequisite did not admit an actual execution task");
return { companyId: ctx.input.fixtures.company.id, agentId: worker.id, issueId: invocation.linkedIssueId, routineId: routine.id, invocationId: invocation.id, ...connection };
};
const failedBase = await invoke("failed");
const failure = await waitHistoryWork(ctx, failedBase, true, -1, 150_000);
const failedIdentity = { ...failedBase, runId: failure.snapshot.run!.id };
const accounting = await pollUntil({ label: "genuine failed work normal provider accounting before separate success invocation", deadlineAt: Date.now() + 60_000, intervalMs: 1000,
load: async () => ({ snapshot: await readHistoryWork(ctx, failedBase), overview: await ctx.input.api.get<BudgetOverview>(`${root(ctx)}/budgets/overview`) }),
accept: value => genuineHistoryWork(failedIdentity, value.snapshot, historyNotesCalls(ctx), true)
&& routineHistoryAccountingReady(failedIdentity, [value.snapshot.run!], value.overview) });
const failedCalls = historyNotesCalls(ctx); const failedPhase = historyNotesPhase(ctx);
if (failedPhase.written !== null) throw new Error("Failed synthetic work unexpectedly changed the notes page");
historyNotesPhase(ctx, true);
const completedBase = await invoke("success");
const success = await waitHistoryWork(ctx, completedBase, false, failedPhase.lastSequence, 180_000, true);
if ((await ctx.allRuns()).length !== 2) throw new Error("Routine history fixture exceeded its two declared real native invocations");
const weekStart = new Date(); weekStart.setUTCHours(0, 0, 0, 0); weekStart.setUTCDate(weekStart.getUTCDate() - (weekStart.getUTCDay() + 6) % 7);
return { project: p, worker: { id: worker.id }, routine, failedInvocationId: failedInvocation!.id, completedInvocationId: completedInvocation.id, failedIssueId: failure.failedIssueId, completedIssueId: completedInvocation.linkedIssueId,
failedModel: failure.failedModel, failedRun: { ...failure.failedRun, runnerProfileJson: { nativeExecutionInput: { provider: failure.failedRun.runnerProfileJson?.nativeExecutionInput?.provider } } },
failedEvents: failure.failedRunEvents.map(event => ({ seq: event.seq, eventType: event.eventType, payload: { prpEvent: { eventType: event.payload?.prpEvent?.eventType } } })),
completedNativeRunId: success.runs.find(run => run.contextSnapshot?.issueId === completedInvocation.linkedIssueId)?.id,
return { project: p, worker: { id: worker.id }, routine, failedInvocationId: failedBase.invocationId, completedInvocationId: completedBase.invocationId,
failedIssueId: failedBase.issueId, completedIssueId: completedBase.issueId, failedIdentity, completedIdentity: { ...completedBase, runId: success.snapshot.run!.id },
failedWork: accounting.snapshot, completedWork: success.snapshot, failedCalls, completedCalls: success.calls,
failedWritten: failedPhase.written, completedWritten: historyNotesPhase(ctx).written, completedNativeRunId: success.snapshot.run!.id,
weekStart: weekStart.getTime(), weekEnd: weekStart.getTime() + 7 * 86400_000 };
}, observeHistory, gradeIntegratorRoutineHistory, async (ctx, state) => {
const routine = state.routine as RoutineDetail;
await inspectResult(ctx, `/${ctx.input.fixtures.company.issuePrefix}/routines/${routine.id}/history`, routine.title);
}, { sourceFiles: ["chat-catalog-domains/integrator-routine-history.ts", "chat-catalog-domains/runtime-native-failure.ts", "chat-catalog-domains/internal-definition.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/integrator-support.ts", "chat-scenarios/routines.yaml"],
expectedRunCount: 3, runPolicy: { kind: "terminal-counts", terminalStatusCounts: { failed: 1, succeeded: 2 } }, attemptTimeoutMs: 720_000,
}, { sourceFiles: ["chat-catalog-domains/integrator-routine-history.ts", "chat-catalog-domains/routine-history-work-failure.ts", "chat-catalog-domains/external-provider.ts", "chat-catalog-domains/external-support.ts", "chat-catalog-domains/internal-definition.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/integrator-support.ts", "chat-scenarios/routines.yaml"],
oracleVersion: "paperclip.chat-catalog.integrator.routine-work-failure.v2", authentication: "human-session", cleanup: cleanupHistoryNotes, expectedRunCount: 3, runPolicy: { kind: "terminal-counts", terminalStatusCounts: { succeeded: 3 } }, attemptTimeoutMs: 720_000,
})];
tests/runner-e2e/chat-catalog-domains/integrator-skill-enable.test.ts
+37 -4
View File
@@ -1,9 +1,9 @@
import { describe, expect, it } from "vitest";
import { SKILL_POLICY_ACTIONS } from "../../../packages/shared/src/validators/skill-policy.js";
import { gradeIntegratorSkillEnable, integratorSkillEnableCases, unsupportedUsePrerequisite } from "./integrator-skill-enable.js";
import { gradeIntegratorSkillEnable, integratorSkillEnableCases, unsupportedUsePrerequisite, skillEnableAuthorityReady } from "./integrator-skill-enable.js";
function valid() {
const skill = { id: "skill", key: "company/accessibility-review", currentVersionId: "revision", markdown: "Check form labels" };
return { scenarioId: "skills.enable-for-agent", origin: "https://fixture.test", companyId: "company", leadId: "ceo", state: { skill: structuredClone(skill), worker: { id: "morgan" }, futureTask: { id: "future" }, agentConfigurationStatus: 200, beforeActivityIds: ["setup-probe"], effectivePolicy: { revision: 1, rules: [] } },
return { scenarioId: "skills.enable-for-agent", origin: "https://fixture.test", companyId: "company", leadId: "ceo", state: { skill: structuredClone(skill), worker: { id: "morgan" }, futureTask: { id: "future" }, agentConfigurationStatus: 200, configurationAuthority: { companyId: "company", principalId: "ceo", targetId: "morgan", targetCompanyId: "company", status: 200, membership: { companyId: "company", principalType: "agent", principalId: "ceo", status: "active" }, grants: [{ permissionKey: "agents:configure", scope: null as unknown }], before: { supported: true, desiredSkills: [] as string[], desiredSkillEntries: [] }, after: { supported: true, desiredSkills: [] as string[], desiredSkillEntries: [] } }, beforeActivityIds: ["setup-probe"], effectivePolicy: { revision: 1, rules: [] } },
tasks: [], projects: [], goals: [], comments: [], documents: [], revisions: [], humanQueries: [], links: [],
runs: [{ id: "source", agentId: "ceo", runtimeMode: "native", status: "succeeded", contextSnapshot: { issueId: "chat" } }, { id: "future-run", agentId: "morgan", companyId: "company", runtimeMode: "native", status: "succeeded", contextSnapshot: { issueId: "future", skillVersions: { [skill.key]: "revision" } } }],
skill, versions: [{ id: "revision", companyId: "company", companySkillId: "skill", fileInventory: [{ path: "SKILL.md", kind: "skill", content: skill.markdown }] }], assignment: { supported: true, desiredSkills: [skill.key], desiredSkillEntries: [{ key: skill.key, versionId: "revision" }] }, effectivePolicy: { revision: 1, rules: [] },
@@ -21,8 +21,8 @@ describe("effective-use prerequisite classification", () => {
});
});
describe("actual permitted skill assignment and future immutable delivery", () => {
it("keeps unsupported grant/use-policy prerequisites partial, never an expected-refusal pass", () => {
expect(integratorSkillEnableCases[0]!.coverageGaps).toHaveLength(2);
it("requires actual CEO configuration while leaving denied-use coverage conditional", () => {
expect(integratorSkillEnableCases[0]!.coverageGaps?.length ?? 0).toBe(0);
expect(gradeIntegratorSkillEnable(valid()).passed).toBe(true);
expect(gradeIntegratorSkillEnable({ ...valid(), state: { ...valid().state, agentConfigurationStatus: 403 } }).passed).toBe(false);
expect(gradeIntegratorSkillEnable(undefined).passed).toBe(false);
@@ -34,6 +34,11 @@ describe("actual permitted skill assignment and future immutable delivery", () =
const probe = valid(); probe.activity[0]!.id = "setup-probe"; expect(gradeIntegratorSkillEnable(probe).passed).toBe(false);
const changed = valid(); changed.effectivePolicy.revision = 2; expect(gradeIntegratorSkillEnable(changed).passed).toBe(false);
});
it("does not turn a completed positive context into permission to omit the actual skill or authority", () => {
const missing = valid(); missing.assignment.desiredSkills = []; expect(gradeIntegratorSkillEnable(missing).passed).toBe(false);
const foreign = valid(); foreign.state.configurationAuthority.targetCompanyId = "foreign"; expect(gradeIntegratorSkillEnable(foreign).passed).toBe(false);
const unauthorized = valid(); unauthorized.state.configurationAuthority.status = 403; expect(gradeIntegratorSkillEnable(unauthorized).passed).toBe(false);
});
it("accepts substantive assignment language but rejects false refusal/global activation", () => {
const o = valid();
expect(gradeIntegratorSkillEnable({ ...o, reply: "Morgan is configured with the already installed accessibility skill. This assignment does not change usage permissions." }).passed).toBe(true);
@@ -76,3 +81,31 @@ it.each([
])("accepts natural current assignment and historical negation: %s", reply => {
const o = valid(); o.reply = reply; expect(gradeIntegratorSkillEnable(o).passed).toBe(true);
});
describe("existing standard CEO authority and actual target no-op", () => {
for (const [field, value] of [["companyId", "foreign"], ["principalId", "board"], ["targetId", "other"], ["targetCompanyId", "foreign"], ["status", 403]] as const) it(`rejects mismatched authority ${field}`, () => {
const o = valid(); Object.assign(o.state.configurationAuthority, { [field]: value }); expect(gradeIntegratorSkillEnable(o).passed).toBe(false);
});
for (const [field, value] of [["principalType", "user"], ["principalId", "other"], ["companyId", "foreign"], ["status", "suspended"]] as const) it(`rejects mismatched membership ${field}`, () => {
const o = valid(); Object.assign(o.state.configurationAuthority.membership, { [field]: value }); expect(gradeIntegratorSkillEnable(o).passed).toBe(false);
});
it("does not infer configuration authority from hiring/job title or substitute setup assignment", () => {
const o = valid(); o.state.configurationAuthority.grants = [{ permissionKey: "agents:create", scope: null }]; expect(gradeIntegratorSkillEnable(o).passed).toBe(false);
const assigned = valid(); assigned.state.configurationAuthority.after.desiredSkills.push(assigned.skill.key); expect(gradeIntegratorSkillEnable(assigned).passed).toBe(false);
const changed = valid(); changed.state.configurationAuthority.after.desiredSkills.push("another-existing-skill"); expect(gradeIntegratorSkillEnable(changed).passed).toBe(false);
});
it("retains target scope as evidence while the real target-specific HTTP decision stays authoritative", () => {
const o = valid(); o.state.configurationAuthority.grants = [{ permissionKey: "agents:configure", scope: { agentIds: ["morgan"] } }];
expect(gradeIntegratorSkillEnable(o).passed).toBe(true);
o.state.configurationAuthority.status = 403; expect(gradeIntegratorSkillEnable(o).passed).toBe(false);
});
it("requires the actual immutable revision and rejects installation or policy substitution", () => {
const o = valid(); o.skill.markdown = "Replacement instructions"; expect(gradeIntegratorSkillEnable(o).passed).toBe(false);
const version = valid(); version.assignment.desiredSkillEntries[0]!.versionId = "wrong"; expect(gradeIntegratorSkillEnable(version).passed).toBe(false);
const policy = valid(); policy.activity.push({ ...policy.activity[0]!, id: "policy-write", action: "skill_policy.updated" }); expect(gradeIntegratorSkillEnable(policy).passed).toBe(false);
});
it("fails closed on missing no-op snapshots and a board-backed status alone", () => {
const o = valid(); Object.assign(o.state, { configurationAuthority: undefined }); expect(gradeIntegratorSkillEnable(o).passed).toBe(false);
expect(skillEnableAuthorityReady(undefined, "company", "ceo", "morgan", "skill")).toBe(false);
});
});
tests/runner-e2e/chat-catalog-domains/integrator-skill-enable.ts
+34 -46
View File
@@ -1,8 +1,6 @@
import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import type { CompanySkillDetail, CompanySkillVersion } from "../../../packages/shared/src/types/company-skill.js";
import type { AgentSkillSnapshot } from "../../../packages/shared/src/types/adapter-skills.js";
import { SKILL_POLICY_ACTIONS, type EffectiveSkillPolicy, type SkillPolicyDecision } from "../../../packages/shared/src/validators/skill-policy.js";
import { SKILL_POLICY_ACTIONS, type EffectiveSkillPolicy } from "../../../packages/shared/src/validators/skill-policy.js";
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
import { pollUntil } from "../api.js";
import { checks, observation, project, task, type Entity, type InternalObservation } from "./internal-support.js";
@@ -17,6 +15,21 @@ async function observeEnable(ctx: ChatCatalogContext, state: Record<string, unkn
assignment: await ctx.input.api.get<AgentSkillSnapshot>(`/api/agents/${(state.worker as Entity).id}/skills`), effectivePolicy: await ctx.input.api.get(`${root(ctx)}/skill-policy`),
versions: await ctx.input.api.get(`${root(ctx)}/skills/${(state.skill as CompanySkillDetail).id}/versions`) };
}
export interface SkillEnableAuthorityProof {
companyId: string; principalId: string; targetId: string; targetCompanyId: string; status: number;
membership?: { companyId: string; principalType: string; principalId: string; status: string } | null;
grants: Array<{ permissionKey: string; scope?: unknown }>;
before: AgentSkillSnapshot; after: AgentSkillSnapshot;
}
export function skillEnableAuthorityReady(proof: SkillEnableAuthorityProof | undefined, companyId: string, principalId: string, targetId: string, skillKey: string) {
return !!proof && proof.status === 200 && proof.companyId === companyId && proof.principalId === principalId && proof.targetId === targetId && proof.targetCompanyId === companyId
&& proof.membership?.companyId === companyId && proof.membership.principalType === "agent" && proof.membership.principalId === principalId && proof.membership.status === "active"
&& Array.isArray(proof.grants) && proof.grants.some(grant => grant.permissionKey === "agents:configure")
&& proof.before?.supported === true && proof.after?.supported === true && Array.isArray(proof.before.desiredSkills) && Array.isArray(proof.after.desiredSkills)
&& !proof.before.desiredSkills.includes(skillKey) && !proof.after.desiredSkills.includes(skillKey)
&& JSON.stringify(proof.before.desiredSkills) === JSON.stringify(proof.after.desiredSkills)
&& JSON.stringify(proof.before.desiredSkillEntries ?? []) === JSON.stringify(proof.after.desiredSkillEntries ?? []);
}
function enableObservation(value: unknown): EnableObservation | undefined { const o = value as EnableObservation; return observation(value) && !!o.skill && Array.isArray(o.assignment?.desiredSkills) && !!o.effectivePolicy && Array.isArray(o.versions) ? o : undefined; }
/** Past lack of access is not a present denial. Keep contrastive clauses
* separate so an installed-library statement cannot cancel a current denial. */
@@ -33,6 +46,8 @@ function truthfulAssignmentReply(reply: string) {
export function gradeIntegratorSkillEnable(value: unknown) {
const c = checks(); const o = enableObservation(value); c.add("real-assignment-policy-and-future-admission", !!o, "Requires actual permitted assignment, unchanged installed revision/policy and a real subsequent native admission."); if (!o) return c.result();
const original = o.state.skill as CompanySkillDetail; const worker = o.state.worker as Entity;
c.add("actual-ceo-target-configuration-and-no-op-readback", skillEnableAuthorityReady(o.state.configurationAuthority as SkillEnableAuthorityProof | undefined, o.companyId, o.leadId, worker.id, original.key),
"The actual same-company CEO membership/default configuration grant and its successful target-specific agent-bearer no-op are read back without assigning the requested skill during setup.");
c.add("actual-configuration-grant-prerequisite", o.state.agentConfigurationStatus === 200, "The actual CEO bearer successfully exercised a no-op public skill configuration probe; title/legacy hire authority is not a configuration grant.");
const future = o.runs.find(run => run.agentId === worker.id && run.companyId === o.companyId && run.contextSnapshot?.issueId === (o.state.futureTask as Entity | undefined)?.id);
const selectedId = (future?.contextSnapshot?.skillVersions as Record<string, unknown> | undefined)?.[original.key];
@@ -56,65 +71,38 @@ export function gradeIntegratorSkillEnable(value: unknown) {
"Current target assignment must be affirmed without a contradictory current-use denial; historical lack of access is not a current denial.");
return c.result();
}
/** This is a public prerequisite experiment, never a substitute for the
* scenario's required model refusal. Lifecycle denials cannot be relabelled
* as a use denial when actual assignment remains allowed. */
export function unsupportedUsePrerequisite(value: { useStatus: number; decisions: Array<{ action: string; allowed: boolean }>; assignmentRetained: boolean }) {
return !(SKILL_POLICY_ACTIONS as readonly string[]).includes("skills.use") && value.useStatus === 400
&& value.decisions.length === SKILL_POLICY_ACTIONS.length
&& SKILL_POLICY_ACTIONS.every(action => value.decisions.some(decision => decision.action === action && !decision.allowed))
&& value.assignmentRetained;
}
async function probeUsePrerequisite(ctx: ChatCatalogContext, workerId: string, skill: CompanySkillDetail, useStatus: number) {
const api = ctx.input.api; const path = `${root(ctx)}/skill-policy`;
const original = await api.get<EffectiveSkillPolicy>(path);
let changed: EffectiveSkillPolicy | undefined;
try {
changed = await api.putSensitive<EffectiveSkillPolicy>(path, { expectedRevision: original.revision, schemaVersion: 1, defaultEffect: "deny", rules: [{ id: "prerequisite-deny-all-skill-lifecycle", priority: 1, effect: "deny", subject: { type: "agents", agentIds: [workerId] }, actions: [...SKILL_POLICY_ACTIONS] }] });
const decisions: SkillPolicyDecision[] = [];
for (const action of SKILL_POLICY_ACTIONS) decisions.push(await api.post<SkillPolicyDecision>(`${path}/evaluate`, { principal: { agentId: workerId }, action, resource: { skillId: skill.id } }));
await api.post<AgentSkillSnapshot>(`/api/agents/${workerId}/skills/sync`, { mode: "add", desiredSkills: [skill.key] });
const assignment = await api.get<AgentSkillSnapshot>(`/api/agents/${workerId}/skills`);
const receipt = { useStatus, decisions: decisions.map(({ action, allowed }) => ({ action, allowed })), assignmentRetained: assignment.desiredSkills.includes(skill.key) };
await ctx.input.evidence("chat-skill-effective-use-prerequisite.json", { ...receipt, unsupportedUsePrerequisite: unsupportedUsePrerequisite(receipt),
querySent: false, modelAdmissions: 0, lifecycleDenialIsNotUseDenial: true,
sourceContracts: ["../../../packages/shared/src/validators/skill-policy.ts", "../../../server/src/routes/agents.ts", "../../../server/src/services/company-skills.ts"].map(relative => { const url = new URL(relative, import.meta.url); return { source: relative.replace("../../../", ""), sha256: createHash("sha256").update(readFileSync(url)).digest("hex") }; }),
runtimeSourceContract: "buildRuntimeSkillConfig/listRuntimeSkillEntries resolve selected skill versions without a skill-use principal/action; no runtime denial is inferred from this assignment probe." });
if (!unsupportedUsePrerequisite(receipt)) throw new Error("Effective-use prerequisite contract changed; re-author the actual denied-use branch before qualification");
return receipt;
} finally {
try { await api.post(`/api/agents/${workerId}/skills/sync`, { mode: "remove", desiredSkills: [skill.key] }); }
finally { if (changed) await api.putSensitive(path, { expectedRevision: changed.revision, schemaVersion: original.schemaVersion, defaultEffect: original.defaultEffect, rules: original.rules }); }
}
}
export const integratorSkillEnableCases = [integratorDefinition("skills", "skills.enable-for-agent", async ctx => {
try {
const p = await project(ctx, "Launch"); const worker = await fixtureWorker(ctx);
const created = await ctx.input.api.post<CompanySkillDetail>(`${root(ctx)}/skills`, { name: "Accessibility review", slug: "accessibility-review", markdown: "---\nname: accessibility-review\ndescription: Check synthetic HTML form labels.\n---\n\nFor supplied synthetic HTML, identify every form input without a visible label. Save concrete evidence in the task comment and finish the task. Do not browse other pages or claim global qualification." });
const unsupportedUseActionStatus = await probeAgentPermission(ctx, `${root(ctx)}/skill-policy/evaluate`, "POST", { principal: { agentId: ctx.input.fixtures.agent.id }, action: "skills.use", resource: { skillId: created.id } });
const usePrerequisite = await probeUsePrerequisite(ctx, worker.id, created, unsupportedUseActionStatus);
const effectivePolicy = await ctx.input.api.get<EffectiveSkillPolicy>(`${root(ctx)}/skill-policy`);
const access = await ctx.input.api.get<{ access?: { membership?: SkillEnableAuthorityProof["membership"]; grants?: SkillEnableAuthorityProof["grants"] } }>(`/api/agents/${ctx.input.fixtures.agent.id}`);
const before = await ctx.input.api.get<AgentSkillSnapshot>(`/api/agents/${worker.id}/skills`);
const agentConfigurationStatus = await probeAgentPermission(ctx, `/api/agents/${worker.id}/skills/sync`, "POST", { mode: "add", desiredSkills: [] });
let grantStatus: number | null = null;
if (agentConfigurationStatus !== 200) {
const access = await ctx.input.api.get<{ access?: { membership?: { id: string; principalType: string; principalId: string }; grants?: Array<{ permissionKey: string }> } }>(`/api/agents/${ctx.input.fixtures.agent.id}`);
const member = access.access?.membership;
if (member?.principalType === "agent" && member.principalId === ctx.input.fixtures.agent.id) {
const response = await ctx.input.page.context().request.patch(`${root(ctx)}/members/${member.id}/permissions`, { data: { grants: [...access.access?.grants?.filter(grant => grant.permissionKey !== "agents:configure") ?? [], { permissionKey: "agents:configure" }] }, headers: { Origin: ctx.input.api.baseURL } }); grantStatus = response.status();
}
}
const effectiveStatus = agentConfigurationStatus === 200 ? 200 : await probeAgentPermission(ctx, `/api/agents/${worker.id}/skills/sync`, "POST", { mode: "add", desiredSkills: [] });
await ctx.input.evidence("chat-skill-configuration-prerequisite.json", { agentConfigurationStatus: effectiveStatus, publicGrantStatus: grantStatus, principalId: ctx.input.fixtures.agent.id,
querySent: false, usagePolicyActionExists: (SKILL_POLICY_ACTIONS as readonly string[]).includes("skills.use"), unsupportedUseActionStatus, unsupportedPrerequisite: effectiveStatus !== 200,
effectiveUsePrerequisite: unsupportedUsePrerequisite(usePrerequisite), useDenialOutcomeCovered: false, lifecycleDenialIsNotUseDenial: true });
if (effectiveStatus !== 200) throw new Error(`Setup blocker: actual agent_config:update remained denied (HTTP ${effectiveStatus}); public agent membership grant attempt HTTP ${grantStatus ?? "unavailable"}. This is not an expected-refusal pass.`);
return { project: p, worker: { id: worker.id }, skill: await ctx.input.api.get<CompanySkillDetail>(`${root(ctx)}/skills/${created.id}`), effectivePolicy: await ctx.input.api.get(`${root(ctx)}/skill-policy`), agentConfigurationStatus: effectiveStatus,
const after = await ctx.input.api.get<AgentSkillSnapshot>(`/api/agents/${worker.id}/skills`);
const configurationAuthority: SkillEnableAuthorityProof = { companyId: ctx.input.fixtures.company.id, principalId: ctx.input.fixtures.agent.id,
targetId: worker.id, targetCompanyId: worker.companyId!, status: agentConfigurationStatus,
membership: access.access?.membership, grants: access.access?.grants ?? [], before, after };
const policyAfterProbe = await ctx.input.api.get<EffectiveSkillPolicy>(`${root(ctx)}/skill-policy`);
await ctx.input.evidence("chat-skill-configuration-prerequisite.json", { configurationAuthority, querySent: false, grantMutationAttempted: false,
requestedSkillAssignedBySetup: after.desiredSkills.includes(created.key), effectiveUseDenialExperimentRun: false, deniedUseCoverageClaimed: false,
conditionalCoverageNote: "This installed-skill/allowed-assignment positive context does not claim conditional effective-use-denial coverage. No lifecycle denial or unsupported skills.use action substitutes for a supported actual use denial.",
originalPolicyRetained: JSON.stringify(effectivePolicy) === JSON.stringify(policyAfterProbe) });
if (!skillEnableAuthorityReady(configurationAuthority, ctx.input.fixtures.company.id, ctx.input.fixtures.agent.id, worker.id, created.key)
|| JSON.stringify(effectivePolicy) !== JSON.stringify(policyAfterProbe)) throw new Error(`Setup blocker: actual CEO target configuration authority/no-op readback was not established (HTTP ${agentConfigurationStatus}); no grant or requested assignment was substituted.`);
return { project: p, worker: { id: worker.id }, skill: await ctx.input.api.get<CompanySkillDetail>(`${root(ctx)}/skills/${created.id}`), effectivePolicy, agentConfigurationStatus, configurationAuthority,
beforeActivityIds: (await ctx.input.api.get<Entity[]>(`${root(ctx)}/activity?limit=1000`)).map(event => event.id) };
} catch (error) { await cleanupIdentityProbes(ctx); throw error; }
}, observeEnable, gradeIntegratorSkillEnable, async (ctx, state) => {
const skill = state.skill as CompanySkillDetail; await inspectResult(ctx, `/${ctx.input.fixtures.company.issuePrefix}/skills/studio/${skill.id}`, skill.name);
}, { sourceFiles: ["chat-catalog-domains/integrator-skill-enable.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-definition.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/integrator-support.ts", "chat-scenarios/skills.yaml", "../../server/src/routes/agents.ts", "../../server/src/routes/company-skill-policy.ts", "../../server/src/services/company-skills.ts", "../../packages/shared/src/validators/skill-policy.ts"],
expectedRunCount: 2, attemptTimeoutMs: 540_000,
coverageGaps: ["Source-qualified prerequisite limitation: the supported membership permission route currently rejects agent principals, so agents:configure cannot be assumed from CEO/legacy hire authority. Actual grant/configuration HTTP probes must establish the runtime blocker or succeed before this scenario can run.", "Unsupported effective-use context: reserved setup probes the actual unsupported use action plus every real lifecycle action under a denied policy and the actual still-permitted board assignment, restoring policy/assignment in finally. Source runtime version selection has no use-policy principal/action. These receipts establish a prerequisite limitation, never the required model refusal: effective-use-denial outcome remains unqualifiable until a supported actual use-denial contract exists."],
expectedRunCount: 2, attemptTimeoutMs: 540_000, oracleVersion: "paperclip.chat-catalog.skill-enable.standard-config.v2",
async act(ctx, state) {
const first = await pollUntil({ label: "actual Morgan assignment before future admission", deadlineAt: Date.now() + 180_000, intervalMs: 1000,
load: () => observeEnable(ctx, state), accept: o => o.runs.length === 1 && o.runs[0]?.status === "succeeded" && ctx.issue()?.conversationState === "waiting" });
tests/runner-e2e/chat-catalog-domains/integrator-status-cards.test.ts
+12 -6
View File
@@ -1,11 +1,12 @@
import { describe, expect, it } from "vitest";
import { cardWorkFixture, successfulCardWorkFixture } from "./status-card-work-failure.test-fixtures.js";
import { gradeIntegratorCards, integratorStatusCardsCases } from "./integrator-status-cards.js";
function valid(id: string) {
const creation = id === "status-cards.create-card"; const archive = id === "status-cards.archive-card";
const card = { id: "card", companyId: "company", createdByAgentId: id === "status-cards.other-authors-card" ? "morgan" : "ceo", title: "Launch blockers", interestPrompt: "Launch's biggest blockers", agentId: "morgan", queries: [{ projectId: "project" }], fingerprint: { source: { status: "blocked" } }, refreshPolicy: { mode: "manual" }, archivedAt: archive ? "2026-10-26T12:00:00Z" : null, archivedByAgentId: archive ? "ceo" : null, summaryBody: "Payment approval blocks Launch.", lastGeneratedAt: "2026-10-26T11:00:00Z" };
const update = { id: "update", kind: "full", status: "ok", runId: "worker-run", generationIssueId: "generation-task" };
const summary = { id: "summary", body: card.summaryBody };
const source = { id: "source", title: "Payment approval blocks launch", status: "blocked" };
const source = { id: "source", companyId: "company", title: "Payment approval blocks launch", status: "blocked" };
const result = { scenarioId: id, origin: "https://fixture.test", companyId: "company", leadId: "ceo", state: { project: { id: "project" }, worker: { id: "morgan" }, sourceTasks: [structuredClone(source)], ...(creation ? {} : { card: { ...card, archivedAt: null, archivedByAgentId: null }, originalUpdates: [structuredClone(update)], originalSummaries: [structuredClone(summary)], ...(archive ? { otherCard: { ...card, id: "other", interestPrompt: "Research blockers", archivedAt: null, archivedByAgentId: null } } : {}) }) },
tasks: [source], projects: [], goals: [], comments: [], documents: [], revisions: [], humanQueries: [], links: ["/ACORN/status/card"],
runs: [{ id: "source-run", agentId: "ceo", status: "succeeded" }, { id: "worker-run", agentId: "morgan", status: "succeeded" }],
@@ -29,10 +30,15 @@ function valid(id: string) {
result.runs.push({ id: "new-worker-run", agentId: "morgan", status: "succeeded" });
result.reply = `${first.reply}\nThe refreshed summary identifies mobile Safari checkout as the remaining blocker.`;
const successful = structuredClone(result);
Object.assign(result.state, { successfulRefreshResult: successful, failedGenerationRunId: "failed-native", failedGenerationIssueId: "failure-task" });
Object.assign(result.cards[0]!, { state: "error", failureReason: "The generation failed", generatingIssueId: null });
result.updates.card.push(Object.assign({ ...update, id: "failed-update", status: "failed", runId: "failed-native", generationIssueId: "failure-task" }, { error: "Unavailable model" }));
result.runs.push(Object.assign({ id: "failed-native", agentId: "morgan", status: "failed" }, { runtimeMode: "native", contextSnapshot: { issueId: "failure-task" } }));
const work = cardWorkFixture();
const initialChecklist = successfulCardWorkFixture("worker-run", "generation-task", "update");
const successfulChecklist = successfulCardWorkFixture("new-worker-run", "refresh-task", "new-update");
Object.assign(result.state, { successfulRefreshResult: successful, failedGenerationRunId: "failed-native", failedGenerationIssueId: "failure-task", cardRecipe: work.evidence.recipe, initialChecklist: initialChecklist.evidence, successfulChecklist: successfulChecklist.evidence });
Object.assign(result.cards[0]!, work.card);
Object.assign(result.updates.card[0]!, initialChecklist.update); Object.assign(result.updates.card[1]!, successfulChecklist.update);
Object.assign((result.state as Record<string, unknown>).originalUpdates as object[], [initialChecklist.update]);
result.updates.card.push(work.update as unknown as typeof update);
Object.assign(result, { cardWork: work.evidence, runs: work.evidence.runs });
result.reply += "\nThe latest generation failed; the prior successful summary was retained.";
}
@@ -88,7 +94,7 @@ describe("saved card state and truthful generation controls", () => {
const o = valid("status-cards.refresh-card");
const missingError = structuredClone(o); Object.assign(missingError.cards[0]!, { state: "active", failureReason: null });
expect(gradeIntegratorCards(missingError).passed).toBe(false);
const unrelated = structuredClone(o); Object.assign(unrelated.runs.at(-1)!, { contextSnapshot: { issueId: "unrelated-task" } });
const unrelated = structuredClone(o); Object.assign(unrelated.runs.find(run => run.id === "failed-native")!, { contextSnapshot: { issueId: "unrelated-task" } });
expect(gradeIntegratorCards(unrelated).passed).toBe(false);
const rewritten = structuredClone(o); rewritten.cards[0]!.summaryBody = "Replacement after failure";
expect(gradeIntegratorCards(rewritten).passed).toBe(false);
tests/runner-e2e/chat-catalog-domains/integrator-status-cards.ts
+65 -36
View File
@@ -4,12 +4,11 @@ import type { StatusCard, StatusCardUpdate, StatusCardSummaryRevision } from "..
import type { ChatCatalogCase, ChatCatalogContext } from "../chat-catalog-contract.js";
import { pollUntil } from "../api.js";
import { checks, observation, project, task, sameTask, hasLink, type Entity, type InternalObservation } from "./internal-support.js";
import { unavailableNativeModelFailure } from "./runtime-native-failure.js";
import type { RuntimeAgent } from "./runtime-support.js";
import { prepareCardChecklist, updateCardChecklistSource, observeCardChecklistSuccess, observeCardWork, failCardChecklist, genuineCardWorkFailure, genuineCardChecklistSuccess, cardAccountingReady, cleanupCardChecklist, type CardRecipe, type CardWorkEvidence } from "./status-card-work-failure.js";
import { readAgentResource, cleanupIdentityProbes } from "./identity-support.js";
import { fixtureWorker } from "./internal-definition.js";
import { groundedStatement, integratorDefinition, inspectResult, resultHref, observeInternal } from "./integrator-support.js";
interface CardsObservation extends InternalObservation { cards: StatusCard[]; updates: Record<string, StatusCardUpdate[]>; summaries: Record<string, StatusCardSummaryRevision[]>; cardsEnabled: boolean }
interface CardsObservation extends InternalObservation { cards: StatusCard[]; updates: Record<string, StatusCardUpdate[]>; summaries: Record<string, StatusCardSummaryRevision[]>; cardsEnabled: boolean; cardWork?: CardWorkEvidence }
const root = (ctx: ChatCatalogContext) => `/api/companies/${ctx.input.fixtures.company.id}`;
async function observeCards(ctx: ChatCatalogContext, state: Record<string, unknown>): Promise<CardsObservation> {
const base = await observeInternal(ctx, state); const api = ctx.input.api;
@@ -18,7 +17,9 @@ async function observeCards(ctx: ChatCatalogContext, state: Record<string, unkno
const cards = await Promise.all(ids.map(id => api.get<StatusCard>(`/api/status-cards/${id}`)));
const updates = Object.fromEntries(await Promise.all(ids.map(async id => [id, await api.get<StatusCardUpdate[]>(`/api/status-cards/${id}/updates`)])));
const summaries = Object.fromEntries(await Promise.all(ids.map(async id => [id, await api.get<StatusCardSummaryRevision[]>(`/api/status-cards/${id}/summary-revisions`)])));
return { ...base, cards, updates, summaries, cardsEnabled: settings.enableStatusCards };
const cardWork = ctx.scenario.id === "status-cards.refresh-card" && state.cardRecipe && typeof state.failedGenerationIssueId === "string"
? await observeCardWork(ctx, state.cardRecipe as CardRecipe, state.failedGenerationIssueId) : undefined;
return { ...base, cards, updates, summaries, cardsEnabled: settings.enableStatusCards, ...(cardWork ? { cardWork } : {}) };
}
function cardsObservation(value: unknown): CardsObservation | undefined {
const o = value as CardsObservation;
@@ -87,10 +88,18 @@ export function gradeIntegratorCards(value: unknown) {
const finalReply = initial && successfulResult?.reply.startsWith(initial.reply) ? successfulResult.reply.slice(initial.reply.length) : "";
c.add("actual-result-reported-after-generation", /mobile Safari/i.test(finalReply) && /\b(?:generated|refreshed|updated|summary|blocker)\b/i.test(finalReply)
&& !/\b(?:still waiting|not refreshed|refresh failed|could not refresh)\b/i.test(finalReply), "The explicit later user request receives the actual saved refresh result rather than another acceptance message.");
const recipe = o.state.cardRecipe as CardRecipe | undefined;
const initialUpdate = (o.state.originalUpdates as StatusCardUpdate[] | undefined)?.find(row => row.kind !== "compile" && row.status === "ok");
c.add("same-required-input-recipe-used-for-successes", !!before && !!recipe && !!initialUpdate && successful.length === 1
&& genuineCardChecklistSuccess(before.id, recipe, initialUpdate, o.state.initialChecklist as CardWorkEvidence | undefined)
&& genuineCardChecklistSuccess(before.id, recipe, successful[0]!, o.state.successfulChecklist as CardWorkEvidence | undefined), "Both real successful generations used the same saved required source-verification recipe and actual authorized current reads before the independent failure.");
const failedRun = o.runs.find(run => run.id === o.state.failedGenerationRunId);
const failureReply = successfulResult && o.reply.startsWith(successfulResult.reply) ? o.reply.slice(successfulResult.reply.length) : "";
c.add("genuine-failed-generation-keeps-last-good-output", !!successfulResult && !!lastGood && !!card && card.state === "error" && !!card.failureReason && !card.generatingIssueId
&& failed.length === 1 && failed[0]?.generationIssueId === o.state.failedGenerationIssueId && !!failed[0]?.error && failedRun?.status === "failed" && failedRun.runtimeMode === "native" && failedRun.contextSnapshot?.issueId === failed[0]?.generationIssueId
&& failed.length === 1 && failed[0]?.generationIssueId === o.state.failedGenerationIssueId && !!failed[0]?.error && !!failedRun
&& genuineCardWorkFailure(card, failed[0]!, failedRun.id, o.leadId, o.cardWork)
&& !!o.cardWork && originals.some(source => sameTask(source, o.cardWork!.checklistSource))
&& o.cardWork?.runs.length === 7 && o.runs.length === 7 && o.runs.every(run => run.status === "succeeded" && o.cardWork?.runs.some(actual => actual.id === run.id && actual.agentId === run.agentId && actual.companyId === run.companyId && actual.contextSnapshot?.issueId === run.contextSnapshot?.issueId))
&& card.summaryBody === lastGood.summaryBody && card.lastGeneratedAt === lastGood.lastGeneratedAt
&& JSON.stringify(o.summaries[card.id]) === JSON.stringify(successfulResult.summaries[card.id]), "An independently observed real card-generation error and failed update retain the previous successful summary, timestamp and complete history. A failed native row without card error does not qualify.");
c.add("actual-generation-error-reported", /refresh|generation/i.test(failureReply) && /failed|error|could not|couldn't|did not finish/i.test(failureReply)
@@ -113,13 +122,13 @@ async function verifyCard(ctx: ChatCatalogContext, _state: Record<string, unknow
}
/** Author identity comes from a real revocable agent key. The compile and first
* summary are produced by an actual admitted summarizer run, never fixture PUTs. */
async function seedCard(ctx: ChatCatalogContext, authorId: string, summarizerId: string, projectId: string, projectName = "Launch") {
async function seedCard(ctx: ChatCatalogContext, authorId: string, summarizerId: string, projectId: string, projectName = "Launch", interestPrompt?: string) {
if (!ctx.input.privateSecrets) throw new Error("Card fixture author key needs private redaction");
const key = await ctx.input.api.postSensitive<{ id: string; token: string }>(`/api/agents/${authorId}/keys`, { name: `card-prerequisite-${ctx.input.nonce}` });
ctx.input.privateSecrets.push(key.token);
try {
const response = await fetch(new URL(`${root(ctx)}/status-cards`, ctx.input.api.baseURL), { method: "POST", redirect: "error", signal: AbortSignal.timeout(20_000),
headers: { Authorization: `Bearer ${key.token}`, "content-type": "application/json" }, body: JSON.stringify({ title: `${projectName} blockers`, interestPrompt: `Summarize the biggest blockers for ${projectName} project ${projectId}, citing actual source tasks.`, agentId: summarizerId, refreshPolicy: { mode: "manual" } }) });
headers: { Authorization: `Bearer ${key.token}`, "content-type": "application/json" }, body: JSON.stringify({ title: `${projectName} blockers`, interestPrompt: interestPrompt ?? `Summarize the biggest blockers for ${projectName} project ${projectId}, citing actual source tasks.`, agentId: summarizerId, refreshPolicy: { mode: "manual" } }) });
if (!response.ok) { await response.body?.cancel(); throw new Error(`Card fixture public creation failed HTTP ${response.status}`); }
const created = await response.json() as StatusCard;
return await pollUntil({ label: "actual prerequisite card generation", deadlineAt: Date.now() + 180_000, intervalMs: 1000,
@@ -154,8 +163,17 @@ function define(id: string, seed: "none" | "ceo" | "morgan", extra: Partial<Chat
// the ordinary shared flow applies the same policy before the human query.
for (const scope of [{ scopeType: "company", scopeId: ctx.input.fixtures.company.id }, { scopeType: "agent", scopeId: worker.id }])
await ctx.input.api.post(`${root(ctx)}/budgets/policies`, { ...scope, amount: 500, hardStopEnabled: true, isActive: true, unpricedUsagePolicy: "allow" });
const card = await seedCard(ctx, seed === "ceo" ? ctx.input.fixtures.agent.id : worker.id, worker.id, p.id);
await pollUntil({ label: "prerequisite summarizer native completion", deadlineAt: Date.now() + 45_000, load: () => ctx.allRuns(), accept: rows => rows.length === 1 && rows.every(row => row.status === "succeeded") });
const cardRecipe = id === "status-cards.refresh-card" ? await prepareCardChecklist(ctx, worker.id, p.id, t) : undefined;
const card = await seedCard(ctx, seed === "ceo" ? ctx.input.fixtures.agent.id : worker.id, worker.id, p.id, "Launch", cardRecipe?.interestPrompt);
const initialComplete = await pollUntil({ label: "prerequisite summarizer native completion", deadlineAt: Date.now() + 45_000,
load: async () => {
const runs = await ctx.allRuns();
const update = cardRecipe ? (await ctx.input.api.get<StatusCardUpdate[]>(`/api/status-cards/${card.id}/updates`)).find(row => row.kind !== "compile" && row.status === "ok") : undefined;
const checklist = cardRecipe && update ? await observeCardChecklistSuccess(ctx, cardRecipe, update) : undefined;
return { runs, update, checklist };
}, accept: value => value.runs.length === 1 && value.runs.every(row => row.status === "succeeded") && (!cardRecipe || !!value.update && genuineCardChecklistSuccess(card.id, cardRecipe, value.update, value.checklist)) });
const initialChecklist = initialComplete.checklist;
if (initialChecklist) await ctx.input.evidence("chat-card-initial-required-checklist.json", initialChecklist);
let otherCard: StatusCard | undefined; const sourceTasks = [t];
if (id === "status-cards.archive-card") {
const research = await project(ctx, "Research"); sourceTasks.push(await task(ctx, "Research follow-up", research.id));
@@ -166,13 +184,14 @@ function define(id: string, seed: "none" | "ceo" | "morgan", extra: Partial<Chat
sourceTasks[0] = await ctx.input.api.patch<Entity>(`/api/issues/${t.id}`, id === "status-cards.explain-card"
? { status: "done", description: "The payment approval has been received; this blocker is resolved." }
: refreshedBlockerTaskInput());
if (cardRecipe) updateCardChecklistSource(ctx, sourceTasks[0]!);
}
const factualContextAttachment = ["status-cards.explain-card", "status-cards.archive-card"].includes(id)
? await uploadFactualQueryAttachment(ctx, { filename: "selected-status-card.md", body: selectedStatusCardReference(card, sourceTasks, ctx.input.fixtures.company.issuePrefix) }) : undefined;
return { project: p, sourceTasks, worker, card, otherCard, factualContextAttachment, originalUpdates: await ctx.input.api.get<StatusCardUpdate[]>(`/api/status-cards/${card.id}/updates`), originalSummaries: await ctx.input.api.get<StatusCardSummaryRevision[]>(`/api/status-cards/${card.id}/summary-revisions`) };
} catch (error) { await ctx.input.api.patch("/api/instance/settings/experimental", { enableStatusCards: previous.enableStatusCards }); experiments.delete(ctx); throw error; }
return { project: p, sourceTasks, worker, card, otherCard, ...(cardRecipe ? { cardRecipe, initialChecklist } : {}), factualContextAttachment, originalUpdates: await ctx.input.api.get<StatusCardUpdate[]>(`/api/status-cards/${card.id}/updates`), originalSummaries: await ctx.input.api.get<StatusCardSummaryRevision[]>(`/api/status-cards/${card.id}/summary-revisions`) };
} catch (error) { try { await cleanupCardChecklist(ctx); } finally { await ctx.input.api.patch("/api/instance/settings/experimental", { enableStatusCards: previous.enableStatusCards }); experiments.delete(ctx); } throw error; }
}, observeCards, gradeIntegratorCards, verifyCard, { expectedRunCount: 2, attemptTimeoutMs: 540_000, ...extra,
async cleanup(ctx) { await cleanupIdentityProbes(ctx); const previous = experiments.get(ctx); if (previous !== undefined) { await ctx.input.api.patch("/api/instance/settings/experimental", { enableStatusCards: previous }); experiments.delete(ctx); } } });
async cleanup(ctx) { try { await cleanupCardChecklist(ctx); } finally { await cleanupIdentityProbes(ctx); const previous = experiments.get(ctx); if (previous !== undefined) { await ctx.input.api.patch("/api/instance/settings/experimental", { enableStatusCards: previous }); experiments.delete(ctx); } } } });
}
export const integratorStatusCardsCases: readonly ChatCatalogCase[] = [
define("status-cards.create-card", "none"),
@@ -198,8 +217,9 @@ export const integratorStatusCardsCases: readonly ChatCatalogCase[] = [
},
}),
define("status-cards.refresh-card", "ceo", { expectedRunCount: 7, expectedHumanMessageCount: 4, attemptTimeoutMs: 1_200_000,
runPolicy: { kind: "terminal-counts", terminalStatusCounts: { failed: 1, succeeded: 6 } },
sourceFiles: ["chat-catalog-domains/integrator-status-cards.ts", "chat-catalog-domains/runtime-native-failure.ts", "chat-catalog-domains/runtime-support.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-definition.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/integrator-support.ts", "chat-scenarios/status-cards.yaml"],
runPolicy: { kind: "terminal-counts", terminalStatusCounts: { succeeded: 7 } },
oracleVersion: "paperclip.chat-catalog.status-card.required-work-failure.v2",
sourceFiles: ["chat-catalog-domains/integrator-status-cards.ts", "chat-catalog-domains/status-card-work-failure.ts", "chat-catalog-domains/external-provider.ts", "chat-catalog-domains/runtime-support.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-definition.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/integrator-support.ts", "chat-scenarios/status-cards.yaml"],
async act(ctx, state) {
const first = await pollUntil({ label: "initial actual refresh acknowledgement", deadlineAt: Date.now() + 180_000, intervalMs: 1000,
@@ -211,30 +231,39 @@ export const integratorStatusCardsCases: readonly ChatCatalogCase[] = [
load: () => observeCards(ctx, state), accept: o => { const card = o.cards.find(row => row.id === before.id); return !!card && card.lastGeneratedAt !== before.lastGeneratedAt && freshSummary(o, card) && o.runs.length === 3 && o.runs.every(run => run.status === "succeeded"); } });
const { sendChatMessage } = await import("../chat-flow.js");
await sendChatMessage(ctx.input.page, "What did the refresh actually produce?", chatCatalogNativePhaseRemainingBudget(ctx));
const succeeded = await pollUntil({ label: "successful generation actually reported before separate failure condition", deadlineAt: Date.now() + 180_000, intervalMs: 1000, load: () => observeCards(ctx, state), accept: o => o.runs.length === 4 && o.runs.every(run => run.status === "succeeded") && ctx.issue()?.conversationState === "waiting" });
const recipe = state.cardRecipe as CardRecipe;
const success = await pollUntil({ label: "successful generation actually reported before separate failure condition", deadlineAt: Date.now() + 180_000, intervalMs: 1000,
load: async () => {
const observed = await observeCards(ctx, state);
const update = (observed.updates[before.id] ?? []).find(row => row.kind !== "compile" && row.status === "ok" && !(state.originalUpdates as StatusCardUpdate[]).some(old => old.id === row.id));
return { observed, update, checklist: update ? await observeCardChecklistSuccess(ctx, recipe, update) : undefined };
}, accept: value => value.observed.runs.length === 4 && value.observed.runs.every(run => run.status === "succeeded") && ctx.issue()?.conversationState === "waiting"
&& !!value.update && genuineCardChecklistSuccess(before.id, recipe, value.update, value.checklist)
&& !!value.checklist && cardAccountingReady(ctx.input.fixtures.company.id, recipe.workerId, ctx.input.fixtures.agent.id, value.checklist.runs, value.checklist.overview) });
const succeeded = success.observed;
state.successfulChecklist = success.checklist;
state.successfulRefreshResult = { ...succeeded, state: structuredClone(state) }; await ctx.input.evidence("chat-card-refresh-last-successful-result.json", state.successfulRefreshResult);
const worker = await ctx.input.api.get<RuntimeAgent>(`/api/agents/${(state.worker as Entity).id}`);
await ctx.input.evidence("chat-card-successful-required-checklist.json", success.checklist);
const beforeFailureIds = new Set((succeeded.updates[before.id] ?? []).map(update => update.id));
// The helper scopes induction accounting to this one actual summarizer.
// The shared oracle still receives every company run, including both
// genuine CEO source turns; nothing is hidden from the declared bound.
const workerContext = { ...ctx, allRuns: async () => (await ctx.allRuns()).filter(run => run.agentId === worker.id) };
const failure = await unavailableNativeModelFailure(workerContext, worker, async () => {
await sendChatMessage(ctx.input.page, "Run another full refresh of that card now.", chatCatalogNativePhaseRemainingBudget(ctx));
const update = await pollUntil({ label: "explicit human full refresh creates actual generation attempt", deadlineAt: Date.now() + 180_000, intervalMs: 1000,
load: () => ctx.input.api.get<StatusCardUpdate[]>(`/api/status-cards/${before.id}/updates`),
accept: updates => updates.some(row => !beforeFailureIds.has(row.id) && row.kind !== "compile" && !!row.generationIssueId) });
const generation = update.find(row => !beforeFailureIds.has(row.id) && row.kind !== "compile" && row.generationIssueId)!;
return { issueId: generation.generationIssueId! };
});
state.failedGenerationRunId = failure.failedRun.id; state.failedGenerationIssueId = failure.failedIssueId;
const observed = await observeCards(ctx, state); const actual = observed.cards.find(row => row.id === before.id);
const failedUpdates = observed.updates[before.id]?.filter(update => update.generationIssueId === failure.failedIssueId && update.status === "failed") ?? [];
const established = actual?.state === "error" && !actual.generatingIssueId && failedUpdates.length === 1;
await ctx.input.evidence("chat-card-generation-failure-outcome.json", { kind: "real-unavailable-model-invocation", nativeFailureObserved: true, failedRunId: failure.failedRun.id,
generationIssueId: failure.failedIssueId, cardId: before.id, observedCardState: actual?.state, observedGeneratingIssueId: actual?.generatingIssueId, failedUpdateIds: failedUpdates.map(update => update.id),
cardErrorObserved: established, expectedCardFailureOutcome: established ? "observed" : "not-observed-failing-product-outcome", taskOrCardStatusPatched: false, cancelledAsFailure: false });
await pollUntil({ label: "actual refresh source turn settled after failed generation", deadlineAt: Date.now() + 180_000, intervalMs: 1000, load: () => observeCards(ctx, state), accept: o => o.runs.length === 6 && o.runs.filter(run => run.status === "succeeded").length === 5 && o.runs.filter(run => run.status === "failed").length === 1 && ctx.issue()?.conversationState === "waiting" });
failCardChecklist(ctx);
await sendChatMessage(ctx.input.page, "Run another full refresh of that card now.", chatCatalogNativePhaseRemainingBudget(ctx));
const created = await pollUntil({ label: "explicit human full refresh creates actual generation attempt", deadlineAt: Date.now() + 180_000, intervalMs: 1000,
load: () => ctx.input.api.get<StatusCardUpdate[]>(`/api/status-cards/${before.id}/updates`),
accept: updates => updates.some(row => !beforeFailureIds.has(row.id) && row.kind !== "compile" && !!row.generationIssueId) });
const generation = created.find(row => !beforeFailureIds.has(row.id) && row.kind !== "compile" && row.generationIssueId)!;
state.failedGenerationIssueId = generation.generationIssueId!;
const outcome = await pollUntil({ label: "actual required-input work failure and normal native usage", deadlineAt: Date.now() + 150_000, intervalMs: 1000,
load: () => observeCards(ctx, state),
accept: o => {
const card = o.cards.find(row => row.id === before.id), update = o.updates[before.id]?.find(row => row.id === generation.id);
const run = o.cardWork?.runs.find(row => row.agentId === recipe.workerId && row.contextSnapshot?.issueId === generation.generationIssueId);
return !!card && !!update && !!run && card.state === "error" && !card.generatingIssueId && genuineCardWorkFailure(card, update, run.id, o.leadId, o.cardWork);
},
reject: o => o.cardWork?.runs.some(run => run.contextSnapshot?.issueId === generation.generationIssueId && ["failed", "cancelled", "timed_out"].includes(run.status)) ? "Card generation did not yield a normally completed provider turn; no work-failure substitution" : undefined });
state.failedGenerationRunId = outcome.cardWork!.runs.find(run => run.agentId === recipe.workerId && run.contextSnapshot?.issueId === generation.generationIssueId)!.id;
await ctx.input.evidence("chat-card-generation-failure-outcome.json", { ...outcome.cardWork, kind: "synthetic-required-current-source-unavailable", cardId: before.id,
updateId: generation.id, generationIssueId: generation.generationIssueId, taskOrCardStatusPatched: false, cancelledAsFailure: false, providerFailureSubstituted: false });
await pollUntil({ label: "actual refresh source turn settled after failed work", deadlineAt: Date.now() + 180_000, intervalMs: 1000, load: () => observeCards(ctx, state), accept: o => o.runs.length === 6 && o.runs.every(run => run.status === "succeeded") && ctx.issue()?.conversationState === "waiting" });
await sendChatMessage(ctx.input.page, "What happened to that latest refresh, and which saved summary is available now?", chatCatalogNativePhaseRemainingBudget(ctx));
},
}),
tests/runner-e2e/chat-catalog-domains/internal-activity.ts
+5 -3
View File
@@ -1,3 +1,4 @@
import { groundedOverviewBlocker, reportsRecordedTaskState } from "./recorded-work-claims.js";
import type { ChatCatalogCase } from "../chat-catalog-contract.js";
import { project, task, goal, putDocument, entity, observation, checks, hasLink, sameTask, type Entity } from "./internal-support.js";
import { makeDefinition, retryDefinition, fixtureWorker } from "./internal-definition.js";
@@ -17,9 +18,10 @@ export function grade_activity(value: unknown) {
const blocked = entity(o.state, "blockedTask"); const done = entity(o.state, "doneTask"); const worker = entity(o.state, "worker");
const expected = [before, blocked, done].filter((row): row is Entity => !!row);
c.add("actual-work-summary", expected.length === 3 && expected.every(row => sameTask(row, o.tasks.find(now => now.id === row.id)) && hasLink(o, "issues", row))
&& /\b(?:active|in.progress|underway)\b/i.test(o.reply) && /\bblocked\b/i.test(o.reply) && /\b(?:completed|done|finished)\b/i.test(o.reply),
&& !!before && reportsRecordedTaskState(o.reply, before, expected, "in_progress")
&& reportsRecordedTaskState(o.reply, blocked, expected, "blocked") && reportsRecordedTaskState(o.reply, done, expected, "done"),
"The overview links the actual active, blocked and completed work without changing its state.");
c.add("grounded-blocker", o.reply.includes(String(o.state.blocker)), "The summary describes the fixture's actual recorded blocker.");
c.add("grounded-blocker", groundedOverviewBlocker(o.reply, blocked, expected, o.state.blocker), "The blocked task carries the actual recorded reason and the answer grounds that reason in the same task without claiming it is resolved. Incidental fixture nonces are not a user requirement.");
c.add("idle-is-not-broken", o.agents?.some(row => row.id === worker.id && row.status === "idle") === true
&& /\bMorgan\b[^.\n]*\b(?:idle|available|not currently working)\b/i.test(o.reply)
&& !/\bMorgan\b\s+(?:is|looks|appears)\s+(?:broken|failed|stuck)/i.test(o.reply),
@@ -107,7 +109,7 @@ function companyOverview(): ChatCatalogCase {
await ctx.input.api.post(`/api/companies/${hiddenCompany.id}/issues`, { title: hiddenMarker, status: "blocked" });
return { project: p, task: active, blockedTask, doneTask, worker, blocker, hiddenCompany, hiddenMarker };
});
return { ...base, async verifySetup(ctx, state) {
return { ...base, oracleVersion: "paperclip.chat-catalog.activity-overview.grounded.v2", sourceFiles: [...base.sourceFiles, "chat-catalog-domains/recorded-work-claims.ts"], async verifySetup(ctx, state) {
const active = await ctx.input.api.get<Entity & { assigneeUserId: string | null }>(`/api/issues/${entity(state, "task").id}`);
const worker = await ctx.input.api.get<Entity>(`/api/agents/${entity(state, "worker").id}`);
const c = checks();
tests/runner-e2e/chat-catalog-domains/internal-privacy.ts
+3 -2
View File
@@ -1,3 +1,4 @@
import { chatCatalogNativePhaseDeadline, chatCatalogNativePhaseIsTimely } from "../chat-catalog-contract.js";
import { expect } from "@playwright/test";
import type { ChatCatalogCase, ChatCatalogContext } from "../chat-catalog-contract.js";
import { createAuthenticatedPerson, enrollPrerequisitePerson } from "./identity-auth.js";
@@ -181,8 +182,8 @@ function shareProject(): ChatCatalogCase {
return { ...base, authentication: "human-session", sourceFiles: [...base.sourceFiles, "chat-catalog-domains/internal-privacy.ts", "chat-catalog-domains/identity-auth.ts"],
async act(ctx, state) {
if (!observer) throw new Error("Missing independent observer");
const first = await pollUntil({ label: "Existing private-project owner handoff", deadlineAt: Date.now() + 150_000,
load: () => observeInternal(ctx, state), accept: o => o.runs.length === 1 && o.runs.every(row => row.status === "succeeded")
const first = await pollUntil({ label: "Existing private-project owner handoff", deadlineAt: chatCatalogNativePhaseDeadline(ctx, Date.now() + 150_000),
load: () => observeInternal(ctx, state), accept: o => chatCatalogNativePhaseIsTimely(ctx) && o.runs.length === 1 && o.runs.every(row => row.status === "succeeded")
&& ctx.issue()?.conversationState === "waiting" });
const p = entity(state, "project"); const protectedTask = entity(state, "protectedTask");
const leadReadStatus = await agentStatus(ctx, first.leadId, `/api/projects/${p.id}`);
tests/runner-e2e/chat-catalog-domains/internal-project-defaults.ts
+4 -3
View File
@@ -1,3 +1,4 @@
import { chatCatalogNativePhaseDeadline, chatCatalogNativePhaseIsTimely } from "../chat-catalog-contract.js";
import { CHAT_NATIVE_TIMEOUT_POLICY } from "../chat-native-timeout-policy.js";
import { expect } from "@playwright/test";
import { createHash } from "node:crypto";
@@ -103,12 +104,12 @@ function defaults(): ChatCatalogCase {
return gradeProjectDefaultsSetup({ lead, future, held, task: t, runs, activeRunId: String(state.activeRunId), gateCalls: gate?.calls ?? [] }); },
sourceFiles: [...base.sourceFiles, "chat-catalog-domains/internal-project-defaults.ts", "chat-catalog-domains/internal-run-brief.ts", "../e2e/fetch-allowed-port.ts", "run-observations.ts"],
async act(ctx, state) {
const settled = await pollUntil({ label: "Canonical project-defaults turn settles", deadlineAt: Date.now() + 150000, load: () => observeInternal(ctx, state),
accept: o => o.runs.some(row => row.agentId === o.leadId && row.status === "succeeded") && ctx.issue()?.conversationState === "waiting" });
const settled = await pollUntil({ label: "Canonical project-defaults turn settles", deadlineAt: chatCatalogNativePhaseDeadline(ctx, Date.now() + 150000), load: () => observeInternal(ctx, state),
accept: o => chatCatalogNativePhaseIsTimely(ctx) && o.runs.some(row => row.agentId === o.leadId && row.status === "succeeded") && ctx.issue()?.conversationState === "waiting" });
state.defaultsReply = settled.reply;
state.futureTask = await task(ctx, "Verify inherited Launch defaults", entity(state, "project").id, { status: "todo", assigneeAgentId: entity(state, "futureWorker").id,
description: "Use the current saved project/task context to report which project checklist applies to this new work and the associated goal. Do not change project defaults or any other task. Complete after recording your factual findings, including if no defaults or goal are present." });
await pollUntil({ label: "Actual new native task reads inherited project context", deadlineAt: Date.now() + 180000, load: () => ctx.allRuns(), accept: rows => rows.length === 3 && rows.some(row => row.agentId === entity(state, "futureWorker").id && row.status === "succeeded") });
await pollUntil({ label: "Actual new native task reads inherited project context", deadlineAt: chatCatalogNativePhaseDeadline(ctx, Date.now() + 180000), load: () => ctx.allRuns(), accept: rows => chatCatalogNativePhaseIsTimely(ctx) && rows.length === 3 && rows.some(row => row.agentId === entity(state, "futureWorker").id && row.status === "succeeded") });
},
async observe(ctx, state): Promise<DefaultsObservation> { const o = await observeInternal(ctx, state); const run = state.futureTask ? o.runs.find(row => row.agentId === entity(state, "futureWorker").id && (row.contextSnapshot?.issueId === entity(state, "futureTask").id || row.contextSnapshot?.taskId === entity(state, "futureTask").id)) : undefined;
const events = run ? await collectRunEvents((after, limit) => ctx.input.api.get(`/api/heartbeat-runs/${run.id}/events?afterSeq=${after}&limit=${limit}`)) : [];
tests/runner-e2e/chat-catalog-domains/internal.test.ts
+3 -3
View File
@@ -304,11 +304,11 @@ function valid(scenario: string): InternalObservation {
o.runs.push({ id: "final", companyId: "company", agentId: "ceo", status: "succeeded" }); Object.assign(o, { phaseReply: "Casey's newer offline research prototype edit is retained in history. I restored the selected revision 2 as current revision 7." }); break;
}
case "activity.company-overview": {
o.tasks[0] = { ...o.tasks[0]!, status: "in_progress" }; o.state.task = { ...o.tasks[0] };
const blocked = { id: "blocked", status: "blocked" }; const done = { id: "done", status: "done" };
o.tasks[0] = { ...o.tasks[0]!, identifier: "ACORN-1", status: "in_progress" }; o.state.task = { ...o.tasks[0] };
const blocked = { id: "blocked", identifier: "ACORN-2", title: "Accessibility review", status: "blocked", description: "Waiting for approved launch colors." }; const done = { id: "done", identifier: "ACORN-3", title: "Launch brief delivered", status: "done" };
o.state.blockedTask = blocked; o.state.doneTask = done; o.tasks.push(blocked, done); o.state.worker = { id: "morgan" }; o.agents = [{ id: "morgan", status: "idle" }];
o.state.blocker = "approved launch colors"; o.state.hiddenMarker = "hidden acquisition";
o.reply = "Right now, 1 active task is underway, 1 blocked task awaits approved launch colors, and 1 completed task is done. Morgan is idle.";
o.reply = "Right now, ACORN-1 Website task is underway (1 active task).\nACORN-2 Accessibility review is blocked, waiting for approved launch colors (1 blocked task).\nACORN-3 Launch brief delivered is done (1 completed task).\nMorgan is idle.";
o.links.push("/ACORN/issues/blocked", "/ACORN/issues/done"); break;
}
case "activity.audit-change": {
tests/runner-e2e/chat-catalog-domains/model-option-catalog.test.ts
+26
View File
@@ -0,0 +1,26 @@
import {it, expect} from "vitest";
import {gradeModelOptions} from "./runtime-runs-adapters.js";
import {openRouterCatalogScope, modelChoiceMatches, publicCatalogCaveat, publicOpenRouterCatalogEndpoint} from "./model-option-catalog.js";
function valid() {
const config = {provider:"opencode",model:"openrouter/deepseek/deepseek-v4.1-flash-1017",env:{OPENROUTER_API_KEY:{type:"secret_ref",secretId:"company-openrouter-reference",version:"latest"}}};
const worker = {id:"morgan",companyId:"company",adapterType:"paperclip_runner",defaultEnvironmentId:"owned",adapterConfig:structuredClone(config)}, lead={...structuredClone(worker),id:"ceo"};
return {scenarioId:"runs-adapters.list-model-options",companyId:"company",leadId:"ceo",state:{worker,admittedModels:[{id:"opencode/big-pickle",label:"builtin"}],discoveryStatus:200,agentRuntimeInventoryStatus:403,secretScanComplete:true,secretLeaked:false,
publicOpenRouterCatalog:[{id:config.model,label:"current"},{id:"openrouter/openai/gpt-5.3-codex",label:"catalog"},{id:"openrouter/anthropic/claude-sonnet-5.5",label:"catalog"}],publicCatalogStatus:200,publicCatalogScope:openRouterCatalogScope("company","ceo",worker,lead)},
agents:[structuredClone(worker),lead],tasks:[],links:[],activity:[],runs:[],humanQueries:[],reply:"Morgan's configured model is `openrouter/deepseek/deepseek-v4.1-flash-1017`. The native options include `opencode/big-pickle`. The public OpenRouter catalog also lists `gpt-5.3-codex` and `anthropic/claude-sonnet-5.5`; availability depends on the selected account/key and environment. Not all catalog models are qualified or tested."};
}
it("accepts actually catalogued options with explicit current-account limits, without testing another model",()=>expect(gradeModelOptions(valid()).passed).toBe(true));
it("uses the existing company-scoped public metadata endpoint",()=>expect(publicOpenRouterCatalogEndpoint("company")).toBe("/api/companies/company/adapters/opencode_local/models?provider=openrouter"));
it.each(["gpt-99-ultra","openrouter/openai/not-a-real-model","wrong-vendor/gpt-5.3-codex","anthropic/gpt-5.3-codex"])("rejects invented/out-of-catalog or wrong-vendor choice %s",id=>{const o=valid();o.reply+=` Another option is \`${id}\`.`;expect(gradeModelOptions(o).passed).toBe(false)});
it.each(["companyId","leadId","workerId","environmentId","credentialReferenceSha256"])("rejects a foreign saved catalog binding %s",field=>{const o=valid();Object.assign(o.state.publicCatalogScope!,{[field]:"foreign"});expect(gradeModelOptions(o).passed).toBe(false)});
it.each(["companyId","defaultEnvironmentId"])("rejects changed actual target scope %s",field=>{const o=valid();Object.assign(o.agents[0]!,{[field]:"foreign"});expect(gradeModelOptions(o).passed).toBe(false)});
it("rejects another actual key reference, plain private key, changed provider and missing public discovery",()=>{for(const mutate of[(o:ReturnType<typeof valid>)=>{o.agents[1]!.adapterConfig.env.OPENROUTER_API_KEY.secretId="other"},(o:ReturnType<typeof valid>)=>{Object.assign(o.agents[1]!.adapterConfig.env,{OPENROUTER_API_KEY:"synthetic-private-value"})},(o:ReturnType<typeof valid>)=>{o.agents[1]!.adapterConfig.provider="codex"},(o:ReturnType<typeof valid>)=>{o.state.publicCatalogStatus=403}]){const o=valid();mutate(o);expect(gradeModelOptions(o).passed).toBe(false)}});
it.each(["All catalog models are tested and ready.","Every model will work with this key.","All models are installed; the account is configured.","All models are available on this key."])("rejects blanket runtime qualification %s",claim=>{const o=valid();o.reply+=" "+claim;expect(gradeModelOptions(o).passed).toBe(false)});
it("requires a real caveat for public catalog choices",()=>{const o=valid();o.reply="The selected configured account has `opencode/big-pickle` and `gpt-5.3-codex` available.";expect(gradeModelOptions(o).passed).toBe(false)});
it("does not treat an explicit not-all-qualified caveat as an all-ready claim",()=>expect(publicCatalogCaveat("Catalog models depend on the account key; not all models are tested or qualified.")).toBe(true));
it("rejects ambiguous bare names across distinct published vendors",()=>expect(modelChoiceMatches("shared-model",[{id:"openrouter/vendor-a/shared-model",label:"a"},{id:"openrouter/vendor-b/shared-model",label:"b"}])).toBe(false));
it("keeps the original private/runtime/configuration negatives",()=>{const leaked=valid();leaked.state.secretLeaked=true;expect(gradeModelOptions(leaked).passed).toBe(false);const runtime=valid();runtime.state.agentRuntimeInventoryStatus=200;expect(gradeModelOptions(runtime).passed).toBe(false);const changed=valid();changed.agents[0]!.adapterConfig.model="different";expect(gradeModelOptions(changed).passed).toBe(false)});
it("binds the entire public secret-reference projection rather than only its secret ID",()=>{const o=valid();Object.assign(o.agents[1]!.adapterConfig.env.OPENROUTER_API_KEY,{projectionAllowlistKey:"another-scope"});expect(gradeModelOptions(o).passed).toBe(false)});
it.each(["openrouter/vendor/not-a-published-model", "openrouter/wrong-vendor/gpt-5.3-codex"])("rejects plain unquoted unsupported model %s",id=>{const o=valid();o.reply+=" Another model option is "+id+".";expect(gradeModelOptions(o).passed).toBe(false)});
it("accepts plain unquoted actual public model with the same entitlement caveat",()=>{const o=valid();o.reply+=" Another public catalog option is openrouter/openai/gpt-5.3-codex.";expect(gradeModelOptions(o).passed).toBe(true)});
tests/runner-e2e/chat-catalog-domains/model-option-catalog.ts
+53
View File
@@ -0,0 +1,53 @@
import { createHash } from "node:crypto";
import type { RuntimeAgent } from "./runtime-support.js";
export interface DiscoveredModel { id: string; label: string }
export interface OpenRouterCatalogScope { companyId: string; leadId: string; workerId: string; environmentId: string; currentModel: string; credentialReferenceSha256: string }
function credentialReference(agent: RuntimeAgent) {
const env = agent.adapterConfig?.env as Record<string, unknown> | undefined;
const value = env?.OPENROUTER_API_KEY as { type?: unknown; secretId?: unknown; version?: unknown; projectionClass?: unknown; projectionAllowlistKey?: unknown } | undefined;
if (!value || value.type !== "secret_ref" || typeof value.secretId !== "string" || !value.secretId
|| !(value.version === "latest" || Number.isSafeInteger(value.version) && Number(value.version) > 0)) return;
if (Object.keys(value).some(key => !["type", "secretId", "version", "projectionClass", "projectionAllowlistKey"].includes(key))
|| [value.projectionClass, value.projectionAllowlistKey].some(field => field !== undefined && field !== null && typeof field !== "string")) return;
return { type: value.type, secretId: value.secretId, version: value.version,
...(value.projectionClass !== undefined ? { projectionClass: value.projectionClass } : {}),
...(value.projectionAllowlistKey !== undefined ? { projectionAllowlistKey: value.projectionAllowlistKey } : {}) };
}
export function openRouterCatalogScope(companyId: string, leadId: string, worker: RuntimeAgent, lead: RuntimeAgent): OpenRouterCatalogScope | undefined {
const a = credentialReference(worker), b = credentialReference(lead), environmentId = worker.defaultEnvironmentId;
if (worker.companyId !== companyId || lead.companyId !== companyId || lead.id !== leadId || worker.id === leadId
|| worker.adapterType !== "paperclip_runner" || lead.adapterType !== "paperclip_runner" || worker.adapterConfig?.provider !== "opencode" || lead.adapterConfig?.provider !== "opencode"
|| !environmentId || environmentId !== lead.defaultEnvironmentId || typeof worker.adapterConfig.model !== "string" || !worker.adapterConfig.model.startsWith("openrouter/")
|| worker.adapterConfig.model !== lead.adapterConfig.model || !a || !b || JSON.stringify(a) !== JSON.stringify(b)) return;
return { companyId, leadId, workerId: worker.id, environmentId, currentModel: worker.adapterConfig.model,
credentialReferenceSha256: createHash("sha256").update(JSON.stringify(a)).digest("hex") };
}
export function publicOpenRouterCatalogEndpoint(companyId: string) {
return `/api/companies/${encodeURIComponent(companyId)}/adapters/opencode_local/models?provider=openrouter`;
}
function normalized(id: string) { return id.toLowerCase().replace(/^openrouter\//, ""); }
export function observedModelChoices(reply: string) {
const vendor = [...reply.matchAll(/\b(?:gpt-\d[\w.-]*|o[134](?:-[\w.-]+)?|claude-(?:sonnet|opus|haiku)-[\w.-]+)/gi)].map(m => m[0].replace(/[.,;:!?]+$/, ""));
const code = [...reply.matchAll(/`([^`\n]+)`/g)].map(m => m[1]!).filter(s => /^[\w.-]+\/[\w./:+-]+$/.test(s) && !s.includes("://"));
const plainOpenRouter = [...reply.matchAll(/(?<![\w/:])openrouter\/[a-z0-9][\w.-]*\/[a-z0-9][\w./:+-]*/gi)].map(m => m[0].replace(/[.,;:!?]+$/, ""));
return [...new Set([...vendor, ...code, ...plainOpenRouter])];
}
export function modelChoiceMatches(id: string, models: DiscoveredModel[]) {
const wanted = normalized(id), exact = models.some(m => normalized(m.id) === wanted);
if (exact) return true;
if (wanted.includes("/")) return false;
const candidates = new Set(models.filter(m => normalized(m.id).endsWith("/" + wanted)).map(m => normalized(m.id)));
if (/^(?:gpt-|o[134](?:-|$))/.test(wanted)) return candidates.size === 1 && [...candidates][0]!.startsWith("openai/");
if (/^claude-/.test(wanted)) return candidates.size === 1 && [...candidates][0]!.startsWith("anthropic/");
return candidates.size === 1;
}
export function unsupportedModelReadinessClaim(reply: string) {
return reply.split(/[.!?;\n]|,|:\s|\bbut\b|\bhowever\b/i).some(clause => /\b(?:all|every)\b/i.test(clause)
&& /\b(?:ready|qualified|tested|guaranteed|installed|available|usable|will work)\b/i.test(clause)
&& !/\b(?:not|no|cannot|can't|aren't|are not|unverified|untested)\b/i.test(clause));
}
export function publicCatalogCaveat(reply: string) {
return /\b(?:account|key|credential|entitlement|credit|plan)\b/i.test(reply)
&& /\b(?:depends?|restricted?|subject to|unverified|not (?:all |yet )?(?:tested|qualified|guaranteed|ready))\b/i.test(reply)
&& !unsupportedModelReadinessClaim(reply);
}
tests/runner-e2e/chat-catalog-domains/recorded-work-claims.test.ts
+72
View File
@@ -0,0 +1,72 @@
import { describe, expect, it } from "vitest";
import { grade_activity } from "./internal-activity.js";
import { gradeIntegratorRelatedWork } from "./integrator-case-related.js";
function overview() {
const active = { id: "active", identifier: "ACN-1", title: "Website draft", status: "in_progress" };
const blocked = { id: "blocked", identifier: "ACN-2", title: "Accessibility review", status: "blocked", description: "Waiting for approved launch colors nonce-1." };
const done = { id: "done", identifier: "ACN-3", title: "Launch brief delivered", status: "done" };
return { scenarioId: "activity.company-overview", origin: "https://fixture.test", companyId: "company", leadId: "ceo", state: { task: active, blockedTask: blocked, doneTask: done, worker: { id: "morgan" }, blocker: "approved launch colors nonce-1", hiddenMarker: "HIDDEN-SENTINEL" },
tasks: [active, blocked, done], projects: [], goals: [], comments: [], activity: [], documents: [], revisions: [], humanQueries: [], runs: [], agents: [{ id: "morgan", status: "idle" }], links: ["/ACN/issues/ACN-1", "/ACN/issues/ACN-2", "/ACN/issues/ACN-3"],
reply: "Here's the picture right now:\n\n- **ACN-1 Website draft** — in progress, assigned to the board.\n- **ACN-2 Accessibility review** — blocked, unassigned. Waiting on approved launch colors, but there are zero approvals on record.\n- **ACN-3 Launch brief delivered** — done, unassigned.\n\nMorgan is idle.\n\nNeeds a decision: ACN-2 can't move until the launch color palette is approved, and no approval card exists." };
}
function related() {
const primary = { id: "primary", identifier: "ACN-1", title: "Investigation", status: "done" };
const followup = { id: "followup", identifier: "ACN-2", title: "Mobile follow-up", status: "backlog" };
const unrelated = { id: "unrelated", identifier: "ACN-3", title: "Mobile follow-up investigation", status: "backlog" };
const row = { id: "case", identifier: "ACN-C1", workIssueId: "primary", status: "draft", handoffVersion: 1 };
return { scenarioId: "cases.related-work", origin: "https://fixture.test", companyId: "company", leadId: "ceo", state: { case: structuredClone(row), followup, unrelated, privateTask: { id: "private", identifier: "PRV-1", title: "Private customer" }, privateMarker: "PRIVATE-SENTINEL" },
tasks: [primary, followup, unrelated], projects: [], goals: [], comments: [], activity: [], documents: [], revisions: [], humanQueries: [], runs: [], links: ["/ACN/issues/ACN-1", "/ACN/issues/ACN-2", "/ACN/issues/ACN-3"],
caseDetail: { ...row, issueLinks: [{ issueId: "primary", role: "work" }, { issueId: "followup", role: "reference" }] }, reverseLinks: { primary: [{ role: "work", case: structuredClone(row) }], followup: [{ role: "reference", case: structuredClone(row) }] }, deniedStatus: 404,
reply: "| Task | Role | Owner | Status |\n|---|---|---|---|\n| ACN-1 Investigation | work | Morgan | done |\n| ACN-2 Mobile follow-up | reference | unassigned | backlog |\n\nNot tied: ACN-3 Mobile follow-up investigation has a similar title but is explicitly unrelated and has no case link.\n\nThe case remains open." };
}
describe("recorded overview blocker and per-task state grounding", () => {
it("accepts the observed bullet/conditional form without requiring an incidental nonce", () => { expect(grade_activity(overview()).passed).toBe(true); });
it("does not treat explicit non-approval or future approval as resolved", () => {
const v = overview(); v.reply += "\nACN-2 is blocked; its colors are not approved."; expect(grade_activity(v).passed).toBe(true);
});
it.each([
["wrong reason", (v: ReturnType<typeof overview>) => { v.reply = v.reply.replace("Waiting on approved launch colors", "Waiting on database access"); }],
["reason on wrong task", (v: ReturnType<typeof overview>) => { v.reply = v.reply.replace("Waiting on approved launch colors", "Waiting for a decision").replace("assigned to the board.", "assigned to the board. Waiting on approved launch colors."); }],
["wrong active state despite a matching count label", (v: ReturnType<typeof overview>) => { v.reply = v.reply.replace("in progress, assigned", "blocked (1 active task), assigned"); }],
["wrong active state", (v: ReturnType<typeof overview>) => { v.reply = v.reply.replace("in progress, assigned", "blocked, assigned"); }],
["wrong done state", (v: ReturnType<typeof overview>) => { v.reply = v.reply.replace("— done, unassigned", "— in progress, unassigned"); }],
["wrong blocked state", (v: ReturnType<typeof overview>) => { v.reply = v.reply.replace("— blocked, unassigned", "— done, unassigned"); }],
["false resolution", (v: ReturnType<typeof overview>) => { v.reply += "\nACN-2 colors are approved and it is ready."; }],
["false readiness after conditional", (v: ReturnType<typeof overview>) => { v.reply += "\nACN-2 can't move until the palette is approved, but is ready now."; }],
["later contradictory state", (v: ReturnType<typeof overview>) => { v.reply += "\nACN-2 is now completed."; }],
["missing task link", (v: ReturnType<typeof overview>) => { v.links.shift(); }],
["hidden work", (v: ReturnType<typeof overview>) => { v.reply += " HIDDEN-SENTINEL"; }],
["wrong recorded blocker", (v: ReturnType<typeof overview>) => { v.state.blockedTask = { ...v.state.blockedTask, description: "Waiting for legal review." }; }],
])("rejects %s", (_name, mutate) => { const v = overview(); mutate(v); expect(grade_activity(v).passed).toBe(false); });
});
describe("actual work/reference roles and explicit negative related-work context", () => {
it("accepts recorded work/reference table roles and a visibly linked but explicitly unrelated accessible task", () => { expect(gradeIntegratorRelatedWork(related()).passed).toBe(true); });
it("retains harmless external references with title-grounded actual work", () => { const v=related();v.reply="The main Investigation task is done after change PR-123.\nIts reference Mobile follow-up remains backlog.\nACN-3 is explicitly unrelated and has no case link."; expect(gradeIntegratorRelatedWork(v).passed).toBe(true); });
it("retains harmless external numeric issue references", () => { const v=related();v.reply="The main Investigation task is done after [a change](https://github.com/acme/repo/issues/123).\nIts reference Mobile follow-up remains backlog.\nACN-3 is explicitly unrelated and has no case link."; expect(gradeIntegratorRelatedWork(v).passed).toBe(true); });
it("accepts a direct denial of case membership", () => { const v=related();v.reply=v.reply.replace("has a similar title but is explicitly unrelated and has no case link", "is not part of this case"); expect(gradeIntegratorRelatedWork(v).passed).toBe(true); });
it.each([
["wrong local identifier despite correct identifier URL", (v: ReturnType<typeof related>) => { v.reply = v.reply.replace("ACN-1 Investigation", "[ACN-9 Investigation](/ACN/issues/ACN-1)"); }],
["wrong local identifier despite copied title and correct URL", (v: ReturnType<typeof related>) => { v.reply = v.reply.replace("ACN-1 Investigation", "[ACN-9 Investigation](/ACN/issues/primary)"); }],
["explicit foreign issue resource with copied title", (v: ReturnType<typeof related>) => { v.reply = v.reply.replace("ACN-1 Investigation", "[ZZZ-9 Investigation](/ZZZ/issues/ZZZ-9)"); }],
["foreign primary identifier with copied title", (v: ReturnType<typeof related>) => { v.reply = v.reply.replace("ACN-1 Investigation", "ACN-9 Investigation"); }],
["foreign reference identifier with copied title", (v: ReturnType<typeof related>) => { v.reply = v.reply.replace("ACN-2 Mobile follow-up", "ACN-9 Mobile follow-up"); }],
["swapped primary role", (v: ReturnType<typeof related>) => { v.reply = v.reply.replace("ACN-1 Investigation | work", "ACN-1 Investigation | reference"); }],
["swapped reference role", (v: ReturnType<typeof related>) => { v.reply = v.reply.replace("ACN-2 Mobile follow-up | reference", "ACN-2 Mobile follow-up | work"); }],
["primary falsely incomplete", (v: ReturnType<typeof related>) => { v.reply = v.reply.replace("Morgan | done", "Morgan | backlog"); }],
["followup falsely done", (v: ReturnType<typeof related>) => { v.reply = v.reply.replace("unassigned | backlog", "unassigned | done"); }],
["unrelated link without denial", (v: ReturnType<typeof related>) => { v.reply = v.reply.replace(/Not tied:[^\n]+/, "ACN-3 is another task."); }],
["unrelated asserted reference", (v: ReturnType<typeof related>) => { v.reply += "\nACN-3 is a reference."; }],
["denial plus contradictory assertion", (v: ReturnType<typeof related>) => { v.reply = v.reply.replace("has no case link", "has no case link but actually is a reference"); }],
["private identifier", (v: ReturnType<typeof related>) => { v.reply += "\nPRV-1 is also involved."; }],
["foreign private link", (v: ReturnType<typeof related>) => { v.links.push("https://other.test/issues/private"); }],
["case-varied private marker", (v: ReturnType<typeof related>) => { v.reply += "\nprivate-sentinel"; }],
["private title", (v: ReturnType<typeof related>) => { v.reply += "\nPrivate customer has work too."; }],
["private marker", (v: ReturnType<typeof related>) => { v.reply += "\nPRIVATE-SENTINEL"; }],
["private link", (v: ReturnType<typeof related>) => { v.links.push("/PRV/issues/PRV-1"); }],
["private access unexpectedly allowed", (v: ReturnType<typeof related>) => { v.deniedStatus = 200; }],
["lost reverse relationship", (v: ReturnType<typeof related>) => { v.reverseLinks.followup = []; }],
["relationship mutation", (v: ReturnType<typeof related>) => { v.caseDetail.handoffVersion++; }],
["new unrelated link relationship", (v: ReturnType<typeof related>) => { v.caseDetail.issueLinks.push({ issueId: "unrelated", role: "reference" }); }],
["later false primary state", (v: ReturnType<typeof related>) => { v.reply += "\n| ACN-1 Investigation | work | Morgan | backlog |"; }],
])("rejects %s", (_name, mutate) => { const v = related(); mutate(v); expect(gradeIntegratorRelatedWork(v).passed).toBe(false); });
});
tests/runner-e2e/chat-catalog-domains/recorded-work-claims.ts
+83
View File
@@ -0,0 +1,83 @@
import type { Entity } from "./internal-support.js";
const escape = (value: string) => value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
const mentions = (value: string, name: string | undefined) => !!name && new RegExp(`(?<![\\w-])${escape(name)}(?![\\w-])`, "i").test(value);
export function recordedTaskUnits(reply: string, target: Entity, knownTasks: Entity[]) {
const units: Array<{ taskId: string; text: string }> = [];
let current: { taskId: string; text: string } | undefined;
for (const line of reply.split(/\n/)) {
if (!line.trim()) { current = undefined; continue; }
if (/^\s*(?:[-*]|\d+\.|#{1,6})\s/.test(line)) current = undefined;
for (const sentence of line.split(/(?<=[.!?;])\s+/)) {
const prose = sentence.replace(/(?:https?:\/\/|(?<![\w])\/)[^\s)\x60\]>]+/gi, "");
const companyPrefixes = knownTasks.flatMap(task => task.identifier?.match(/^(.+-)\d+$/)?.[1]?.toLowerCase() ?? []);
const explicitIdentifiers = (prose.match(/(?<![\w-])[A-Z][A-Z0-9]*-\d+(?![\w-])/gi) ?? [])
.filter(value => companyPrefixes.some(prefix => value.toLowerCase().startsWith(prefix)));
const textIdentifiers = knownTasks.filter(task => mentions(prose, task.identifier));
const urlIdentifiers = knownTasks.filter(task => [task.id, task.identifier].some(id => !!id && new RegExp(`/issues/${escape(id)}(?![\\w-])`, "i").test(sentence)));
const unknownIssueResource = /\/issues\/(?:[a-f0-9]{8}-[a-f0-9-]{27}|[A-Z][A-Z0-9]*-\d+)(?![\w-])/i.test(sentence) && !urlIdentifiers.length;
const identifiers = unknownIssueResource ? [] : textIdentifiers.length ? textIdentifiers : explicitIdentifiers.length ? [] : urlIdentifiers;
let subjects = identifiers.length ? identifiers : explicitIdentifiers.length || unknownIssueResource ? [] : knownTasks.filter(task => mentions(prose, task.title));
if (!identifiers.length) subjects = subjects.filter(task => !subjects.some(other => other.id !== task.id && other.title !== task.title && other.title?.includes(task.title ?? "")));
const ids = [...new Set(subjects.map(task => task.id))];
if (ids.length === 1) { current = { taskId: ids[0]!, text: sentence }; units.push(current); }
else if (ids.length > 1 || explicitIdentifiers.length || unknownIssueResource) current = undefined;
else if (current) current.text += ` ${sentence}`;
}
}
return units.filter(unit => unit.taskId === target.id).map(unit => unit.text);
}
const states = {
in_progress: { yes: /\b(?:active|in.progress|underway)\b/i, no: /\b(?:not (?:active|in.progress|underway)|(?:is|remains|now) (?:blocked|done|completed|backlog))\b/i, otherCells: /\|\s*(?:blocked|done|completed|backlog)\s*\|/i },
blocked: { yes: /\bblocked\b/i, no: /\b(?:not blocked|no longer blocked|unblocked|(?:is|now) (?:done|completed|ready))\b/i, otherCells: /\|\s*(?:done|completed|backlog|in.progress)\s*\|/i },
done: { yes: /\b(?:done|completed|finished)\b/i, no: /\b(?:not (?:done|completed|finished)|still (?:running|pending)|(?:is|remains) (?:backlog|blocked|in.progress))\b/i, otherCells: /\|\s*(?:backlog|blocked|in.progress)\s*\|/i },
backlog: { yes: /\bbacklog\b/i, no: /\b(?:not (?:in )?backlog|done|completed|finished)\b/i, otherCells: /\|\s*(?:blocked|in.progress|active)\s*\|/i },
} as const;
export function reportsRecordedTaskState(reply: string, task: Entity, knownTasks: Entity[], state: keyof typeof states) {
const units = recordedTaskUnits(reply, task, knownTasks), rule = states[state];
const contrary = {
in_progress: /(?:[—:]|\b(?:is|remains|now))\s*(?:blocked|done|completed|backlog)\b/i,
blocked: /(?:[—:]|\b(?:is|remains|now))\s*(?:in.progress|done|completed|backlog|ready)\b/i,
done: /(?:[—:]|\b(?:is|remains|now))\s*(?:in.progress|blocked|backlog)\b/i,
backlog: /(?:[—:]|\b(?:is|remains|now))\s*(?:in.progress|blocked|done|completed)\b/i,
}[state];
const wrong = (unit: string) => rule.no.test(unit) || rule.otherCells.test(unit) || contrary.test(unit);
return units.some(unit => rule.yes.test(unit) && !wrong(unit)) && !units.some(wrong);
}
export function groundedOverviewBlocker(reply: string, blocked: Entity, knownTasks: Entity[], recordedBlocker: unknown) {
const reason = "approved launch colors";
if (typeof recordedBlocker !== "string" || !(recordedBlocker === reason || recordedBlocker.startsWith(`${reason} `))
|| blocked.status !== "blocked" || blocked.description !== `Waiting for ${recordedBlocker}.`) return false;
const units = recordedTaskUnits(reply, blocked, knownTasks);
const resolved = /\b(?:not|no longer) (?:blocked|waiting|awaiting)\b|\b(?:is|now|already) (?:ready|unblocked|resolved)\b|\bcan (?:proceed|start)\b|\b(?:colors|palette) (?:are|is|were|was|have been|has been) (?:already )?approved\b/i;
const claimsResolution = (unit: string) => resolved.test(unit.replace(/\b(?:until|once|when|if|after)\b[^.!?;]{0,100}\b(?:colors|palette) (?:are|is|were|was|have been|has been) (?:already )?approved\b/gi, ""));
return units.some(unit => unit.includes(reason) && /\b(?:blocked|waiting|awaits?|awaiting|pending|needs?)\b/i.test(unit) && !claimsResolution(unit))
&& !units.some(claimsResolution);
}
export function reportsRecordedCaseRole(reply: string, task: Entity, linkedTasks: Entity[], role: "work" | "reference") {
const units = recordedTaskUnits(reply, task, linkedTasks);
const work = /\b(?:primary|main|initial investigation)\b|\|\s*work\s*\|/i;
const reference = /\b(?:linked|reference)\b|\|\s*reference\s*\|/i;
const yes = role === "work" ? work : reference;
const other = role === "work" ? /\|\s*reference\s*\||\b(?:is|as|role[: ]+) (?:a |the )?reference\b/i : /\|\s*work\s*\||\b(?:is|as|role[: ]+) (?:a |the )?(?:primary|main)\b/i;
const denied = role === "work" ? /\bnot (?:the |a )?(?:primary|main|work task)\b/i : /\bnot (?:the |a )?(?:linked|reference)\b/i;
return units.some(unit => yes.test(unit) && !other.test(unit) && !denied.test(unit)) && !units.some(unit => other.test(unit) || denied.test(unit));
}
export function explicitlyUnrelatedWork(reply: string, unrelated: Entity, knownTasks: Entity[]) {
const units = recordedTaskUnits(reply, unrelated, knownTasks);
const negative = /\b(?:not tied|explicitly unrelated|not (?:linked|related|connected)|no case link|not part of (?:the |this )?case)\b/i;
const affirmative = /\b(?:is|actually|now|also) (?:a |the )?(?:linked|related|reference|primary|main)\b|\b(?:part of|belongs to) (?:the |this )?case\b|\|\s*(?:work|reference)\s*\|/i;
return units.length > 0 && units.every(unit => negative.test(unit) && !affirmative.test(unit.replace(/\b(?:not tied|explicitly unrelated|not (?:linked|related|connected)|no case link|not part of (?:the |this )?case)\b/gi, "")));
}
export function privateWorkUndisclosed(reply: string, links: string[], task: Entity, marker: unknown) {
if (!task.id || !task.title || typeof marker !== "string" || !marker) return false;
const lower = reply.toLowerCase();
if (lower.includes(task.title.toLowerCase()) || lower.includes(marker.toLowerCase())) return false;
const opaqueId = /^[a-f0-9]{8}-[a-f0-9-]{27}$/i.test(task.id) ? task.id : undefined;
if (mentions(reply, opaqueId) || mentions(reply, task.identifier)) return false;
return links.every(href => {
try { const decoded = decodeURIComponent(href); return !mentions(decoded, task.id) && !mentions(decoded, task.identifier) && !decoded.toLowerCase().includes(marker.toLowerCase()); }
catch { return false; }
});
}
tests/runner-e2e/chat-catalog-domains/routine-history-work-failure.ts
+115
View File
@@ -0,0 +1,115 @@
import { createHash } from "node:crypto";
import type { BudgetOverview } from "../../../packages/shared/src/types/budget.js";
import type { RoutineRunSummary } from "../../../packages/shared/src/types/routine.js";
import type { ToolRunDecisionLookup } from "../../../packages/shared/src/types/tool-access.js";
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
import type { ChatRun } from "../chat-flow.js";
import { pollUntil } from "../api.js";
import { connectNotes, startNotesProvider, type NotesProvider, type ProviderCall } from "./external-provider.js";
import type { Entity } from "./internal-support.js";
export interface HistoryWorkIdentity { companyId: string; agentId: string; issueId: string; runId: string; routineId: string; invocationId: string; connectionId: string; writeEntryId: string }
export interface AccountedHistoryRun extends ChatRun { finishedAt?: string | null; usageJson?: Record<string, unknown> | null; costAccountingPending?: boolean; costAccountedAt?: string | null }
export interface HistoryWorkSnapshot { run?: AccountedHistoryRun; task: Entity; invocation?: RoutineRunSummary; decisions: ToolRunDecisionLookup; comments: Entity[]; activity: Array<Entity & { runId?: string }> }
export type WorkCall = Pick<ProviderCall, "method" | "name" | "authorized" | "sequence" | "requestIdSha256" | "responseHttpStatus" | "toolError">;
const providers = new WeakMap<ChatCatalogContext, { provider: NotesProvider; connectionId?: string }>();
const root = (ctx: ChatCatalogContext) => `/api/companies/${ctx.input.fixtures.company.id}`;
const positiveInteger = (value: unknown) => typeof value === "number" && Number.isSafeInteger(value) && value > 0;
const nonnegativeInteger = (value: unknown) => typeof value === "number" && Number.isSafeInteger(value) && value >= 0;
const record = (value: unknown): Record<string, unknown> => value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
const text = (value: unknown) => typeof value === "string" && value.length > 0 && value !== "unknown";
export function completeHistoryUsage(run: AccountedHistoryRun | undefined) {
const u = run?.usageJson;
return !!run && !!u && run.status === "succeeded" && run.runtimeMode === "native" && !!run.finishedAt
&& run.costAccountingPending === false && typeof run.costAccountedAt === "string" && Number.isFinite(Date.parse(run.costAccountedAt))
&& record(record(run.resultJson).executionRecovery).providerWorkStarted !== false
&& u.accountingReceiptReady === true && u.accountingCaptureFailed !== true && u.accountingProviderWorkStarted !== false
&& text(u.accountingReceiptId) && text(u.accountingReceiptSourceId) && positiveInteger(u.accountingReceiptSequence)
&& typeof u.accountingReceiptReceivedAt === "string" && Number.isFinite(Date.parse(u.accountingReceiptReceivedAt))
&& u.usageSource === "per_run" && text(u.provider) && text(u.biller) && text(u.model)
&& ["metered_api", "subscription_included"].includes(String(u.billingType))
&& nonnegativeInteger(u.inputTokens) && nonnegativeInteger(u.outputTokens) && Number(u.inputTokens) + Number(u.outputTokens) > 0;
}
export function routineHistoryAccountingReady(expected: Pick<HistoryWorkIdentity, "companyId" | "agentId" | "issueId" | "runId">, runs: AccountedHistoryRun[], overview: BudgetOverview) {
const run = runs.find(row => row.id === expected.runId);
if (!run || run.companyId !== expected.companyId || run.agentId !== expected.agentId || run.contextSnapshot?.issueId !== expected.issueId
|| !completeHistoryUsage(run) || overview.companyId !== expected.companyId) return false;
const policies = overview.policies.filter(p => p.isActive && (p.scopeType === "company" && p.scopeId === expected.companyId || p.scopeType === "agent" && p.scopeId === expected.agentId));
return ["company", "agent"].every(scope => policies.some(p => p.scopeType === scope)) && policies.every(p => p.companyId === expected.companyId
&& p.hardStopEnabled && p.amount > 0 && p.unpricedUsagePolicy === "allow" && p.pendingRunCount === 0 && !p.paused && p.status !== "hard_stop" && p.remainingAmount > 0);
}
export function genuineHistoryWork(expected: HistoryWorkIdentity, snapshot: HistoryWorkSnapshot | undefined, calls: WorkCall[], failed: boolean) {
if (!expected || !snapshot) return false;
const { run, task, invocation, decisions, comments, activity } = snapshot;
const status = failed ? "blocked" : "done";
if (!run || run.id !== expected.runId || run.companyId !== expected.companyId || run.agentId !== expected.agentId
|| run.status !== "succeeded" || run.runtimeMode !== "native" || !run.startedAt || !run.finishedAt || run.contextSnapshot?.issueId !== expected.issueId
|| task.id !== expected.issueId || task.companyId !== expected.companyId || task.status !== status || task.assigneeAgentId !== expected.agentId
|| !invocation || invocation.id !== expected.invocationId || invocation.companyId !== expected.companyId || invocation.routineId !== expected.routineId
|| invocation.linkedIssueId !== expected.issueId || invocation.status !== (failed ? "failed" : "completed") || !invocation.completedAt || invocation.source !== "manual") return false;
const writes = calls.filter(call => call.method === "tools/call" && call.name === "launch_update");
if (!writes.length || !writes.every(call => call.authorized && call.responseHttpStatus === 200 && call.toolError === failed)) return false;
const receipt = decisions.runId === expected.runId && decisions.decisions.some(({ invocation: v, auditEvents }) =>
v.companyId === expected.companyId && v.agentId === expected.agentId && v.issueId === expected.issueId && v.runId === expected.runId
&& v.connectionId === expected.connectionId && v.catalogEntryId === expected.writeEntryId && v.policyDecision === "allow"
&& v.status === (failed ? "failed" : "succeeded") && (failed ? v.errorCode === "tool_error" : v.errorCode === null)
&& !!v.startedAt && !!v.completedAt && !!v.argumentsHash && (failed || !!v.resultHash)
&& auditEvents.some(a => a.invocationId === v.id && a.companyId === expected.companyId && a.agentId === expected.agentId && a.issueId === expected.issueId
&& a.runId === expected.runId && a.connectionId === expected.connectionId && a.catalogEntryId === expected.writeEntryId
&& a.outcome === (failed ? "failure" : "success") && a.decision === (failed ? "deny" : "allow")
&& a.reasonCode === (failed ? "tool_error" : "tool_completed") && record(record(a.metadata).execution).response != null
&& record(record(record(a.metadata).execution).response).httpStatus === 200
&& record(a.metadata).execution != null && record(record(a.metadata).execution).transport === "mcp_remote"
&& record(record(record(a.metadata).execution).request).dispatched === true
&& record(record(record(a.metadata).execution).request).upstreamToolName === "launch_update"
&& record(record(record(a.metadata).execution).request).mcpMethod === "tools/call"
&& typeof record(record(record(a.metadata).execution).request).requestId === "string"
&& writes.some(call => call.requestIdSha256 === createHash("sha256").update(String(record(record(record(a.metadata).execution).request).requestId)).digest("hex"))));
return !!receipt && comments.some(c => c.issueId === task.id && c.authorAgentId === expected.agentId && c.createdByRunId === run.id && !!c.body?.trim()
&& (!failed || /\b(?:fail\w*|error|blocked)\b/i.test(c.body)))
&& activity.some(a => a.companyId === expected.companyId && a.entityId === expected.issueId && a.runId === expected.runId && a.action === "issue.updated"
&& (a.actorType === "agent" && a.actorId === expected.agentId && a.details?.status === status
|| a.actorType === "system" && a.actorId === "native-status-committer" && a.details?.source === "native_status_decision" && a.details?.toStatus === status));
}
export async function prepareHistoryNotes(ctx: ChatCatalogContext, workerId: string) {
const provider = await startNotesProvider("Synthetic Launch project: payment approval remains pending; no other active work.");
const owned = { provider, connectionId: undefined as string | undefined }; providers.set(ctx, owned); provider.failWrites();
try {
const connection = await connectNotes(ctx, provider, "all"); owned.connectionId = connection.connectionId;
const agentIds = [ctx.input.fixtures.agent.id, workerId];
await ctx.input.api.post(`${root(ctx)}/tools/apps/${connection.connectionId}/finish`, { enabledCatalogEntryIds: [...connection.read, ...connection.write], askFirstCatalogEntryIds: [], access: { agentIds } });
await ctx.input.api.putSensitive(`/api/tool-connections/${connection.connectionId}/installs`, { installs: agentIds.map(targetId => ({ targetType: "agent", targetId })) });
return { connectionId: connection.connectionId, writeEntryId: connection.write[0]! };
} catch (error) { await cleanupHistoryNotes(ctx); throw error; }
}
export function historyNotesCalls(ctx: ChatCatalogContext, afterSequence = -1): WorkCall[] {
return providers.get(ctx)!.provider.calls.filter(c => c.sequence > afterSequence).map(({ method, name, authorized, sequence, requestIdSha256, responseHttpStatus, toolError }) => ({ method, name, authorized, sequence, requestIdSha256, responseHttpStatus, toolError }));
}
export function historyNotesPhase(ctx: ChatCatalogContext, recover = false) {
const provider = providers.get(ctx)!.provider;
if (recover) provider.allowWrites();
return { lastSequence: provider.calls.at(-1)?.sequence ?? -1, written: provider.written() ?? null };
}
export async function readHistoryWork(ctx: ChatCatalogContext, expected: Omit<HistoryWorkIdentity, "runId">): Promise<HistoryWorkSnapshot> {
const runs = (await ctx.allRuns()).filter(run => run.contextSnapshot?.issueId === expected.issueId);
if (runs.length > 1) throw new Error("Routine work exceeded its single declared native invocation");
const run = runs[0] ? await ctx.input.api.get<AccountedHistoryRun>(`/api/heartbeat-runs/${runs[0].id}`) : undefined;
const [task, history, comments, activity, decisions] = await Promise.all([
ctx.input.api.get<Entity>(`/api/issues/${expected.issueId}`), ctx.input.api.get<RoutineRunSummary[]>(`/api/routines/${expected.routineId}/runs`),
ctx.input.api.get<Entity[]>(`/api/issues/${expected.issueId}/comments?order=asc`), ctx.input.api.get<Array<Entity & { runId?: string }>>(`/api/issues/${expected.issueId}/activity`),
run ? ctx.input.api.get<ToolRunDecisionLookup>(`${root(ctx)}/tools/runs/${run.id}/decisions`) : Promise.resolve({ runId: "", decisions: [] }),
]);
return { run, task, invocation: history.find(row => row.id === expected.invocationId), comments, activity, decisions };
}
export async function waitHistoryWork(ctx: ChatCatalogContext, expected: Omit<HistoryWorkIdentity, "runId">, failed: boolean, afterSequence: number, durationMs: number, accounting = false) {
return pollUntil({ label: failed ? "actual failed routine work and native blocked decision" : "actual recovered routine work and complete normal usage", deadlineAt: Date.now() + durationMs, intervalMs: 1000,
load: async () => ({ snapshot: await readHistoryWork(ctx, expected), calls: historyNotesCalls(ctx, afterSequence), overview: accounting ? await ctx.input.api.get<BudgetOverview>(`${root(ctx)}/budgets/overview`) : undefined }),
accept: v => !!v.snapshot.run && genuineHistoryWork({ ...expected, runId: v.snapshot.run.id }, v.snapshot, v.calls, failed)
&& (!accounting || routineHistoryAccountingReady({ ...expected, runId: v.snapshot.run!.id }, [v.snapshot.run!], v.overview!)),
reject: v => v.snapshot.run && ["failed", "cancelled", "timed_out"].includes(v.snapshot.run.status) ? "Routine provider prerequisite did not yield a normally completed native turn" : undefined });
}
export async function cleanupHistoryNotes(ctx: ChatCatalogContext) {
const owned = providers.get(ctx); if (!owned) return; providers.delete(ctx);
try { if (owned.connectionId) await ctx.input.api.delete(`/api/tool-connections/${owned.connectionId}`); }
finally { await owned.provider.close(); }
}
tests/runner-e2e/chat-catalog-domains/runtime-dependency.ts
+8 -7
View File
@@ -1,3 +1,4 @@
import { chatCatalogNativePhaseDeadline, chatCatalogNativePhaseIsTimely, chatCatalogNativePhaseRemainingBudget } from "../chat-catalog-contract.js";
import { expect, type Page } from "@playwright/test";
import type { Issue } from "../../../packages/shared/src/types/issue.js";
import type { HeartbeatRun } from "../../../packages/shared/src/types/heartbeat.js";
@@ -42,9 +43,9 @@ export const runtimeDependencyCases = [runtimeDefinition("task-coordination.bloc
expectedHumanMessageCount: 2, expectedRunCount: 3, attemptTimeoutMs: 720_000,
sourceFiles: ["chat-catalog-domains/runtime-dependency.ts", "chat-catalog-domains/runtime-support.ts", "chat-scenarios/task-coordination.yaml"],
async act(ctx, state) {
await pollUntil({ label: "Saved dependency and canonical reply before human approval", deadlineAt: Date.now() + 150_000,
await pollUntil({ label: "Saved dependency and canonical reply before human approval", deadlineAt: chatCatalogNativePhaseDeadline(ctx, Date.now() + 150_000),
load: async () => { await ctx.refreshIssue(); return { source: ctx.issue(), runs: await ctx.allRuns() }; },
accept: v => v.source?.conversationState === "waiting" && v.runs.length === 1 && v.runs[0]?.status === "succeeded" });
accept: v => chatCatalogNativePhaseIsTimely(ctx) && v.source?.conversationState === "waiting" && v.runs.length === 1 && v.runs[0]?.status === "succeeded" });
state.queryReply = (await observeRuntime(ctx, state)).reply;
const source = ctx.input.page.url(), task = state.task as Entity, design = state.design as Entity, websiteRoot = `/api/issues/${task.id}`;
state.waitingTask = await ctx.input.api.get<Issue>(websiteRoot);
@@ -68,9 +69,9 @@ export const runtimeDependencyCases = [runtimeDefinition("task-coordination.bloc
await ctx.input.page.goto(`/${ctx.input.fixtures.company.issuePrefix}/issues/${task.identifier ?? task.id}`, { waitUntil: "domcontentloaded" });
await chooseStatus(ctx.input.page, "Todo");
}
const result = await pollUntil({ label: "One ordinary Website admission after Design approval", deadlineAt: Date.now() + 240_000,
const result = await pollUntil({ label: "One ordinary Website admission after Design approval", deadlineAt: chatCatalogNativePhaseDeadline(ctx, Date.now() + 240_000),
load: async () => ({ runs: await ctx.allRuns(), task: await ctx.input.api.get<Issue>(websiteRoot) }),
accept: v => v.runs.length === 2 && v.runs.every(row => row.status === "succeeded") && v.task.status === "done",
accept: v => chatCatalogNativePhaseIsTimely(ctx) && v.runs.length === 2 && v.runs.every(row => row.status === "succeeded") && v.task.status === "done",
reject: v => v.runs.length > 2 || v.runs.some(row => ["failed", "cancelled", "timed_out"].includes(row.status)) ? "Eligible Website admission failed or duplicated native work" : undefined });
const worker = result.runs.find(row => row.agentId === (state.worker as Entity).id)!;
const designActivity = await ctx.input.api.get<Entity[]>(`/api/issues/${design.id}/activity`);
@@ -79,10 +80,10 @@ export const runtimeDependencyCases = [runtimeDefinition("task-coordination.bloc
state.workerAdmissionAfterApproval = !!approval && Date.parse(String(workerDetail.createdAt)) >= Date.parse(approval.createdAt ?? ""); state.websiteRunId = worker.id;
await ctx.input.page.goto(source, { waitUntil: "domcontentloaded" });
const followup = `I approved Design ${design.identifier ?? design.id}. What is the actual result of Website ${task.identifier ?? task.id} now?`;
await sendChatMessage(ctx.input.page, followup);
const final = await pollUntil({ label: "Own native reply reporting actual post-approval Website outcome", deadlineAt: Date.now() + 150_000,
await sendChatMessage(ctx.input.page, followup, chatCatalogNativePhaseRemainingBudget(ctx));
const final = await pollUntil({ label: "Own native reply reporting actual post-approval Website outcome", deadlineAt: chatCatalogNativePhaseDeadline(ctx, Date.now() + 150_000),
load: async () => { await ctx.refreshIssue(); return { source: ctx.issue(), runs: await ctx.allRuns() }; },
accept: v => v.source?.conversationState === "waiting" && v.runs.length === 3 && v.runs.every(row => row.status === "succeeded") });
accept: v => chatCatalogNativePhaseIsTimely(ctx) && v.source?.conversationState === "waiting" && v.runs.length === 3 && v.runs.every(row => row.status === "succeeded") });
const run = final.runs.find(row => !result.runs.some(prior => prior.id === row.id))!, comments = await ctx.input.api.get<Entity[]>(`/api/issues/${ctx.issue()!.id}/comments?order=asc`);
const human = comments.filter(row => !row.authorAgentId && row.body === followup), replies = comments.filter(row => row.authorAgentId === ctx.input.fixtures.agent.id && row.createdByRunId === run.id);
const detail = await ctx.input.api.get<{ runnerProfileJson?: { nativeExecutionInput?: { task?: { prompt?: string } } } }>(`/api/heartbeat-runs/${run.id}`);
tests/runner-e2e/chat-catalog-domains/runtime-preview.ts
+6 -5
View File
@@ -1,3 +1,4 @@
import { chatCatalogNativePhaseDeadline, chatCatalogNativePhaseIsTimely, chatCatalogNativePhaseRemainingBudget } from "../chat-catalog-contract.js";
import { expect } from "@playwright/test";
import { readFile, realpath, writeFile, unlink } from "node:fs/promises";
import path from "node:path";
@@ -59,9 +60,9 @@ export const runtimePreviewCases = [runtimeDefinition("workspaces.preview-servic
sourceFiles: ["chat-catalog-domains/runtime-preview.ts", "chat-catalog-domains/runtime-workspaces.ts", "chat-catalog-domains/runtime-support.ts", "chat-scenarios/workspaces.yaml"],
expectedHumanMessageCount: 2, expectedRunCount: 3, attemptTimeoutMs: 720_000,
async act(ctx, state) {
await pollUntil({ label: "Actual canonical preview service readiness", deadlineAt: Date.now() + 180_000,
await pollUntil({ label: "Actual canonical preview service readiness", deadlineAt: chatCatalogNativePhaseDeadline(ctx, Date.now() + 180_000),
load: async () => { await ctx.refreshIssue(); return { source: ctx.issue(), runs: await ctx.allRuns(), workspace: await ctx.input.api.get<PreviewWorkspace>(`/api/execution-workspaces/${(state.workspace as Entity).id}`) }; },
accept: value => value.source?.conversationState === "waiting" && value.runs.length === 2 && value.runs.every(run => run.status === "succeeded")
accept: value => chatCatalogNativePhaseIsTimely(ctx) && value.source?.conversationState === "waiting" && value.runs.length === 2 && value.runs.every(run => run.status === "succeeded")
&& value.workspace.runtimeServices?.some(service => service.status === "running" && service.healthStatus === "healthy") === true });
const ws = await ctx.input.api.get<PreviewWorkspace>(`/api/execution-workspaces/${(state.workspace as Entity).id}`);
const service = ws.runtimeServices?.find(row => row.serviceName === "Website preview" && row.status === "running")!;
@@ -80,10 +81,10 @@ export const runtimePreviewCases = [runtimeDefinition("workspaces.preview-servic
const beforeRuns = (await ctx.allRuns()).map(run => run.id);
const beforeOperations = (await ctx.input.api.get<WorkspaceOperation[]>(`/api/execution-workspaces/${ws.id}/workspace-operations`)).map(row => row.id);
await writeFile(String(state.failureFlag), "Deliberate bounded owned process-start failure fixture", "utf8");
await sendChatMessage(ctx.input.page, "Restart that same website preview now and tell me whether it actually comes back. If startup fails, keep that failure visible and explain the next repair step; do not claim a working preview.");
const settled = await pollUntil({ label: "Explicit preview restart reports its genuine startup failure", deadlineAt: Date.now() + 180_000,
await sendChatMessage(ctx.input.page, "Restart that same website preview now and tell me whether it actually comes back. If startup fails, keep that failure visible and explain the next repair step; do not claim a working preview.", chatCatalogNativePhaseRemainingBudget(ctx));
const settled = await pollUntil({ label: "Explicit preview restart reports its genuine startup failure", deadlineAt: chatCatalogNativePhaseDeadline(ctx, Date.now() + 180_000),
load: async () => { await ctx.refreshIssue(); return { source: ctx.issue(), runs: await ctx.allRuns(), operations: await ctx.input.api.get<WorkspaceOperation[]>(`/api/execution-workspaces/${ws.id}/workspace-operations`) }; },
accept: value => value.source?.conversationState === "waiting" && value.runs.length === 3 && value.runs.every(run => run.status === "succeeded")
accept: value => chatCatalogNativePhaseIsTimely(ctx) && value.source?.conversationState === "waiting" && value.runs.length === 3 && value.runs.every(run => run.status === "succeeded")
&& value.operations.some(row => !beforeOperations.includes(row.id) && row.status === "failed") });
const ids = settled.runs.filter(run => !beforeRuns.includes(run.id)).map(run => run.id); state.failureRunIds = ids;
state.failureOperation = settled.operations.find(row => !beforeOperations.includes(row.id) && row.status === "failed" && ids.includes(row.heartbeatRunId ?? ""));
tests/runner-e2e/chat-catalog-domains/runtime-runs-adapters.ts
+27 -5
View File
@@ -2,7 +2,7 @@ import { expect } from "@playwright/test";
import { checks } from "./internal-support.js";
import { readAgentResource } from "./identity-support.js";
import { runtimeDefinition, runtimeWorker, runtimeObservation, unchangedAgent, type RuntimeAgent } from "./runtime-support.js";
interface Model { id: string; label: string }
import { openRouterCatalogScope, publicOpenRouterCatalogEndpoint, observedModelChoices, modelChoiceMatches, publicCatalogCaveat, unsupportedModelReadinessClaim, type DiscoveredModel as Model, type OpenRouterCatalogScope } from "./model-option-catalog.js";
export function gradeModelOptions(value: unknown) {
const c = checks(); const o = runtimeObservation(value);
c.add("authorized-model-discovery-evidence", !!o, "Actual agent-authorized discovery and saved account/runtime scope are independently read."); if (!o) return c.result();
@@ -10,9 +10,18 @@ export function gradeModelOptions(value: unknown) {
const current = o.agents.find(row => row.id === worker?.id);
c.add("at-least-one-current-discovered-option", Array.isArray(models) && models.length > 0 && o.state.discoveryStatus === 200
&& models.some(model => o.reply.includes(model.id)), "At least one returned coding option comes from the current admitted model catalog.");
const mentionedIds = [...o.reply.matchAll(/\b(?:gpt-\d[\w.-]*|o[134](?:-[\w.-]+)?|claude-(?:sonnet|opus|haiku)-[\w.-]+)/gi)].map(match => match[0].replace(/[.,;:!?]+$/, ""));
c.add("no-unqualified-model-inventory", !!models && mentionedIds.every(id => models.some(model => model.id.toLowerCase() === id.toLowerCase()))
&& !/(?:all|every) (?:public |vendor )?models? (?:are |is )?(?:installed|ready|available)|all models.*(?:ready|work)/i.test(o.reply),
const publicCatalog = o.state.publicOpenRouterCatalog as Model[] | undefined, savedScope = o.state.publicCatalogScope as OpenRouterCatalogScope | undefined;
const lead = o.agents.find(row => row.id === o.leadId), currentScope = current && lead ? openRouterCatalogScope(o.companyId, o.leadId, current, lead) : undefined;
const catalogBound = Array.isArray(publicCatalog) && publicCatalog.length > 0 && publicCatalog.every(model => typeof model.id === "string" && model.id.startsWith("openrouter/"))
&& o.state.publicCatalogStatus === 200 && !!savedScope && !!currentScope && JSON.stringify(savedScope) === JSON.stringify(currentScope);
c.add("same-binding-public-catalog-provenance", publicCatalog === undefined || catalogBound,
"Public catalog choices require the actual unchanged company/CEO/Morgan/environment/OpenCode/OpenRouter key-reference binding; metadata never proves entitlement.");
const known = [...(models ?? []), ...(catalogBound ? publicCatalog! : [])], mentionedIds = observedModelChoices(o.reply);
const publicChoiceUsed = catalogBound && mentionedIds.some(id => !modelChoiceMatches(id, models ?? []) && !modelChoiceMatches(id, [{ id: currentScope!.currentModel, label: "current" }]) && modelChoiceMatches(id, publicCatalog!));
c.add("catalog-visibility-not-account-entitlement", !publicChoiceUsed || publicCatalogCaveat(o.reply),
"A public catalog choice is conditional on the selected key/account/runtime and cannot be presented as universally tested, qualified or ready.");
c.add("no-unqualified-model-inventory", !!models && mentionedIds.every(id => modelChoiceMatches(id, known))
&& !unsupportedModelReadinessClaim(o.reply),
"The answer does not add unsupported vendor models or equate a public catalog with universal runtime readiness.");
c.add("availability-not-account-qualification", /account|binding|connection|connected|configured/i.test(o.reply)
&& /available|option|catalog|model/i.test(o.reply) && /valid|qualif|restrict|depends|selected|configured|bound/i.test(o.reply),
@@ -31,8 +40,21 @@ export const runtimeRunsAdaptersCases = [runtimeDefinition("runs-adapters.list-m
const discovered = await readAgentResource(ctx, endpoint);
const restricted = await readAgentResource(ctx, `/api/agents/${worker.id}/runtime-state`);
if (discovered.status !== 200 || !Array.isArray(discovered.data)) throw new Error("Scenario requires admitted real coding model discovery");
return { worker, admittedModels: discovered.data, discoveryStatus: discovered.status, agentRuntimeInventoryStatus: restricted.status };
let publicOpenRouterCatalog: Model[] | undefined, publicCatalogScope: OpenRouterCatalogScope | undefined, publicCatalogStatus: number | undefined;
if (worker.adapterType === "paperclip_runner" && worker.adapterConfig?.provider === "opencode" && String(worker.adapterConfig.model).startsWith("openrouter/")) {
const lead = await ctx.input.api.get<RuntimeAgent>(`/api/agents/${ctx.input.fixtures.agent.id}`);
publicCatalogScope = openRouterCatalogScope(ctx.input.fixtures.company.id, ctx.input.fixtures.agent.id, worker, lead);
if (!publicCatalogScope) throw new Error("Model catalog setup requires the actual same-company OpenRouter account reference and native environment binding");
const catalog = await readAgentResource(ctx, publicOpenRouterCatalogEndpoint(ctx.input.fixtures.company.id));
publicCatalogStatus = catalog.status;
if (catalog.status !== 200 || !Array.isArray(catalog.data) || !catalog.data.length || !catalog.data.every((row: Model) => typeof row.id === "string" && row.id.startsWith("openrouter/") && typeof row.label === "string")) throw new Error("Official public OpenRouter model metadata unavailable; no model options were invented");
publicOpenRouterCatalog = catalog.data;
}
return { worker, admittedModels: discovered.data, discoveryStatus: discovered.status, agentRuntimeInventoryStatus: restricted.status,
...(publicOpenRouterCatalog ? { publicOpenRouterCatalog, publicCatalogScope, publicCatalogStatus } : {}) };
}, gradeModelOptions, {
oracleVersion: "paperclip.chat-catalog.model-options.scoped-catalog.v3",
sourceFiles: ["chat-catalog-domains/runtime-support.ts", "chat-catalog-domains/runtime-runs-adapters.ts", "chat-scenarios/runs-adapters.yaml", "chat-catalog-domains/model-option-catalog.ts"],
async observe(ctx, state) {
const { observeRuntime } = await import("./runtime-support.js");
const o = await observeRuntime(ctx, state);
tests/runner-e2e/chat-catalog-domains/runtime-workspace-repositories-setup.test.ts
+72
View File
@@ -0,0 +1,72 @@
import { readFileSync } from "node:fs";
import { describe, expect, it } from "vitest";
import { updateIssueSchema } from "../../../packages/shared/src/validators/issue.js";
import { repositoryWaitingOwnerInput, repositoryWaitingOwnerRetained } from "./runtime-workspace-repositories.js";
type Context = Parameters<typeof repositoryWaitingOwnerInput>[0];
function valid(): Context {
return { companyId: "company", personId: "person", workerId: "worker", taskId: "task", workspaceId: "workspace", fixtureRunId: "run",
members: [{ companyId: "company", principalType: "user", principalId: "person", status: "active" }],
task: { id: "task", companyId: "company", status: "done", assigneeAgentId: "worker", assigneeUserId: null, executionWorkspaceId: "workspace" },
run: { id: "run", companyId: "company", agentId: "worker", status: "succeeded", runtimeMode: "native", contextSnapshot: { issueId: "task" } } };
}
function saved(input = valid()) {
return { ...input.task, ...repositoryWaitingOwnerInput(input) };
}
const service = readFileSync(new URL("../../../server/src/services/issues.ts", import.meta.url), "utf8");
const guard = service.match(/if\s*\(\s*(patch\.status === "in_progress"\s*&&\s*!nextAssigneeAgentId\s*&&\s*!nextAssigneeUserId)\s*\)\s*\{\s*throw unprocessable\("in_progress issues require an assignee"\)/)?.[1];
if (!guard) throw new Error("Actual public issue-update ownership guard changed; review repository setup contract");
const rejectsOwner = new Function("patch", "nextAssigneeAgentId", "nextAssigneeUserId", `return (${guard});`) as (patch: { status: string }, agentId: string | null, userId: string | null) => boolean;
describe("repository waiting task public ownership", () => {
it("satisfies the actual service predicate with a real human owner and rejects the original unassigned active patch", () => {
const input = valid(), patch = repositoryWaitingOwnerInput(input);
expect(updateIssueSchema.parse(patch)).toEqual(patch);
expect(patch).toEqual({ status: "in_progress", assigneeAgentId: null, assigneeUserId: "person" });
expect(rejectsOwner(patch, patch.assigneeAgentId, patch.assigneeUserId)).toBe(false);
expect(rejectsOwner({ status: "in_progress" }, null, null)).toBe(true);
expect(input.task.status).toBe("done");
expect(input.run.status).toBe("succeeded");
expect(input.task.assigneeAgentId).toBe("worker");
});
it("requires the signed-in active human membership in the exact company", () => {
for (const change of [
(v: Context) => { v.personId = undefined; },
(v: Context) => { v.members = []; },
(v: Context) => { v.members[0]!.principalType = "agent"; },
(v: Context) => { v.members[0]!.principalId = "another-person"; },
(v: Context) => { v.members[0]!.companyId = "foreign-company"; },
(v: Context) => { v.members[0]!.status = "suspended"; },
]) { const input = valid(); change(input); expect(() => repositoryWaitingOwnerInput(input)).toThrow(); }
});
it("preserves actual completed worker, task, workspace and native-run provenance", () => {
for (const change of [
(v: Context) => { v.task.id = "other-task"; },
(v: Context) => { v.task.companyId = "foreign"; },
(v: Context) => { v.task.status = "in_progress"; },
(v: Context) => { v.task.assigneeAgentId = "another-worker"; },
(v: Context) => { v.task.assigneeUserId = "another-person"; },
(v: Context) => { v.task.executionWorkspaceId = "another-workspace"; },
(v: Context) => { v.run.id = "invented-run"; },
(v: Context) => { v.run.companyId = "foreign"; },
(v: Context) => { v.run.agentId = "another-worker"; },
(v: Context) => { v.run.status = "running"; },
(v: Context) => { v.run.status = "cancelled"; },
(v: Context) => { v.run.status = "failed"; },
(v: Context) => { v.run.runtimeMode = "legacy"; },
(v: Context) => { v.run.contextSnapshot = { issueId: "another-task" }; },
]) { const input = valid(); change(input); expect(() => repositoryWaitingOwnerInput(input)).toThrow(); }
});
it("requires independent saved ownership and no extra or cancelled native runs", () => {
const input = valid();
expect(repositoryWaitingOwnerRetained(input, saved(), ["run"], [{ id: "run", status: "succeeded" }])).toBe(true);
const changes: Array<Partial<Context["task"]>> = [
{ assigneeAgentId: "worker" }, { assigneeUserId: "other" }, { assigneeUserId: null }, { companyId: "foreign" },
{ id: "other-task" }, { executionWorkspaceId: "other-workspace" }, { status: "done" },
];
for (const change of changes) expect(repositoryWaitingOwnerRetained(input, { ...saved(), ...change }, ["run"], [{ id: "run", status: "succeeded" }])).toBe(false);
for (const runs of [[], [{ id: "other-run", status: "succeeded" }], [{ id: "run", status: "cancelled" }], [{ id: "run", status: "succeeded" }, { id: "extra", status: "running" }]] as const)
expect(repositoryWaitingOwnerRetained(input, saved(), ["run"], [...runs])).toBe(false);
expect(repositoryWaitingOwnerRetained(input, saved(), ["run", "earlier-unaccounted"], [{ id: "run", status: "succeeded" }])).toBe(false);
});
});
tests/runner-e2e/chat-catalog-domains/runtime-workspace-repositories.ts
+46 -2
View File
@@ -4,6 +4,7 @@ import type { Issue, ConnectionIntentInteraction } from "../../../packages/share
import type { ConnectionIntentSetupOptions } from "../../../packages/shared/src/types/connection-intent.js";
import type { ConnectionGrant } from "../../../packages/shared/src/types/tool-access.js";
import type { HeartbeatRun } from "../../../packages/shared/src/types/heartbeat.js";
import type { CompanyMembership } from "../../../packages/shared/src/types/access.js";
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
import { pollUntil } from "../api.js";
import { collectRunEvents } from "../run-observations.js";
@@ -14,6 +15,28 @@ import { runtimeDefinition, runtimeObservation, observeRuntime, type RuntimeObse
import { savedTaskWorkspace } from "./runtime-workspaces.js";
import { inspectCheckout, protectedWorkload, sourceProtectionReceipt, workloadBoundaryCommand, workloadBoundaryReceipts, type RepositoryWorkspace } from "./runtime-workspace-repositories-support.js";
type Fixture = Awaited<ReturnType<typeof connectRepositoryFixture>>;
type WaitingOwnerContext = {
companyId: string; personId?: string; workerId: string; taskId: string; workspaceId: string; fixtureRunId: string;
members: Pick<CompanyMembership, "companyId" | "principalType" | "principalId" | "status">[];
task: Pick<Issue, "id" | "companyId" | "status" | "assigneeAgentId" | "assigneeUserId" | "executionWorkspaceId">;
run: Pick<HeartbeatRun, "id" | "companyId" | "agentId" | "status" | "contextSnapshot"> & { runtimeMode?: string };
};
export function repositoryWaitingOwnerInput(input: WaitingOwnerContext) {
const { companyId, personId, workerId, taskId, workspaceId, fixtureRunId, task, run } = input;
if (!personId || !input.members.some(member => member.companyId === companyId && member.principalType === "user" && member.principalId === personId && member.status === "active"))
throw new Error("Repository setup requires the actual signed-in active company member");
if (task.id !== taskId || task.companyId !== companyId || task.status !== "done" || task.assigneeAgentId !== workerId || task.assigneeUserId != null || task.executionWorkspaceId !== workspaceId
|| run.id !== fixtureRunId || run.companyId !== companyId || run.agentId !== workerId || run.status !== "succeeded" || run.runtimeMode !== "native" || run.contextSnapshot?.issueId !== taskId)
throw new Error("Repository setup must retain its actual completed worker task, native run and workspace");
return { status: "in_progress" as const, assigneeAgentId: null, assigneeUserId: personId };
}
export function repositoryWaitingOwnerRetained(input: WaitingOwnerContext, saved: WaitingOwnerContext["task"], beforeRunIds: string[], afterRuns: Array<{ id: string; status: string }>) {
const expected = repositoryWaitingOwnerInput(input);
return saved.id === input.taskId && saved.companyId === input.companyId && saved.executionWorkspaceId === input.workspaceId
&& saved.status === expected.status && saved.assigneeAgentId === null && saved.assigneeUserId === expected.assigneeUserId
&& beforeRunIds.length === 1 && beforeRunIds[0] === input.fixtureRunId && afterRuns.length === 1
&& afterRuns[0]?.id === input.fixtureRunId && afterRuns[0].status === "succeeded";
}
function site(fixture: Fixture) { if (!("siteRepository" in fixture)) throw new Error("Wrong selected repository fixture"); return fixture.siteRepository; }
interface RepositoryObservation extends RuntimeObservation {
activeTask: Issue; workspace: RepositoryWorkspace; interactions: ConnectionIntentInteraction[];
@@ -49,11 +72,32 @@ async function setup(ctx: ChatCatalogContext) {
if (grants.length || connection.installs?.length) throw new Error("Setup blocker: disposable metadata discovery must not install a broad managed identity into this workload");
if (ctx.scenario.id === "workspaces.prepare-repository") await grantServerAcquisition(ctx);
const prepared = await savedTaskWorkspace(ctx);
const [session, members, completedTask, completedRun, beforeRuns] = await Promise.all([
ctx.input.api.get<{ user?: { id: string } }>("/api/auth/get-session"),
ctx.input.api.get<{ members: WaitingOwnerContext["members"] }>(`/api/companies/${ctx.input.fixtures.company.id}/members`),
ctx.input.api.get<Issue>(`/api/issues/${prepared.task.id}`),
ctx.input.api.get<WaitingOwnerContext["run"]>(`/api/heartbeat-runs/${prepared.fixtureRunId}`),
ctx.allRuns(),
]);
const waitingOwner = { companyId: ctx.input.fixtures.company.id, personId: session.user?.id, workerId: prepared.worker.id,
taskId: prepared.task.id, workspaceId: prepared.workspaceBefore.workspace.id, fixtureRunId: prepared.fixtureRunId, members: members.members, task: completedTask, run: completedRun };
const ownerInput = repositoryWaitingOwnerInput(waitingOwner);
expect(beforeRuns.map(run => run.id)).toEqual([prepared.fixtureRunId]);
expect(beforeRuns[0]?.status).toBe("succeeded");
// This is an actual public input-state transition, never a forged run or
// preparation result. Its admitted task workspace and sentinel survive.
const activeTask = await ctx.input.api.patch<Issue>(`/api/issues/${prepared.task.id}`, { status: "in_progress", assigneeAgentId: null,
const activeTask = await ctx.input.api.patch<Issue>(`/api/issues/${prepared.task.id}`, { ...ownerInput,
description: `This existing Website task awaits the person's repository choice. Keep its admitted workspace and existing website-fixture.txt. Once authorized, use the intended repository; do not substitute a public namesake or copy host credentials. ${ctx.scenario.id === "workspaces.private-repository-no-access" ? "The requested work is to fix its header locally; no remote push or deployment is requested." : "Once repository preparation is ready, inspect actual tracked files and record the available source in a task comment; no implementation or deployment is requested."}\nSecurity qualification, before declaring completion: execute this harmless own-run capability check exactly; do not print any credential or response body: ${workloadBoundaryCommand}` });
const workspace = await ctx.input.api.get<RepositoryWorkspace>(`/api/issues/${activeTask.id}/workspace`);
const [workspace, savedTask, afterRuns] = await Promise.all([
ctx.input.api.get<RepositoryWorkspace>(`/api/issues/${activeTask.id}/workspace`),
ctx.input.api.get<Issue>(`/api/issues/${activeTask.id}`), ctx.allRuns(),
]);
if (!repositoryWaitingOwnerRetained(waitingOwner, savedTask, beforeRuns.map(run => run.id), afterRuns))
throw new Error("Repository waiting setup changed ownership, workspace or native scheduling");
await ctx.input.evidence("workspace-repository-waiting-owner.json", { companyId: waitingOwner.companyId, taskId: savedTask.id,
workspaceId: waitingOwner.workspaceId, assigneeAgentId: savedTask.assigneeAgentId, assigneeUserId: savedTask.assigneeUserId,
fixtureRunId: prepared.fixtureRunId, fixtureRunStatus: completedRun.status, fixtureRunAgentId: completedRun.agentId,
additionalNativeRuns: 0, ownerSource: "actual-signed-in-active-company-member" });
if (workspace.issueId !== activeTask.id || !workspace.workspace?.cwd || workspace.repositories.length) throw new Error("Setup blocker: actual active task must have a valid admitted workspace and no acquired repository");
return { ...fixture, worker: prepared.worker, task: activeTask, fixtureRunId: prepared.fixtureRunId, workspaceBefore: workspace,
requestedUrl: `https://github.com/${site(fixture).fullName}`, initialRunIds: (await ctx.allRuns()).map(run => run.id), accessGranted: ctx.scenario.id === "workspaces.prepare-repository" };
tests/runner-e2e/chat-catalog-domains/status-card-work-failure.test-fixtures.ts
+39
View File
@@ -0,0 +1,39 @@
import { createHash } from "node:crypto";
import { cardChecklistSourceData } from "./status-card-work-failure.js";
import { notesReadResponse } from "./external-provider.js";
import type { CardWorkEvidence } from "./status-card-work-failure.js";
import type { StatusCard, StatusCardUpdate } from "../../../packages/shared/src/validators/status-card.js";
const sha = (s: string) => createHash("sha256").update(s).digest("hex");
export function cardWorkFixture() {
const t = (seconds: number) => new Date(Date.parse("2026-10-28T12:00:00Z") + seconds * 1000).toISOString();
const recipe = { companyId: "company", workerId: "morgan", connectionId: "connection", readEntryId: "read-action", instructionRevisionId: "revision", instructionSha256: sha("saved static recipe"), interestPrompt: "Launch blockers and current synthetic checklist verification" };
const run = { id: "failed-native", companyId: "company", agentId: "morgan", status: "succeeded", runtimeMode: "native", startedAt: t(20), finishedAt: t(30), contextSnapshot: { issueId: "failure-task" }, costAccountingPending: false, costAccountedAt: t(31),
usageJson: { accountingReceiptReady: true, accountingCaptureFailed: false, accountingProviderWorkStarted: true, accountingReceiptId: "receipt", accountingReceiptSourceId: "source", accountingReceiptSequence: 1, accountingReceiptReceivedAt: t(30), usageSource: "per_run", provider: "openrouter", biller: "openrouter", model: "openrouter/deepseek/deepseek-v4.1-flash-1017", billingType: "metered_api", inputTokens: 10, outputTokens: 3, accountingCostCoverage: "unpriced" } };
const invocation = { id: "invocation", companyId: "company", agentId: "morgan", issueId: "failure-task", runId: run.id, connectionId: "connection", catalogEntryId: "read-action", policyDecision: "allow", status: "failed", errorCode: "tool_error", startedAt: t(21), completedAt: t(24), argumentsHash: sha("arguments"), resultHash: null };
const audit = { invocationId: "invocation", companyId: "company", agentId: "morgan", issueId: "failure-task", runId: run.id, connectionId: "connection", catalogEntryId: "read-action", outcome: "failure", decision: "deny", reasonCode: "tool_error", metadata: { execution: { transport: "mcp_remote", request: { dispatched: true, upstreamToolName: "launch_read", mcpMethod: "tools/call", requestId: "request" }, response: { httpStatus: 200 } } } };
const workerRun = (id: string, issueId: string, offset: number) => ({ ...structuredClone(run), id, startedAt: t(offset), finishedAt: t(offset + 5), contextSnapshot: { issueId } });
const runs = [workerRun("worker-run", "generation-task", 0), workerRun("source-run", "chat", 6), workerRun("new-worker-run", "refresh-task", 10), workerRun("ceo-two", "chat", 16), workerRun("ceo-three", "chat", 18), run, workerRun("ceo-four", "chat", 32)];
for (const row of runs) if (row.contextSnapshot.issueId === "chat") row.agentId = "ceo";
const checklistSource = { id: "source", companyId: "company", title: "Payment approval blocks launch", status: "blocked" };
const evidence = { recipe, checklistSource, instruction: { path: "AGENTS.md", content: "saved static recipe", revision: { id: "revision", companyId: "company", agentId: "morgan", source: "board" } }, runs,
overview: { companyId: "company", policies: [{ scopeType: "company", scopeId: "company" }, { scopeType: "agent", scopeId: "morgan" }, { scopeType: "agent", scopeId: "ceo" }].map(p => ({ ...p, companyId: "company", isActive: true, hardStopEnabled: true, amount: 500, unpricedUsagePolicy: "allow", pendingRunCount: 0, paused: false, status: "ok", remainingAmount: 400 })) },
generation: { id: "failure-task", companyId: "company", assigneeAgentId: "morgan", status: "blocked", description: '```json\n'+JSON.stringify({ companyId: "company", statusCardId: "card", generationIssueId: "failure-task" })+'\n```' },
decisions: { runId: run.id, decisions: [{ invocation, auditEvents: [audit] }] },
calls: [{ method: "tools/call", name: "launch_read", authorized: true, sequence: 4, requestIdSha256: sha("request"), responseHttpStatus: 200, toolError: true, responseSha256: sha(JSON.stringify(notesReadResponse(cardChecklistSourceData(checklistSource), true))), receivedAt: Date.parse(t(22)), respondedAt: Date.parse(t(23)) }],
comments: [{ id: "comment", companyId: "company", issueId: "failure-task", authorAgentId: "morgan", createdByRunId: run.id, body: "The required current checklist is unavailable; generation is blocked." }],
activity: [{ id: "audit", companyId: "company", entityId: "failure-task", runId: run.id, action: "issue.updated", actorType: "system", actorId: "native-status-committer", details: { source: "native_status_decision", toStatus: "blocked" } }],
} as unknown as CardWorkEvidence;
const card = { id: "card", companyId: "company", agentId: "morgan", interestPrompt: recipe.interestPrompt, state: "error", failureReason: "Generation task blocked before writing a summary", generatingIssueId: null } as StatusCard;
const update = { id: "failed-update", cardId: "card", generationIssueId: "failure-task", runId: null, kind: "full", status: "failed", error: card.failureReason, startedAt: t(19), finishedAt: t(25) } as StatusCardUpdate;
return { evidence, card, update, runId: run.id };
}
export function successfulCardWorkFixture(runId: string, issueId: string, updateId: string) {
const value = cardWorkFixture(), e = value.evidence, run = e.runs.find(r => r.id === runId)!;
e.generation.id = issueId; e.generation.status = "done"; e.generation.description = '```json\n'+JSON.stringify({ companyId: "company", statusCardId: "card", generationIssueId: issueId })+'\n```';
const v = e.decisions.decisions[0]!.invocation, a = e.decisions.decisions[0]!.auditEvents[0]!;
Object.assign(v, { runId, issueId, status: "succeeded", errorCode: null, resultHash: sha("result"), startedAt: run.startedAt, completedAt: run.finishedAt });
Object.assign(a, { runId, issueId, outcome: "success", decision: "allow", reasonCode: "tool_completed" });
e.decisions.runId = runId; e.calls[0]!.toolError = false; e.calls[0]!.responseSha256 = sha(JSON.stringify(notesReadResponse(cardChecklistSourceData(e.checklistSource), false))); e.calls[0]!.receivedAt = Date.parse(run.startedAt!) + 1000; e.calls[0]!.respondedAt = Date.parse(run.startedAt!) + 2000;
const update = { ...value.update, id: updateId, generationIssueId: issueId, runId, status: "ok", error: null } as StatusCardUpdate;
return { evidence: e, update };
}
tests/runner-e2e/chat-catalog-domains/status-card-work-failure.test.ts
+110
View File
@@ -0,0 +1,110 @@
import { createHash } from "node:crypto";
import { describe, expect, it } from "vitest";
import { cardAccountingReady, cardChecklistInstructions, cardChecklistPurpose, cardUsageComplete, genuineCardChecklistSuccess, genuineCardWorkFailure } from "./status-card-work-failure.js";
import { notesReadResponse } from "./external-provider.js";
import { cardWorkFixture, successfulCardWorkFixture } from "./status-card-work-failure.test-fixtures.js";
const grade = (v: ReturnType<typeof cardWorkFixture>) => genuineCardWorkFailure(v.card, v.update, v.runId, "ceo", v.evidence);
describe("real required-input work failure, not a provider failure", () => {
it("accepts the exact normally completed native work, authorized read error and blocked card-generation projection", () => expect(grade(cardWorkFixture())).toBe(true));
it("rejects a card projection before the actual required read's completed tool receipt", () => {
const v = cardWorkFixture(), invocation = v.evidence.decisions.decisions[0]!.invocation;
for (const finishedAt of [v.evidence.runs.find(r => r.id === v.runId)!.startedAt!, new Date(v.evidence.calls[0]!.respondedAt!).toISOString(), new Date(new Date(invocation.completedAt!).getTime() - 1).toISOString()]) {
v.update.finishedAt = finishedAt; expect(grade(v)).toBe(false);
}
});
it("accepts exact same-millisecond causal completion for Date and serialized receipt fields", () => {
for (const asDate of [false, true]) {
const v = cardWorkFixture(), invocation = v.evidence.decisions.decisions[0]!.invocation;
v.update.finishedAt = new Date(invocation.completedAt!).toISOString();
if (asDate) { invocation.startedAt = new Date(invocation.startedAt!); invocation.completedAt = new Date(invocation.completedAt!); }
expect(grade(v)).toBe(true);
}
});
it("keeps the original causal failure when a separately bound later read also fails", () => {
const v = cardWorkFixture(), first = v.evidence.decisions.decisions[0]!;
const next = structuredClone(first); next.invocation.id = "later-invocation";
next.invocation.startedAt = new Date("2026-10-28T12:00:26Z"); next.invocation.completedAt = new Date("2026-10-28T12:00:28Z");
next.auditEvents[0]!.invocationId = next.invocation.id;
((next.auditEvents[0]!.metadata!.execution as {request:{requestId:string}}).request).requestId = "later-request";
v.evidence.decisions.decisions.push(next);
v.evidence.calls.push({ ...v.evidence.calls[0]!, sequence: 5, receivedAt: Date.parse("2026-10-28T12:00:26Z"), respondedAt: Date.parse("2026-10-28T12:00:27Z"), requestIdSha256: createHash("sha256").update("later-request").digest("hex") });
expect(grade(v)).toBe(true);
v.update.finishedAt = "2026-10-28T12:00:20Z"; expect(grade(v)).toBe(false);
});
for (const key of ["companyId", "agentId", "id"] as const) it(`rejects foreign actual run ${key}`, () => {
const v = cardWorkFixture(); Object.assign(v.evidence.runs.find(r => r.id === v.runId)!, { [key]: "foreign" }); expect(grade(v)).toBe(false);
});
for (const status of ["failed", "cancelled", "timed_out", "running"]) it(`rejects provider status ${status}`, () => {
const v = cardWorkFixture(); Object.assign(v.evidence.runs.find(r => r.id === v.runId)!, { status }); expect(grade(v)).toBe(false);
});
for (const key of ["companyId", "workerId", "connectionId", "readEntryId", "instructionRevisionId", "instructionSha256", "interestPrompt"] as const) it(`rejects wrong recipe ${key}`, () => {
const v = cardWorkFixture(); v.evidence.recipe[key] = "foreign"; expect(grade(v)).toBe(false);
});
for (const key of ["id", "companyId", "assigneeAgentId", "status", "description"] as const) it(`rejects wrong generation ${key}`, () => {
const v = cardWorkFixture(); v.evidence.generation[key] = "foreign"; expect(grade(v)).toBe(false);
});
for (const key of ["companyId", "statusCardId", "generationIssueId"]) it(`rejects foreign generation payload ${key}`, () => {
const v = cardWorkFixture(); const p = { companyId: "company", statusCardId: "card", generationIssueId: "failure-task", [key]: "foreign" };
v.evidence.generation.description = '```json\n'+JSON.stringify(p)+'\n```'; expect(grade(v)).toBe(false);
});
for (const key of ["cardId", "generationIssueId", "status", "error", "finishedAt"] as const) it(`rejects absent/wrong failed update ${key}`, () => {
const v = cardWorkFixture(); Object.assign(v.update, { [key]: key === "error" || key === "finishedAt" ? null : "foreign" }); expect(grade(v)).toBe(false);
});
it("requires actual card error/cleared claim, independent of a blocked task", () => {
for (const patch of [{ state: "active" }, { failureReason: null }, { generatingIssueId: "failure-task" }]) { const v = cardWorkFixture(); Object.assign(v.card, patch); expect(grade(v)).toBe(false); }
});
for (const patch of [{ authorized: false }, { responseHttpStatus: 403 }, { toolError: false }, { requestIdSha256: "foreign" }, { responseSha256: "foreign" }, { name: "launch_update" }, { receivedAt: 0 }, { respondedAt: Date.now() + 1e9 }]) it(`rejects wrong provider evidence ${Object.keys(patch)}`, () => {
const v = cardWorkFixture(); Object.assign(v.evidence.calls[0]!, patch); expect(grade(v)).toBe(false);
});
it("rejects an unmatched additional read and reversed call chronology", () => {
const v = cardWorkFixture(); v.evidence.calls.push({ ...v.evidence.calls[0]!, sequence: 5, requestIdSha256: "foreign" }); expect(grade(v)).toBe(false);
v.evidence.calls.pop(); v.evidence.calls[0]!.respondedAt = v.evidence.calls[0]!.receivedAt! - 1; expect(grade(v)).toBe(false);
});
for (const patch of [{ policyDecision: "deny" }, { status: "succeeded" }, { errorCode: "permission_denied" }, { runId: "foreign" }, { connectionId: "foreign" }, { catalogEntryId: "foreign" }, { issueId: "foreign" }, { argumentsHash: null }]) it(`rejects wrong invocation ${Object.keys(patch)}`, () => {
const v = cardWorkFixture(); Object.assign(v.evidence.decisions.decisions[0]!.invocation, patch); expect(grade(v)).toBe(false);
});
it("rejects fake dispatch/HTTP/error receipts, missing worker explanation and unrelated status changes", () => {
const v = cardWorkFixture();
for (const mutate of [(x: typeof v) => { x.evidence.decisions.decisions[0]!.auditEvents = []; },
(x: typeof v) => { x.evidence.comments[0]!.createdByRunId = "foreign"; },
(x: typeof v) => { x.evidence.comments[0]!.body = "Everything is fresh"; },
(x: typeof v) => { x.evidence.activity[0]!.runId = "foreign"; },
(x: typeof v) => { x.evidence.activity[0]!.details = { status: "blocked" }; }]) { const copy = structuredClone(v); mutate(copy); expect(grade(copy)).toBe(false); }
const e = v.evidence.decisions.decisions[0]!.auditEvents[0]!.metadata!.execution as { request: { dispatched: boolean }; response: { httpStatus: number } };
e.request.dispatched = false; expect(grade(v)).toBe(false); e.request.dispatched = true; e.response.httpStatus = 500; expect(grade(v)).toBe(false);
});
});
describe("complete settled usage with explicit unpriced allow", () => {
it("accepts complete unpriced receipts without inventing zero dollar cost", () => {
const v = cardWorkFixture(); expect(cardUsageComplete(v.evidence.runs[0]!)).toBe(true); expect(grade(v)).toBe(true);
expect(v.evidence.runs[0]!.usageJson).not.toHaveProperty("costUsd");
});
for (const patch of [{ costAccountingPending: true }, { costAccountedAt: null }, { finishedAt: null }]) it(`rejects incomplete settlement ${Object.keys(patch)}`, () => {
const v = cardWorkFixture(); Object.assign(v.evidence.runs[0]!, patch); expect(grade(v)).toBe(false);
});
for (const patch of [{ accountingReceiptReady: false }, { accountingCaptureFailed: true }, { accountingProviderWorkStarted: false }, { accountingReceiptId: "" }, { accountingReceiptSourceId: "" }, { accountingReceiptSequence: 0 }, { usageSource: "aggregate" }, { inputTokens: 0, outputTokens: 0 }]) it(`rejects incomplete usage ${Object.keys(patch)}`, () => {
const v = cardWorkFixture(); Object.assign(v.evidence.runs[0]!.usageJson!, patch); expect(grade(v)).toBe(false);
});
for (const patch of [{ unpricedUsagePolicy: "deny" }, { pendingRunCount: 1 }, { amount: 0 }, { isActive: false }, { paused: true }, { status: "hard_stop" }, { remainingAmount: 0 }]) it(`rejects policy/accounting boundary ${Object.keys(patch)}`, () => {
const v = cardWorkFixture(); Object.assign(v.evidence.overview.policies[0]!, patch); expect(grade(v)).toBe(false);
});
it("requires active scoped company, worker and CEO policies", () => {
const v = cardWorkFixture(); v.evidence.overview.policies.pop(); expect(cardAccountingReady("company", "morgan", "ceo", v.evidence.runs, v.evidence.overview)).toBe(false);
});
});
it("requires the same real successful generation read before the later failure", () => {
const v = successfulCardWorkFixture("new-worker-run", "refresh-task", "new-update");
const ok = () => genuineCardChecklistSuccess("card", v.evidence.recipe, v.update, v.evidence); expect(ok()).toBe(true);
v.evidence.calls[0]!.toolError = true; expect(ok()).toBe(false); v.evidence.calls[0]!.toolError = false;
v.evidence.generation.status = "blocked"; expect(ok()).toBe(false); v.evidence.generation.status = "done";
v.update.cardId = "foreign"; expect(ok()).toBe(false);
});
it("keeps the recipe static and scope-compatible and changes only synthetic read availability", () => {
const p = cardChecklistPurpose("project"); const instructions = cardChecklistInstructions("connection");
expect(p).toContain("saved Launch tasks"); expect(p).toContain("Synthetic Launch Notes");
expect(instructions).toContain("bounded changed-issue facts"); expect(instructions).toContain("required query/summary write-back sequence");
expect(instructions).not.toMatch(/fail now|set.*blocked|PATCH|cancel|expected|canonical.query|Refresh the launch status card now/i);
expect(notesReadResponse("actual source data", false)).toEqual({ content: [{ type: "text", text: "actual source data" }] });
const failed = notesReadResponse("actual source data", true); expect(failed).toHaveProperty("isError", true); expect(JSON.stringify(failed)).not.toContain("actual source data");
expect(JSON.stringify(failed)).not.toMatch(/block|status|summary|tool.call|PATCH|fail now/i);
});
tests/runner-e2e/chat-catalog-domains/status-card-work-failure.ts
+149
View File
@@ -0,0 +1,149 @@
import { createHash } from "node:crypto";
import type { BudgetOverview } from "../../../packages/shared/src/types/budget.js";
import type { ToolRunDecisionLookup } from "../../../packages/shared/src/types/tool-access.js";
import type { StatusCard, StatusCardUpdate } from "../../../packages/shared/src/validators/status-card.js";
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
import type { ChatRun } from "../chat-flow.js";
import { connectNotes, startNotesProvider, type NotesProvider, type ProviderCall, notesReadResponse } from "./external-provider.js";
import type { Entity } from "./internal-support.js";
import type { InstructionFile } from "./runtime-support.js";
const hash = (value: string) => createHash("sha256").update(value).digest("hex");
const record = (v: unknown): Record<string, unknown> => v && typeof v === "object" && !Array.isArray(v) ? v as Record<string, unknown> : {};
const text = (v: unknown) => typeof v === "string" && v.length > 0 && v !== "unknown";
const millis = (v: unknown) => v instanceof Date ? v.getTime() : typeof v === "string" ? Date.parse(v) : NaN;
const time = (v: unknown) => Number.isFinite(millis(v));
const natural = (v: unknown) => typeof v === "number" && Number.isSafeInteger(v) && v >= 0;
const root = (ctx: ChatCatalogContext) => `/api/companies/${ctx.input.fixtures.company.id}`;
export interface CardRecipe { companyId: string; workerId: string; connectionId: string; readEntryId: string; instructionRevisionId: string; instructionSha256: string; interestPrompt: string }
export interface CardWorkRun extends ChatRun { usageJson?: Record<string, unknown>; finishedAt?: string | null; costAccountingPending?: boolean; costAccountedAt?: string | null }
export interface CardWorkEvidence { recipe: CardRecipe; instruction: InstructionFile; runs: CardWorkRun[]; overview: BudgetOverview; generation: Entity; decisions: ToolRunDecisionLookup; comments: Entity[]; activity: Array<Entity & { runId?: string }>; calls: ProviderCall[]; checklistSource: Entity }
const owned = new WeakMap<ChatCatalogContext, { provider: NotesProvider; connectionId?: string; recipe?: CardRecipe; source: Entity }>();
export function cardChecklistSourceData(source: Entity) {
return JSON.stringify({ syntheticDependency: "Synthetic Launch Notes", companyId: source.companyId, sourceTaskId: source.id, title: source.title, status: source.status, description: source.description });
}
export function updateCardChecklistSource(ctx: ChatCatalogContext, source: Entity) {
const state = owned.get(ctx);
if (!state || source.id !== state.source.id || source.companyId !== state.source.companyId) throw new Error("Card checklist source identity changed");
state.source = structuredClone(source);
}
export function cardChecklistPurpose(projectId: string) {
return `Summarize the biggest blockers for Launch project ${projectId}, citing actual source tasks. This card requires current Synthetic Launch Notes checklist verification of the same source-task facts; older verification is not current. Include only blockers grounded in the saved Launch tasks.`;
}
export function cardChecklistInstructions(connectionId: string) {
return `\n\n## Launch card source verification\nThis card uses an explicitly synthetic test dependency, Synthetic Launch Notes. For every generation, use its authorized read action on connection ${connectionId} to obtain current checklist verification of the saved Launch task facts. A previous generation's verification is not current input. If the required current source cannot be read, explain the actual blocker rather than presenting an unverified summary as fresh. Preserve the assigned task's company-search scope, bounded changed-issue facts, source visibility and required query/summary write-back sequence; do not add unrelated work or writes.\n`;
}
export function cardUsageComplete(run: CardWorkRun) {
const u = run.usageJson;
return run.status === "succeeded" && run.runtimeMode === "native" && time(run.finishedAt) && run.costAccountingPending === false && time(run.costAccountedAt)
&& record(record(run.resultJson).executionRecovery).providerWorkStarted !== false && !!u
&& u.accountingReceiptReady === true && u.accountingCaptureFailed !== true && u.accountingProviderWorkStarted !== false
&& text(u.accountingReceiptId) && text(u.accountingReceiptSourceId) && natural(u.accountingReceiptSequence) && Number(u.accountingReceiptSequence) > 0
&& time(u.accountingReceiptReceivedAt) && u.usageSource === "per_run" && text(u.provider) && text(u.biller) && text(u.model)
&& ["metered_api", "subscription_included"].includes(String(u.billingType)) && natural(u.inputTokens) && natural(u.outputTokens) && Number(u.inputTokens) + Number(u.outputTokens) > 0;
}
export function cardAccountingReady(companyId: string, workerId: string, leadId: string, runs: CardWorkRun[], overview: BudgetOverview) {
if (!runs.length || runs.some(run => run.companyId !== companyId || ![workerId, leadId].includes(run.agentId ?? "") || !cardUsageComplete(run)) || overview.companyId !== companyId) return false;
const scopes = [{ scopeType: "company", scopeId: companyId }, { scopeType: "agent", scopeId: workerId }, { scopeType: "agent", scopeId: leadId }];
return scopes.every(scope => {
const policies = overview.policies.filter(p => p.isActive && p.scopeType === scope.scopeType && p.scopeId === scope.scopeId);
return policies.length > 0 && policies.every(p => p.companyId === companyId && p.hardStopEnabled && p.amount > 0 && p.unpricedUsagePolicy === "allow" && p.pendingRunCount === 0 && !p.paused && p.status !== "hard_stop" && p.remainingAmount > 0);
});
}
function generationPayload(task: Entity) {
try { return JSON.parse(task.description?.match(/```json\n([\s\S]*?)\n```/)?.[1] ?? "null") as Record<string, unknown> | null; } catch { return null; }
}
export function boundCardChecklistRead(recipe: CardRecipe, run: CardWorkRun, decisions: ToolRunDecisionLookup, calls: ProviderCall[], failed: boolean, source: Entity, failedProjectionAt?: unknown) {
const relevant = calls.filter(call => call.method === "tools/call" && call.name === "launch_read" && typeof call.receivedAt === "number" && typeof call.respondedAt === "number"
&& Number.isFinite(call.receivedAt) && Number.isFinite(call.respondedAt) && call.receivedAt <= call.respondedAt
&& call.receivedAt >= millis(run.startedAt!) && call.respondedAt <= millis(run.finishedAt!));
if (source.companyId !== recipe.companyId || !source.id || !relevant.length || !relevant.every(call => call.authorized && call.responseHttpStatus === 200 && call.toolError === failed
&& call.responseSha256 === hash(JSON.stringify(notesReadResponse(cardChecklistSourceData(source), failed))))) return false;
let causalReadCompleted = false;
const bound = decisions.runId === run.id && relevant.every(call => decisions.decisions.some(({ invocation: v, auditEvents }) => v.companyId === recipe.companyId && v.agentId === recipe.workerId
&& v.issueId === run.contextSnapshot?.issueId && v.runId === run.id && v.connectionId === recipe.connectionId && v.catalogEntryId === recipe.readEntryId
&& v.policyDecision === "allow" && v.status === (failed ? "failed" : "succeeded") && v.errorCode === (failed ? "tool_error" : null)
&& time(v.startedAt) && time(v.completedAt) && !!v.argumentsHash && (failed || !!v.resultHash)
&& auditEvents.some(a => {
const e = record(record(a.metadata).execution), request = record(e.request), response = record(e.response);
const matched = a.invocationId === v.id && a.companyId === recipe.companyId && a.agentId === recipe.workerId && a.issueId === v.issueId && a.runId === run.id
&& a.connectionId === recipe.connectionId && a.catalogEntryId === recipe.readEntryId && a.outcome === (failed ? "failure" : "success")
&& a.decision === (failed ? "deny" : "allow") && a.reasonCode === (failed ? "tool_error" : "tool_completed")
&& e.transport === "mcp_remote" && request.dispatched === true && request.upstreamToolName === "launch_read" && request.mcpMethod === "tools/call" && response.httpStatus === 200
&& typeof request.requestId === "string" && call.requestIdSha256 === hash(request.requestId)
&& call.receivedAt! >= millis(v.startedAt!) && call.respondedAt! <= millis(v.completedAt!);
if (matched && time(failedProjectionAt) && millis(failedProjectionAt) >= millis(v.completedAt) && millis(failedProjectionAt) >= call.respondedAt!) causalReadCompleted = true;
return matched;
})));
return bound && (!failed || causalReadCompleted);
}
export function genuineCardWorkFailure(card: StatusCard, update: StatusCardUpdate, runId: string, leadId: string, evidence: CardWorkEvidence | undefined) {
if (!evidence) return false;
const { recipe, instruction, generation, runs, decisions, calls, comments, activity } = evidence;
const run = runs.find(row => row.id === runId), payload = generationPayload(generation);
if (!run || !cardUsageComplete(run) || !time(run.startedAt) || card.state !== "error" || !card.failureReason || card.generatingIssueId || card.companyId !== recipe.companyId || card.agentId !== recipe.workerId || card.interestPrompt !== recipe.interestPrompt
|| instruction.revision.companyId !== recipe.companyId || instruction.revision.agentId !== recipe.workerId || instruction.revision.id !== recipe.instructionRevisionId || hash(instruction.content) !== recipe.instructionSha256
|| generation.id !== update.generationIssueId || generation.companyId !== recipe.companyId || generation.assigneeAgentId !== recipe.workerId || generation.status !== "blocked"
|| payload?.companyId !== recipe.companyId || payload.statusCardId !== card.id || payload.generationIssueId !== generation.id
|| update.cardId !== card.id || update.status !== "failed" || !update.error || !time(update.finishedAt) || !time(update.startedAt)
|| millis(update.startedAt) > millis(run.startedAt) || millis(update.finishedAt!) < millis(run.startedAt)
|| run.companyId !== recipe.companyId || run.agentId !== recipe.workerId || run.contextSnapshot?.issueId !== generation.id
|| !boundCardChecklistRead(recipe, run, decisions, calls, true, evidence.checklistSource, update.finishedAt) || !cardAccountingReady(recipe.companyId, recipe.workerId, leadId, runs, evidence.overview)) return false;
return comments.some(c => c.companyId === recipe.companyId && c.issueId === generation.id && c.authorAgentId === recipe.workerId && c.createdByRunId === run.id
&& /checklist|source/i.test(c.body ?? "") && /unavailable|could not|couldn't|fail|error|blocked/i.test(c.body ?? ""))
&& activity.some(a => a.companyId === recipe.companyId && a.entityId === generation.id && a.runId === run.id && a.action === "issue.updated"
&& (a.actorType === "agent" && a.actorId === recipe.workerId && a.details?.status === "blocked"
|| a.actorType === "system" && a.actorId === "native-status-committer" && a.details?.source === "native_status_decision" && a.details?.toStatus === "blocked"));
}
export async function prepareCardChecklist(ctx: ChatCatalogContext, workerId: string, projectId: string, source: Entity) {
const current = { source: structuredClone(source) };
const provider = await startNotesProvider(cardChecklistSourceData(source), { readContent: () => cardChecklistSourceData(current.source) });
const state = { provider, connectionId: undefined as string | undefined, recipe: undefined as CardRecipe | undefined, get source() { return current.source; }, set source(value: Entity) { current.source = value; } }; owned.set(ctx, state);
try {
const connection = await connectNotes(ctx, provider, "read-only"); state.connectionId = connection.connectionId;
await ctx.input.api.post(`${root(ctx)}/tools/apps/${connection.connectionId}/finish`, { enabledCatalogEntryIds: connection.read, askFirstCatalogEntryIds: [], access: { agentIds: [workerId] } });
await ctx.input.api.putSensitive(`/api/tool-connections/${connection.connectionId}/installs`, { installs: [{ targetType: "agent", targetId: workerId }] });
const url = `/api/agents/${workerId}/instructions-bundle/file`, current = await ctx.input.api.get<InstructionFile>(`${url}?path=AGENTS.md`);
const saved = await ctx.input.api.putSensitive<InstructionFile>(url, { path: "AGENTS.md", content: current.content + cardChecklistInstructions(connection.connectionId), baseRevisionId: current.revision.id });
const readback = await ctx.input.api.get<InstructionFile>(`${url}?path=AGENTS.md`);
if (saved.revision.id !== readback.revision.id || saved.content !== readback.content || !readback.content.startsWith(current.content)) throw new Error("Card recipe instructions were not saved through the public revision path");
state.recipe = { companyId: ctx.input.fixtures.company.id, workerId, connectionId: connection.connectionId, readEntryId: connection.read[0]!, instructionRevisionId: readback.revision.id, instructionSha256: hash(readback.content), interestPrompt: cardChecklistPurpose(projectId) };
return state.recipe;
} catch (error) { await cleanupCardChecklist(ctx); throw error; }
}
export function cardChecklistCalls(ctx: ChatCatalogContext) {
return owned.get(ctx)?.provider.calls.map(({ method, name, authorized, sequence, requestIdSha256, responseHttpStatus, toolError, receivedAt, respondedAt, responseSha256 }) => ({ method, name, authorized, sequence, requestIdSha256, responseHttpStatus, toolError, receivedAt, respondedAt, responseSha256 })) ?? [];
}
export function failCardChecklist(ctx: ChatCatalogContext) { const state = owned.get(ctx); if (!state?.recipe) throw new Error("Missing saved card recipe"); state.provider.failReads(); }
export async function observeCardWork(ctx: ChatCatalogContext, recipe: CardRecipe, issueId: string): Promise<CardWorkEvidence> {
const all = await ctx.allRuns();
const runs = await Promise.all(all.map(run => ctx.input.api.get<CardWorkRun>(`/api/heartbeat-runs/${run.id}`)));
const workerRuns = runs.filter(run => run.agentId === recipe.workerId && run.contextSnapshot?.issueId === issueId);
if (workerRuns.length > 1) throw new Error("Card generation exceeded its single declared native turn");
const run = workerRuns[0];
const [instruction, overview, generation, decisions, comments, activity] = await Promise.all([
ctx.input.api.get<InstructionFile>(`/api/agents/${recipe.workerId}/instructions-bundle/file?path=AGENTS.md`), ctx.input.api.get<BudgetOverview>(`${root(ctx)}/budgets/overview`),
ctx.input.api.get<Entity>(`/api/issues/${issueId}`), run ? ctx.input.api.get<ToolRunDecisionLookup>(`${root(ctx)}/tools/runs/${run.id}/decisions`) : Promise.resolve({ runId: "", decisions: [] }),
ctx.input.api.get<Entity[]>(`/api/issues/${issueId}/comments?order=asc`), ctx.input.api.get<Array<Entity & { runId?: string }>>(`/api/issues/${issueId}/activity`),
]);
return { recipe, instruction, runs, overview, generation, decisions, comments, activity, calls: cardChecklistCalls(ctx), checklistSource: structuredClone(owned.get(ctx)!.source) };
}
export function genuineCardChecklistSuccess(cardId: string, recipe: CardRecipe, update: StatusCardUpdate, evidence: CardWorkEvidence | undefined) {
if (!evidence || !update.runId || !update.generationIssueId || update.status !== "ok" || update.cardId !== cardId) return false;
const run = evidence.runs.find(row => row.id === update.runId), payload = generationPayload(evidence.generation);
return !!run && cardUsageComplete(run) && time(run.startedAt) && run.companyId === recipe.companyId && run.agentId === recipe.workerId && run.contextSnapshot?.issueId === update.generationIssueId
&& evidence.generation.companyId === recipe.companyId && evidence.generation.id === update.generationIssueId && evidence.generation.assigneeAgentId === recipe.workerId && evidence.generation.status === "done"
&& payload?.companyId === recipe.companyId && payload.statusCardId === cardId && payload.generationIssueId === update.generationIssueId
&& boundCardChecklistRead(recipe, run, evidence.decisions, evidence.calls, false, evidence.checklistSource)
&& evidence.instruction.revision.companyId === recipe.companyId && evidence.instruction.revision.agentId === recipe.workerId
&& evidence.instruction.revision.id === recipe.instructionRevisionId && hash(evidence.instruction.content) === recipe.instructionSha256;
}
export async function observeCardChecklistSuccess(ctx: ChatCatalogContext, recipe: CardRecipe, update: StatusCardUpdate) {
if (!update?.generationIssueId) throw new Error("Card checklist success needs actual saved generation work");
return observeCardWork(ctx, recipe, update.generationIssueId);
}
export async function cleanupCardChecklist(ctx: ChatCatalogContext) {
const state = owned.get(ctx); if (!state) return; owned.delete(ctx);
try { if (state.connectionId) await ctx.input.api.delete(`/api/tool-connections/${state.connectionId}`); }
finally { await state.provider.close(); }
}
tests/runner-e2e/chat-catalog-flow.ts
+2
View File
@@ -1,3 +1,4 @@
import { assertSdkCanaryFingerprintMatches } from "./sdk-canary-prefixture.js";
import { collectConversationLinks, type ConversationLinkInput } from "./chat-rendered-links.js";
import { chatCatalogAgentIsReady, chatCatalogNativePhaseIsTimely, chatCatalogNativePhaseRemainingBudget } from "./chat-catalog-contract.js";
import { chatNativeTimeoutEvidence } from "./chat-native-timeout-policy.js";
@@ -22,6 +23,7 @@ export async function runChatCatalogFlow(context: ChatCatalogContext) {
const descriptor = findChatCatalogCase(scenario.id);
if (!descriptor) throw new Error(`No executable chat scenario: ${scenario.id}`);
const fingerprint = chatCatalogCaseFingerprint(descriptor, normalizedChatCatalogExecution(input.execution));
assertSdkCanaryFingerprintMatches(input.execution, fingerprint);
const implementation = await loadChatCatalogImplementation(descriptor);
if (implementation.companionSurface) return runScopedCompanionSurface(context, implementation, fingerprint);
const slack = implementation.journeyKind === "native-slack-thread";
tests/runner-e2e/chat-catalog-preservation.json
+107
View File
@@ -0,0 +1,107 @@
{
"schema": "paperclip.chat-catalog-preservation/v1",
"productionBaseline": "39e811c9c5226979fce42e052a27cfb28301018a",
"canonicalEvalBase": "0098da56f4c10f3154ed4ed96b1404fdf7a021ca",
"scenarioCount": 244,
"lazyCatalogEntries": 242,
"standaloneFlows": [
"tasks.create-backlog",
"tasks.delegate-announcement"
],
"purpose": "Eval source preservation only; no qualification, new attempt, budget reset or live admission.",
"components": [
{
"name": "SDK",
"sourceRevision": "9754e46d89bfa99e913efaf1769fe2d8b403fc4d"
},
{
"name": "overview-related",
"sourceRevision": "a62a023a5544ffcab98ee6624b0a583a6e8e6775"
},
{
"name": "status-card",
"sourceRevision": "97bcf2476d9f03bec70bec4ebd788e113fcc4425"
},
{
"name": "skills",
"sourceRevision": "bdca7305bf4c64fea033ae6ceec07927d2a754e2"
},
{
"name": "Ready2",
"sourceRevision": "819b7cf649c5551c32ba913509790ad16c1d5008"
},
{
"name": "repo",
"sourceRevision": "2a630b6b09efe4711dbbc42d0e0fc070067bbb17"
},
{
"name": "formal-model",
"sourceRevision": "687efafc61f841342c52b7d2a8c920b75c218d16"
},
{
"name": "formal-clock",
"sourceRevision": "f09af3cdebd58376d095d6f49d53aae80b635b6e"
},
{
"name": "new-company",
"sourceRevision": "6726c40f01342081557c265944ab8783f9ee82b4"
},
{
"name": "history-eval-only",
"sourceRevision": "2b84e38c15b10e10ef6c063984d375a14b307c4a"
},
{
"name": "ordinary-original-clock-multipart",
"sourceRevision": "73856cebe07cab32df42fdec7ff6ea7192cdd474"
},
{
"name": "new-company-original-clock",
"sourceRevision": "7c846a27a0b1ac1908d929209b60d7942f4246fb"
}
],
"productionPrerequisites": [
{
"scenarioId": "onboarding.new-company",
"corePullRequest": 18103,
"reason": "Actual local OpenCode native authentication/hello probe, absent from frozen production baseline."
},
{
"scenarioId": "questions-and-approvals.formal-hire-revision",
"corePullRequest": 18105,
"reason": "Permission-qualified, truthful responseText pre-dispatch recovery guidance, absent from frozen production baseline."
},
{
"scenarioId": "routines.read-history",
"corePullRequest": 18033,
"reason": "Actual local-native task status must finalize public routine history; merged after the frozen production baseline."
}
],
"coreChangesIncluded": false,
"candidateOutcomesQualifyFrozenBaseline": false,
"historicalResultsCarriedToThisHead": 0,
"knownUnqualifiedFixtureFindings": [
{
"scenarioIds": [
"plugins.configure-plugin",
"plugins.disable-plugin"
],
"finding": "The preserved fixture sends a secret_ref object to an apiKey schema declared string with secret-ref format. The public validator requires the secure secret UUID string. These spent failures are retained; this preservation change does not repair or rerun them."
}
],
"readinessLimits": {
"declaredUnsupportedFreshContext": [
"access.invite-teammate",
"access.manage-membership",
"activity.routine-audit",
"agents.restore-instructions",
"budgets-costs.subscription-fees"
],
"currentProfileOrPublicCapabilityGaps": [
"assistant-connections.use-scoped-client",
"companies.configure-decision-model",
"steering.steer-active"
],
"otherRequirements": "Official provider resources, selected credentials, authorized operators and actual remote targets still require their declared public prerequisites. Importability is not runtime readiness.",
"attempts": "All original raw grades, source fingerprints, ordinals, usage, billing, privacy and cleanup remain bound to their original measurements. Exhausted cases remain exhausted."
}
}
tests/runner-e2e/chat-multipart-required-ui.test.ts
+45
View File
@@ -0,0 +1,45 @@
import { expect, it, vi } from "vitest";
vi.mock("@playwright/test", () => ({ expect: (target: { isDisabled?: () => boolean; isVisible?: () => boolean }) => ({
toBeDisabled: async () => { if (!target.isDisabled?.()) throw new Error("Expected actual disabled Next"); },
toBeVisible: async () => { if (!target.isVisible?.()) throw new Error("Expected current first question visible"); },
}) }));
import { questionIdentityCases } from "./chat-catalog-domains/identity-questions-and-approvals.js";
import type { ChatCatalogContext } from "./chat-catalog-contract.js";
function fixture(options: { nextEnabled?: boolean; wrongPrompt?: boolean; changedPayload?: boolean; prematurelyResolved?: boolean } = {}) {
let page = 0, answered = false, reads = 0, submitted = false;
const clicks: string[] = [], answers: Record<string, string> = {};
const original = { id: "question", kind: "ask_user_questions", status: "pending", payload: { questionSet: { questions: [
{ id: "audience", prompt: "Who is the audience?", required: true }, { id: "tone", prompt: "What tone should we use?", required: true }, { id: "headline", prompt: "What is your custom headline?", required: true },
] } } };
const form: any = { isVisible: () => true, getByText: (prompt: string) => ({ isVisible: () => !options.wrongPrompt && prompt === original.payload.questionSet.questions[page]?.prompt }),
getByRole: (_role: string, { name }: { name: string | RegExp }) => ({
isDisabled: () => !options.nextEnabled && !answered,
isChecked: () => answers.audience === "Researchers",
click: async () => { const label = String(name); clicks.push(label);
if (label === "Next") { if (!answered) throw new Error("Disabled Next cannot be clicked"); page++; answered = !!answers[original.payload.questionSet.questions[page]?.id ?? ""]; }
else if (label === "Back") { page--; answered = !!answers[original.payload.questionSet.questions[page]!.id]; }
else if (label === "Researchers" || label === "Clear") { answers[original.payload.questionSet.questions[page]!.id] = label; answered = true; }
else { if (page !== 2 || !answers.headline) throw new Error("Cannot submit incomplete questions"); submitted = true; }
},
}),
getByTestId: () => ({ isVisible: async () => page === 2, locator: () => ({ first: () => ({ fill: async (value: string) => { answers.headline = value; } }) }) }),
};
const ctx = { issue: () => ({ id: "source", conversationState: "waiting" }), refreshIssue: async () => {}, allRuns: async () => [{ status: "succeeded" }],
input: { fixtures: { company: { id: "company" } }, page: { locator: () => ({ click: async () => {} }), getByTestId: () => form }, api: { get: async (url: string) => {
if (url.endsWith("interactions")) { reads++; return [{ ...original, status: options.prematurelyResolved ? "resolved" : "pending", payload: options.changedPayload && reads >= 2 ? { replaced: true } : original.payload }]; }
return [];
} } },
} as unknown as ChatCatalogContext;
return { ctx, state: { original, answer: "Acorn unchanged human answer" } as Record<string, unknown>, clicks, answers, submitted: () => submitted };
}
it("actual multipart act observes required blocking without clicking disabled Next, retains earlier choice and submits the exact supplied answer", async () => {
const f = fixture(), c = questionIdentityCases.find(row => row.scenarioId === "questions-and-approvals.multi-part-question")!;
await c.act!(f.ctx, f.state);
expect(f.state.validationObserved).toBe(true); expect(f.state.retainedAudience).toBe(true); expect(f.state.unsubmittedPending).toBe(true);
expect(f.clicks).toEqual(["Researchers", "Next", "Back", "Next", "Clear", "Next", "/^(Submit|Send) answers$/"]);
expect(f.answers).toEqual({ audience: "Researchers", tone: "Clear", headline: "Acorn unchanged human answer" }); expect(f.submitted()).toBe(true);
});
it.each([{ nextEnabled: true }, { wrongPrompt: true }, { changedPayload: true }, { prematurelyResolved: true }])("rejects invalid current required-question proof %j before any answer", async options => {
const f = fixture(options), c = questionIdentityCases.find(row => row.scenarioId === "questions-and-approvals.multi-part-question")!;
await expect(c.act!(f.ctx, f.state)).rejects.toThrow(); expect(f.clicks).toEqual([]); expect(f.submitted()).toBe(false);
});
tests/runner-e2e/chat-ordinary-original-clock-group.test.ts
+31
View File
@@ -0,0 +1,31 @@
import { afterEach, expect, it, vi } from "vitest";
import type { ChatCatalogContext } from "./chat-catalog-contract.js";
import { chatCatalogNativePhaseDeadline, chatCatalogNativePhaseIsTimely, chatCatalogOriginalPhaseDeadline } from "./chat-catalog-contract.js";
import { waitForReply } from "./chat-catalog-domains/identity-support.js";
const ids = ["companies.export-company", "projects.project-defaults", "questions-and-approvals.ambiguous-yes", "questions-and-approvals.review-plan", "questions-and-approvals.necessary-question", "task-coordination.block-on-design", "gateways-profiles.rotate-token", "gateways-profiles.create-gateway", "workspaces.preview-service", "assistant-connections.consent-read-only", "navigation-preferences.quick-search", "navigation-preferences.edit-profile", "navigation-preferences.broken-link", "projects.share-project"];
const context = (id: string, deadline: number | undefined = 630000) => ({ scenario: { id }, input: { execution: { id: `chat-features.runner-opencode.local.${id}` } }, observationDeadlineAt: deadline,
refreshIssue: async () => {}, issue: () => ({ conversationState: "waiting" }), allRuns: async () => [{ status: Date.now() >= 200000 ? "succeeded" : "running" }] }) as unknown as ChatCatalogContext;
afterEach(() => vi.useRealTimers());
it.each(ids)("%s reuses the original setup-inclusive deadline and never grants a fresh phase", id => {
vi.useFakeTimers(); vi.setSystemTime(200000); const c = context(id);
expect(chatCatalogNativePhaseDeadline(c, 380000)).toBe(630000); expect(chatCatalogOriginalPhaseDeadline(c)).toBe(630000);
vi.setSystemTime(629999); expect(chatCatalogNativePhaseDeadline(c, 900000)).toBe(630000);
vi.setSystemTime(630000); expect(chatCatalogNativePhaseIsTimely(c)).toBe(false); expect(() => chatCatalogOriginalPhaseDeadline(c)).toThrow(/exhausted/);
});
it.each(["companies.export-company", "navigation-preferences.quick-search", "questions-and-approvals.ambiguous-yes"])("%s actual identity waiter can finish after the incidental150s cap", async id => {
vi.useFakeTimers(); vi.setSystemTime(0); const c = context(id, 270000); let completed = false;
const pending = waitForReply(c, chatCatalogOriginalPhaseDeadline(c)).then(() => { completed = true; });
await vi.advanceTimersByTimeAsync(151000); expect(completed).toBe(false);
await vi.advanceTimersByTimeAsync(50000); await pending; expect(completed).toBe(true); expect(c.observationDeadlineAt).toBe(270000);
});
it("missing, infinite, expired clocks and delayed late successes fail closed", async () => {
vi.useFakeTimers(); vi.setSystemTime(1000);
for (const deadline of [undefined, NaN, Infinity, 1000, 999]) { const c = context(ids[0]!, deadline); Object.assign(c, { observationDeadlineAt: deadline }); expect(() => chatCatalogOriginalPhaseDeadline(c)).toThrow(/exhausted/); }
const c = context(ids[0]!, 1100); c.refreshIssue = async () => { vi.setSystemTime(1101); }; c.allRuns = async () => [{ status: "succeeded" }] as never;
const pending = waitForReply(c, chatCatalogOriginalPhaseDeadline(c)).then(() => "accepted", () => "expired"); await vi.advanceTimersByTimeAsync(1000); expect(await pending).toBe("expired");
});
it("unselected cases and other provider/environment lanes retain authored phase caps", () => {
vi.useFakeTimers(); vi.setSystemTime(1000);
for (const id of ["goals.create-goal-tree", "skills.author-skill", "questions-and-approvals.reject-and-revise", "questions-and-approvals.multi-part-question"]) { const c = context(id, 0); expect(chatCatalogOriginalPhaseDeadline(c)).toBeUndefined(); expect(chatCatalogNativePhaseDeadline(c, 151000)).toBe(151000); }
for (const id of ids) for (const lane of ["runner-codex.local", "runner-opencode.daytona"]) { const c = context(id, 0); Object.assign(c.input.execution, { id: `chat-features.${lane}.${id}` }); expect(chatCatalogOriginalPhaseDeadline(c)).toBeUndefined(); }
});
tests/runner-e2e/chat-ordinary-phase-callers.test.ts
+24
View File
@@ -0,0 +1,24 @@
import { afterEach, expect, it, vi } from "vitest";
const captured = vi.hoisted(() => ({ deadlineAt: undefined as number | undefined }));
vi.mock("./api.js", async original => ({ ...await original<object>(), pollUntil: async (options: { deadlineAt: number }) => { captured.deadlineAt = options.deadlineAt; throw new Error("closed pure phase boundary"); } }));
vi.mock("./chat-catalog-domains/identity-auth.js", async original => ({ ...await original<object>(), createAuthenticatedPerson: async () => ({ id: "viewer", dispose: async () => {} }), enrollPrerequisitePerson: async () => {} }));
import { chatCatalogCases, loadChatCatalogImplementation } from "./chat-catalog-registry.js";
import type { ChatCatalogContext } from "./chat-catalog-contract.js";
const ids = ["companies.export-company", "projects.project-defaults", "questions-and-approvals.ambiguous-yes", "questions-and-approvals.review-plan", "questions-and-approvals.necessary-question", "task-coordination.block-on-design", "gateways-profiles.rotate-token", "gateways-profiles.create-gateway", "workspaces.preview-service", "assistant-connections.consent-read-only", "navigation-preferences.quick-search", "navigation-preferences.edit-profile", "navigation-preferences.broken-link", "projects.share-project"];
afterEach(() => { vi.useRealTimers(); captured.deadlineAt = undefined; });
it.each(ids)("actual %s callback enters its first native observation with the original deadline", async id => {
vi.useFakeTimers(); vi.setSystemTime(200000);
const ctx = { scenario: { id }, observationDeadlineAt: 630000, issue: () => ({ id: "source", conversationState: "waiting" }), refreshIssue: async () => {}, allRuns: async () => [],
input: { nonce: "synthetic", execution: { id: `chat-features.runner-opencode.local.${id}` }, fixtures: { company: { id: "company" }, agent: { id: "ceo" } }, privateSecrets: [],
page: { context: () => ({ browser: () => ({}) }) }, api: { get: async () => ({ user: { id: "owner" } }), post: async (_path: string, body: object) => ({ id: "saved", companyId: "company", ...body }) } },
} as unknown as ChatCatalogContext;
const selected = await loadChatCatalogImplementation(chatCatalogCases.find(row => row.scenarioId === id)!);
const state = id === "projects.share-project" ? await selected.setup(ctx) : {};
await expect(selected.act!(ctx, state)).rejects.toThrow("closed pure phase boundary"); expect(captured.deadlineAt).toBe(630000);
});
it("the shared plan module retains the unselected reject/revise180s phase", async () => {
vi.useFakeTimers(); vi.setSystemTime(200000); const id = "questions-and-approvals.reject-and-revise";
const selected = await loadChatCatalogImplementation(chatCatalogCases.find(row => row.scenarioId === id)!);
await expect(selected.act!({ input: { execution: { id: `chat-features.runner-opencode.local.${id}` } }, observationDeadlineAt: 630000 } as unknown as ChatCatalogContext, {})).rejects.toThrow("closed pure phase boundary");
expect(captured.deadlineAt).toBe(380000);
});
tests/runner-e2e/launch.ts
+2 -2
View File
@@ -923,7 +923,7 @@ async function runAttempt(input: {
});
}
await copyHealthHelloDetail(temporaryRoot, candidate.id, uploadDir, credentials);
await copyRetainedSourceSdk(temporaryRoot, candidate.id, result.source?.sha ?? null, uploadDir, credentials);
await copyRetainedSourceSdk(temporaryRoot, candidate.id, result.source?.sha ?? null, uploadDir, credentials, repositoryRoot);
console.log(
`${result.status === "passed" ? "PASS" : "FAIL"} ${candidate.id} attempt ${attempt} -> ${uploadDir}`,
);
@@ -956,7 +956,7 @@ async function runAttempt(input: {
schemaVersion: 1, executionId: candidate.id, originalResult: original,
packaging: "failed; raw diagnostics retained privately", error: error instanceof Error ? error.message : String(error),
}, attemptSecrets), null, 2)}\n`, { mode: 0o600 });
await copyRetainedSourceSdk(temporaryRoot, candidate.id, result.source?.sha ?? null, uploadDir, attemptSecrets).catch(() => {});
await copyRetainedSourceSdk(temporaryRoot, candidate.id, result.source?.sha ?? null, uploadDir, attemptSecrets, repositoryRoot).catch(() => {});
if (rawCleanupResults.length < executions.length) rawCleanupResults.push({ cleanup: original.cleanup, synthetic: original === fallback, status: original.status });
publishedResults.push(result); publishedResultPaths.set(candidate.id, resultPath);
}
tests/runner-e2e/redaction.ts
+11
View File
@@ -13,6 +13,17 @@ const SECRET_SHAPES = [
/\b(?:openrouter|daytona)[-_]?(?:api)?[-_]?key["'=:\s]+[A-Za-z0-9._-]{12,}\b/gi,
] as const;
export function secretShapeObservations(text: string) {
const matches: Array<{ detector: number; start: number; end: number }> = [];
for (const [detector, pattern] of SECRET_SHAPES.entries()) {
for (const match of text.matchAll(new RegExp(pattern.source, pattern.flags))) {
if (matches.length >= 32) return { bounded: false, matches };
matches.push({ detector, start: match.index!, end: match.index! + match[0].length });
}
}
return { bounded: true, matches: matches.sort((a, b) => a.start - b.start || a.detector - b.detector) };
}
const SENSITIVE_JSON_KEY =
/^(?:api[-_]?key|access[-_]?token|refresh[-_]?token|auth(?:orization)?|bearer|client[-_]?secret|cookie|password|secret|token)$/i;
tests/runner-e2e/runner.spec.ts
+7 -2
View File
@@ -1,3 +1,4 @@
import { withSdkCanaryPreFixture, captureSdkCanaryPreFixture } from "./sdk-canary-prefixture.js";
import { withholdPrivateScreenshots } from "./screenshot-privacy.js";
import { ordinaryChatCapture } from "./chat-screenshot-capture.js";
import { checkSetupHealthCredential } from "./chat-setup-credential.js";
@@ -1022,7 +1023,11 @@ for (const execution of executions) {
if (execution.suite.id === "daytona-git-streaming") {
await setupGitStreamingWorkspace(workspacePath);
}
fixtures = execution.task.flow === "first_task"
fixtures = await withSdkCanaryPreFixture({ execution, sourceRevision: resolveRunnerE2ESource().sha,
capture: () => captureSdkCanaryPreFixture({ execution, repositoryRoot: path.resolve(import.meta.dirname, "../.."), temporaryRoot,
sourceRevision: resolveRunnerE2ESource().sha, secrets }),
retain: proof => writeSanitizedJson(snapshotsDir, "sdk-canary-pre-fixture.json", proof, secrets),
fixtures: async () => execution.task.flow === "first_task"
? await setupFirstTaskFixtures({ page, api, execution, nonce, credentials, daytonaImage: process.env.PAPERCLIP_E2E_DAYTONA_IMAGE, deadlineAt: startedAtMs + deadlineMs, observe: value => { fixtures = value; }, onCompanyCreated })
: await setupLiveFixtures({
api,
@@ -1032,7 +1037,7 @@ for (const execution of executions) {
credentials,
daytonaImage: process.env.PAPERCLIP_E2E_DAYTONA_IMAGE,
onCompanyCreated,
});
}) });
await finishBudgetedEvalSetup({ executionId: execution.id, environmentId: execution.environment.id, agentId: fixtures.agent.id,
companyId: fixtures.company.id, deadlineAt: startedAtMs + deadlineMs, api,
observe: observations => writeSanitizedJson(snapshotsDir, "budgeted-setup.json", { observations }, secrets) });
tests/runner-e2e/sdk-canary-build-pins.json
+28
View File
@@ -0,0 +1,28 @@
{
"schema": "paperclip.sdk-canary-build-pins/v1",
"authenticatedMetadataSha256": "56e130cadb510a7bbbfc9aea2386a655d83ad7a13e84a326e1019ad856f2068c",
"frozenInputManifestSha256": "88980780c834e4bcc186eac46eb5cde8a63b77d36242f56192dffe0885d547aa",
"copiedInputManifestSha256": "19a20f8e78cd0d07fcb56390b5ac2cb8945e73b8659f4a98beeca83f6976373b",
"helperSha256": "4e8c21ea34550b5308009cbc4cebfebfa99d1c728b641600964bde5237f372ae",
"lockSha256": "c7f23021335b60da0cff1dd4c75cf8b948333ef3165d7d1703eb79a589edb798",
"compilerPackageSha256": "3722b30210616a13a3213ded11575ba6b2dbab10c32a5ef67afca8513e27017e",
"publicInputsFileSha256": "4330369fd34812273db11762c82ce4c427bae2353991eae354e7923e229b4a4a",
"inputFiles": 501,
"gitInputFiles": 500,
"compiledFiles": 1611,
"compiledManifestSha256": "49391ba801b596df6866325b7338a0945773312024c9bdc65537a3d47acce261",
"certificates": [
{
"target": "packages/shared/dist",
"sources": "45f45bba15a88e69ebf492592c92856af68ea08d8c6e242c002ab2568f4929fe",
"outputs": "b3f2c23d77d986eec5ff63d3f642fd55d0884426dd87fc14432a96f3ee3f9674",
"markerSha256": "a5324336d8bf33e86fb083cd505ea37f8d0c23b63fae3ce880b135e52c54475e"
},
{
"target": "packages/plugins/sdk/dist",
"sources": "913f4864e89451a84ceacf33f69b40eec174d0fbc5761cd833e49ae362a53fb6",
"outputs": "2dbee883799e3cef21744a9f45ec5ae1fa3b6bec6d1257ca240fe5612a12fe71",
"markerSha256": "f7e9a10281aa5bde04745ffd86c0b0ddba1c2687d2d104dbbc30e4db77532bb5"
}
]
}
tests/runner-e2e/sdk-canary-prefixture.test.ts
+85
View File
@@ -0,0 +1,85 @@
import { expect, it, vi } from "vitest";
import { readFile, writeFile, mkdir, mkdtemp, realpath, symlink, rm } from "node:fs/promises";
import path from "node:path"; import os from "node:os"; import { createHash } from "node:crypto";
import pins from "./sdk-canary-build-pins.json" with { type: "json" };
import { assertSdkCanaryBuild, sdkCanaryOutputManifest, withSdkCanaryPreFixture, assertSdkCanaryFingerprintMatches, SDK_CANARY, type SdkCanaryBuildObservation } from "./sdk-canary-prefixture.js";
import type { MatrixExecution } from "./types.js";
const hash = (value: string | Buffer) => createHash("sha256").update(value).digest("hex");
const revision = "a".repeat(40);
const build = (): SdkCanaryBuildObservation => ({ ...pins, sourceRevision: revision, retainedManifestSha256: pins.compiledManifestSha256 });
const execution = () => ({ id: SDK_CANARY, profile: { id: "runner-opencode", provider: "opencode", adapterType: "paperclip_runner", credential: "OPENROUTER_API_KEY", generation: "native", model: "openrouter/deepseek/deepseek-v4.1-flash-1017" }, suite: { id: "chat-features" }, environment: { id: "local" }, task: { id: "plugins.use-plugin-tool", flow: "agent_chat", turnTimeoutMs: 0, expectedRunCount: 2, attemptTimeoutMs: { local: 480_000 } } }) as MatrixExecution;
type Fingerprint = Parameters<typeof assertSdkCanaryFingerprintMatches>[1];
// A synthetic value for lifecycle controls only; never computed or reported as an actual case fingerprint.
const fingerprint = () => ({ fingerprintKind: "complete-execution-and-oracle-closure", digest: "b".repeat(64), definitionDigest: "b".repeat(64), graderDigest: "b".repeat(64), sources: [{ file: "packages/plugins/sdk/dist/index.js", sha256: "c".repeat(64) }] }) as unknown as Fingerprint;
it("binds authenticated output pins to the diagnostic and500Git+lock-overlay contract", () => {
expect(pins.authenticatedMetadataSha256).toBe("56e130cadb510a7bbbfc9aea2386a655d83ad7a13e84a326e1019ad856f2068c");
expect(pins.inputFiles).toBe(501); expect(pins.gitInputFiles).toBe(500); expect(pins.compiledFiles).toBe(1611);
expect(() => assertSdkCanaryBuild(build())).not.toThrow();
});
it.each(["inputFiles", "gitInputFiles", "frozenInputManifestSha256", "copiedInputManifestSha256", "publicInputsFileSha256", "helperSha256", "lockSha256", "compilerPackageSha256", "compiledFiles", "compiledManifestSha256", "retainedManifestSha256", "sourceRevision", "certificates"] as const)("blocks mismatched %s", field => {
const wrong = build(); Object.assign(wrong, { [field]: typeof wrong[field] === "number" ? 0 : field === "certificates" ? [] : "mismatch" });
expect(() => assertSdkCanaryBuild(wrong)).toThrow();
});
it("captures and privately retains the non-null complete fingerprint before allowing fixtures once", async () => {
const order: string[] = [], selected = execution(), fp = fingerprint();
const run = () => withSdkCanaryPreFixture({ execution: selected, sourceRevision: revision,
capture: async () => { order.push("capture-after-existing-startup-build-retention"); return { build: build(), fingerprint: fp }; },
retain: async () => { order.push("private-proof"); }, fixtures: async () => { order.push("fixtures-and-possible-hello"); return 7; } });
expect(await run()).toBe(7); expect(order).toEqual(["capture-after-existing-startup-build-retention", "private-proof", "fixtures-and-possible-hello"]);
expect(() => assertSdkCanaryFingerprintMatches(selected, structuredClone(fp))).not.toThrow();
await expect(run()).rejects.toThrow(); expect(order).toHaveLength(3);
expect(() => assertSdkCanaryFingerprintMatches(selected, { ...fp, digest: "d".repeat(64) } as never)).toThrow();
});
it.each(["startup-build-or-retention", "missing-output", "missing-fingerprint", "deferred-fingerprint", "private-retention", "wrong-source"])("never admits fixtures when %s fails", async reason => {
const fixtures = vi.fn(), fp = fingerprint();
await expect(withSdkCanaryPreFixture({ execution: execution(), sourceRevision: revision,
capture: async () => {
if (["startup-build-or-retention", "missing-output"].includes(reason)) throw new Error("Synthetic startup prerequisite failed");
return { build: { ...build(), ...(reason === "wrong-source" ? { sourceRevision: "e".repeat(40) } : {}) },
fingerprint: reason === "missing-fingerprint" ? null as never : reason === "deferred-fingerprint" ? { ...fp, digest: null } as never : fp };
}, retain: async () => { if (reason === "private-retention") throw new Error("Synthetic private custody failure"); }, fixtures })).rejects.toThrow();
expect(fixtures).not.toHaveBeenCalled();
});
it("rejects a later fingerprint without pre-fixture custody and leaves nonselected cases unchanged", async () => {
expect(() => assertSdkCanaryFingerprintMatches(execution(), fingerprint())).toThrow();
const other = { ...execution(), id: "chat-features.runner-opencode.local.plugins.install-plugin" }, capture = vi.fn(), retain = vi.fn(), fixtures = vi.fn(async () => 3);
expect(await withSdkCanaryPreFixture({ execution: other, sourceRevision: null, capture, retain, fixtures })).toBe(3);
expect(capture).not.toHaveBeenCalled(); expect(retain).not.toHaveBeenCalled(); expect(fixtures).toHaveBeenCalledOnce();
});
it("reads owned synthetic compiled manifests, detects missing output and rejects symlinks without a compiler", async () => {
const root = await realpath(await mkdtemp(path.join(os.tmpdir(), "sdk-canary-synthetic-")));
try {
const rows = [];
for (const target of pins.certificates) {
await mkdir(path.join(root, target.target), { recursive: true });
for (const [file, text] of [["index.js", "export const syntheticUnit=true;\n"], [".paperclip-build-complete", JSON.stringify({ sources: "a".repeat(64), outputs: "b".repeat(64) })+'\n']]) {
await writeFile(path.join(root, target.target, file!), text!);
rows.push({ file: `${target.target}/${file}`, size: Buffer.byteLength(text!), sha256: hash(text!) });
}
}
rows.sort((a, b) => a.file.localeCompare(b.file));
expect(await sdkCanaryOutputManifest(root)).toEqual(rows);
const synthetic = { ...build(), compiledFiles: rows.length, compiledManifestSha256: hash(JSON.stringify(rows)), retainedManifestSha256: hash(JSON.stringify(rows)) };
expect(() => assertSdkCanaryBuild(synthetic, { ...pins, compiledFiles: rows.length, compiledManifestSha256: synthetic.compiledManifestSha256 })).not.toThrow();
expect(() => assertSdkCanaryBuild(synthetic)).toThrow(); // Live immutable1611-file expectations never accept toy outputs.
await writeFile(path.join(root, pins.certificates[0]!.target, "index.js"), "synthetic mutation");
expect(hash(JSON.stringify(await sdkCanaryOutputManifest(root)))).not.toBe(synthetic.compiledManifestSha256);
await rm(path.join(root, pins.certificates[0]!.target), { recursive: true }); await expect(sdkCanaryOutputManifest(root)).rejects.toThrow();
await symlink(path.join(root, pins.certificates[1]!.target), path.join(root, pins.certificates[0]!.target)); await expect(sdkCanaryOutputManifest(root)).rejects.toThrow();
} finally { await rm(root, { recursive: true, force: true }); }
});
it.each(["model", "credential", "generation", "suite", "environment", "flow", "turn-clock", "count", "overall-clock"])("rejects changed selected %s before capture or fixtures", async field => {
const selected = execution(), capture = vi.fn(), fixtures = vi.fn();
if (field === "model") selected.profile.model = "foreign/model";
if (field === "credential") selected.profile.credential = "OPENAI_API_KEY";
if (field === "generation") selected.profile.generation = "legacy";
if (field === "suite") selected.suite.id = "another-suite";
if (field === "environment") selected.environment.id = "daytona";
if (field === "flow") selected.task.flow = "first_task";
if (field === "turn-clock") selected.task.turnTimeoutMs = 120000;
if (field === "count") selected.task.expectedRunCount = 3;
if (field === "overall-clock") selected.task.attemptTimeoutMs = { ...selected.task.attemptTimeoutMs, local: 960000 };
await expect(withSdkCanaryPreFixture({ execution: selected, sourceRevision: revision, capture, retain: vi.fn(), fixtures })).rejects.toThrow();
expect(capture).not.toHaveBeenCalled(); expect(fixtures).not.toHaveBeenCalled();
});
tests/runner-e2e/sdk-canary-prefixture.ts
+112
View File
@@ -0,0 +1,112 @@
import { createHash } from "node:crypto";
import { lstat, readFile, readdir, realpath } from "node:fs/promises";
import path from "node:path";
import pins from "./sdk-canary-build-pins.json" with { type: "json" };
import inputs from "./source-sdk-public-inputs.json" with { type: "json" };
import { verifySdkPublicSource } from "./source-sdk-public-type.js";
import { verifyRetainedSourceSdk } from "./source-sdk-evidence.js";
import { chatCatalogCaseFingerprint, normalizedChatCatalogExecution } from "./chat-catalog-fingerprint.js";
import { findChatCatalogCase } from "./chat-catalog-registry.js";
import { CHAT_OPENCODE_MODEL, CHAT_OPENCODE_PROFILE, reviewedChatProfile } from "./chat-runtime-profile.js";
import type { MatrixExecution } from "./types.js";
const hash = (value: string | Buffer) => createHash("sha256").update(value).digest("hex");
const hex = (value: unknown) => typeof value === "string" && /^[a-f0-9]{64}$/.test(value);
function fail(): never { throw new Error("SDK canary pre-fixture prerequisite is unqualified"); }
export const SDK_CANARY = "chat-features.runner-opencode.local.plugins.use-plugin-tool";
type Fingerprint = ReturnType<typeof chatCatalogCaseFingerprint>;
interface FileIdentity { file: string; size: number; sha256: string }
export interface SdkCanaryBuildObservation {
sourceRevision: string; inputFiles: number; gitInputFiles: number; frozenInputManifestSha256: string; copiedInputManifestSha256: string;
publicInputsFileSha256: string; helperSha256: string; lockSha256: string; compilerPackageSha256: string;
compiledFiles: number; compiledManifestSha256: string; retainedManifestSha256: string;
certificates: typeof pins.certificates;
}
export function assertSdkCanaryBuild(observed: SdkCanaryBuildObservation, expected = pins) {
if (!/^[a-f0-9]{40}$/.test(observed.sourceRevision)) fail();
for (const key of ["inputFiles", "gitInputFiles", "frozenInputManifestSha256", "copiedInputManifestSha256", "publicInputsFileSha256", "helperSha256", "lockSha256", "compilerPackageSha256", "compiledFiles", "compiledManifestSha256"] as const)
if (observed[key] !== expected[key]) fail();
if (observed.retainedManifestSha256 !== expected.compiledManifestSha256 || JSON.stringify(observed.certificates) !== JSON.stringify(expected.certificates)) fail();
}
export async function sdkCanaryOutputManifest(root: string) {
const files: FileIdentity[] = []; let total = 0;
const visit = async (relative: string) => {
const file = path.join(root, relative), info = await lstat(file);
if (info.isSymbolicLink() || await realpath(file) !== file) fail();
if (info.isDirectory()) { for (const child of await readdir(file)) await visit(`${relative}/${child}`); }
else {
if (!info.isFile() || info.size > 8 * 1024 * 1024 || files.length >= 4096 || (total += info.size) > 64 * 1024 * 1024) fail();
const bytes = await readFile(file); files.push({ file: relative, size: bytes.length, sha256: hash(bytes) });
}
};
for (const certificate of pins.certificates) await visit(certificate.target);
return files.sort((a, b) => a.file.localeCompare(b.file));
}
export async function captureSdkCanaryBuild(input: { repositoryRoot: string; temporaryRoot: string; sourceRevision: string | null; secrets: readonly string[] }) {
if (!input.sourceRevision || await realpath(input.repositoryRoot) !== input.repositoryRoot) fail();
const source = await verifySdkPublicSource(input.repositoryRoot);
const publicInputsFileSha256 = hash(await readFile(path.join(input.repositoryRoot, "tests/runner-e2e/source-sdk-public-inputs.json")));
const copied = [];
for (const [file, gitBlobSha1] of Object.entries(inputs.inputs)) {
const bytes = await readFile(path.join(input.repositoryRoot, file));
copied.push({ path: file, bytes: bytes.length, sha256: hash(bytes), gitBlobSha1, restoredLockOverlay: file === "pnpm-lock.yaml" });
}
copied.sort((a, b) => a.path.localeCompare(b.path));
const retained = await verifyRetainedSourceSdk(input.temporaryRoot, SDK_CANARY, input.sourceRevision, input.secrets, input.repositoryRoot);
if (retained.receipt.schema !== "paperclip.source-plugin-sdk/v2" || retained.receipt.scanAudit?.bounded !== true || retained.receipt.scanAudit.suppressedFindings !== 0
|| !Array.isArray(retained.receipt.certifiedTargets) || retained.receipt.certifiedTargets.length !== pins.certificates.length) fail();
const files = await sdkCanaryOutputManifest(input.repositoryRoot);
const retainedFiles = retained.files.filter(row => row.file !== "receipt.json").map(({ file, size, sha256 }) => ({ file, size, sha256 })).sort((a, b) => a.file.localeCompare(b.file));
const certificates = [];
for (const expected of pins.certificates) {
const marker = await readFile(path.join(input.repositoryRoot, expected.target, ".paperclip-build-complete"));
const certificate = JSON.parse(marker.toString("utf8"));
const saved = retained.receipt.certifiedTargets.find(row => row.directory === expected.target);
if (saved?.sources !== certificate.sources || saved?.outputs !== certificate.outputs) fail();
certificates.push({ target: expected.target, sources: certificate.sources, outputs: certificate.outputs, markerSha256: hash(marker) });
}
const observed: SdkCanaryBuildObservation = { sourceRevision: input.sourceRevision, inputFiles: copied.length, gitInputFiles: copied.length - 1,
frozenInputManifestSha256: hash(JSON.stringify({ frozenRevision: "cb31ebccf0411f3e260159b1d39e6c047a39f4e9", inputs: inputs.inputs, schema: "paperclip.sdk-diagnostic-frozen-inputs/v1" })),
copiedInputManifestSha256: hash(JSON.stringify(copied)), publicInputsFileSha256,
helperSha256: hash(await readFile(path.join(input.repositoryRoot, "scripts/ensure-plugin-build-deps.mjs"))),
lockSha256: source.identity.lockSha256, compilerPackageSha256: source.identity.compilerPackageSha256,
compiledFiles: files.length, compiledManifestSha256: hash(JSON.stringify(files)), retainedManifestSha256: hash(JSON.stringify(retainedFiles)), certificates };
if (retained.receipt.buildHelperSha256 !== observed.helperSha256 || retained.receipt.lockSha256 !== observed.lockSha256) fail();
assertSdkCanaryBuild(observed); return observed;
}
function completeFingerprint(fingerprint: Fingerprint) {
if (!fingerprint || fingerprint.fingerprintKind !== "complete-execution-and-oracle-closure" || !hex(fingerprint.digest)
|| fingerprint.definitionDigest !== fingerprint.digest || fingerprint.graderDigest !== fingerprint.digest || !Array.isArray(fingerprint.sources) || !fingerprint.sources.length
|| fingerprint.sources.some(row => !row.file || !hex(row.sha256)) || new Set(fingerprint.sources.map(row => row.file)).size !== fingerprint.sources.length
|| !fingerprint.sources.some(row => row.file === "packages/plugins/sdk/dist/index.js")) fail();
}
const entered = new WeakSet<MatrixExecution>();
const fingerprints = new WeakMap<MatrixExecution, Fingerprint>();
export async function withSdkCanaryPreFixture<T>(input: { execution: MatrixExecution; sourceRevision: string | null;
capture(): Promise<{ build: SdkCanaryBuildObservation; fingerprint: Fingerprint }>;
retain(proof: unknown): Promise<void>; fixtures(): Promise<T>;
}) {
if (input.execution.id !== SDK_CANARY) return input.fixtures();
if (entered.has(input.execution)) fail(); entered.add(input.execution);
if (!reviewedChatProfile(input.execution.profile) || input.execution.profile.id !== CHAT_OPENCODE_PROFILE || input.execution.profile.provider !== "opencode"
|| input.execution.profile.model !== CHAT_OPENCODE_MODEL || input.execution.profile.generation !== "native" || input.execution.suite.id !== "chat-features"
|| input.execution.environment.id !== "local" || input.execution.task.flow !== "agent_chat" || input.execution.task.turnTimeoutMs !== 0
|| input.execution.task.id !== "plugins.use-plugin-tool" || input.execution.task.expectedRunCount !== 2 || input.execution.task.attemptTimeoutMs.local !== 480_000) fail();
const captured = await input.capture(); assertSdkCanaryBuild(captured.build); completeFingerprint(captured.fingerprint);
if (captured.build.sourceRevision !== input.sourceRevision) fail();
await input.retain({ schema: "paperclip.sdk-canary-prefixture/v1", executionId: SDK_CANARY, sourceRevision: input.sourceRevision,
authenticatedMetadataSha256: pins.authenticatedMetadataSha256, build: captured.build, fingerprint: captured.fingerprint });
fingerprints.set(input.execution, structuredClone(captured.fingerprint));
return input.fixtures();
}
export async function captureSdkCanaryPreFixture(input: { execution: MatrixExecution; repositoryRoot: string; temporaryRoot: string; sourceRevision: string | null; secrets: readonly string[] }) {
const build = await captureSdkCanaryBuild(input);
const descriptor = findChatCatalogCase(input.execution.task.id);
if (!descriptor || descriptor.expectedHumanMessageCount !== 2 || descriptor.expectedRunCount !== 2 || descriptor.attemptTimeoutMs !== 480_000) fail();
return { build, fingerprint: chatCatalogCaseFingerprint(descriptor, normalizedChatCatalogExecution(input.execution)) };
}
export function assertSdkCanaryFingerprintMatches(execution: MatrixExecution, fingerprint: Fingerprint) {
if (execution.id !== SDK_CANARY) return;
completeFingerprint(fingerprint);
const before = fingerprints.get(execution);
if (!before || JSON.stringify(before) !== JSON.stringify(fingerprint)) fail();
}
tests/runner-e2e/source-sdk-evidence.ts
+36 -7
View File
@@ -1,3 +1,4 @@
import { createSdkArtifactScanner } from "./source-sdk-public-type.js";
import { createHash } from "node:crypto";
import { chmod, lstat, mkdir, readFile, readdir, realpath, rm, writeFile } from "node:fs/promises";
import path from "node:path";
@@ -13,7 +14,7 @@ const staging = (root: string) => path.join(root, "artifacts-private", "source-s
interface BuiltFile { file: string; size: number; sha256: string }
const retainedPath = (file: string) => file.replace(/\/\.paperclip-build-complete$/, "/build-completion-receipt.json");
interface RetainedBuild {
schema: "paperclip.source-plugin-sdk/v1"; executionId: string; sourceRevision: string; retainedAt: string;
schema: "paperclip.source-plugin-sdk/v1" | "paperclip.source-plugin-sdk/v2"; sourceRootSha256?: string; scanAudit?: Awaited<ReturnType<ReturnType<typeof createSdkArtifactScanner>["audit"]>>; executionId: string; sourceRevision: string; retainedAt: string;
buildHelperSha256: string; lockSha256: string; files: Array<BuiltFile & { retainedPath: string; retainedSha256: string }>;
certifiedTargets: Array<{ directory: string; sources: string; outputs: string }>;
}
@@ -21,6 +22,13 @@ function selected(ids: readonly string[]) { return ids.length === 1 && selectedI
function checkSecret(bytes: Buffer, secrets: readonly string[]) {
if (findSecretLeak(bytes.toString("utf8"), secrets)) throw new Error("Source SDK evidence privacy check failed; bytes withheld");
}
async function retainScanAudit(root: string, executionId: string, audit: unknown, complete: boolean, secrets: readonly string[]) {
const bytes = Buffer.from(JSON.stringify({ schema: "paperclip.source-sdk-privacy-observation/v1", complete, audit }, null, 2) + "\n");
checkSecret(bytes, secrets);
const directory = path.join(root, "artifacts-private", "cases", executionId.slice("chat-features.runner-opencode.local.".length), "snapshots");
await mkdir(directory, { recursive: true, mode: 0o700 });
await writeFile(path.join(directory, "source-sdk-privacy-observations.json"), bytes, { flag: "wx", mode: 0o600 });
}
async function boundedTree(root: string) {
const files: Array<BuiltFile & { bytes: Buffer }> = []; let total = 0;
async function visit(directory: string) {
@@ -53,6 +61,7 @@ export async function retainPreparedSourceSdk(input: { repositoryRoot: string; t
const root = await realpath(input.temporaryRoot), repository = await realpath(input.repositoryRoot);
if (await realpath(path.join(root, "artifacts-private")) !== path.join(root, "artifacts-private")) throw new Error("Source SDK staging custody invalid");
const secrets = diagnosticSecrets(readPrivateDiagnosticSecrets(path.join(root, "selected-private-inputs", "diagnostic-redaction.json"), root)).values;
const scanner = createSdkArtifactScanner(repository, secrets);
const buffers: Array<BuiltFile & { bytes: Buffer }> = [], certifiedTargets: RetainedBuild["certifiedTargets"] = [];
for (const directory of targets) {
const files = await boundedTree(path.join(repository, directory));
@@ -61,10 +70,19 @@ export async function retainPreparedSourceSdk(input: { repositoryRoot: string; t
const certificate = JSON.parse(marker.bytes.toString("utf8"));
if (!hex(certificate.sources) || !hex(certificate.outputs) || outputHash(files) !== certificate.outputs) throw new Error("Source SDK build output no longer matches its completed compiler receipt");
certifiedTargets.push({ directory, sources: certificate.sources, outputs: certificate.outputs });
for (const file of files) { checkSecret(file.bytes, secrets); buffers.push({ ...file, file: `${directory}/${file.file}` }); }
for (const file of files) {
if (!await scanner.scan(`${directory}/${file.file}`, file.bytes)) {
await retainScanAudit(root, input.executionIds[0]!, await scanner.audit(), false, secrets);
throw new Error("Source SDK evidence privacy check failed; bytes withheld");
}
buffers.push({ ...file, file: `${directory}/${file.file}` });
}
}
if (buffers.length >= 4096 || buffers.reduce((total, file) => total + file.size, 0) > 63 * 1024 * 1024) throw new Error("Source SDK output exceeds combined evidence bound");
const receipt: RetainedBuild = { schema: "paperclip.source-plugin-sdk/v1", executionId: input.executionIds[0]!, sourceRevision: input.sourceRevision!, retainedAt: new Date().toISOString(),
const scanAudit = await scanner.audit();
await retainScanAudit(root, input.executionIds[0]!, scanAudit, true, secrets);
const syntax = scanAudit.observations.some(row => row.classification === "public_schema_type");
const receipt: RetainedBuild = { schema: syntax ? "paperclip.source-plugin-sdk/v2" : "paperclip.source-plugin-sdk/v1", ...(syntax ? { sourceRootSha256: hash(repository), scanAudit } : {}), executionId: input.executionIds[0]!, sourceRevision: input.sourceRevision!, retainedAt: new Date().toISOString(),
buildHelperSha256: hash(await readFile(path.join(repository, "scripts/ensure-plugin-build-deps.mjs"))), lockSha256: hash(await readFile(path.join(repository, "pnpm-lock.yaml"))),
certifiedTargets, files: buffers.map(({ bytes: _, ...row }) => ({ ...row, retainedPath: retainedPath(row.file), retainedSha256: row.sha256 })) };
if (new Set(receipt.files.map(file => file.retainedPath)).size !== receipt.files.length) throw new Error("Source SDK retained filename collision");
@@ -78,18 +96,29 @@ export async function retainPreparedSourceSdk(input: { repositoryRoot: string; t
await writeFile(path.join(destination, "receipt.json"), receiptBytes, { flag: "wx", mode: 0o600 });
} catch (error) { await rm(destination, { recursive: true, force: true }); throw error; }
}
export async function copyRetainedSourceSdk(root: string, executionId: string, sourceRevision: string | null, uploadDir: string, knownSecrets: readonly string[]) {
if (!selected([executionId])) return;
export async function verifyRetainedSourceSdk(root: string, executionId: string, sourceRevision: string | null, knownSecrets: readonly string[], repositoryRoot = path.resolve(import.meta.dirname, "../..")) {
if (!selected([executionId])) throw new Error("Source SDK retained selection mismatch");
if (await realpath(staging(root)) !== staging(root)) throw new Error("Source SDK retained custody invalid");
const secrets = diagnosticSecrets(knownSecrets).values, files = await boundedTree(staging(root));
const receiptFile = files.find(row => row.file === "receipt.json"); if (!receiptFile) throw new Error("Source SDK retained receipt missing");
const receipt = JSON.parse(receiptFile.bytes.toString("utf8")) as RetainedBuild;
if (receipt.schema !== "paperclip.source-plugin-sdk/v1" || receipt.executionId !== executionId || receipt.sourceRevision !== sourceRevision
if (!["paperclip.source-plugin-sdk/v1", "paperclip.source-plugin-sdk/v2"].includes(receipt.schema) || receipt.executionId !== executionId || receipt.sourceRevision !== sourceRevision
|| !hex(receipt.buildHelperSha256) || !hex(receipt.lockSha256) || !Array.isArray(receipt.files)) throw new Error("Source SDK retained identity mismatch");
const actual = files.filter(row => row.file !== "receipt.json");
if (actual.length !== receipt.files.length || actual.some(row => !targets.some(target => row.file.startsWith(`${target}/`))
|| !receipt.files.some(expected => expected.file === row.file && expected.retainedPath === retainedPath(row.file) && expected.size === row.size && expected.sha256 === row.sha256 && expected.retainedSha256 === row.sha256))) throw new Error("Source SDK retained bytes changed");
for (const file of files) checkSecret(file.bytes, secrets);
if (receipt.schema === "paperclip.source-plugin-sdk/v2") {
if (!receipt.scanAudit || receipt.sourceRootSha256 !== hash(await realpath(repositoryRoot))) throw new Error("Source SDK public source custody changed");
const scanner = createSdkArtifactScanner(await realpath(repositoryRoot), secrets);
for (const file of actual) if (!await scanner.scan(file.file, file.bytes)) throw new Error("Source SDK evidence privacy check failed; bytes withheld");
if (JSON.stringify(await scanner.audit()) !== JSON.stringify(receipt.scanAudit)) throw new Error("Source SDK public observations changed");
checkSecret(receiptFile.bytes, secrets);
} else for (const file of files) checkSecret(file.bytes, secrets);
return { receipt, files };
}
export async function copyRetainedSourceSdk(root: string, executionId: string, sourceRevision: string | null, uploadDir: string, knownSecrets: readonly string[], repositoryRoot = path.resolve(import.meta.dirname, "../..")) {
if (!selected([executionId])) return;
const { files } = await verifyRetainedSourceSdk(root, executionId, sourceRevision, knownSecrets, repositoryRoot);
const destination = path.join(uploadDir, SOURCE_SDK_DIAGNOSTICS); await mkdir(destination, { recursive: true, mode: 0o700 });
for (const file of files) {
const filename = path.join(destination, retainedPath(file.file)); await mkdir(path.dirname(filename), { recursive: true, mode: 0o700 });
tests/runner-e2e/source-sdk-public-inputs.json
+516
View File
@@ -0,0 +1,516 @@
{
"inputs": {
"package.json": "abff9ed16d555128f06f0dc5f74a5208b0e51d8d",
"packages/plugins/sdk/package.json": "1dd9473bdc6296126230ffd9320d480e0ae596b8",
"packages/plugins/sdk/src/bundlers.ts": "883bf3d9f2bd42877cfc0c3707cd882a6f6248d0",
"packages/plugins/sdk/src/define-plugin.ts": "18edbc1db9da4a731364023c2673be2dc201bceb",
"packages/plugins/sdk/src/dev-cli.ts": "7c1b0e2b6e4b7fd82027cca5321066e473a03d20",
"packages/plugins/sdk/src/dev-server.ts": "2eadff81b74ca0706aec53aaf8637797fe4b4093",
"packages/plugins/sdk/src/environment-creation-cleanup.ts": "35f23cd19ab0dd4148731846419a1c9b9942786b",
"packages/plugins/sdk/src/environment-sync-error.ts": "a630bb126e1c93935a21c46cd471c656d03bfe10",
"packages/plugins/sdk/src/environment-tasks.ts": "f4f86e26b5306c7cbb6af44974dee0e346f2eeb3",
"packages/plugins/sdk/src/host-client-factory.ts": "9e6a6cf8ce8a035b04e4bc59250ca7cc2b7788fb",
"packages/plugins/sdk/src/idle-drain.test.ts": "cc59f7881e7d46c89395a1d93d74f6c6b9b0a64a",
"packages/plugins/sdk/src/idle-drain.ts": "ae6e4c6be4657a3d541396922b3bc551b8db557f",
"packages/plugins/sdk/src/index.ts": "4cfb3ff1dc3d0c13068d22cfb2b53994c64ee6f8",
"packages/plugins/sdk/src/protocol.duplex-channel.test.ts": "18255eb3d63038dc459240c111cbcb29fcba25cc",
"packages/plugins/sdk/src/protocol.ts": "e76615d1687818bc3b4e98998c93375e2acdde9b",
"packages/plugins/sdk/src/testing.ts": "eb496adf22c62b6d68ef467c25465afd0973d236",
"packages/plugins/sdk/src/types.ts": "9f0ea69b7fea29d816641704b4406f494ca0b49e",
"packages/plugins/sdk/src/ui/clipboard.ts": "1f42069c97d731e636a6ad5481dad22d20c0ed63",
"packages/plugins/sdk/src/ui/components.ts": "3f0594408a1bb5ac3a8d00e709f91877dd260359",
"packages/plugins/sdk/src/ui/hooks.ts": "686d02cf4b35f1cddb79a83a4c4d81fd52f0100e",
"packages/plugins/sdk/src/ui/index.ts": "f7d1cee64a8f0c924bd10a1bd20745a52fe2060e",
"packages/plugins/sdk/src/ui/runtime.ts": "998428e9701eba3b762d3417236d32c2d0d5763d",
"packages/plugins/sdk/src/ui/types.ts": "5a5644acf6554d2e5e1d000bfd03b97d57001eb2",
"packages/plugins/sdk/src/worker-rpc-host.ts": "daad66769c026aaf849a593a2d0f6a28e6aa2429",
"packages/plugins/sdk/src/workspace-provider.ts": "bd28c5610fd1a5fbf0036e5a963a359f06d0b529",
"packages/plugins/sdk/tsconfig.json": "d62ebdff94bc4b75dabf1fcad187e1e7a90fc794",
"packages/shared/package.json": "b29212ed2555cdd22640404079d525ced76c7f42",
"packages/shared/src/__snapshots__/slack-app-manifest.test.ts.snap": "79020ba3aa6662923f70aa44e2c483844e9901d9",
"packages/shared/src/account-handle.test.ts": "fb77e08a68b8deb3ab5ebf271724a268ac6bae56",
"packages/shared/src/account-handle.ts": "4bc5d6ec16b854497e83ce781bd900dfec599cbd",
"packages/shared/src/accounting.ts": "86e29530871acb9b03fdba853039ad109c29d01f",
"packages/shared/src/adapter-agnostic-keys.test.ts": "d4977904a18ec068dfe2a995357009cf15385bdc",
"packages/shared/src/adapter-auth-check-code.test.ts": "5f466b2887051ce24ca92f2505ebf5d7cea3cd6c",
"packages/shared/src/adapter-auth-check-code.ts": "a8995b2d4875834ece1a9d71af04ca2a2c7c742c",
"packages/shared/src/adapter-auth-session.ts": "1bde62818a6e2a127ceb6e947882ded311798892",
"packages/shared/src/adapter-type.ts": "5af29dfc580ef18d53404b679cbe32a53587098e",
"packages/shared/src/adapter-types.test.ts": "49646cb8f85f86cb45d9dd0efda6b84ed0599341",
"packages/shared/src/agent-appearance.test.ts": "95cb8b22a18bd68932d618bb97ddde24d0f8ee23",
"packages/shared/src/agent-appearance.ts": "d45525d26e5899afe673aa4ff52ddc1b6cf67b11",
"packages/shared/src/agent-avatar-upload.ts": "7146d88adf363161494bddca0f1074c1aa4d7b52",
"packages/shared/src/agent-commentary.test.ts": "d8eb09c29f30023705de111639054e35d98bc5bf",
"packages/shared/src/agent-eligibility.test.ts": "ee85624d2ec13b50fb75bc675bf27b40fe2c8115",
"packages/shared/src/agent-eligibility.ts": "bf658da7e9de17719fd1cf1e151c5d231e766805",
"packages/shared/src/agent-observability.test.ts": "36cd74a8a8d1947364ec96fa30deabff14658df2",
"packages/shared/src/agent-observability.ts": "4e71f2ca92ff7eb5720b01b5568b26f46f21c5e3",
"packages/shared/src/agent-runner.ts": "413eb0adfb432de653e1c0dc4e606d8a04e9fe49",
"packages/shared/src/agent-url-key.ts": "d0ec20b3ea1636a960bc5372a2c1e5a124bc5511",
"packages/shared/src/aggregator-app-catalog.test.ts": "39ee377e55cc9565fce004b928bbe0998268e004",
"packages/shared/src/aggregator-app-catalog.ts": "d6778b07a41356878c549d039ae316553ca9a22b",
"packages/shared/src/aggregator-apps.ts": "2a69a54ee0cb305170275b6f4e97689c107defb1",
"packages/shared/src/ai-connection-router.ts": "9fbd89c385a4d5827bc68e1c7f6a66e00453d413",
"packages/shared/src/ai-connection-usage.ts": "4b51573ee92f8a2ae0f80dd15288a0e59b4dddc7",
"packages/shared/src/ai-connections.ts": "250eece62122cea327abe6cd6379d80584e523a1",
"packages/shared/src/ai-provider-routing.ts": "7095664442f2684b62cb232cedfb064d9049776d",
"packages/shared/src/announcements.test.ts": "121ed97b560e0dd271fe6602c97e2415ee4e4bf7",
"packages/shared/src/announcements.ts": "1405e849eeeeb67239bacc1bbb3903385a5faa6f",
"packages/shared/src/api.ts": "1d61cdff51433c4136368d52ee7cf2f16aa820ed",
"packages/shared/src/app-definitions-url.test.ts": "7378d29fd8428e4807dd234b495aa9043f40e6d7",
"packages/shared/src/app-definitions.generated.ts": "d74319a5c9684633b4a0a23c8bfa682e0d3bf267",
"packages/shared/src/app-definitions.ingestion-report.json": "a7b60c7c16a484a411989315e9f08d48258fa647",
"packages/shared/src/app-definitions.test.ts": "e0584a8d9eb648d3cc42d594a2e5617240475dab",
"packages/shared/src/app-definitions.ts": "eaadd6b7831cc23998a1a3ec6fd1d3a5032a503b",
"packages/shared/src/app-definitions/1password.json": "c11a4777de2c78805b2d391da5eb075a9511e0f8",
"packages/shared/src/app-definitions/agentmail.json": "edd0799b1980d0f5f43b89c8f7a1173dc623f7fa",
"packages/shared/src/app-definitions/airtable.json": "c9b216a2bbe693bc6167413255d41c33512d418d",
"packages/shared/src/app-definitions/anthropic.json": "52eb85f36005f031cf83e2f92e4e53cec9966d50",
"packages/shared/src/app-definitions/api-key-generic.json": "09e6d4295513ac9f1002eaf37c34e7c2d00a3c05",
"packages/shared/src/app-definitions/arcade.json": "c8379450055d4ea226d229b1cbf99c7ed2e52beb",
"packages/shared/src/app-definitions/asana.json": "4cc2859043a30037fd5cdc3ebd0909938b9344a8",
"packages/shared/src/app-definitions/bedrock.json": "962d8a722ac6e2e4db00abc34f71947a3fdb733a",
"packages/shared/src/app-definitions/beehiiv.json": "47aec8fdfbf5aed0919b6ac9bddb435de4988db3",
"packages/shared/src/app-definitions/bitly.json": "81f4937e92f2971f2011acb50929adf5287315e4",
"packages/shared/src/app-definitions/box.json": "f8c45a35b14709f458850df298149e73fc33274e",
"packages/shared/src/app-definitions/braintrust.json": "c7b52e8588bed8692e7e8fc02394f5f31a5ec027",
"packages/shared/src/app-definitions/brex.json": "08fce55d3993b5368e90289432b209517c0675cc",
"packages/shared/src/app-definitions/browser-use-cloud.json": "28ae41d20c1ed6750d8767679e6167ceed56ed5e",
"packages/shared/src/app-definitions/calendly.json": "cdedf7bed6940ec375042b6e1982b7af6e2d7284",
"packages/shared/src/app-definitions/candid.json": "7d9668da949918163ef3d4b4be85bf7b559613d5",
"packages/shared/src/app-definitions/chat-completions-api.json": "021e0df41be5bdc3801f33586adf91c22bb080ae",
"packages/shared/src/app-definitions/clickhouse.json": "f7ede2aa309c5607b008309ba32397924c73950d",
"packages/shared/src/app-definitions/clickup.json": "2367ba3afc2f22bd49572467643bd81620b3f350",
"packages/shared/src/app-definitions/cloudflare.json": "693237c0a9b09b66c4fc3b265efb17520b40df31",
"packages/shared/src/app-definitions/cloudinary.json": "64231fb870438fa5659b8bd5fc54f24e83acf8b5",
"packages/shared/src/app-definitions/coda.json": "f0f78684ae3e9110d395aa1769f0df84596135fc",
"packages/shared/src/app-definitions/cognee.json": "2019b44563fb22cd20f819efb8062496e3733f51",
"packages/shared/src/app-definitions/composio.json": "19bc349f24e3f76a0c46cc5d8fd4a3e0a75593af",
"packages/shared/src/app-definitions/context7.json": "b7e78083051913703a2e1819c24b8aef0d7e7c7f",
"packages/shared/src/app-definitions/daytona.json": "5ee359cd655d65bf98df7d1279175f0f847c6a95",
"packages/shared/src/app-definitions/discord.json": "a7a6f7031b3ca22401f704086f7ac80b365aaeb0",
"packages/shared/src/app-definitions/ditto.json": "5976f435edf01ef6493f63d753d41106145fa5e4",
"packages/shared/src/app-definitions/egnyte.json": "72a50f331359d1100a56c2d8a108613c4b090a31",
"packages/shared/src/app-definitions/embat.json": "f29752d20e8a9ef19a0b71af2f480deb190aa2a3",
"packages/shared/src/app-definitions/enterpret.json": "3725b17e2a18adc9dcdac055b918a7dd3f5c891e",
"packages/shared/src/app-definitions/exa.json": "f5e9a384c03e9b7e923f5a3e5609786c5cf78a97",
"packages/shared/src/app-definitions/executor.json": "f01e0cd4e11d966fc486f064d24e8986b448d6c0",
"packages/shared/src/app-definitions/firecrawl.json": "09f884d3dee406aa7f805b03553b770acf859d8b",
"packages/shared/src/app-definitions/fireflies.json": "2c4605a6044ccdd4e6cd66d2763f80a2fc5c88c0",
"packages/shared/src/app-definitions/gauge.json": "da9d1f2bd1592689f7363462c7c432d6f20a156f",
"packages/shared/src/app-definitions/github-code-review-bot.json": "c5897ebcbbbf4f0c2d243574748d4e682f14e57e",
"packages/shared/src/app-definitions/github.json": "1e7b359b584309c5e6e7c56f46d782c8bdbcd885",
"packages/shared/src/app-definitions/gmail.json": "a0f2dcfaaeca5bf54318d8cad82a0cb188e72bea",
"packages/shared/src/app-definitions/google-calendar.json": "ab9e93999dd69a5d432ae66195e517c66aad2b74",
"packages/shared/src/app-definitions/google-chat.json": "f3cf1f37b4b9f0e937b2af2e8d86ebf5df83ac7c",
"packages/shared/src/app-definitions/google-docs.json": "71a66e01f79f815a8c3091a3b413ff098c92a347",
"packages/shared/src/app-definitions/google-drive.json": "6a944df5532898a88269fbb3a0cadb0c241f6f3e",
"packages/shared/src/app-definitions/google-people.json": "07121347fb388b8258456505276975502dacc170",
"packages/shared/src/app-definitions/google-sheets.json": "de87b58007fca0b719d242148a838e2b5c26ddca",
"packages/shared/src/app-definitions/google-slides.json": "289366009d46be6b770eee6cd0bc650eb59a240c",
"packages/shared/src/app-definitions/google-workspace-search.json": "f3ec9d00ed0116b7cb8b9b1b71353cb8d6dc8cf4",
"packages/shared/src/app-definitions/google-workspace.json": "68e401fa299a85ad64426b66cf0f546522ee4730",
"packages/shared/src/app-definitions/google.json": "794282359d2a2b64585b421fb8ed0a55043ad586",
"packages/shared/src/app-definitions/gsc-wizard.json": "a4ba6f183d5f35a25d0a420ade18f00d3fb64534",
"packages/shared/src/app-definitions/honcho.json": "17185eaf5189fc8c443388997c90ebf833bc9327",
"packages/shared/src/app-definitions/hugging-face.json": "6567ad1e62c007da1702fc314736c2d3aa1d4499",
"packages/shared/src/app-definitions/imessage-photon.json": "b5f018a9fca83bfde88a714c3b09ceecf1be5e28",
"packages/shared/src/app-definitions/jira.json": "750e15c2695f9db4a6f67dcfa385f099d0bd286c",
"packages/shared/src/app-definitions/kernel.json": "bdf3d17edd364d4cc127edbcfc4bc907c994f48a",
"packages/shared/src/app-definitions/klaviyo.json": "839d800d676aa97ba3b4538302dcebe30096ab75",
"packages/shared/src/app-definitions/linear.json": "552c1ea4ef8a0ef122b9250a4ff3132752b010fc",
"packages/shared/src/app-definitions/local-falcon.json": "f069afe88ca681da8b51c221c83b5333589720d9",
"packages/shared/src/app-definitions/local.json": "c9cd6dab3631b349be94d7ea05541a13ac9613a0",
"packages/shared/src/app-definitions/make.json": "a8cd0bce733ee384a60d2fef1da55f5b6f518a99",
"packages/shared/src/app-definitions/manufact.json": "8973ddd3656bb138d0b899c004f093f12b864409",
"packages/shared/src/app-definitions/mem0.json": "ff74c42d3ed360c61119da76047fb13b930809b8",
"packages/shared/src/app-definitions/messages-api.json": "c5c8d855ee51eff560fa0cd1e98d7d72711e3dac",
"packages/shared/src/app-definitions/microsoft-teams.json": "12d0fa5eec8cbd2f325b1b43efdc31721d542917",
"packages/shared/src/app-definitions/miro.json": "93191223dbb5f8914d619386248b7d0c878b7d37",
"packages/shared/src/app-definitions/mixpanel.json": "b6f4c355fb80c4014d16200b20d88fc4ec009803",
"packages/shared/src/app-definitions/monday.json": "fa8e7d607faae9b4cbc13ea48d857829ce2d07eb",
"packages/shared/src/app-definitions/neon.json": "8e9f20eaa1290aba14c18c822c666a17a36e3762",
"packages/shared/src/app-definitions/netlify.json": "b94365447e8f890ea9534dfa455cd122b5f5cb71",
"packages/shared/src/app-definitions/notion.json": "54d07923fc2f79fc95c6bbf0098eb3c179af3a7d",
"packages/shared/src/app-definitions/oauth-generic.json": "b9a7ece3c237bd152c6870e0df1e286b822d3b91",
"packages/shared/src/app-definitions/openai.json": "14708e9bacff1e65b4041979f4c27aeb6c1c322b",
"packages/shared/src/app-definitions/openrouter.json": "d4d1be16fce8f5e33ad1597d0f02076e40d485b8",
"packages/shared/src/app-definitions/oreilly.json": "297f25e790f61c72b0190970ee4b4b3aabfbc6da",
"packages/shared/src/app-definitions/pagerduty.json": "47dd41d1cf64b5d409ec3632092cc1d39d29d836",
"packages/shared/src/app-definitions/paperclip-notebook.json": "4bdf194d6ba5eb94788ea34a61bc6253ada635fc",
"packages/shared/src/app-definitions/parallel-search.json": "cbc9843fb9649b620c745fc43e7d8228628fbe8c",
"packages/shared/src/app-definitions/planetscale.json": "96d64734ee4b5b918461c6169ce97e9027bdfdc1",
"packages/shared/src/app-definitions/posthog.json": "6042a291a24354e3eaba2ca283ec59c9cc7a258e",
"packages/shared/src/app-definitions/postman.json": "bcee846eabd7d8592342324b2f6c0a825a912e3f",
"packages/shared/src/app-definitions/railway.json": "96193b680a11a41e1fa34fd5c96233b70c608ed8",
"packages/shared/src/app-definitions/razorpay.json": "e2abe5d6f56864c2dae26b34c738c175307339d5",
"packages/shared/src/app-definitions/resend.json": "38f7595265f9ff6afdc84b9bce18602cd2c6ebfa",
"packages/shared/src/app-definitions/responses-api.json": "29ad92e4c41a0dc0e3a3b07f518200bdc8e2e6ff",
"packages/shared/src/app-definitions/sanity.json": "15968d52eb75008b170f45a82e79e0b77516b50d",
"packages/shared/src/app-definitions/sentry.json": "5a85dd024f11382a802557b0c5374b856e29b56a",
"packages/shared/src/app-definitions/shopify.json": "f6bff7db55083d96060d8bd46a08aa38b33c3080",
"packages/shared/src/app-definitions/similarweb.json": "37a912287b7ad323cf783eb32d01b943a93e83fd",
"packages/shared/src/app-definitions/slack.json": "208c7aaccb863784f450071e89499f0cfd32560b",
"packages/shared/src/app-definitions/speko.json": "6f25f776269b435e8cbde3c3249d35d3063156ce",
"packages/shared/src/app-definitions/stripe.json": "708fb6d3498dbb57902dd5ebc45c8fded5d6b578",
"packages/shared/src/app-definitions/supabase.json": "9a92a2f30af01e1c9f4cebdf7d4ba0101353c1c6",
"packages/shared/src/app-definitions/superagent.json": "6e1a847cc0c754a326250caa8dc93401b5c823f1",
"packages/shared/src/app-definitions/supermemory.json": "0120fc518187af3b8bf6d19be946486da20e71b4",
"packages/shared/src/app-definitions/tavily.json": "3cbc727ad0e051993b20e5afeb63ab4f7553c6e8",
"packages/shared/src/app-definitions/telegram.json": "60b158b98d308815daecabce26748c3cf4beb9d8",
"packages/shared/src/app-definitions/telem.json": "1ddd27834c9a928ca880835c3732a122d1ae9575",
"packages/shared/src/app-definitions/ticket-tailor.json": "6fd269f205410307c7bc00e83795023cdf3c0fd1",
"packages/shared/src/app-definitions/ticktick.json": "9366a275b002fc94b398b7becb0c9d51d12d22d1",
"packages/shared/src/app-definitions/todoist.json": "25ee5e88dcaa06339afef3becd5f9f653e296a8e",
"packages/shared/src/app-definitions/unikraft.json": "2a930d3e373d06dfa0b6af62e01ac965e950215b",
"packages/shared/src/app-definitions/vercel.json": "b4b5e95003b0afe2556a6e722f567982c99ff2af",
"packages/shared/src/app-definitions/webflow.json": "f77c24e479249765ce3d25d6bcb4e9499ace8dc6",
"packages/shared/src/app-definitions/windsor-ai.json": "89ba85b5dc11d5d9d13250e46ad65d131ed8befe",
"packages/shared/src/app-definitions/wix.json": "978c310f36686b4401521dd414cc64da678bddf6",
"packages/shared/src/app-definitions/xai.json": "6b60b0211124fc49cbba6ee44e66950fc55e1e81",
"packages/shared/src/app-definitions/xero.json": "3a9de0e9aa9223d15379eb0cdece9ac6707f73c3",
"packages/shared/src/app-definitions/youcom.json": "2993fbbda98c7afd899ca3ecd7a7a907b280a72c",
"packages/shared/src/app-definitions/zapier.json": "3e18a1fdacbd559b758b163310c85220d723f89b",
"packages/shared/src/app-definitions/zep.json": "d5e2021e9b4ed7ba9abb933242f1ccb26a67a638",
"packages/shared/src/arcade-app-catalog.json": "d0b5221ace19eabefb90cda401a8018384a62993",
"packages/shared/src/asana-connectors.ts": "6a585a7cdd6e364f2f4cd40c67775d185f4a662b",
"packages/shared/src/browser-use.ts": "9b02bc40c340eca46453ea9e92074e53a492c91f",
"packages/shared/src/case-handoffs.test.ts": "78a8115c35ab1719af288427dafcd62ca7080f25",
"packages/shared/src/case-handoffs.ts": "1db7e288c4b4a912407d9f23b6d19f5504967291",
"packages/shared/src/cliplab/LICENSE": "07a83c8665b8f5b6d80fea455bbc2622245ef5d9",
"packages/shared/src/cliplab/PROVENANCE.md": "66a7b9b123dd60482c86ca1bc1393f9e5907a7fb",
"packages/shared/src/cliplab/character.ts": "f338262f28ffc1930a1b94e973f97a9c9f32f27c",
"packages/shared/src/cliplab/definition.ts": "0c6f0789c694c3996601b70660e98c8720197eec",
"packages/shared/src/cliplab/face-morph.ts": "b46ea3c50e9221fbd5c2b92d353f92bf39ab4f4b",
"packages/shared/src/cliplab/gaze.ts": "66e58f5b47592ab4f2e9ea99176f2ced981bc203",
"packages/shared/src/cliplab/model.ts": "8dd02d7ada5be6e5da9118e1a12803356208a8bc",
"packages/shared/src/cliplab/palette-tokens.ts": "e8a547c9f4e0fe0bff548114cb5942e1a948d0bf",
"packages/shared/src/cliplab/particles.ts": "8ee5c63d1b8fc52367171b5fb6b63b3b3c554df5",
"packages/shared/src/cliplab/renderer.ts": "47a1154009e4071740204962615af43078b14862",
"packages/shared/src/cliplab/runtime.ts": "a197f0f0eb28d7b06e1b956fa2feb1e396a35a8d",
"packages/shared/src/cliplab/static.ts": "14d3fb114014ff4b30c18936133e132ea8b90c78",
"packages/shared/src/cliplab/svg-canvas.ts": "2b1bb631749524487d05a61df435da7effb5d607",
"packages/shared/src/cliplab/svg-path.ts": "69ab5fc506a0ad9965837ddb41ecc245b2df1916",
"packages/shared/src/cliplab/svg-snapshot.ts": "f1ef8e5d8e4d69db0d1cdcf80ba9b54a7ad3d0c7",
"packages/shared/src/company-import-transfer.test.ts": "213169db79a3f4b6788f4107d6a7b49cd0beb6b9",
"packages/shared/src/company-import-transfer.ts": "376482a8c28feba53b1e4973db0ba2e158b65b41",
"packages/shared/src/composio-app-setup.ts": "75f544f5647f227214b8a9d0f61d0196bcbbab55",
"packages/shared/src/composio-search-catalog.json": "483d391b1dbd68f46848dffa83f2e735375835c1",
"packages/shared/src/config-schema.test.ts": "5810f2c152e612326bc377aa47f23f623b206ec8",
"packages/shared/src/config-schema.ts": "f9c8d41cb09fbc51c3e7a038089b54376cc9150c",
"packages/shared/src/connection-instructions.test.ts": "5095a9eeaa378b8d881feafa699c1705f207717f",
"packages/shared/src/connection-instructions.ts": "61b3e102838bd8bcab20eb9d0497a182f86eaac0",
"packages/shared/src/connection-intent-guidance.test.ts": "76047c037508beb098f914d231cc239a9b1d256a",
"packages/shared/src/connection-intent-guidance.ts": "7178d4afddd09dac4eaa5b7203b44fc1c6a499cf",
"packages/shared/src/connection-routing.test.ts": "95f6d30b790a2ef69f409fb72e4ba83256d7771e",
"packages/shared/src/connection-routing.ts": "fa29ecccbed3f38497ed7f6eadcb8f7c2b193177",
"packages/shared/src/connection-search.test.ts": "fc7a3ee0df22b19afe1d58f4d9bfbf8213ed0092",
"packages/shared/src/connection-search.ts": "6aa86e3fb28a8e32da1aa99b841aacede055e0bf",
"packages/shared/src/connection-setup-state.test.ts": "252f1a1627b80f322385a415df4656f8aa9148e9",
"packages/shared/src/connection-setup-state.ts": "becd04545f04d601b4292c9b5cec4b737adc5e69",
"packages/shared/src/connector-availability.test.ts": "a1e17eb407249d53953c9b9fe94f6d97ad062883",
"packages/shared/src/connector-availability.ts": "1e1b5c0a3404220456818f27608f84b3342210d9",
"packages/shared/src/connector-identity.test.ts": "d58357707faefa7c98bb906cdaeb4db39d31081a",
"packages/shared/src/connector-identity.ts": "5ede83c9c5225dbe8af9cf3a49c74e477ef5b955",
"packages/shared/src/constants.ts": "574d033e66504ebb860685fe2ba8e914731b6a0a",
"packages/shared/src/customer-success.ts": "a2d02aaa8223164c0eae4267b2a6da82242eaa6f",
"packages/shared/src/decision-models.ts": "a07aefd87d2fae34283ce21a0334d57e2108aa7e",
"packages/shared/src/decision.test.ts": "d0783d18613c73102f662e8a5da795ff6e7faed5",
"packages/shared/src/document-anchors.test.ts": "6805d149ad66d91f2e3ba4841f82e9af05bf5d83",
"packages/shared/src/document-anchors.ts": "2efc96cdb0bb37daea7a26bf945985fbfa3b1408",
"packages/shared/src/dot-invitations.ts": "88cd4e0dc64edb571eafb252fd75a59406222d10",
"packages/shared/src/easy-mcp-connectors.test.ts": "31d14b574c491d7e1b6caf9b823836f31b01d7a8",
"packages/shared/src/env-file.test.ts": "57f2cd76173d95c037adc4caaef9e90b29eb1736",
"packages/shared/src/env-file.ts": "ee15f8dfa6610d3fd1eea3ca0846cc2c583f9070",
"packages/shared/src/environment-custom-images.test.ts": "80d00f3581f13a782480700296b1295d8159ddae",
"packages/shared/src/environment-custom-images.ts": "a81814b7f99e5c8871f50ffcf9dd93dc435dee27",
"packages/shared/src/environment-support.test.ts": "f0bc2520cd51b568cd7a2eeab94450c027e392bf",
"packages/shared/src/environment-support.ts": "3a61f86a51bf37bbfe3395fedcdd7d7423852477",
"packages/shared/src/execution-workspace-guards.ts": "5428546ab16ef5cb4d23edd914d0a15a83a94668",
"packages/shared/src/external-agent-guidance.ts": "d815fc60b170717f0fdcff61bc30c8a1289f7fcc",
"packages/shared/src/external-objects-server.ts": "f323ba3e5e0b1a2ad415bdb5e0ca99bf9d449821",
"packages/shared/src/external-objects.test.ts": "aa55bd97d918b3bfc8ea3b47526a71684ef61cbd",
"packages/shared/src/external-objects.ts": "a9953c4e3f77f2bad03cb7832914ffc0b1a1266a",
"packages/shared/src/fast-response.ts": "a81332a2b46b59c1f6953cfc4da659eae2326beb",
"packages/shared/src/feature-catalog.test.ts": "d3e000c039492daad1fe0b7302b34e7c3816a8ce",
"packages/shared/src/feature-catalog.ts": "48e282e76918245bee14e92f1492e32619b24cd5",
"packages/shared/src/frontmatter-runner-contract.test.ts": "cd121330e10aaa9da36c16293bcd9760561b6b41",
"packages/shared/src/frontmatter.test.ts": "ee9e36feda9469f6c16fb3dc4dbf7b39c7ea37cc",
"packages/shared/src/frontmatter.ts": "59cfd440d528bc0f2c21da3f7208abe8bfd9ae01",
"packages/shared/src/github-connectors.ts": "b8448b1eddac9a9d1308b94e4d136c761cbd4a00",
"packages/shared/src/github-skill-repository.ts": "b00a9963472723bcb7fcd433aae442f75d7a138b",
"packages/shared/src/gitignore-runtime.test.ts": "e334ab0eca8faeab5807171983ec9102a3f1c734",
"packages/shared/src/google-workspace-connectors.test.ts": "2a3eae7217dc2c499281791d643ea7c0f6f6d24d",
"packages/shared/src/google-workspace-connectors.ts": "f2890ae7d6c2e126379b9e165c13989367561659",
"packages/shared/src/grok-bot.ts": "9de68f00e7926c397729ff62bf8e277f5a41c525",
"packages/shared/src/heartbeat-inbox.ts": "227dc89705c20460ca02194091cb162e11503186",
"packages/shared/src/home-paths.test.ts": "81235c17c376224e1be3abd3f034a8400f079c66",
"packages/shared/src/home-paths.ts": "fa4f1c440653b76b3dbb7a51026d557287deb45d",
"packages/shared/src/humanize-connection.test.ts": "ba9e90a44c63452b0bf9f733adb90241921e8eca",
"packages/shared/src/humanize-connection.ts": "fbd414476a0ef7106a7c80d8572208781b091699",
"packages/shared/src/index.ts": "c698a2904aaed44caa1d8d7eb3296622bc4195cb",
"packages/shared/src/issue-attribution.test.ts": "84563685e7f3c48029243cf1cc24a3a09ee3cde9",
"packages/shared/src/issue-attribution.ts": "9c85194fb8739aeb5d70c8e128db6127a3057569",
"packages/shared/src/issue-creation-source.test.ts": "afd00ce1ad8357daf860bc7e8e79340c765805f7",
"packages/shared/src/issue-creation-source.ts": "48bd43f183b1a9f7bc233128ad0fe84c82e04c17",
"packages/shared/src/issue-references.test.ts": "d4af275ae8c615ccdc587dfbfd675cea8626e530",
"packages/shared/src/issue-references.ts": "bfdd2126fb12797f2c8177113285b2c3233aecef",
"packages/shared/src/issue-thread-interactions.test.ts": "6f516055744fcedd2e0c06ce82c6a1a019120762",
"packages/shared/src/issue-write-denial.test.ts": "2cf4d4977fda5f2894f7349594dd4a63abcd6962",
"packages/shared/src/issue-write-denial.ts": "737bab7bd7a29e2f686cdca7655c93eefc306e8c",
"packages/shared/src/markdown-work-products.test.ts": "7293d14ad45ec2b11b5deaab4ce070c36f7c9010",
"packages/shared/src/markdown-work-products.ts": "16a88cfb717f325c247a516e78b47d53871b52ed",
"packages/shared/src/mcp-config-help-prompt.ts": "fa58effb7cd995198909315d9ba2a93dedde0552",
"packages/shared/src/mcp-remote-headers.test.ts": "b59cd4fc197145d400d2eca7998a09b6fb834cb6",
"packages/shared/src/mcp-remote-headers.ts": "d4522440255ddbb5b117bf1575e7fb1a6744f505",
"packages/shared/src/mcp-setup.ts": "a774458f00a85aacd0ba4c7f4700eca02011ee24",
"packages/shared/src/memory-connectors.test.ts": "a0cfd02a7915d6ef2d975df2e7ba8f7d0ef159af",
"packages/shared/src/memory-connectors.ts": "c224910a12bcfda947bd560384887325a70a4720",
"packages/shared/src/money.ts": "156663aa931e5227fe22d813d1e3faa7173c94c6",
"packages/shared/src/muse-invitations.ts": "fe464a54c374d5621137225773cb2635c2e3eb01",
"packages/shared/src/muse-protocol.test.ts": "6dbb45d82a5ff73953003d7304d6a356cb72521d",
"packages/shared/src/muse-protocol.ts": "56f3121fd1972bb9c84db0f6696449557111ace9",
"packages/shared/src/network-bind.ts": "eeda9c5028670dc928115c7355ea9e2df2551cb1",
"packages/shared/src/node-version.ts": "9a7766a8324122ce41738918bb0125398b9657e1",
"packages/shared/src/oauth-endpoint-url.test.ts": "7f7ad634b3de7f25513e6c8b6426e1285d079e12",
"packages/shared/src/oauth-endpoint-url.ts": "5ea73648dc5df103881e40e8ed920e38d901760a",
"packages/shared/src/openrouter-attribution.test.ts": "ddc980f9d06fac708d02dcef8a8962e9887d6854",
"packages/shared/src/openrouter-attribution.ts": "b77e887721cbf6bd882004f07df2405b6e58e52c",
"packages/shared/src/portability-fidelity.test.ts": "3cb1d7740cd016d8f2000cf400b87971c571e5ed",
"packages/shared/src/portability-fidelity.ts": "3465dc57777f0f74bcc018ecc9fa955d3c92f4a3",
"packages/shared/src/portability-hash.ts": "4465c125a08bef6cd78ac4fdff7062989a2d8a25",
"packages/shared/src/portability-zip.test.ts": "f95df784e114bc43f83ba0a62f128dcbff82cb7a",
"packages/shared/src/portability-zip.ts": "f4b77e7e5dea009ebf31dfcf2813f91b9fd6e4ce",
"packages/shared/src/primary-agent.ts": "63d7841ed6228ff01f7639c8bb9648e10f5acae0",
"packages/shared/src/project-mentions.test.ts": "1bb3bceb1a05f702ecbf2f81e7aa715edfa547fa",
"packages/shared/src/project-mentions.ts": "5d61889358640ee2ebe168beae12e0dd5432f3a9",
"packages/shared/src/project-url-key.ts": "a367d41709f0bf80828da8dad585b2f5e2384de4",
"packages/shared/src/prompt-injection-prevention.ts": "3aa87403de724425efe04471650615cac68a4351",
"packages/shared/src/public-mcp.ts": "13bd01e5016d5a4a2948fe19d9a6796c5f483b95",
"packages/shared/src/question-set.ts": "4d46f35c480097159aa2279e11f677106bde292d",
"packages/shared/src/railway-connection.ts": "6b63076ca3e464da9179e469afe90770f162446f",
"packages/shared/src/remote-mcp-connectors.test.ts": "19a9fcef237fe6f64aa9402e1863087584e9816f",
"packages/shared/src/remote-mcp-connectors.ts": "a32e4d9ffff7f8b402d800c5a063569a1c8f0fd8",
"packages/shared/src/resource-memberships.test.ts": "8b4231689917a0f7a0e5d16f3ae72d30d8ad6c37",
"packages/shared/src/responsible-user-denial.test.ts": "bbc4faf914a0216b53625267fe5b3a3e2285b09e",
"packages/shared/src/responsible-user-denial.ts": "fb5ac5d47a49e1229f1f74d93c214561f0c9fe0f",
"packages/shared/src/retired-composio.ts": "ae048fd6b72fa4022220f1b553bfc23dd4d3b0a8",
"packages/shared/src/routine-variables.test.ts": "3cd4ea5e1a279eacdc525a3eb8e7bf317170c1ec",
"packages/shared/src/routine-variables.ts": "43d57551aaf8fcd32c10733af3198b6932b0d5d4",
"packages/shared/src/runner-configuration-warnings.ts": "81db25b1b3ac273ba9b21911672a659b10129917",
"packages/shared/src/runner-goal.ts": "c8469908d47944e1e489c4f9ce7d7c2c974afce1",
"packages/shared/src/runtime-exposure/loopback-bind.test.ts": "91e49c143f2ec84d369699fbec71d63d763a1ec8",
"packages/shared/src/runtime-exposure/loopback-bind.ts": "696dcbc1fc8f7e64f37314bc574360cf06bbae95",
"packages/shared/src/runtime-exposure/ports.test.ts": "de27b2c17db88c65f9f4e7240d9fae969dd1551e",
"packages/shared/src/runtime-exposure/ports.ts": "6566d780a36c4eb37fab5584db19714f596654b7",
"packages/shared/src/self-serve-mcp-research.json": "e9f6f6216273917205162b82a71993003225ebc2",
"packages/shared/src/self-serve-mcp-research.ts": "312abfecb0dbac8a0e5bf80fb8e3467ca94f500f",
"packages/shared/src/setting-defaults.test.ts": "f5554686f7bf9cc34f14cdd4505257dcc562d410",
"packages/shared/src/setting-defaults.ts": "672726c2ea0c9608829a1514d7ae3567b53d88aa",
"packages/shared/src/settings-visibility-catalog-growth.test.ts": "3606f66ed62f437c0762609d85c577e742710221",
"packages/shared/src/settings-visibility.test.ts": "709ca672c36b8c769e58f4fb46c8fb53c9cef5c7",
"packages/shared/src/settings-visibility.ts": "4651724fe80c66b49c26101e65c13b96b062c727",
"packages/shared/src/slack-app-manifest.test.ts": "780a0306deb2a67d7ed89060dac9449934453478",
"packages/shared/src/slack-app-manifest.ts": "e6d3f2514242efe539eee70303d00e63308ddebb",
"packages/shared/src/slack-tools.ts": "c53a75cd3099c6d8dcd1475e755aae589d4c88f6",
"packages/shared/src/subscriptions.ts": "ef70e63395ca967bbc30776b71560e96eb354a75",
"packages/shared/src/summary-slot.test.ts": "5f8e222dea425f0d5bff0adb5c683087bd7cf7fb",
"packages/shared/src/telemetry/README.md": "da0403c260b2a17b8be947cd5ba957b70f569289",
"packages/shared/src/telemetry/client-types.test.ts": "3b630edda4972b929ba4decd7457f226eb9cfc5b",
"packages/shared/src/telemetry/client.test.ts": "3ba900cdab5a1ac1dcc1dcea382bb790af32d895",
"packages/shared/src/telemetry/client.ts": "03a10a5550b542487543fbfd4a417e6eb316e11e",
"packages/shared/src/telemetry/config.ts": "dfcd4d692cc06e71cb77f70b44fc05721eb330d1",
"packages/shared/src/telemetry/events.ts": "71f78bb2f3f9f2fd9d0485c9c2f57bb72af4d302",
"packages/shared/src/telemetry/generated/paperclip-telemetry.ts": "0502b6e2a1cdb35a225c4050768126a89f14ca25",
"packages/shared/src/telemetry/index.ts": "61f945ad834841fa9a4d221680af1f005382223c",
"packages/shared/src/telemetry/proposed-connector-events.test.ts": "a8a6826e9decc86f1ccb47baae8699b913a87f33",
"packages/shared/src/telemetry/readme-contract.test.ts": "eec9d4511ddc3085a9af1a2dab6bf8487e49dfd7",
"packages/shared/src/telemetry/retention.test.ts": "9d54db1371bf20f22b8d2f8373f84433107a7408",
"packages/shared/src/telemetry/retention.ts": "543e1085298e2dfa8d9ee4dd268c76e6d529b440",
"packages/shared/src/telemetry/state.ts": "a060af26a8e9a82a8afb0e2fccbfe6a59e9d0a8a",
"packages/shared/src/telemetry/types.ts": "083d4341a72ac47bc1a3bbff163965de8b06f90a",
"packages/shared/src/trust-policy.ts": "88001a8d629a2af66dcd8d75b7b4d36298455ef6",
"packages/shared/src/types/access.ts": "015d51792da07acaebe59e0e395698d207811d0d",
"packages/shared/src/types/activity.ts": "9d383f85775bf3657a7f9bdfb32a6e58e58c12a3",
"packages/shared/src/types/adapter-registry.ts": "bd9b0a1f7980bf444f7f3d75a4687874014c4bca",
"packages/shared/src/types/adapter-skills.ts": "ac77f7e013e8fb8de6fcbe916b7f7923554de815",
"packages/shared/src/types/agent-lifecycle.ts": "13a7a829c8059f9e50a23849540338621137e7a2",
"packages/shared/src/types/agent.adapter-auth-session.test.ts": "2b6af7a2cf8e7898077224ad819c05b19fdfdfe6",
"packages/shared/src/types/agent.ts": "ba4b594389b4f546a76c002153e9ac69883b2418",
"packages/shared/src/types/app-definition.ts": "f55fad80e9c765e16a3a59f694bdb0764940d968",
"packages/shared/src/types/approval.ts": "4554d36f804cb74b825b29711ed4c622df07ff6d",
"packages/shared/src/types/artifact.ts": "6d022d34a718196df5c109a6164e7a70ede287e8",
"packages/shared/src/types/asset.ts": "d148d6cba0e9a8d2e3aa6cb9182df6f59f909fba",
"packages/shared/src/types/attention.ts": "8c9318962af93c3b41202abb8f62b169cdeafc1b",
"packages/shared/src/types/budget.ts": "7b3fe924d571008d67093c051a13d2c7e67f3ebe",
"packages/shared/src/types/chat-channels.ts": "103eb76c5c517102e999b49b4bb726e0ea228902",
"packages/shared/src/types/chat-github.ts": "be6e7c951620ee6d107af651fba440938f973ddc",
"packages/shared/src/types/company-portability.ts": "76ce2cd19c05d788f57cde743c83b663a584648b",
"packages/shared/src/types/company-skill.ts": "9ab84634f21756922414b746d5a7d3dbd8b204a1",
"packages/shared/src/types/company.ts": "40b5858ebabe3df694af955bd68d2222069a3993",
"packages/shared/src/types/connection-intent.ts": "fe314b98f4ca9ab095c42fb7df49faafb3a0124d",
"packages/shared/src/types/cost.ts": "098ff480e4491ebcc95e771dfac310c7a17cb3cc",
"packages/shared/src/types/dashboard.ts": "cdeaebdb7774b3b039dc5199a5ed8bc6cf32ce40",
"packages/shared/src/types/decision-queue.ts": "029b8c252913700287892b277d5353f00d2ef484",
"packages/shared/src/types/decision-training.ts": "23a6485b82e1cd1125ec2002285e0a29cd73d485",
"packages/shared/src/types/decision.ts": "b0d40077d481ce6180de6b02bad6df5bb0fbf977",
"packages/shared/src/types/document-annotation.ts": "7769a42f1ccd707a2259774e7fe6aa536b5d183f",
"packages/shared/src/types/email.ts": "556ff6a0dcefa01eb704d6d00d3b191de81ff4dc",
"packages/shared/src/types/environment.ts": "9f929fcb71be9dbfb66baa3186e62b1d84f34f60",
"packages/shared/src/types/execution-continuation.ts": "0546b3b4707cabcd69e055f12ca792095ba8feaf",
"packages/shared/src/types/execution-projection.ts": "ff28fd01dc115b171aba15dd3830f0d3cc2a2308",
"packages/shared/src/types/external-object.ts": "7a225e0f5a6777d449019a30530c858b753eb4c1",
"packages/shared/src/types/feedback.ts": "c5eb31545c7f67c04ad9afd9db45a6b4774bcb35",
"packages/shared/src/types/finance.ts": "5c2208c7950020f1005aa0dd86a1680c54dd4354",
"packages/shared/src/types/folder.ts": "451856a6720f91525b846d91ec25f7eff110dba5",
"packages/shared/src/types/github-app-setup.ts": "d36e0b87233d662998c49d2a2d0709bb59bd97e8",
"packages/shared/src/types/goal.ts": "29100192bcc3e0c4c4904e59b5d84874952e67a2",
"packages/shared/src/types/heartbeat.ts": "a14cefbbdd17a473f73c091ae73f1f71f2e85565",
"packages/shared/src/types/inbox-agent-policy.ts": "cb9d41d9681e80b834889268ad96015a39df92f6",
"packages/shared/src/types/inbox-dismissal.ts": "5d2c71ce44ad77266d286ec1b1119ab1f0390a9d",
"packages/shared/src/types/index.ts": "fe0ed6ec356c8956ff2c166ebf490bf7a374f48c",
"packages/shared/src/types/instance.ts": "ae4187c6f0e89ae1d448892205cfb2af0ac782ee",
"packages/shared/src/types/issue-tree-control.ts": "75489fffd491b8350b05caf57d7796419797d9df",
"packages/shared/src/types/issue.ts": "00bfb4fab21691e1219479d663749f50e135c2d6",
"packages/shared/src/types/live.ts": "b4da9a874b9d5a49f4768ac847875cc150973705",
"packages/shared/src/types/native-finalization.ts": "c026a73003ae232fea4f7624d561493b383df358",
"packages/shared/src/types/plugin.ts": "9b8e5f24844ff9444683d13f66e3cc3bea3d568c",
"packages/shared/src/types/project.ts": "346aede02c410608eb3c6df6870c8972f54f6e34",
"packages/shared/src/types/quota.ts": "a0686a9afb3111a4de093f66200dc0c9152e883e",
"packages/shared/src/types/resource-memberships.ts": "9eab6fad50facf3e489f8c40e5bedd92e1d1faf8",
"packages/shared/src/types/routine.ts": "f87b1dda2a097dcaa0b2975fe3b10c2b291df55b",
"packages/shared/src/types/runtime-exposure.ts": "785f053a7ad5448cde94f36738cdb90585bbc92e",
"packages/shared/src/types/search.ts": "825d4d4d1c4d7a19a09bad23854f7b18050e4f0c",
"packages/shared/src/types/secrets.ts": "05b07bce0550c1b3a160615bed661bc7f9c26156",
"packages/shared/src/types/server-info.ts": "5c9ae793bb912f68f5dc22a43735605a142b85dd",
"packages/shared/src/types/sidebar-badges.ts": "d1ba9a64271eaee1c66ec9cc43c44f26ed769d8b",
"packages/shared/src/types/sidebar-preferences.ts": "457b5f0bee99a0a2d42f671fd827930693de1c00",
"packages/shared/src/types/skill-source.ts": "f92df553fb8b3b967d62dce4d93fbe5a3aef1b30",
"packages/shared/src/types/smoke-lab.ts": "534b1a1ef3f8fda6ae9a15e009c4f247a6f0b9e3",
"packages/shared/src/types/summary-slot.ts": "9698bb7d628be3f2310d050037e30157d474ed48",
"packages/shared/src/types/task-workspace.ts": "a23afa13d2cb58dc2eb72692ed28c16ed6fb84fb",
"packages/shared/src/types/teams-catalog.ts": "a088b07eb0b54344f57b074a2bb4cc16659c8f46",
"packages/shared/src/types/tool-access.ts": "0fbd422fdadfa53000a8538f7b1df4d1564a6604",
"packages/shared/src/types/user-profile.ts": "d961a4074852e170c4e61599bb4e293d5708ec16",
"packages/shared/src/types/voice-sessions.ts": "95e3b62cdd51f62290e68536728ddd7297bc0459",
"packages/shared/src/types/work-product.ts": "26f9129f606a207ce7a218ccef40f96747283086",
"packages/shared/src/types/work-timeline.ts": "8ebfc5cd6cb664b2944c20b7b1b947d71e4f3c46",
"packages/shared/src/types/workspace-file-resource.ts": "02f1bd181300b21330933450e72092023c5fae88",
"packages/shared/src/types/workspace-operation.ts": "8fcb52c7088ae9da4417cf01e55632897f17645e",
"packages/shared/src/types/workspace-readiness.ts": "c979f99c3dccb684028f8a6ab07660a018a4a3ea",
"packages/shared/src/types/workspace-runtime.ts": "00ab237577d3ea3a1fa946ce6ab6ef8b389b1739",
"packages/shared/src/types/workspace-source-verification.ts": "647f5c8817590629439813df5dc63750b316206d",
"packages/shared/src/validators/access.test.ts": "0df0e096658a31c4f21462a992e38cf8bd7082b1",
"packages/shared/src/validators/access.ts": "c046b84d56ddda6df3594a39c73bfecef26d38f6",
"packages/shared/src/validators/adapter-auth-session.ts": "b98b6c4c70d8284e94cab5cf9e8bbaf07197ca9b",
"packages/shared/src/validators/adapter-registry.test.ts": "567c113e50dd0c0d259b095375f8f8f403ffa893",
"packages/shared/src/validators/adapter-registry.ts": "6fb9f655d49f774ef7f082bf842f388f1187e545",
"packages/shared/src/validators/adapter-skills.ts": "ae43f7b180f632d5f54dfa821b8dc5ef6c6aa53d",
"packages/shared/src/validators/agent-commentary.ts": "8330e2c2f99db6960a50ae1fe7aefc1403425ccd",
"packages/shared/src/validators/agent.test.ts": "d38767b0c1fc1439871855ed55dca3c8c7b645c6",
"packages/shared/src/validators/agent.ts": "1e9e42e28761d1ac4fd84cafd84f409d4ade89cf",
"packages/shared/src/validators/app-definition.ts": "9fb660249eaf49a71adeab222cdb29a6db41ce28",
"packages/shared/src/validators/approval.test.ts": "051d29b27400fdd1e311466587b276264cf072e4",
"packages/shared/src/validators/approval.ts": "c1c074dc2f06d46adae0df8fa236d43c7aa01a25",
"packages/shared/src/validators/artifact.test.ts": "2f0c138a6fa46d68507515d3932e9bc4cef44374",
"packages/shared/src/validators/artifact.ts": "64871e29aeb6d3c6340b720a6861809b0eb77407",
"packages/shared/src/validators/asset.test.ts": "cc6b4a18b1b05c6cb130594b2c197d0e738ce0bd",
"packages/shared/src/validators/asset.ts": "93b9a747b0e7a2bce216db4ac3ab82ee42eb5cc3",
"packages/shared/src/validators/budget.ts": "76ad91791c89d5be0f64cb23dd5587c493a7ead5",
"packages/shared/src/validators/chat-channels.test.ts": "eb157fcb00128a62d381e3e06630ed38c9a80314",
"packages/shared/src/validators/chat-channels.ts": "9d0adf5ce89655d22071061d5e085d2ed84625d6",
"packages/shared/src/validators/chat-github.ts": "56763088c66e6fecad99a44e6adabb2859b802b6",
"packages/shared/src/validators/claude-setup-token-session.test.ts": "2e672819bfbbe752c888e5fcff319971c68cf7d8",
"packages/shared/src/validators/claude-setup-token-session.ts": "581303dd7af607c07f3242b573b9271eb0387e71",
"packages/shared/src/validators/company-portability.ts": "6045d608a05e83ccf632b1225f01d352984340eb",
"packages/shared/src/validators/company-skill.test.ts": "ec28e5f9331f98bfc45234b7b22ef61b25963211",
"packages/shared/src/validators/company-skill.ts": "7e64c15846481edec316d2bae71b6a2cb313e251",
"packages/shared/src/validators/company.test.ts": "038157bb5dded4081959754d12b1edee6f66fc41",
"packages/shared/src/validators/company.ts": "35e2e2feb6d5ab980409defde051f311de1e67fe",
"packages/shared/src/validators/connection-intent.test.ts": "fdc314617d5a458a64271647bd0cb9d00696001c",
"packages/shared/src/validators/connection-intent.ts": "33b4ceb713a0a131ebda6f9804571d42a98037a2",
"packages/shared/src/validators/cost.ts": "d86b9c83943825118d07cbcf6b51ec4266be3157",
"packages/shared/src/validators/decision-queue.ts": "0d686360f577c7a7738ff85e29f23ab423be4d09",
"packages/shared/src/validators/decision.ts": "02af23ad448183bc7fc023d80885067cfb4756f6",
"packages/shared/src/validators/document-annotation.ts": "d60dca3192d61a42aff5503915fed954b7aeac72",
"packages/shared/src/validators/email.ts": "661624d121772acfc7376ad7d933d6d54c56f4eb",
"packages/shared/src/validators/environment-custom-images.ts": "f79a98e19c5398a29c9932c89898379a5fd95c78",
"packages/shared/src/validators/environment.ts": "2639bb5c43a8017ad147b35a6f99f1f031ebbf62",
"packages/shared/src/validators/execution-workspace.ts": "debae3727ae481c7772a30b8a1057cb887d34bd4",
"packages/shared/src/validators/external-object.ts": "80a58f89a7e326007b730497feae848fc79d82d0",
"packages/shared/src/validators/feedback.ts": "25d83bcc09292e0770c01de223770819df2b8eb1",
"packages/shared/src/validators/finance.ts": "ded2b0bdf24b9ab6de9b52548de0b7219b5dd1b8",
"packages/shared/src/validators/folder.ts": "b486444ab09ec3d5a9afec7dbd2e71c1bc2beb67",
"packages/shared/src/validators/goal.ts": "602724134915e1e83c50115abba36b288c0abd3d",
"packages/shared/src/validators/inbox-agent-policy.ts": "29cef157f4d9b564fd813868038aa8c813df4a7b",
"packages/shared/src/validators/index.ts": "8f1ba1bf907113fc934dafc20d4d8f6c8f6141be",
"packages/shared/src/validators/instance.test.ts": "f1db216ab7b22aefc329b8f8494716a0e3bc64ad",
"packages/shared/src/validators/instance.ts": "7dbaf078db6dbc63e136f1c8339c200dd5431a0c",
"packages/shared/src/validators/issue-tree-control.ts": "7144ef86a83e008ac0626e1d62a1d6164c46f438",
"packages/shared/src/validators/issue.test.ts": "669a1f7df9aedbfc384920800981375c6ee64efa",
"packages/shared/src/validators/issue.ts": "98f88a5af22bc9155bd9a1f84ab7f927a5f2ff0d",
"packages/shared/src/validators/native-finalization.test.ts": "40c68bdd1d0e8a77cd70c4576fae53334fe5be0d",
"packages/shared/src/validators/native-finalization.ts": "d4e674de6ac7d36385e6f6c2172fa16ffc9b1830",
"packages/shared/src/validators/onboarding-seed.ts": "de302a8400d4199053f379d9ed853522a3ac6d32",
"packages/shared/src/validators/partial.ts": "fa60ec7481fc14175365e9e500c814b159f1ab53",
"packages/shared/src/validators/plugin.test.ts": "f5ce7de33157ab882ce6c5e6124917584373a4ab",
"packages/shared/src/validators/plugin.ts": "258991ee76485d21503188afa5a35734bbaf1c19",
"packages/shared/src/validators/project.ts": "61839f7a1b530f4d4e15df7546acf8b5f4a9843f",
"packages/shared/src/validators/provider-trace.ts": "b38b6200a2e51513f5dca0f8b008f2fe68175e82",
"packages/shared/src/validators/resource-memberships.ts": "b124ac8a8f24b4dbb03ae82bcc87541f6b4ecdfd",
"packages/shared/src/validators/routine.test.ts": "64ea2f49f369d38a4bbd2f71968e49620bff93bc",
"packages/shared/src/validators/routine.ts": "4e4e261f2d2ccf38671385c49f31559c898b20e1",
"packages/shared/src/validators/runtime-exposure.test.ts": "31175388dbab9f3d5dba920cd711ea6e0e6c6ece",
"packages/shared/src/validators/runtime-exposure.ts": "362cfc4721581beb4f495987bde60028e6f390f0",
"packages/shared/src/validators/search.ts": "839f4cb5296bbc534b0456ec216213fa261010e5",
"packages/shared/src/validators/secret.test.ts": "f25c85d91374c77885a775c8100555f2ebc92d36",
"packages/shared/src/validators/secret.ts": "746be54ac08ea676e608308d49dd75bcf2b83806",
"packages/shared/src/validators/sidebar-preferences.ts": "7d950abbd95fcdbf9f7e25191a349e29000a7a2d",
"packages/shared/src/validators/skill-policy.test.ts": "b9d74af4d8aecfc08db9be8993231e0e10a45179",
"packages/shared/src/validators/skill-policy.ts": "7a6233776c17c4e83cbe7d6a9552638428c1c53c",
"packages/shared/src/validators/skill-source.ts": "94e418769a493e75b58fba2e17c4e11826b114c5",
"packages/shared/src/validators/smoke-lab.ts": "9672d163b711124cdfd0d32d919ef0122301d557",
"packages/shared/src/validators/status-card.test.ts": "26194a5d39ab8924a8676af85cc93fa8bf159245",
"packages/shared/src/validators/status-card.ts": "0dccd38bed2678f9309f1cc9ab28ac4e928af127",
"packages/shared/src/validators/summary-slot.ts": "44005a160d02756fb34025d57f9d6ac85d4bb5fb",
"packages/shared/src/validators/task-workspace.ts": "28774b510b3f4b23a2be98fdae964612693ea633",
"packages/shared/src/validators/teams-catalog.ts": "7bddf40c091a52aaa54e79de784bea5ae496f9aa",
"packages/shared/src/validators/text.ts": "99c753fd63d0774d3b6defd096dceaba54d6f835",
"packages/shared/src/validators/tool-access.test.ts": "b5838320115946a936b256919a5bd3a2294c6070",
"packages/shared/src/validators/tool-access.ts": "bafe22e80e13232b0b90f78ee1370b91f580c294",
"packages/shared/src/validators/trust-policy.test.ts": "1fc4818984eb0e14f034d56ad147600224496d4e",
"packages/shared/src/validators/trust-policy.ts": "f16b002db940b06bd882d51df8b5712c576e1d7b",
"packages/shared/src/validators/voice-sessions.ts": "018a71f9ac544645041ac3cc9919e59c95fbe998",
"packages/shared/src/validators/work-product.test.ts": "82ccb7118296f5340805343103438d4b88d9eb12",
"packages/shared/src/validators/work-product.ts": "0934852449e44d779662def3c37be6cd0d0878ae",
"packages/shared/src/validators/workspace-file-resource.ts": "8ee89c63567df29bfc1175628acfefa2aeed079a",
"packages/shared/src/validators/workspace-repository.ts": "e753942ce573ad0389a61b27a40a976a31949684",
"packages/shared/src/work-product.test.ts": "0fdfb875549416b2fe9347e3ba625bdc44bd6034",
"packages/shared/src/workflows.ts": "5e07086a4c22e224ee152a62505d3aecabc81923",
"packages/shared/src/workspace-commands.test.ts": "ee609120591e981c679cf50f84ff020ab8c9fde1",
"packages/shared/src/workspace-commands.ts": "c567b4f8d934dc13f9081cd8ce03656a08471e20",
"packages/shared/src/workspace-file-resource.test.ts": "1a64f06528356e9695eab61faeeb4bfaafd9989d",
"packages/shared/src/workspace-provider.ts": "d1451e2b5caf2bd41baf3464bc2ed0515231d7a2",
"packages/shared/src/workspace-restore.ts": "65da97089693c54ddfff21dbc8d0ebac800aa2c2",
"packages/shared/src/worktree-port-registry.test.ts": "ad52c4b3e552378929b56448b216144b00c44c9f",
"packages/shared/src/worktree-port-registry.ts": "4c3a0a2b40d0598fe057ab9f9f0cc9ce2974a5b0",
"packages/shared/src/worktree-seed-source.test.ts": "852856b1b5b8f4d0d3dada6501886fe50a2c2dde",
"packages/shared/src/worktree-seed-source.ts": "3d1a09bed2d969feabef3400ff84fd499c1408f4",
"packages/shared/tsconfig.json": "5a24989cd30dcf3749ca7e60d4e83d6ddbb14714",
"pnpm-lock.yaml": "cc58386e5c2b63e332ffdb5f818e6226539c8ff8",
"scripts/ensure-plugin-build-deps.mjs": "356e0e0f80e50890067027da3003485cf5d2ccbe",
"tsconfig.base.json": "2d0e41f723add0c678936eb9ae1c60eab710f8e0",
"tsconfig.json": "994ee490fb97be7cc0954f4dfd88cf0723290e6c"
},
"lockSha256": "c7f23021335b60da0cff1dd4c75cf8b948333ef3165d7d1703eb79a589edb798",
"compilerPackageSha256": "3722b30210616a13a3213ded11575ba6b2dbab10c32a5ef67afca8513e27017e",
"parserPackageSha256": "6d6a65b879cd7cd249a4367c0b57cce00e78c728de45056ac2a5ca1d1f528d1a",
"parserEntrySha256": "6969920ae0610df927b6b3e675d1309372c268e36d391652af8e3e0183cbe8f8",
"zodPackageSha256": "c630bd10b52dcf71c112a2bf78dbf2734b9db58d62de663b8d86c2ec2c8cda2e",
"zodEntrySha256": "2a3455cecff4f7021c92d0a3e2e5fc170c0448dc5cc261160127e08f4888a9c2",
"zodTypeFiles": {
"index.d.cts": "29f823cbe0166e10e7176a94afe609a24b9e5af3858628c541ff8ce1727023cd",
"v4/classic/external.d.cts": "268fd6d9f2e807a39a6c5aa654b00f949feb63d3faa7dd0f9bba7dde9172159c",
"v4/classic/schemas.d.cts": "aef26cf95593c8ace1c62c4724f9afac77bdfa756fb8a00613cd152117cb2f43"
}
}
tests/runner-e2e/source-sdk-public-type.test.ts
+106
View File
@@ -0,0 +1,106 @@
import { createHash } from "node:crypto";
import { copyFile, mkdir, mkdtemp, readFile, realpath, rm, symlink, writeFile } from "node:fs/promises";
import path from "node:path";
import os from "node:os";
import { beforeAll, afterAll, expect, it } from "vitest";
import pins from "./source-sdk-public-inputs.json" with { type: "json" };
import { createSdkArtifactScanner, sdkPublicTypeSyntax } from "./source-sdk-public-type.js";
import { findSecretLeakMatch, normalizedSecrets } from "./redaction.js";
import { diagnosticSecrets } from "./private-diagnostics.js";
const hash = (v: string | Buffer) => createHash("sha256").update(v).digest("hex");
const blob = (v: string | Buffer) => createHash("sha1").update(`blob ${Buffer.byteLength(v)}\0`).update(v).digest("hex");
const originalRoot = path.resolve(import.meta.dirname, "../.."), file = "packages/shared/dist/validators/synthetic-unit.d.ts", inputFile = file.replace("/dist/", "/src/").replace(/\.d\.ts$/, ".ts");
const source = 'import {z} from "zod";export const schema=z.object({daytona_api_key:z.string().min(1)});';
const declaration = 'import {z} from "zod";export declare const schema:{daytona_api_key:z.ZodOptional<z.ZodString>};';
let root: string; let fixtureLockSha256: string;
beforeAll(async () => {
root = await realpath(await mkdtemp(path.join(os.tmpdir(), "sdk-public-type-synthetic-")));
for (const [relative] of Object.entries(pins.inputs)) { const dest = path.join(root, relative); await mkdir(path.dirname(dest), { recursive: true }); await copyFile(path.join(originalRoot, relative), dest); }
await copyFile(path.join(originalRoot, "pnpm-lock.yaml"), path.join(root, "pnpm-lock.yaml")); fixtureLockSha256 = hash(await readFile(path.join(root, "pnpm-lock.yaml")));
await symlink(await realpath(path.join(originalRoot, "node_modules")), path.join(root, "node_modules"));
await symlink(await realpath(path.join(originalRoot, "packages/shared/node_modules")), path.join(root, "packages/shared/node_modules"));
});
afterAll(async () => { if (root) await rm(root, { recursive: true, force: true }); });
async function fixture(sourceText = source) {
await mkdir(path.dirname(path.join(root, inputFile)), { recursive: true }); await writeFile(path.join(root, inputFile), sourceText);
return { ...pins, lockSha256: fixtureLockSha256, inputs: { ...pins.inputs, [inputFile]: blob(sourceText) } };
}
it("classifies varied genuinely public schema types while retaining the raw observation", async () => {
for (const text of [declaration, declaration.replaceAll("z.", "schemaLibrary.").replace("{z}", "{z as schemaLibrary}"), declaration.replace("ZodString", "ZodNumber")]) {
const scanner = createSdkArtifactScanner(root, [], await fixture()); expect(await scanner.scan(file, Buffer.from(text))).toBe(true);
const audit = await scanner.audit(); expect(audit.suppressedFindings).toBe(0); expect(audit.observations).toHaveLength(1); expect(audit.observations[0]!.classification).toBe("public_schema_type");
}
});
it("protects an exact private value even when equal to a public schema symbol", async () => {
const scanner = createSdkArtifactScanner(root, ["ZodString"], await fixture()); expect(await scanner.scan(file, Buffer.from(declaration))).toBe(false);
expect((await scanner.audit()).observations[0]!.classification).toBe("private_exact");
});
it("keeps direct token shapes unconditional inside declarations", async () => {
const scanner = createSdkArtifactScanner(root, [], await fixture()); expect(await scanner.scan(file, Buffer.from(declaration+'declare const privateValue:"sk-ABCDEFGHIJKLMNOPQRSTUVWX123456789";'))).toBe(false);
expect((await scanner.audit()).observations.some(row => row.classification === "direct_token")).toBe(true);
});
it.each([
declaration.replace("z.ZodOptional<z.ZodString>", '"syntheticCredentialValue123456789"'),
declaration.replace("z.ZodString", '"syntheticCredentialValue123456789"'),
declaration.replace('from "zod"', 'from "unverified-library"'),
declaration.replace("z.ZodString", "UnboundPrivateType"),
declaration.replace("z.ZodString", "typeof privateValue"),
declaration+'declare namespace z {}',
declaration.replace("<z.ZodString>", "</* synthetic context */z.ZodString>"),
'const daytona_api_key="syntheticCredentialValue123456789";',
'// daytona_api_key: syntheticCredentialValue123456789',
declaration.slice(0, -2),
])("keeps unknown/literal/initializer/comment/shadow/parse evidence strict", async text => {
const scanner = createSdkArtifactScanner(root, [], await fixture()); expect(await scanner.scan(file, Buffer.from(text))).toBe(false);
});
it("does not classify unproven files or a forged proof object", async () => {
const expectations = await fixture(), scanner = createSdkArtifactScanner(root, [], expectations);
expect(await scanner.scan("arbitrary/credential.d.ts", Buffer.from(declaration))).toBe(false);
expect(await sdkPublicTypeSyntax({} as never, file, declaration, { start: 0, end: 1 })).toBe(false);
});
it("preserves unknown credential roots/private enum-equal leaves and their encoded values", async () => {
const raw = JSON.stringify({ unknown: { kind: "identity" }, token: "synthetic-private-value-71983521" });
const scanner = createSdkArtifactScanner(root, normalizedSecrets([raw]), await fixture());
for (const text of [declaration+'type PublicKind="identity";', declaration+'type Value="synthetic-private-value-71983521";']) expect(await scanner.scan(file, Buffer.from(text))).toBe(false);
});
it.each(["lockSha256", "compilerPackageSha256", "parserPackageSha256", "parserEntrySha256", "zodPackageSha256", "zodEntrySha256"] as const)("rejects mismatched %s without classifying syntax", async field => {
const expectations = await fixture(), scanner = createSdkArtifactScanner(root, [], { ...expectations, [field]: "0".repeat(64) });
expect(await scanner.scan(file, Buffer.from(declaration))).toBe(false); expect((await scanner.audit()).publicSource).toBeNull();
});
it("rejects changed source and unknown source initializers", async () => {
const expectations = await fixture(); await writeFile(path.join(root, inputFile), source+'\n');
const scanner = createSdkArtifactScanner(root, [], expectations); expect(await scanner.scan(file, Buffer.from(declaration))).toBe(false);
const other = createSdkArtifactScanner(root, [], await fixture(source.replace("z.string().min(1)", "privateFactory()"))); expect(await other.scan(file, Buffer.from(declaration))).toBe(false);
});
it("keeps shape overflow blocked with bounded retained observations", async () => {
const scanner = createSdkArtifactScanner(root, [], await fixture()); expect(await scanner.scan(file, Buffer.from((declaration+'\n').repeat(40)))).toBe(false);
const audit = await scanner.audit(); expect(audit.bounded).toBe(false); expect(audit.observations).toHaveLength(32);
});
it("requires the authenticated public type export graph", async () => {
const expected = await fixture();
const scanner = createSdkArtifactScanner(root, [], { ...expected, zodTypeFiles: { ...expected.zodTypeFiles, "v4/classic/schemas.d.cts": "0".repeat(64) } });
expect(await scanner.scan(file, Buffer.from(declaration))).toBe(false);
expect((await scanner.audit()).publicSource).toBeNull();
});
it("rejects non-exported source factories and declaration constructors independently", async () => {
const unknownFactory = createSdkArtifactScanner(root, [], await fixture(source.replace("z.string()", "z.syntheticUnknownFactory()")));
expect(await unknownFactory.scan(file, Buffer.from(declaration))).toBe(false);
const unknownType = createSdkArtifactScanner(root, [], await fixture());
expect(await unknownType.scan(file, Buffer.from(declaration.replace("ZodString", "syntheticUnknownType")))).toBe(false);
});
it("does not apply typed syntax classification to API, stream, JavaScript or ZIP-shaped content", async () => {
for (const untyped of ["api-state.json", "persisted-run.ndjson", "entry.js", "blob.zip"]) {
const scanner = createSdkArtifactScanner(root, [], await fixture());
expect(await scanner.scan(untyped, Buffer.from(declaration))).toBe(false);
}
expect(findSecretLeakMatch(declaration, [])).toBeTruthy();
});
it("keeps raw and encoded credential leaves protected during a synthetic scan roundtrip", async () => {
const roots = diagnosticSecrets([JSON.stringify({ private: { kind: "identity", value: "synthetic-private-value-71983521" } })]).values;
for (const value of ["identity", "synthetic-private-value-71983521", Buffer.from("synthetic-private-value-71983521").toString("base64"), encodeURIComponent("synthetic-private-value-71983521")]) {
const scanner = createSdkArtifactScanner(root, roots, await fixture());
const bytes = Buffer.from(declaration+'type SyntheticPrivateValue="'+value+'";');
expect(await scanner.scan(file, bytes)).toBe(false);
}
});
tests/runner-e2e/source-sdk-public-type.ts
+161
View File
@@ -0,0 +1,161 @@
import defaultInputs from "./source-sdk-public-inputs.json" with { type: "json" };
import { createHash } from "node:crypto";
import { createRequire } from "node:module";
import { lstat, readFile, readdir, realpath } from "node:fs/promises";
import path from "node:path";
import { findSecretLeakMatch, secretShapeObservations } from "./redaction.js";
const hash = (v: Buffer | string) => createHash("sha256").update(v).digest("hex");
export interface SdkPublicInputs {
inputs: Record<string, string>; lockSha256: string; compilerPackageSha256: string;
parserPackageSha256: string; parserEntrySha256: string; zodPackageSha256: string; zodEntrySha256: string; zodTypeFiles: Record<string, string>;
}
interface PublicSource {
root: string; parser: { parse(text: string, options: unknown): any }; constructors: Record<string, unknown>; factories: Record<string, unknown>;
inputs: SdkPublicInputs; identity: { inputs: number; manifestSha256: string; lockSha256: string; compilerPackageSha256: string; parserPackageSha256: string; parserEntrySha256: string; zodPackageSha256: string; zodEntrySha256: string; zodTypeGraphSha256: string };
}
const verified = new WeakSet<PublicSource>();
export async function verifySdkPublicSource(root: string, expectations?: SdkPublicInputs): Promise<PublicSource> {
const supplied = expectations ?? defaultInputs;
const inputs: SdkPublicInputs = Object.freeze({ ...supplied, inputs: Object.freeze({ ...supplied.inputs }), zodTypeFiles: Object.freeze({ ...supplied.zodTypeFiles }) });
if (await realpath(root) !== root) throw new Error("SDK public source proof unavailable");
for (const [file, expected] of Object.entries(inputs.inputs)) {
if (!/^[A-Za-z0-9._/-]+$/.test(file) || file.split("/").includes("..")) throw new Error("SDK public source proof unavailable");
const filename = path.join(root, file), stat = await lstat(filename);
const bytes = await readFile(filename);
// The authenticated restored lock is the sole overlay on frozen Git inputs.
if (!stat.isFile() || stat.isSymbolicLink() || await realpath(filename) !== filename
|| (file === "pnpm-lock.yaml" ? hash(bytes) !== inputs.lockSha256 : createHash("sha1").update(`blob ${bytes.length}\0`).update(bytes).digest("hex") !== expected)) throw new Error("SDK public source proof unavailable");
}
const actualSources: string[] = [];
const collect = async (directory: string) => {
for (const entry of await readdir(path.join(root, directory), { withFileTypes: true })) {
const file = `${directory}/${entry.name}`;
if (entry.isDirectory()) await collect(file);
else if (entry.isFile()) actualSources.push(file);
else throw new Error("SDK public source proof unavailable");
}
};
for (const directory of ["packages/shared/src", "packages/plugins/sdk/src"]) await collect(directory);
const expectedSources = Object.keys(inputs.inputs).filter(file => file.startsWith("packages/shared/src/") || file.startsWith("packages/plugins/sdk/src/"));
if (JSON.stringify(actualSources.sort()) !== JSON.stringify(expectedSources.sort())) throw new Error("SDK public source proof unavailable");
const matches = async (file: string, expected: string) => { if (hash(await readFile(file)) !== expected) throw new Error("SDK public source proof unavailable"); };
await matches(path.join(root, "pnpm-lock.yaml"), inputs.lockSha256);
await matches(path.join(root, "node_modules/typescript/package.json"), inputs.compilerPackageSha256);
const parserRoot = path.join(root, "node_modules/.pnpm/@babel+parser@7.29.8/node_modules/@babel/parser");
await matches(path.join(parserRoot, "package.json"), inputs.parserPackageSha256); await matches(path.join(parserRoot, "lib/index.js"), inputs.parserEntrySha256);
const require = createRequire(path.join(root, "packages/shared/package.json")), zodPackage = require.resolve("zod/package.json"), zodEntry = require.resolve("zod");
await matches(zodPackage, inputs.zodPackageSha256); await matches(zodEntry, inputs.zodEntrySha256);
const parser = createRequire(path.join(parserRoot, "package.json"))("./lib/index.js");
const publicTypes = new Map<string, any>();
for (const [relative, expected] of Object.entries(inputs.zodTypeFiles)) {
if (!/^[A-Za-z0-9._/-]+$/.test(relative) || relative.split("/").includes("..")) throw new Error("SDK public source proof unavailable");
const file = path.join(path.dirname(zodPackage), relative); await matches(file, expected);
const ast = parser.parse(await readFile(file, "utf8"), { sourceType: "module", plugins: [["typescript", { dts: true }]], errorRecovery: true });
if (ast.errors.length) throw new Error("SDK public source proof unavailable"); publicTypes.set(relative, ast);
}
const resolves = (parent: string, specifier: unknown, child: string) => typeof specifier === "string" && path.posix.normalize(path.posix.join(path.posix.dirname(parent), specifier)).replace(/\.(?:cjs|js)$/, ".d.cts") === child;
const entry = publicTypes.get("index.d.cts"), external = publicTypes.get("v4/classic/external.d.cts"), schemas = publicTypes.get("v4/classic/schemas.d.cts");
if (!entry || !external || !schemas || !entry.program.body.some((n: any) => n.type === "ExportAllDeclaration" && resolves("index.d.cts", n.source?.value, "v4/classic/external.d.cts"))
|| !external.program.body.some((n: any) => n.type === "ExportAllDeclaration" && resolves("v4/classic/external.d.cts", n.source?.value, "v4/classic/schemas.d.cts"))) throw new Error("SDK public source proof unavailable");
const namespaces = entry.program.body.filter((n: any) => n.type === "ImportDeclaration" && resolves("index.d.cts", n.source?.value, "v4/classic/external.d.cts")).flatMap((n: any) => n.specifiers.filter((s: any) => s.type === "ImportNamespaceSpecifier").map((s: any) => s.local.name));
if (!entry.program.body.some((n: any) => n.type === "ExportNamedDeclaration" && n.specifiers?.some((s: any) => s.exported?.name === "z" && namespaces.includes(s.local?.name)))) throw new Error("SDK public source proof unavailable");
const declaredTypes = new Set(schemas.program.body.filter((n: any) => n.type === "ExportNamedDeclaration" && ["TSInterfaceDeclaration", "TSTypeAliasDeclaration", "ClassDeclaration"].includes(n.declaration?.type)).map((n: any) => n.declaration.id.name));
const declaredFactories = new Set(schemas.program.body.filter((n: any) => n.type === "ExportNamedDeclaration" && ["TSDeclareFunction", "FunctionDeclaration"].includes(n.declaration?.type)).map((n: any) => n.declaration.id.name));
const zod = require("zod"), namespace = zod.z ?? zod;
const constructors = Object.freeze(Object.fromEntries(Object.entries(namespace).filter(([name, value]) => declaredTypes.has(name) && typeof value === "function")));
const factories = Object.freeze(Object.fromEntries(Object.entries(namespace).filter(([name, value]) => declaredFactories.has(name) && typeof value === "function")));
const source = { root, parser, constructors, factories, inputs, identity: { inputs: Object.keys(inputs.inputs).length, manifestSha256: hash(JSON.stringify(inputs)),
lockSha256: inputs.lockSha256, compilerPackageSha256: inputs.compilerPackageSha256, parserPackageSha256: inputs.parserPackageSha256,
parserEntrySha256: inputs.parserEntrySha256, zodPackageSha256: inputs.zodPackageSha256, zodEntrySha256: inputs.zodEntrySha256, zodTypeGraphSha256: hash(JSON.stringify(inputs.zodTypeFiles)) } };
Object.freeze(source.identity); Object.freeze(source); verified.add(source); return source;
}
function nodes(node: any, fn: (node: any) => void) {
if (!node || typeof node !== "object") return;
if (Array.isArray(node)) { for (const child of node) nodes(child, fn); return; }
if (typeof node.type !== "string") return;
fn(node);
for (const [key, value] of Object.entries(node)) if (!["loc", "extra", "errors", "tokens", "comments", "leadingComments", "trailingComments", "innerComments"].includes(key)) nodes(value, fn);
}
function aliases(ast: any): string[] {
const found: string[] = [];
for (const row of ast.program.body) if (row.type === "ImportDeclaration" && row.source.value === "zod") {
for (const s of row.specifiers) if (s.type === "ImportNamespaceSpecifier" || s.type === "ImportSpecifier" && s.imported.name === "z") found.push(s.local.name);
}
if (new Set(found).size !== found.length) return [];
let shadowed = false;
nodes(ast, n => {
if (n.type === "TSModuleDeclaration") shadowed = true;
if (["VariableDeclarator", "FunctionDeclaration", "ClassDeclaration", "TSModuleDeclaration", "TSTypeAliasDeclaration", "TSInterfaceDeclaration", "TSEnumDeclaration"].includes(n.type) && found.includes(n.id?.name)) shadowed = true;
if (n.type === "TSTypeParameter" && found.includes(typeof n.name === "string" ? n.name : n.name?.name)) shadowed = true;
if (Array.isArray(n.params) && n.params.some((p: any) => p.type === "Identifier" && found.includes(p.name))) shadowed = true;
});
return shadowed ? [] : found;
}
function parsed(source: PublicSource, text: string, declaration: boolean) {
const ast = source.parser.parse(text, { sourceType: "module", plugins: [["typescript", { dts: declaration }]], errorRecovery: true });
if (ast.errors.length) throw new Error("SDK public source parse unavailable"); return ast;
}
const primitive = (node: any) => node?.type === "NumericLiteral" && Number.isFinite(node.value) || ["BooleanLiteral", "NullLiteral"].includes(node?.type) || node?.type === "StringLiteral" && node.value === "";
function publicSourceProperty(source: PublicSource, ast: any, key: string) {
const publicAliases = aliases(ast), properties: any[] = [];
nodes(ast, n => { if (n.type === "ObjectProperty" && !n.computed && (n.key?.name ?? n.key?.value) === key) properties.push(n); });
if (properties.length !== 1) return false;
let value = properties[0].value, factory: string | undefined;
while (value?.type === "CallExpression" && value.callee?.type === "MemberExpression" && !value.callee.computed && value.callee.property?.type === "Identifier") {
if (!value.arguments.every(primitive)) return false;
factory = value.callee.property.name; value = value.callee.object;
}
return value?.type === "Identifier" && publicAliases.includes(value.name) && !!factory && Object.hasOwn(source.factories, factory) && typeof source.factories[factory] === "function";
}
export async function sdkPublicTypeSyntax(source: PublicSource, file: string, text: string, span: { start: number; end: number }) {
if (!verified.has(source) || Buffer.byteLength(text) > 8 * 1024 * 1024) return false;
const original = file.replace("/dist/", "/src/").replace(/\.d\.ts$/, ".ts");
if (original === file || !source.inputs.inputs[original]) return false;
const filename = path.join(source.root, original), sourceText = await readFile(filename, "utf8");
if (createHash("sha1").update(`blob ${Buffer.byteLength(sourceText)}\0`).update(sourceText).digest("hex") !== source.inputs.inputs[original]) return false;
try {
const ast = parsed(source, text, true), publicAliases = aliases(ast), candidates: any[] = [];
nodes(ast, n => { if (n.type === "TSPropertySignature" && n.start <= span.start && n.end >= span.end) candidates.push(n); });
candidates.sort((a, b) => (a.end - a.start) - (b.end - b.start)); const property = candidates[0], type = property?.typeAnnotation?.typeAnnotation;
if (!property || property.computed || property.key?.type !== "Identifier" || property.initializer || property.value || type?.type !== "TSTypeReference"
|| span.start < property.key.start || span.start >= property.key.end || type.typeName?.type !== "TSQualifiedName" || span.end !== type.typeName.end
|| !/^\s*:\s*$/.test(text.slice(property.key.end, type.typeName.start))
|| (ast.comments ?? []).some((c: any) => c.start < property.end && c.end > property.start)) return false;
let bound = true, references = 0;
nodes(property.typeAnnotation, n => {
if (!["TSTypeAnnotation", "TSTypeReference", "TSQualifiedName", "TSTypeParameterInstantiation", "Identifier"].includes(n.type)) bound = false;
if (n.type === "TSTypeReference") {
references++; const name = n.typeName;
if (name?.type !== "TSQualifiedName" || name.left?.type !== "Identifier" || name.right?.type !== "Identifier" || !publicAliases.includes(name.left.name)
|| !Object.hasOwn(source.constructors, name.right.name) || typeof source.constructors[name.right.name] !== "function") bound = false;
}
});
return bound && references > 0 && publicSourceProperty(source, parsed(source, sourceText, false), property.key.name);
} catch { return false; }
}
export interface SdkScanObservation { pathSha256: string; fileSha256: string; detector: number; start: number; end: number; classification: "private_exact" | "direct_token" | "unresolved_context" | "public_schema_type" }
export function createSdkArtifactScanner(repositoryRoot: string, secrets: readonly string[], expectations?: SdkPublicInputs) {
const observations: SdkScanObservation[] = []; let bounded = true; let proof: Promise<PublicSource | null> | undefined;
const source = () => proof ??= verifySdkPublicSource(repositoryRoot, expectations).catch(() => null);
return {
async scan(file: string, bytes: Buffer) {
const base = { pathSha256: hash(file), fileSha256: hash(bytes) };
if (findSecretLeakMatch(bytes, secrets, { includeShapes: false })) { observations.push({ ...base, detector: -1, start: 0, end: bytes.length, classification: "private_exact" }); return false; }
const raw = secretShapeObservations(bytes.toString("utf8")); if (!raw.bounded) { bounded = false; for (const span of raw.matches) observations.push({ ...base, ...span, classification: span.detector === 3 ? "unresolved_context" : "direct_token" }); return false; }
if (raw.matches.some(span => span.detector !== 3)) {
for (const span of raw.matches) observations.push({ ...base, ...span, classification: span.detector === 3 ? "unresolved_context" : "direct_token" });
return false;
}
let safe = true;
for (const span of raw.matches) {
const publicSource = span.detector === 3 ? await source() : null;
const syntax = !!publicSource && await sdkPublicTypeSyntax(publicSource, file, bytes.toString("utf8"), span);
const classification = span.detector !== 3 ? "direct_token" : syntax ? "public_schema_type" : "unresolved_context";
observations.push({ ...base, ...span, classification }); if (!syntax) safe = false;
}
return safe;
},
async audit() { const p = proof ? await proof : null; return { schema: "paperclip.source-sdk-scan/v1", suppressedFindings: 0, bounded, publicSource: p?.identity ?? null, observations: structuredClone(observations).sort((a, b) => a.pathSha256.localeCompare(b.pathSha256) || a.detector - b.detector || a.start - b.start) }; },
};
}