mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog checks real chat journeys against saved product state > - Full cohort 12 (OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017`) failed three cases where the agent did the correct thing > - In each case the harness, not the product, broke the journey: it left a page too early, could not answer a choice card, or pressed keys faster than the drag library listens > - A harness that breaks the journey cannot measure the product > - This pull request makes the harness act like a person in these three journeys, counts a verified card choice as a human action, and keeps each check able to fail a broken product > - The benefit is that these three cases measure the product again ## Linked Issues or Issue Description **What happened?** In full cohort 12, three cases failed for harness reasons: - `agents.choose-primary`: the CEO answered correctly. The harness clicked the crown and confirmed. Then it waited for "My primary agent" with `.first()`. The sidebar already showed the CEO's crown, so this wait passed at once. The harness then navigated away before the browser sent `PUT /primary-agent/me`. The server log has no PUT. The read-back still showed the CEO. - `tasks.ambiguous-task`: the CEO changed nothing and asked a correct question card. The card linked both tasks and offered "COM-1 (Launch project)", "COM-2 (Experiments project)" and "Both tasks". The harness could only answer with a text box or an "Other" option. It threw "The linked question has no supported explicit clarification input". - `steering.edit-queued-message`: the person's edit saved (PATCH 200). The keyboard reorder (Space, ArrowUp, Space) sent no `PUT /queued-comments/order` in 2 gestures. Cohort 9 had the same failure. **Expected behavior** Each journey completes as a person does it, and the oracle grades the saved result. **Steps to reproduce** 1. Use master `f7ddab8e94`. 2. Run the Runner Full-Stack E2E workflow with `all=false` and the three case ids `chat-features.runner-opencode.local.<case>`. 3. Read each case's `playwright.log` and `server.log`. **Additional context** - Primary agent: the crown fills at once (optimistic update) and the save starts on confirm. The mutation lives in the app-level `PrimaryAgentProvider`, so in-app navigation does not cancel it. In the failed run the confirm and the full-page `goto` happened within the same second, and the PUT never reached the server. Only a full page load in the first milliseconds can lose the save. A person cannot do that in practice, so this is not a product bug. No UI change. - Keyboard reorder: dnd-kit's `KeyboardSensor` starts the drag on Space, adds its keydown listener in a `setTimeout`, and `sortableKeyboardCoordinates` ignores arrows until the rows are measured on a later render. Keys pressed back to back can arrive before this. In the production Storybook story (headless Chromium), back-to-back `locator.press` moved the item in 6 of 8 tries. Keys about 150 ms apart moved it 5 of 5 times, also after an Edit and Save. With the drag-start wait and a paint between keys it moved 16 of 16 times, also at 6x CPU throttle. The keyboard reorder works for people, so this is not a product bug. No UI change. ## What Changed - `runtime-primary.ts`: wait for the browser's own `PUT /primary-agent/me` that names Morgan, require 2xx, and poll the read-back until Morgan is primary before leaving the page. Look for the "My primary agent" crown only in the header that holds Morgan's avatar. Add `isPrimarySaveRequest`. Oracle `runtime-primary.v6`. - `chat-linked-question.ts` / `internal-tasks.ts`: add `chosenTaskOption`. It finds the one option of the bound saved question that names the Launch task's identifier (a setup literal) and no other candidate's. It uses the option id for the rendered DOM id and the payload's submit label. It does not read option wording. The harness selects that option, checks `aria-checked`, submits, and waits for the `respond` POST. The old text and "Other" paths stay for other cards. The checks that nothing changes before the answer and that only COM-1 changes after it are unchanged. - `chat-catalog-contract.ts` / `internal-tasks.ts`: a chosen option saves no comment, so the flow counted only one human action and failed `catalog-original-human-query`. `tasks.ambiguous-task` now joins the cases that may count a bound UI answer. The case declares `expectedHumanQuestionAnswer` only after the harness answered through the choice, and names the chosen question and option. The flow then verifies the answered card independently: answered by this person, exactly the Launch option and no typed text, one audit event, and one continuation run after it. Oracle `internal.ambiguous-task.v8`. - `chat-catalog-contract.ts`: `chatCatalogHumanQuestionPolicies` takes the answering person and mirrors the server's user rule in `evaluateIssueThreadInteractionResolverAudience`. `anyone` and `human_only` admit the person. `not_creator` admits a person who did not create the card. With no resolver (a pending card) `not_creator` stays rejected, so other callers keep their behavior. For the ambiguous-task choice, an unaddressed card is accepted, as the server allows. - `runtime-queue.ts`: `keyboardMoveQueuedMessageUp` waits for the handle's `aria-pressed="true"` after Space. It lets the page paint before ArrowUp and before the closing Space. It records whether each gesture started a drag. Oracle `edit-queued-message.structural.v7`. - No scenario YAML changed, so the TestOps manifest needs no entry. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts` for `chat-linked-question.test.ts`, `chat-catalog-consistency.test.ts`, `runtime-primary.test.ts`, `runtime-queue.test.ts`, `internal.test.ts`, `external-company-secret.test.ts`, `external-personal-secret.test.ts`, `turn-response-card.test.ts`, `runtime-task-coordination.test.ts`: pass. - Red/green: the two new keyboard tests fail on the old `keyboardMoveQueuedMessageUp` and pass on the new one. The resolver-policy test fails on the old `chatCatalogHumanQuestionPolicies` (`not_creator` answered by a person who is not the creator) and passes on the new one; `not_creator` answered by its creator and a pending card stay rejected. The choice-accounting test fails on the old allow list. - `chosenTaskOption` tests use the cohort 12 card shape, a reworded card, and cards with no single choice (both, duplicate, neither, `COM-10`, a different card). - `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit`: pass. `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 pass. - Paid narrow run 1 (commit `33399f8370`): https://github.com/paperclipai/paperclip/actions/runs/43446445525. `agents.choose-primary` passed (browser `PUT /primary-agent/me` 200). `steering.edit-queued-message` passed (browser `PUT /queued-comments/order` 200). `tasks.ambiguous-task` selected COM-1 (`respond` 200, only COM-1 became high) but failed `catalog-original-human-query`, because the flow did not count the choice. Later commits fix that. - Paid narrow run 2 (`c90c2aa26a`) was cancelled by run 3 before its case finished. - Paid narrow run 3 (`tasks.ambiguous-task` only, final verifier `fee3bbd453`): https://github.com/paperclipai/paperclip/actions/runs/43452010284 ## Risks - Low risk. Test harness only. - The ambiguous-task harness now answers through a structured choice when one exists. If the agent asks a card with no option that names only the Launch task, the old text and "Other" paths still apply. - `chatCatalogHumanQuestionPolicies` accepts `not_creator` only when a resolver is passed. Callers that check pending cards pass none, so their behavior is unchanged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
472 lines
47 KiB
TypeScript
472 lines
47 KiB
TypeScript
import { isDeepStrictEqual } from "node:util";
|
|
import { createHash } from "node:crypto";
|
|
import { requestConfirmationPayloadSchema, askUserQuestionsPayloadSchema } from "../../packages/shared/src/validators/issue.js";
|
|
import type { RequestConfirmationInteraction, AskUserQuestionsInteraction } from "../../packages/shared/src/types/issue.js";
|
|
import type { ChatFlowInput, ChatIssue, ChatRun } from "./chat-flow.js";
|
|
import type { CredentialName } from "./types.js";
|
|
import type { ChatExternalInputRequirement } from "./chat-private-inputs.js";
|
|
import type { ChatScenario } from "./chat-feature-cases.js";
|
|
|
|
/** Lifecycle readiness permits an already-busy worker; this is not run completion proof. */
|
|
export interface ChatCatalogAgentReadiness { lifecycleState?: string | null; status?: string | null; lifecycleError?: unknown;
|
|
lifecycleHolds?: unknown; pauseReason?: string | null; lifecycleOperation?: Record<string, unknown> | null }
|
|
export function chatCatalogAgentIsReady(value: ChatCatalogAgentReadiness): boolean {
|
|
return value.lifecycleState === "ready" && !value.lifecycleError
|
|
&& ["idle", "active", "running"].includes(value.status ?? "");
|
|
}
|
|
/** Names only closed lifecycle/status fields of the last observation, so a timeout says which half was unready. */
|
|
export function chatCatalogAgentReadinessDetail(value: ChatCatalogAgentReadiness | undefined): string {
|
|
if (!value) return "no agent state was observed";
|
|
const word = (input: unknown) => typeof input === "string" && /^[a-z_]{1,40}$/.test(input) ? input : input == null ? "none" : "other";
|
|
const holds = Array.isArray(value.lifecycleHolds) ? value.lifecycleHolds.map(word).join(",") || "none" : "none";
|
|
const operation = value.lifecycleOperation ?? {};
|
|
const flags = ["waitingForSignal", "retryRequired", "recheckRequested", "hostComplete"].filter(key => operation[key] === true).join(",") || "none";
|
|
return `last observed lifecycleState=${word(value.lifecycleState)} status=${word(value.status)} holds=${holds} pauseReason=${word(value.pauseReason)}`
|
|
+ ` operationFlags=${flags} attempts=${typeof operation.attempts === "number" ? operation.attempts : "none"}`;
|
|
}
|
|
|
|
export interface ChatCatalogCheck { id: string; passed: boolean; detail: string }
|
|
export interface ChatCatalogGrade { passed: boolean; checks: ChatCatalogCheck[] }
|
|
export interface ChatCatalogContext {
|
|
/** Original flow observation deadline; optional opt-in for bounded human followup acts. */
|
|
readonly observationDeadlineAt?: number;
|
|
input: ChatFlowInput;
|
|
scenario: ChatScenario;
|
|
issue(): ChatIssue | undefined;
|
|
allRuns(): Promise<ChatRun[]>;
|
|
refreshIssue(): Promise<void>;
|
|
/** Available only to the two allowlisted native Slack ingress cases. */
|
|
selectSlackSource?(issueId: string): Promise<void>;
|
|
}
|
|
|
|
/** These exact selected OpenCode/local journeys use the original flow clock for
|
|
* native observation phases. Other cases/lanes keep their authored phase caps. */
|
|
const originalClockExecutionIds = new Set([
|
|
"chat-features.runner-opencode.local.companies.export-company",
|
|
"chat-features.runner-opencode.local.projects.project-defaults",
|
|
"chat-features.runner-opencode.local.questions-and-approvals.ambiguous-yes",
|
|
"chat-features.runner-opencode.local.questions-and-approvals.review-plan",
|
|
"chat-features.runner-opencode.local.questions-and-approvals.necessary-question",
|
|
"chat-features.runner-opencode.local.task-coordination.block-on-design",
|
|
"chat-features.runner-opencode.local.gateways-profiles.rotate-token",
|
|
"chat-features.runner-opencode.local.gateways-profiles.create-gateway",
|
|
"chat-features.runner-opencode.local.workspaces.preview-service",
|
|
"chat-features.runner-opencode.local.assistant-connections.consent-read-only",
|
|
"chat-features.runner-opencode.local.navigation-preferences.quick-search",
|
|
"chat-features.runner-opencode.local.navigation-preferences.edit-profile",
|
|
"chat-features.runner-opencode.local.navigation-preferences.broken-link",
|
|
"chat-features.runner-opencode.local.navigation-preferences.dismiss-announcement",
|
|
"chat-features.runner-opencode.local.projects.share-project",
|
|
"chat-features.runner-opencode.local.access.cli-authorization",
|
|
"chat-features.runner-opencode.local.access.other-company",
|
|
"chat-features.runner-opencode.local.access.private-resource",
|
|
"chat-features.runner-opencode.local.agent-chat.conference-room",
|
|
"chat-features.runner-opencode.local.agent-chat.track-delegated-work",
|
|
"chat-features.runner-opencode.local.agents.agent-api-key",
|
|
"chat-features.runner-opencode.local.agents.hire-needs-approval",
|
|
"chat-features.runner-opencode.local.agents.pause-agent",
|
|
"chat-features.runner-opencode.local.agents.retire-agent",
|
|
"chat-features.runner-opencode.local.app-permissions.new-quarantined-action",
|
|
"chat-features.runner-opencode.local.app-permissions.read-allowed-write-off",
|
|
"chat-features.runner-opencode.local.app-permissions.test-as-agent",
|
|
"chat-features.runner-opencode.local.assistant-connections.device-login",
|
|
"chat-features.runner-opencode.local.budgets-costs.explain-hard-stop",
|
|
"chat-features.runner-opencode.local.budgets-costs.set-budget",
|
|
"chat-features.runner-opencode.local.cases.case-history",
|
|
"chat-features.runner-opencode.local.inbox-search.archive-without-grant",
|
|
"chat-features.runner-opencode.local.inbox-search.show-my-work",
|
|
"chat-features.runner-opencode.local.tasks.ambiguous-task",
|
|
"chat-features.runner-opencode.local.secrets.secret-proposal",
|
|
"chat-features.runner-opencode.local.developer-labs.long-thread-performance",
|
|
"chat-features.runner-opencode.local.questions-and-approvals.approve-tool-once",
|
|
"chat-features.runner-opencode.local.runs-adapters.reset-session",
|
|
"chat-features.runner-opencode.local.tasks.website-without-project",
|
|
"chat-features.runner-opencode.local.inbox-search.archive-partial-failure",
|
|
"chat-features.runner-opencode.local.cli-operations.feedback-bundle",
|
|
"chat-features.runner-opencode.local.cli-operations.inspect-run-cli",
|
|
"chat-features.runner-opencode.local.cli-operations.owned-dev-instance",
|
|
"chat-features.runner-opencode.local.developer-labs.component-reference",
|
|
"chat-features.runner-opencode.local.developer-labs.gated-lab",
|
|
"chat-features.runner-opencode.local.developer-labs.interaction-fixture",
|
|
"chat-features.runner-opencode.local.recovery.browser-reconnect",
|
|
"chat-features.runner-opencode.local.recovery.controller-restart",
|
|
"chat-features.runner-opencode.local.recovery.explain-stall",
|
|
"chat-features.runner-opencode.local.recovery.retry-exhausted",
|
|
"chat-features.runner-opencode.local.recovery.source-action",
|
|
"chat-features.runner-opencode.local.routines.pause-owned",
|
|
"chat-features.runner-opencode.local.routines.run-now",
|
|
"chat-features.runner-opencode.local.steering.queue-follow-up",
|
|
"chat-features.runner-opencode.local.steering.stop-versus-pause",
|
|
"chat-features.runner-opencode.local.task-coordination.delegate-with-context",
|
|
"chat-features.runner-opencode.local.task-coordination.independent-review",
|
|
"chat-features.runner-opencode.local.task-coordination.pause-tree",
|
|
"chat-features.runner-opencode.local.tasks.edit-properties",
|
|
"chat-features.runner-opencode.local.steering.edit-queued-message",
|
|
"chat-features.runner-opencode.local.steering.discard-queued-message",
|
|
"chat-features.runner-opencode.local.instance-operations.backup-and-restore",
|
|
"chat-features.runner-opencode.local.instance-operations.restart-service",
|
|
"chat-features.runner-opencode.local.routines.webhook-trigger",
|
|
"chat-features.runner-opencode.local.status-cards.explain-card",
|
|
"chat-features.runner-opencode.local.secrets.configure-company-secret",
|
|
"chat-features.runner-opencode.local.teams.configure-team",
|
|
"chat-features.runner-opencode.local.steering.interrupt-with-direction",
|
|
"chat-features.runner-opencode.local.companies.fast-responses",
|
|
]);
|
|
function usesOriginalNativePhaseClock(ctx: ChatCatalogContext): boolean {
|
|
return originalClockExecutionIds.has(ctx.input?.execution?.id);
|
|
}
|
|
export function chatCatalogNativePhaseDeadline(ctx: ChatCatalogContext, legacyDeadlineAt: number): number {
|
|
if (!usesOriginalNativePhaseClock(ctx)) return legacyDeadlineAt;
|
|
const deadline = ctx.observationDeadlineAt, now = Date.now();
|
|
if (!Number.isFinite(deadline) || deadline! <= now) throw new Error("Original native observation budget exhausted or missing");
|
|
return deadline!;
|
|
}
|
|
/** Recheck the same clock before each declared human send operation. */
|
|
export function chatCatalogNativePhaseRemainingBudget(ctx: ChatCatalogContext): (() => number) | undefined {
|
|
if (!usesOriginalNativePhaseClock(ctx)) return undefined;
|
|
return () => {
|
|
const now = Date.now(), deadline = ctx.observationDeadlineAt;
|
|
if (!Number.isFinite(deadline) || deadline! <= now) throw new Error("Original native observation budget exhausted or missing");
|
|
return deadline! - now;
|
|
};
|
|
}
|
|
/** Only explicitly selected lanes opt identity helpers into the original clock. */
|
|
export function chatCatalogOriginalPhaseDeadline(ctx: ChatCatalogContext): number | undefined {
|
|
return usesOriginalNativePhaseClock(ctx) ? chatCatalogNativePhaseDeadline(ctx, 0) : undefined;
|
|
}
|
|
export function chatCatalogNativePhaseIsTimely(ctx: ChatCatalogContext): boolean {
|
|
return !usesOriginalNativePhaseClock(ctx) || Number.isFinite(ctx.observationDeadlineAt) && Date.now() < ctx.observationDeadlineAt!;
|
|
}
|
|
|
|
export type ChatCatalogTerminalRunStatus = "succeeded" | "cancelled" | "failed" | "timed_out";
|
|
export type ChatCatalogRunPolicy =
|
|
| { kind: "terminal-counts"; terminalStatusCounts: Partial<Record<ChatCatalogTerminalRunStatus, number>>;
|
|
/** Optional upper bounds: a status listed here may occur from its terminalStatusCounts value up to this count.
|
|
* The case oracle must bind every run above the minimum to a declared role. */
|
|
maximumTerminalStatusCounts?: Partial<Record<ChatCatalogTerminalRunStatus, number>> }
|
|
| { kind: "source-settled-with-active-work"; maximumActiveWorkerRuns: number;
|
|
expectedWorkerCancellation?: "owned-task-reassignment";
|
|
/** Lead runs outside the source woken by a case handoff back to the lead (a worker returning its case).
|
|
* Defaults to 0; the case oracle must bind each one to the worker's actual handoff. */
|
|
maximumLeadCaseHandbackRuns?: number };
|
|
export interface ChatCatalogExpectedWorkerCancellation {
|
|
runId: string; taskId: string; workerId: string; sourceRunId: string;
|
|
}
|
|
export interface ChatCatalogSetupPolicy {
|
|
funding?: "standard" | "preserve-scenario";
|
|
readiness?: "all-ready" | "preserve-scenario";
|
|
/** Required independent prerequisite checks when preserving a scenario-specific policy. */
|
|
invariantCheckIds: readonly string[];
|
|
}
|
|
export type ChatCatalogQueryBinding = { catalogToken: string; stateKey: string } & (
|
|
| { resourceKind: "issue"; valueField: "identifier" }
|
|
| { resourceKind: "project-repository"; valueField: "fullName" }
|
|
| { resourceKind: "agentmail-recipient"; valueField: "email" }
|
|
);
|
|
export interface ChatCatalogQueryAttachment {
|
|
/** A genuine UI upload receipt returned by setup; never arbitrary body text. */
|
|
stateKey: string;
|
|
filename: string;
|
|
}
|
|
export interface ChatCompanionSurface { kind: "scoped-external-assistant"; model: string; maximumModelTurns: 6; maximumToolCalls: 4; maximumOutputTokens: 2048 }
|
|
export interface ChatCompanionResult {
|
|
query: string; reply: string; model: string; usage: { inputTokens: number; outputTokens: number };
|
|
messages: Array<{ role: "system" | "user" | "assistant" | "tool"; content: string }>;
|
|
providerResponseIds: string[]; toolCalls: Array<{ name: string; arguments: unknown; result: unknown }>; links: string[];
|
|
}
|
|
export interface ChatSlackIngress { kind: "native-slack-thread"; maximumProviderCalls: 20; ingressTimeoutMs: 120000; setupThread?: "same-agent-distractor" }
|
|
export interface ChatSlackIngressReceipt {
|
|
query: string; endpointId: string; conversationId: string; issueId: string; humanCommentId: string; linkedPaperclipUserId: string; providerCalls: number;
|
|
provider: { workspaceId: string; channelId: string; threadId: string; canonicalMessageId: string; senderUserId: string };
|
|
}
|
|
/** One concrete scenario, with production API setup and a case-specific independent oracle. */
|
|
export interface NativeStopBoundaryReceipt { runId: string; issueId: string; intentId: string; intentRecordedAt: string }
|
|
export interface ChatCatalogCase {
|
|
/** Only post-stop-message: Stop the genuine same-source setup turn after bookkeeping, then send the unchanged canonical query. */
|
|
canonicalBoundary?: "native-stop-cleanup";
|
|
beforeCanonicalSend?(context: ChatCatalogContext, state: Record<string, unknown>): Promise<NativeStopBoundaryReceipt>;
|
|
scenarioId: string;
|
|
/** Real browser locale context for the canonical person's schedule. */
|
|
browserTimezone?: "America/Chicago";
|
|
respondingAgent?: { kind: "case-primary-owner"; name: "Morgan"; role: "engineer"; canAssignTasks: true };
|
|
requiredCredentials?: readonly CredentialName[];
|
|
externalInput?: ChatExternalInputRequirement;
|
|
installedSource?: "published-global-npm";
|
|
journeyKind?: "agent-chat" | "scoped-external-assistant" | "native-slack-thread";
|
|
slackIngress?: ChatSlackIngress;
|
|
/** Only reply-in-thread: one real other-source setup exchange, included in total native bounds. */
|
|
dispatchSlackOtherThread?(context: ChatCatalogContext, state: Record<string, unknown>, input: { query: string; signal: AbortSignal }): Promise<ChatSlackIngressReceipt & { runId: string }>;
|
|
dispatchSlackQuery?(context: ChatCatalogContext, state: Record<string, unknown>, input: { query: string; signal: AbortSignal }): Promise<ChatSlackIngressReceipt>;
|
|
companionSurface?: ChatCompanionSurface;
|
|
runCompanionSurface?(context: ChatCatalogContext, state: Record<string, unknown>, input: { query: string; signal: AbortSignal }): Promise<ChatCompanionResult>;
|
|
/** Only channel-file: independently settle the supported transfer before a second actual upload. */
|
|
verifySlackSupportedTransfer?(context: ChatCatalogContext, state: Record<string, unknown>): Promise<ChatCatalogGrade>;
|
|
dispatchSlackUnsupportedFile?(context: ChatCatalogContext, state: Record<string, unknown>, input: { query: string; signal: AbortSignal }): Promise<ChatSlackIngressReceipt>;
|
|
authentication?: "local-trusted" | "human-session";
|
|
setupPolicy?: ChatCatalogSetupPolicy;
|
|
runPolicy?: ChatCatalogRunPolicy;
|
|
/** Only edit-properties: independent observation proof of the exact reassigned worker. */
|
|
expectedCancelledWorkerRun?(observation: unknown): ChatCatalogExpectedWorkerCancellation | undefined;
|
|
queryOutcome?: "persisted" | "rejected";
|
|
/** Audited substitution of an incidental catalog fixture token, never other prose. */
|
|
queryBindings?: readonly ChatCatalogQueryBinding[];
|
|
queryAttachments?: readonly ChatCatalogQueryAttachment[];
|
|
/** All case/grader helpers must be listed; the suite also fingerprints its complete source tree. */
|
|
sourceFiles: readonly string[];
|
|
oracleVersion: string;
|
|
/** Known missing catalog conditions keep this executable diagnostic case from acceptance. */
|
|
coverageGaps?: readonly string[];
|
|
expectedRunCount: number;
|
|
/** Persisted human comments; explicit followups must be declared before execution. */
|
|
expectedHumanMessageCount?: number;
|
|
expectedHumanQuestionAnswer?(observation: unknown): ChatCatalogExpectedHumanQuestionAnswer | undefined;
|
|
optionalCompanySecretBindingApproval?: true;
|
|
expectedCompanySecretBindingApproval?(observation: unknown): ChatCatalogExpectedCompanySecretBindingApproval | undefined;
|
|
/** Genuine persisted human setup turns before the canonical query; included in the total above. */
|
|
setupHumanMessageCount?: number;
|
|
minimumRunCount?: number;
|
|
turnTimeoutMs: number;
|
|
attemptTimeoutMs: number;
|
|
verifySetup?(context: ChatCatalogContext, state: Record<string, unknown>): Promise<ChatCatalogGrade>;
|
|
verifyQueryRejection?(context: ChatCatalogContext, state: Record<string, unknown>): Promise<ChatCatalogGrade>;
|
|
setup(context: ChatCatalogContext): Promise<Record<string, unknown>>;
|
|
/** Only explicit scenario user followups/approvals through visible production UI. */
|
|
act?(context: ChatCatalogContext, state: Record<string, unknown>): Promise<void>;
|
|
observe(context: ChatCatalogContext, state: Record<string, unknown>): Promise<unknown>;
|
|
grade(observation: unknown): ChatCatalogGrade;
|
|
/** Open the actual linked result through the browser and verify its identity. */
|
|
verifyResult(context: ChatCatalogContext, state: Record<string, unknown>, observation: unknown): Promise<void>;
|
|
cleanup?(context: ChatCatalogContext, state: Record<string, unknown>): Promise<void>;
|
|
}
|
|
|
|
export function validateChatCatalogGrade(grade: ChatCatalogGrade): ChatCatalogGrade {
|
|
if (!grade || !Array.isArray(grade.checks) || grade.checks.length === 0
|
|
|| grade.checks.some(check => !check.id || typeof check.passed !== "boolean" || !check.detail)
|
|
|| new Set(grade.checks.map(check => check.id)).size !== grade.checks.length
|
|
|| grade.passed !== grade.checks.every(check => check.passed)) {
|
|
throw new Error("Invalid chat catalog oracle result: checks must be nonempty, unique and agree with the verdict");
|
|
}
|
|
return grade;
|
|
}
|
|
|
|
export interface ChatCatalogSetupBoundary { humanCommentIds: string[]; runIds: string[] }
|
|
export function validChatCatalogHumanCounts(implementation: Pick<ChatCatalogCase, "setupHumanMessageCount" | "expectedHumanMessageCount" | "queryOutcome"> & Partial<Pick<ChatCatalogCase,"scenarioId"|"optionalCompanySecretBindingApproval">>): boolean {
|
|
const setup = implementation.setupHumanMessageCount ?? 0, total = implementation.expectedHumanMessageCount ?? 1;
|
|
return (!implementation.optionalCompanySecretBindingApproval || implementation.scenarioId==="secrets.configure-company-secret"&&total===2&&setup===0)
|
|
&& Number.isSafeInteger(setup) && setup >= 0 && Number.isSafeInteger(total)
|
|
&& total >= setup + (implementation.queryOutcome === "rejected" ? 0 : 1);
|
|
}
|
|
export function chatCatalogAllowsHumanQuestionAnswer(scenarioId: string): boolean {
|
|
// tasks.ambiguous-task: the person can answer the clarification by choosing one of the card's saved options.
|
|
return ["secrets.personal-secret", "secrets.configure-company-secret", "tasks.ambiguous-task"].includes(scenarioId);
|
|
}
|
|
export function chatCatalogHumanHistoryMatches(input: {
|
|
implementation: Pick<ChatCatalogCase, "setupHumanMessageCount" | "expectedHumanMessageCount" | "queryOutcome"> & Partial<Pick<ChatCatalogCase, "scenarioId" | "optionalCompanySecretBindingApproval">>;
|
|
questionAnswer?: ChatCatalogVerifiedHumanQuestionAnswer;
|
|
secretApproval?: ChatCatalogVerifiedCompanySecretBindingApproval;
|
|
human: Array<{ id: string; body: string }>; runIds: string[];
|
|
boundary: ChatCatalogSetupBoundary; renderedQuery: string;
|
|
}): boolean {
|
|
const count = input.implementation.setupHumanMessageCount ?? 0;
|
|
return (!input.secretApproval || input.implementation.scenarioId === "secrets.configure-company-secret" && input.implementation.optionalCompanySecretBindingApproval === true)
|
|
&& (!input.questionAnswer || chatCatalogAllowsHumanQuestionAnswer(input.implementation.scenarioId ?? ""))
|
|
&& (!input.questionAnswer?.personalSecretConfirmation || input.implementation.scenarioId === "secrets.personal-secret")
|
|
&& (!input.questionAnswer?.companySecretConfirmation || input.implementation.scenarioId === "secrets.configure-company-secret")
|
|
&& (!(input.questionAnswer?.companyEnvironmentNameAnswer||input.questionAnswer?.companySavedSecretNameAnswer) || input.implementation.scenarioId === "secrets.configure-company-secret")
|
|
&& validChatCatalogHumanCounts(input.implementation)
|
|
&& input.boundary.humanCommentIds.length === count
|
|
&& input.boundary.humanCommentIds.every((id, index) => input.human[index]?.id === id)
|
|
&& input.boundary.runIds.every(id => input.runIds.includes(id))
|
|
&& (input.implementation.queryOutcome === "rejected"
|
|
? input.human.length === count
|
|
: input.human[count]?.body.trim() === input.renderedQuery.trim()
|
|
&& input.human.length + (input.questionAnswer ? 1 : 0) === (input.implementation.expectedHumanMessageCount ?? 1));
|
|
}
|
|
|
|
/**
|
|
* Whether a person may resolve this card under its resolver policies, mirroring the server's user-actor rule in
|
|
* evaluateIssueThreadInteractionResolverAudience: `anyone` and `human_only` admit the person, and `not_creator`
|
|
* admits anyone except the card's creator. Without a resolver (a card still pending) `not_creator` is not accepted.
|
|
*/
|
|
export function chatCatalogHumanQuestionPolicies(row: { resolverPolicy?: unknown; requestedResolverPolicy?: unknown; effectiveResolverPolicy?: unknown; createdByUserId?: unknown },
|
|
resolver?: { userId: string }): boolean {
|
|
return [row.resolverPolicy, row.requestedResolverPolicy ?? row.resolverPolicy, row.effectiveResolverPolicy ?? row.resolverPolicy]
|
|
.every(policy => policy === "anyone" || policy === "human_only"
|
|
|| policy === "not_creator" && !!resolver?.userId && row.createdByUserId !== resolver.userId);
|
|
}
|
|
|
|
export function chatCatalogHasExactDefinitionId(text: string, definitionId: string): boolean {
|
|
return !!definitionId && new RegExp(`(?<![A-Za-z0-9_-])${definitionId.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}(?![A-Za-z0-9_-])`).test(text);
|
|
}
|
|
|
|
export function chatCatalogSavedSecretNameIntent(text:string):boolean {
|
|
const prompt=text.replace(/\bnot (?:the )?value\b/gi,"");
|
|
const recipients=[...prompt.replace(/Deployment API key/g,"credential").matchAll(/\b(?:[Ff]or|[Tt]o)\s+(?:[Aa]gent\s+)?([A-Z][A-Za-z0-9_-]*)\b/g)];
|
|
return recipients.every(match=>match[1]==="Morgan")&&prompt.length<=400&&/\b(?:secret|credential|key)\b/i.test(prompt)
|
|
&&(/\b(?:name|alias|identifier|label)\b/i.test(prompt)||/\bkey\b[^.!?]{0,50}\bunder\b/i.test(prompt))
|
|
&&/\b(?:saved?|stored?|created?|added|used?|under)\b/i.test(prompt)
|
|
&&!/\b(?:value|plaintext|password|CEO|Riley|Jordan|another|other person)\b/i.test(prompt);
|
|
}
|
|
export function chatCatalogCompanyEnvironmentNameQuestion(q:{prompt:string;required?:boolean;answerMode:string;initialText?:string;options?:readonly unknown[];customAnswer?:{enabled?:boolean};textValidation?:{minLength?:number;maxLength?:number;pattern?:string;inputType?:string}},answer="DEPLOYMENT_API_KEY"):boolean {
|
|
const prompt=q.prompt.replace(/\(Do not include the key value\.\)/gi,"").trim(),validation=q.textValidation;
|
|
if(!q.required||q.answerMode!=="text"||q.initialText!==undefined||q.options?.length||q.customAnswer?.enabled||prompt.length>400||!/\b(?:env|environment)\s+variable\s+name\b/i.test(prompt)||!/\bMorgan\b/i.test(prompt)||/\b(?:value|password|plaintext|CEO|Riley|Jordan|another|other person)\b/i.test(prompt)
|
|
||!/^[A-Za-z_][A-Za-z0-9_]{0,119}$/.test(answer)||!validation?.maxLength||validation.maxLength>120||validation.maxLength<answer.length||(validation.minLength??0)>answer.length||(validation.inputType!==undefined&&validation.inputType!=="text"))return false;
|
|
try{return !validation.pattern||new RegExp(validation.pattern).test(answer);}catch{return false;}
|
|
}
|
|
export interface ChatCatalogExpectedHumanQuestionAnswer { interactionId: string; sourceRunId: string; userId: string; personalSecretConfirmation?: { definitionId: string; secretId: string; payloadHash: string; continuationPolicy: "wake_assignee" | "wake_assignee_on_accept" }; companySecretConfirmation?: { secretId: string; key: string; payloadHash: string; continuationPolicy: "wake_assignee" | "wake_assignee_on_accept" }; companyEnvironmentNameAnswer?: { questionId: string; payloadHash: string; value: "DEPLOYMENT_API_KEY" }; companySavedSecretNameAnswer?: { questionId: string; payloadHash: string; value: string; secretId: string };
|
|
/** A clarification answered by choosing one saved option of the card's single question. */
|
|
clarificationChoice?: { questionId: string; optionId: string } }
|
|
export interface ChatCatalogVerifiedHumanQuestionAnswer extends ChatCatalogExpectedHumanQuestionAnswer { continuationRunId: string; resolvedAt: string }
|
|
export interface ChatCatalogHumanQuestionAudit {
|
|
companyId?: string; entityId?: string; action?: string; actorType?: string; actorId?: string;
|
|
details?: Record<string, unknown> | null; createdAt?: string;
|
|
}
|
|
export function verifyChatCatalogHumanQuestionAnswer(input: {
|
|
expected: ChatCatalogExpectedHumanQuestionAnswer; companyId: string; issueId: string; agentId: string;
|
|
canonicalComment: { id: string; authorUserId?: string };
|
|
personalSecrets?: Array<{ definition: { id: string }; secret: { id: string; companyId: string; scope: string; ownerUserId?: string | null; userSecretDefinitionId?: string | null; status: string; createdAt: string } | null }>;
|
|
companySecrets?: Array<{id:string;companyId:string;scope:string;key:string;status:string;createdByUserId?:string|null;createdAt:string}>;
|
|
interactions: Array<Partial<AskUserQuestionsInteraction>>; activity: ChatCatalogHumanQuestionAudit[];
|
|
runs: Array<{ id: string; companyId?: string; agentId: string; status: string; runtimeMode?: string; exitCode?: number | null; startedAt?: string | null; contextSnapshot?: Record<string, unknown> | null }>;
|
|
}): ChatCatalogVerifiedHumanQuestionAnswer | undefined {
|
|
if (input.expected.companySecretConfirmation) {
|
|
const b=input.expected,binding=b.companySecretConfirmation!,rows=input.interactions.filter(row=>row.id===b.interactionId),row=rows[0] as unknown as Partial<RequestConfirmationInteraction>|undefined;
|
|
const source=input.runs.find(run=>run.id===b.sourceRunId),continuations=input.runs.filter(run=>run.contextSnapshot?.interactionId===b.interactionId),continuation=continuations[0];
|
|
const secrets=input.companySecrets?.filter(secret=>secret.id===binding.secretId),secret=secrets?.[0],resolvedAt=Date.parse(String(row?.resolvedAt)),createdAt=Date.parse(String(row?.createdAt));
|
|
const audits=input.activity.filter(event=>event.action==="issue.thread_interaction_accepted"&&event.details?.interactionId===b.interactionId),audit=audits[0];
|
|
const saves=input.activity.filter(event=>event.action==="secret.created"&&event.entityId===binding.secretId),save=saves[0];
|
|
if(b.personalSecretConfirmation||b.companyEnvironmentNameAnswer||b.companySavedSecretNameAnswer||rows.length!==1||!row||row.kind!=="request_confirmation"||row.status!=="accepted"||row.companyId!==input.companyId||row.issueId!==input.issueId||row.sourceRunId!==b.sourceRunId||row.createdByAgentId!==input.agentId||row.createdByUserId||row.addresseeAgentId||(row.addresseeUserId!=null&&row.addresseeUserId!==b.userId)||!row.originCommentIds?.includes(input.canonicalComment.id)||b.userId!==input.canonicalComment.authorUserId||!chatCatalogHumanQuestionPolicies(row,{userId:b.userId})||(row.continuationPolicy!=="wake_assignee"&&row.continuationPolicy!=="wake_assignee_on_accept")||row.continuationPolicy!==binding.continuationPolicy||row.resolvedByUserId!==b.userId||row.resolvedByAgentId||row.result?.outcome!=="accepted"||row.result.resumeFailure||row.result.toolAction||row.result.secretProposal||row.dismissedAt
|
|
||!row.payload||!requestConfirmationPayloadSchema.safeParse(row.payload).success||row.payload.target||row.payload.toolAction||row.payload.secretProposal||row.payload.connectionAuthorization||row.payload.qualityReview||createHash("sha256").update(JSON.stringify(row.payload)).digest("hex")!==binding.payloadHash
|
|
||secrets?.length!==1||!secret||secret.companyId!==input.companyId||secret.scope!=="company"||secret.status!=="active"||secret.createdByUserId!==b.userId||secret.key!==binding.key||!/^[a-zA-Z0-9_.-]{1,120}$/.test(binding.key)||![createdAt,resolvedAt,Date.parse(secret.createdAt)].every(Number.isFinite)||Date.parse(secret.createdAt)<=createdAt||Date.parse(secret.createdAt)>resolvedAt
|
|
||audits.length!==1||audit?.companyId!==input.companyId||audit.entityId!==input.issueId||audit.actorType!=="user"||audit.actorId!==b.userId||audit.details?.interactionKind!=="request_confirmation"||audit.details.interactionStatus!=="accepted"
|
|
||saves.length!==1||save?.companyId!==input.companyId||save.actorType!=="user"||save.actorId!==b.userId||!Number.isFinite(Date.parse(String(save.createdAt)))||Date.parse(String(save.createdAt))<createdAt||Date.parse(String(save.createdAt))>resolvedAt
|
|
||!source||source.companyId!==input.companyId||source.agentId!==input.agentId||source.status!=="succeeded"||source.exitCode!==0||source.runtimeMode!=="native"||source.contextSnapshot?.issueId!==input.issueId||source.contextSnapshot.commentId!==input.canonicalComment.id||!Number.isFinite(Date.parse(source.startedAt??""))||Date.parse(source.startedAt!)>createdAt
|
|
||continuations.length!==1||!continuation||continuation.id===source.id||continuation.companyId!==input.companyId||continuation.agentId!==input.agentId||continuation.status!=="succeeded"||continuation.exitCode!==0||continuation.runtimeMode!=="native"||continuation.contextSnapshot?.issueId!==input.issueId||!Number.isFinite(Date.parse(continuation.startedAt??""))||Date.parse(continuation.startedAt!)<resolvedAt)return;
|
|
return {...b,continuationRunId:continuation.id,resolvedAt:new Date(resolvedAt).toISOString()};
|
|
}
|
|
if (input.expected.personalSecretConfirmation) {
|
|
const b = input.expected, binding = b.personalSecretConfirmation!, rows = input.interactions.filter(row => row.id === b.interactionId);
|
|
const row = rows[0] as unknown as Partial<RequestConfirmationInteraction> | undefined;
|
|
const source = input.runs.find(run => run.id === b.sourceRunId), continuations = input.runs.filter(run => run.contextSnapshot?.interactionId === b.interactionId), continuation = continuations[0];
|
|
const entries = input.personalSecrets?.filter(entry => entry.definition.id === binding.definitionId), secret = entries?.[0]?.secret;
|
|
const audits = input.activity.filter(event => event.action === "issue.thread_interaction_accepted" && event.details?.interactionId === b.interactionId), audit = audits[0];
|
|
const resolvedAt = Date.parse(String(row?.resolvedAt));
|
|
if (b.companySecretConfirmation || b.companyEnvironmentNameAnswer || b.companySavedSecretNameAnswer || rows.length !== 1 || !row || row.kind !== "request_confirmation" || row.status !== "accepted"
|
|
|| row.companyId !== input.companyId || row.issueId !== input.issueId || row.sourceRunId !== b.sourceRunId || row.createdByAgentId !== input.agentId || row.createdByUserId
|
|
|| (row.addresseeUserId != null && row.addresseeUserId !== b.userId) || row.addresseeAgentId || !row.originCommentIds?.includes(input.canonicalComment.id)
|
|
|| b.userId !== input.canonicalComment.authorUserId || !chatCatalogHumanQuestionPolicies(row, { userId: b.userId }) || (row.continuationPolicy !== "wake_assignee" && row.continuationPolicy !== "wake_assignee_on_accept") || row.continuationPolicy !== binding.continuationPolicy
|
|
|| row.resolvedByUserId !== b.userId || row.resolvedByAgentId || row.result?.outcome !== "accepted" || row.result.resumeFailure || row.dismissedAt
|
|
|| !row.payload || row.payload.target || row.payload.toolAction || row.payload.secretProposal || row.payload.connectionAuthorization || row.payload.qualityReview
|
|
|| !chatCatalogHasExactDefinitionId([row.payload.prompt, row.payload.detailsMarkdown].filter(Boolean).join("\n"), binding.definitionId) || createHash("sha256").update(JSON.stringify(row.payload)).digest("hex") !== binding.payloadHash
|
|
|| entries?.length !== 1 || !secret || secret.id !== binding.secretId || secret.companyId !== input.companyId || secret.scope !== "user" || secret.status !== "active"
|
|
|| secret.ownerUserId !== b.userId || secret.userSecretDefinitionId !== binding.definitionId || !Number.isFinite(Date.parse(secret.createdAt)) || !Number.isFinite(resolvedAt) || Date.parse(secret.createdAt) > resolvedAt || Date.parse(secret.createdAt) < Date.parse(source?.startedAt ?? "")
|
|
|| audits.length !== 1 || audit?.companyId !== input.companyId || audit.entityId !== input.issueId || audit.actorType !== "user" || audit.actorId !== b.userId || audit.details?.interactionKind !== "request_confirmation" || audit.details.interactionStatus !== "accepted"
|
|
|| !source || source.companyId !== input.companyId || source.agentId !== input.agentId || source.status !== "succeeded" || source.runtimeMode !== "native" || source.contextSnapshot?.issueId !== input.issueId || source.contextSnapshot.commentId !== input.canonicalComment.id
|
|
|| !Number.isFinite(Date.parse(source.startedAt ?? "")) || Date.parse(source.startedAt!) > resolvedAt
|
|
|| continuations.length !== 1 || !continuation || continuation.id === source.id || continuation.companyId !== input.companyId || continuation.agentId !== input.agentId || continuation.status !== "succeeded" || continuation.runtimeMode !== "native" || continuation.contextSnapshot?.issueId !== input.issueId
|
|
|| !Number.isFinite(Date.parse(continuation.startedAt ?? "")) || Date.parse(continuation.startedAt!) < resolvedAt) return;
|
|
return { ...b, continuationRunId: continuation.id, resolvedAt: new Date(resolvedAt).toISOString() };
|
|
}
|
|
const b = input.expected, rows = input.interactions.filter(row => row.id === b.interactionId), row = rows[0], choice = b.clarificationChoice;
|
|
// As the server allows: an unaddressed card may be answered by the person; an addressed one only by its addressee.
|
|
const addressed = choice ? row?.addresseeUserId == null || row.addresseeUserId === b.userId : row?.addresseeUserId === b.userId;
|
|
const policies = !!row && chatCatalogHumanQuestionPolicies(row, { userId: b.userId });
|
|
if (rows.length !== 1 || !row || b.userId !== input.canonicalComment.authorUserId
|
|
|| row.companyId !== input.companyId || row.issueId !== input.issueId || row.sourceRunId !== b.sourceRunId
|
|
|| row.createdByAgentId !== input.agentId || row.createdByUserId || !addressed || row.addresseeAgentId
|
|
|| row.kind !== "ask_user_questions" || row.status !== "answered" || !policies
|
|
|| row.resolvedByUserId !== b.userId || row.resolvedByAgentId || row.result?.answers?.length !== 1) return;
|
|
const source = input.runs.find(run => run.id === b.sourceRunId);
|
|
const environmentName = b.companyEnvironmentNameAnswer, savedName=b.companySavedSecretNameAnswer;
|
|
if((environmentName?1:0)+(savedName?1:0)+(choice?1:0)>1)return;
|
|
if (choice) {
|
|
const answer = row.result.answers[0];
|
|
if (answer?.questionId !== choice.questionId || answer.optionIds.length !== 1 || answer.optionIds[0] !== choice.optionId || answer.otherText) return;
|
|
}
|
|
if (environmentName) {
|
|
const parsed = askUserQuestionsPayloadSchema.safeParse(row.payload), qs = parsed.success ? parsed.data.questionSet : undefined;
|
|
const q = qs?.questions[0], answer = row.result.answers[0];
|
|
if (!qs || qs.questions.length !== 1 || !q || q.id !== environmentName.questionId || q.answerMode !== "text" || !q.required
|
|
|| !chatCatalogCompanyEnvironmentNameQuestion(q,environmentName.value)
|
|
|| environmentName.value !== "DEPLOYMENT_API_KEY" || answer?.questionId !== q.id || answer.optionIds.length !== 0
|
|
|| answer.otherText !== environmentName.value || createHash("sha256").update(JSON.stringify(row.payload)).digest("hex") !== environmentName.payloadHash) return;
|
|
} else if(savedName){
|
|
const parsed=askUserQuestionsPayloadSchema.safeParse(row.payload),qs=parsed.success?parsed.data.questionSet:undefined,q=qs?.questions[0],answer=row.result.answers[0];
|
|
const secrets=input.companySecrets?.filter(secret=>secret.id===savedName.secretId),secret=secrets?.[0];
|
|
if(!qs||qs.questions.length!==1||!q||q.id!==savedName.questionId||q.answerMode!=="text"||!q.required||q.options?.length||q.customAnswer?.enabled||(q.initialText!==undefined&&q.initialText!==""&&!/^[a-zA-Z0-9_.-]{1,120}$/.test(q.initialText))
|
|
||!chatCatalogSavedSecretNameIntent(q.prompt)
|
|
||secrets?.length!==1||!secret||secret.companyId!==input.companyId||secret.scope!=="company"||secret.status!=="active"||secret.createdByUserId!==b.userId
|
|
||!Number.isFinite(Date.parse(String(row.createdAt)))||!Number.isFinite(Date.parse(source?.startedAt??""))||Date.parse(secret.createdAt)<=Math.max(Date.parse(String(row.createdAt)),Date.parse(source?.startedAt??""))
|
|
||secret.key!==savedName.value||!/^[a-zA-Z0-9_.-]{1,120}$/.test(secret.key)||!Number.isFinite(Date.parse(secret.createdAt))||Date.parse(secret.createdAt)>Date.parse(String(row.resolvedAt))
|
|
||answer?.questionId!==q.id||answer.optionIds.length!==0||answer.otherText!==secret.key||createHash("sha256").update(JSON.stringify(row.payload)).digest("hex")!==savedName.payloadHash)return;
|
|
} else if (row.result.answers.some(answer => !!answer.otherText)) return;
|
|
const resolvedAt = Date.parse(String(row.resolvedAt));
|
|
const audits = input.activity.filter(event => event.details?.interactionId === b.interactionId && event.action === "issue.thread_interaction_answered");
|
|
const audit = audits[0];
|
|
if (!Number.isFinite(resolvedAt) || audits.length !== 1 || !audit || audit.companyId !== input.companyId || audit.entityId !== input.issueId
|
|
|| audit.actorType !== "user" || audit.actorId !== b.userId || audit.details?.interactionKind !== "ask_user_questions" || audit.details.interactionStatus !== "answered"
|
|
|| ((environmentName||savedName) && audit.details.answeredQuestionCount !== 1)) return;
|
|
const continuations = input.runs.filter(run => run.contextSnapshot?.interactionId === b.interactionId), run = continuations[0];
|
|
if (!source || source.companyId !== input.companyId || source.agentId !== input.agentId || source.runtimeMode !== "native" || source.status !== "succeeded"
|
|
|| source.contextSnapshot?.issueId !== input.issueId || source.contextSnapshot.commentId !== input.canonicalComment.id
|
|
|| !Number.isFinite(Date.parse(source.startedAt ?? "")) || Date.parse(source.startedAt!) > resolvedAt
|
|
|| continuations.length !== 1 || !run || run.id === source.id || run.companyId !== input.companyId || run.agentId !== input.agentId
|
|
|| run.runtimeMode !== "native" || run.status !== "succeeded" || run.contextSnapshot?.issueId !== input.issueId
|
|
|| !Number.isFinite(Date.parse(run.startedAt ?? "")) || Date.parse(run.startedAt!) < resolvedAt) return;
|
|
return { ...b, continuationRunId: run.id, resolvedAt: new Date(resolvedAt).toISOString() };
|
|
}
|
|
|
|
export interface ChatCatalogExpectedCompanySecretBindingApproval {
|
|
interactionId:string;proposalId:string;sourceRunId:string;initialRunId:string;userId:string;secretId:string;targetAgentId:string;configPath:string;payloadHash:string;humanInputId:string;humanInputKind:"comment"|"question"|"confirmation";humanInputPayloadHash?:string;humanInputSecretKey?:string;humanInputContinuationPolicy?:"wake_assignee"|"wake_assignee_on_accept";
|
|
}
|
|
export interface ChatCatalogVerifiedCompanySecretBindingApproval extends ChatCatalogExpectedCompanySecretBindingApproval {continuationRunId:string;resolvedAt:string}
|
|
export function verifyChatCatalogCompanySecretBindingApproval(input:{
|
|
scenarioId:string;expected:ChatCatalogExpectedCompanySecretBindingApproval;companyId:string;issueId:string;agentId:string;
|
|
interactions:Array<Partial<RequestConfirmationInteraction|AskUserQuestionsInteraction>>;proposals:Array<Partial<import("../../packages/shared/src/types/secrets.js").SecretProposalView>>;activity:Array<ChatCatalogHumanQuestionAudit&{createdAt?:string}>;
|
|
comments:Array<{id:string;companyId?:string;issueId?:string;authorUserId?:string;authorAgentId?:string;createdByRunId?:string;createdAt:string}>;
|
|
secrets:Array<{id:string;companyId:string;scope:string;status:string;key?:string;createdByUserId?:string|null;createdAt:string}>;
|
|
bindings:Array<{secretId:string;targetType:string;targetId:string;configPath:string;versionSelector:string|number}>;
|
|
runs:Array<{id:string;companyId?:string;agentId:string;status:string;runtimeMode?:string;startedAt?:string|null;contextSnapshot?:Record<string,unknown>|null}>;
|
|
}):ChatCatalogVerifiedCompanySecretBindingApproval|undefined {
|
|
const b=input.expected,row=input.interactions.find(q=>q.id===b.interactionId) as Partial<RequestConfirmationInteraction>|undefined,proposal=input.proposals.find(q=>q.id===b.proposalId),payload=row?.payload?.secretProposal;
|
|
const first=input.runs.find(r=>r.id===b.initialRunId),source=input.runs.find(r=>r.id===b.sourceRunId),secret=input.secrets.find(r=>r.id===b.secretId);
|
|
const human=b.humanInputKind==="comment"?input.comments.find(r=>r.id===b.humanInputId):undefined,question=b.humanInputKind!=="comment"?input.interactions.find(r=>r.id===b.humanInputId):undefined;
|
|
const inputAt=Date.parse(String(human?.createdAt??question?.resolvedAt)),resolvedAt=Date.parse(String(row?.resolvedAt));
|
|
const initialHuman=input.comments.find(c=>c.id===first?.contextSnapshot?.commentId),sourceAt=Date.parse(String(source?.startedAt)),cardAt=Date.parse(String(row?.createdAt)),firstAt=Date.parse(String(first?.startedAt));
|
|
if(input.scenarioId!=="secrets.configure-company-secret"||![sourceAt,cardAt,firstAt,Date.parse(String(payload?.expiresAt))].every(Number.isFinite)||!requestConfirmationPayloadSchema.safeParse(row?.payload).success||!row||input.interactions.filter(q=>q.id===row.id).length!==1||row.kind!=="request_confirmation"||row.status!=="accepted"
|
|
||row.companyId!==input.companyId||row.issueId!==input.issueId||row.sourceRunId!==b.sourceRunId||row.createdByAgentId!==input.agentId||row.createdByUserId||row.addresseeAgentId||(row.addresseeUserId&&row.addresseeUserId!==b.userId)
|
|
||row.requestedResolverPolicy!=="human_only"||row.effectiveResolverPolicy!=="human_only"||row.continuationPolicy!=="wake_assignee"||row.idempotencyKey!==`secret-proposal:${b.proposalId}`
|
|
||row.resolvedByUserId!==b.userId||row.resolvedByAgentId||row.result?.outcome!=="accepted"||row.result?.secretProposal?.status!=="executed"||row.result.secretProposal.errorCode
|
|
||!payload||payload.proposalId!==b.proposalId||payload.targetAgentId!==b.targetAgentId||payload.configPath!==b.configPath||payload.targetAgentName!=="Morgan"||!/^env\.[_A-Za-z][_A-Za-z0-9]*$/.test(b.configPath)
|
|
||createHash("sha256").update(JSON.stringify(row.payload)).digest("hex")!==b.payloadHash||!Number.isFinite(resolvedAt)||resolvedAt<Date.parse(String(row.createdAt))||resolvedAt>Date.parse(payload.expiresAt)
|
|
||!proposal||input.proposals.filter(p=>p.id===b.proposalId).length!==1||proposal.companyId!==input.companyId||proposal.kind!=="binding"||proposal.status!=="approved"||proposal.secretId!==b.secretId||proposal.secretProposalId
|
|
||proposal.targetType!=="agent"||proposal.target?.id!==b.targetAgentId||proposal.configPath!==b.configPath||proposal.appliedBindingConfigPath!==b.configPath||proposal.originIssue?.id!==input.issueId||proposal.originRunId!==b.sourceRunId||proposal.proposedBy?.id!==input.agentId||proposal.resolvedByUserId!==b.userId
|
|
||!initialHuman||initialHuman.authorUserId!==b.userId||initialHuman.authorAgentId||initialHuman.createdByRunId||initialHuman.companyId!==input.companyId||initialHuman.issueId!==input.issueId||!first||first.companyId!==input.companyId||first.agentId!==input.agentId||first.status!=="succeeded"||first.runtimeMode!=="native"||first.contextSnapshot?.issueId!==input.issueId
|
|
||!secret||secret.companyId!==input.companyId||secret.scope!=="company"||secret.status!=="active"||secret.createdByUserId!==b.userId||!first||Date.parse(secret.createdAt)<=Date.parse(String(first.startedAt))||Date.parse(secret.createdAt)>=inputAt
|
|
||!Number.isFinite(inputAt)||(b.humanInputKind==="confirmation"?!verifyChatCatalogHumanQuestionAnswer({expected:{interactionId:b.humanInputId,sourceRunId:b.initialRunId,userId:b.userId,companySecretConfirmation:{secretId:b.secretId,key:b.humanInputSecretKey??"",payloadHash:b.humanInputPayloadHash??"",continuationPolicy:b.humanInputContinuationPolicy!}},companyId:input.companyId,issueId:input.issueId,agentId:input.agentId,canonicalComment:initialHuman!,interactions:input.interactions as Array<Partial<AskUserQuestionsInteraction>>,activity:input.activity,companySecrets:input.secrets.map(secret=>({...secret,key:secret.key??""})),runs:input.runs}):(human?(human.companyId!==input.companyId||human.issueId!==input.issueId||human.authorUserId!==b.userId||!!human.authorAgentId||!!human.createdByRunId):!question||question.kind!=="ask_user_questions"||question.sourceRunId!==b.initialRunId||question.createdByAgentId!==input.agentId||question.status!=="answered"||question.resolvedByUserId!==b.userId||question.companyId!==input.companyId||question.issueId!==input.issueId))
|
|
||!source||source.id===first.id||source.companyId!==input.companyId||source.agentId!==input.agentId||source.runtimeMode!=="native"||source.status!=="succeeded"||source.contextSnapshot?.issueId!==input.issueId
|
|
||source.contextSnapshot[b.humanInputKind==="comment"?"commentId":"interactionId"]!==b.humanInputId||Date.parse(String(source.startedAt))<inputAt||Date.parse(String(row.createdAt))<Date.parse(String(source.startedAt)))return;
|
|
const continuations=input.runs.filter(r=>r.contextSnapshot?.interactionId===b.interactionId),run=continuations[0];
|
|
const accepted=input.activity.filter(a=>a.action==="issue.thread_interaction_accepted"&&a.details?.interactionId===b.interactionId);
|
|
const approved=input.activity.filter(a=>a.action==="secret.proposal.approved"&&a.entityId===b.proposalId);
|
|
const mutations=input.activity.filter(a=>/^agent\.(?:updated|config_updated|created|deleted)$/.test(a.action??"")&&Date.parse(String(a.createdAt))>=inputAt);
|
|
if(!mutations.length||mutations.some(a=>a.companyId!==input.companyId||a.entityId!==b.targetAgentId||a.actorType!=="user"||a.actorId!==b.userId||a.details?.proposalId!==b.proposalId||a.details?.configPath!==b.configPath||Date.parse(String(a.createdAt))<resolvedAt)
|
|
||input.runs.length!==3||continuations.length!==1||!run||run.id===source.id||run.companyId!==input.companyId||run.agentId!==input.agentId||run.runtimeMode!=="native"||run.status!=="succeeded"||run.contextSnapshot?.issueId!==input.issueId||Date.parse(String(run.startedAt))<resolvedAt
|
|
||accepted.length!==1||accepted[0]?.companyId!==input.companyId||accepted[0].entityId!==input.issueId||accepted[0].actorType!=="user"||accepted[0].actorId!==b.userId
|
|
||approved.length!==1||approved[0]?.companyId!==input.companyId||approved[0].actorType!=="user"||approved[0].actorId!==b.userId||approved[0].details?.appliedBindingConfigPath!==b.configPath
|
|
||input.bindings.length!==1||input.bindings[0]?.secretId!==b.secretId||input.bindings[0].targetType!=="agent"||input.bindings[0].targetId!==b.targetAgentId||input.bindings[0].configPath!==b.configPath||input.bindings[0].versionSelector!=="latest")return;
|
|
return {...b,continuationRunId:run.id,resolvedAt:String(row.resolvedAt)};
|
|
}
|
|
|
|
export async function collectChatCatalogCompanySecretApprovalActivity(api:Pick<ChatFlowInput["api"],"get">,companyId:string,issueId:string){
|
|
const [issue,company]=await Promise.all([api.get<Array<ChatCatalogHumanQuestionAudit&{id:string;createdAt:string}>>(`/api/issues/${issueId}/activity`),api.get<Array<ChatCatalogHumanQuestionAudit&{id:string;createdAt:string}>>(`/api/companies/${companyId}/activity?limit=1000`)]);
|
|
const rows=new Map<string,ChatCatalogHumanQuestionAudit&{id:string;createdAt:string}>();
|
|
for(const row of [...issue,...company]){
|
|
if(!row.id||row.companyId!==companyId)throw new Error("Company approval audit scope mismatch");
|
|
const before=rows.get(row.id),project=(v:typeof row)=>({companyId:v.companyId,entityId:v.entityId,action:v.action,actorType:v.actorType,actorId:v.actorId,createdAt:v.createdAt,details:v.details});
|
|
if(before&&!isDeepStrictEqual(project(before),project(row)))throw new Error("Company approval audit identity changed");
|
|
rows.set(row.id,row);
|
|
}
|
|
return [...rows.values()];
|
|
}
|