mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
fix(evals): name stale SDK pins and fix four chat-eval harness defects (#20579)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat feature eval catalog (`tests/runner-e2e`) runs real chat journeys against a real server and grades the saved state, the run's tool events and the rendered reply > - The full-cohort run on master `f4337bb5fb` had 12 failures that come from the harness, not from the product or the model > - Six plugin and secrets cells stopped in 7–9 s, before the server started, because the SDK public source pins were stale > - Four more cells failed on harness timing, a first-visit prompt, or a read receipt that ignored a dedicated tool > - This pull request fixes those four harness defects and adds a cheap guard that names stale SDK pins > - The benefit is that these cells now report on the product and the model again ## Linked Issues or Issue Description Refs #19614, #20550, #20453, #20417, #20483 **What happened** On master `f4337bb5fb`, these chat eval cells failed for harness reasons: - `plugins.configure-plugin`, `plugins.disable-plugin`, `plugins.install-plugin`, `plugins.use-plugin-tool`, `routines.plugin-managed-routine` and `secrets.least-scope-delivery` stopped before the server started. The source SDK privacy scan refused, because `tests/runner-e2e/source-sdk-public-inputs.json` did not match `packages/shared/src`. #20550 and #20362 already resynced the pins on master. The pins are current at `ce1ee9e89f`. - `agent-chat.track-delegated-work` and `navigation-preferences.edit-profile`: the reply was correct and had the right link. The case read the rendered links immediately after the runs settled, before the reply rendered, so it recorded no links. - `inbox-search.archive-done` and `inbox-search.restore-archive`: setup completes five tasks, so the GitHub star prompt from #20453 opened over the chat and blocked the composer. - `status-cards.explain-card`: the follow-up run read the deleted source with `get_task` and received a 404. The reply said so. `apiReadStatuses` counted only `call_api` reads, so the grader recorded no unavailable read. Why the existing pin check did not stop the drift: `ci / Runner support suite` runs the source SDK tests and is part of the required `ci / verify`. On #19614 it passed at 03:43Z on a merge ref from before #20417 (06:12Z) and #20483 (08:14Z). #19614 merged at 14:02Z without a new CI run on current master. After that, the suite failed on every PR until #20550. So the check exists and fires; the gap was a stale green result, and the failure message was opaque (`SDK public source proof unavailable`). **Expected behavior** - A change to a pinned SDK input without a pin refresh fails with the changed files and the command that fixes it. - Cases read rendered links only after the saved reply renders. - The eval browser is a returning user's browser that has already seen the star prompt. The product does not change. - A dedicated task read tool (`get_task`, `list_task_documents`, `read_task_document`, also when namespaced) counts as a read of the record it names, with the status its receipt reports. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.<case>` for one of the cases above on master `f4337bb5fb`. 2. Read `result.json` and `snapshots/chat-catalog-observation.json` in the evidence bundle. 3. See the failures described above. ## What Changed - `tests/runner-e2e/source-sdk-public-inputs.mjs`: add `staleSdkPublicInputs(root)`. It compares the registry with `git ls-tree HEAD` for every pinned input (the roots and build inputs that the pins file lists), and returns each changed, unpinned or removed file. - `tests/runner-e2e/source-sdk-public-inputs.test.mjs`: a test fails when the committed pins do not match the committed tree. The message names the files and gives the `--write` command. The Runner support suite runs it for every `packages/**` change. A synthetic test covers changed, unpinned and removed inputs. - `tests/runner-e2e/chat-catalog-domains/runtime-handoffs.ts` and `identity-profile.ts`: call the existing `waitForRenderedReply` before they read rendered links. - `tests/runner-e2e/chat-returning-browser.ts` (new) and `chat-catalog-flow.ts`: inside the flow's failure boundary, before setup, mark `paperclip.github-star-prompt.seen` in the eval browser. An init script covers later documents of the app origin, and the open page is marked directly. A test checks that the key equals `STAR_PROMPT_SEEN_KEY` in `ui/src/components/GitHubStarPrompt.tsx`. - `tests/runner-e2e/chat-catalog-domains/api-read-receipts.ts`: bind dedicated task read tools by the digest of the `call_api` input that the server builds for them (`taskBridgeApiRequest`). The result must carry the same `apiOperationId`. Other tools, other records, mismatched receipts and unpaired results do not count. - Bump `oracleVersion` in code and descriptor for every case that uses `api-read-receipts.ts` (shared grader change) and for `navigation-preferences.edit-profile`. Add `internal-comment-reference.ts` to the source files of the two cases that now import it. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` (whole config): 11301 passed; 1 local-only failure in `controller-replacement-smoke.test.ts` (server spawn, no changed imports). - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts --testTimeout=180000 tests/runner-e2e/chat-catalog-domains tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts tests/runner-e2e/chat-returning-browser.test.ts` - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts source-sdk` - `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --check` reports `changed: false`. - Red checks: with the `f4337bb5fb` registry, the new pin test fails and names 15 inputs. Without the new waits, the two rendered-reply tests fail. A regrade of the retained `explain-card` run events now gives the follow-up run `unavailable: true, live: false` and the first run `live: true`. - Paid runs of the affected cells (`chat-features.runner-opencode.local.*`): - Run 42852698879 (head `d54c04d5f0`): `track-delegated-work`, `restore-archive`, `explain-card`, `configure-plugin` and `plugin-managed-routine` pass. I read each reply and saved state. `track-delegated-work` now records the rendered `/COM/issues/COM-2` link. `explain-card` records `laterUnavailable: true, laterLive: false` from the follow-up `get_task` 404. - `plugins.install-plugin` now starts its server (the pin refusal is gone). It fails a later grading check, `truthful-installation-handoff`: the reply sends the person to the plugin settings page but does not name the package. This is outside this PR. - `inbox-search.archive-done`: the star prompt no longer blocks the composer, and the query is sent. The agent then archives the 501 Done tasks one at a time (about 2 s each, 57 in the 240 s window), because only the per-task archive route exists. This is outside this PR. - `navigation-preferences.edit-profile` did not start in that run (the AWS credentials step timed out). Run 42857020103 (head `2f0b80aa84`): pass. The agent write gets 401, the reply links `/COM/company/settings/instance/profile`, and the person follows the rendered link and saves the name. ## Risks - Low risk. Test harness only; no product code changes. - The receipt change counts more reads as record reads. It only adds dedicated reads bound to the exact record id and operation, so a grader that needs a read can pass on a genuine `get_task` read. The affected cases have new oracle versions. - The star prompt is not tested by any chat catalog case. Product e2e tests for the prompt do not use this flow. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file editing, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed
1 parent
84007892e9
commit
8e332ccb5e
21 files changed
+270
-35
No files matched your search
@@ -209,7 +209,7 @@
|
||||
"../e2e/fetch-allowed-port.ts",
|
||||
"run-observations.ts"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.run-history.v3",
|
||||
"oracleVersion": "paperclip.chat-catalog.run-history.v4",
|
||||
"coverageGaps": [],
|
||||
"expectedRunCount": 3,
|
||||
"minimumRunCount": 3,
|
||||
@@ -263,7 +263,7 @@
|
||||
"../../server/src/services/work-timeline.ts",
|
||||
"../../server/src/services/native-runtime/runner-api-catalog.ts"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.work-timeline.v15",
|
||||
"oracleVersion": "paperclip.chat-catalog.work-timeline.v16",
|
||||
"coverageGaps": [],
|
||||
"expectedRunCount": 3,
|
||||
"minimumRunCount": 3,
|
||||
@@ -380,6 +380,7 @@
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/runtime-handoffs.ts",
|
||||
"chat-catalog-domains/api-read-receipts.ts",
|
||||
"chat-catalog-domains/internal-comment-reference.ts",
|
||||
"chat-catalog-domains/runtime-busy-worker.ts",
|
||||
"chat-catalog-domains/runtime-support.ts",
|
||||
"completion-update-flow.ts",
|
||||
@@ -387,7 +388,7 @@
|
||||
"run-observations.ts",
|
||||
"chat-scenarios/agent-chat.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.track-delegated-work.v2",
|
||||
"oracleVersion": "paperclip.chat-catalog.track-delegated-work.v3",
|
||||
"coverageGaps": [],
|
||||
"expectedRunCount": 4,
|
||||
"setupHumanMessageCount": 1,
|
||||
@@ -3605,6 +3606,7 @@
|
||||
"scenarioId": "navigation-preferences.edit-profile",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/identity-profile.ts",
|
||||
"chat-catalog-domains/internal-comment-reference.ts",
|
||||
"chat-catalog-domains/identity-auth.ts",
|
||||
"chat-catalog-domains/identity-private-browser.ts",
|
||||
"chat-catalog-domains/identity-private-browser-worker.ts",
|
||||
@@ -3613,7 +3615,7 @@
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/navigation-preferences.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.identity-profile.v2",
|
||||
"oracleVersion": "paperclip.chat-catalog.identity-profile.v3",
|
||||
"expectedRunCount": 1,
|
||||
"turnTimeoutMs": 120000,
|
||||
"attemptTimeoutMs": 300000,
|
||||
@@ -4378,7 +4380,7 @@
|
||||
"../../server/src/vendor/paperclip-runner/index.ts",
|
||||
"chat-scenarios/recovery.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.runtime.infrastructure-recovery.v4",
|
||||
"oracleVersion": "paperclip.chat-catalog.runtime.infrastructure-recovery.v5",
|
||||
"coverageGaps": [],
|
||||
"expectedRunCount": 2,
|
||||
"turnTimeoutMs": 120000,
|
||||
@@ -4457,7 +4459,7 @@
|
||||
"../../server/src/vendor/paperclip-runner/index.ts",
|
||||
"chat-scenarios/recovery.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.runtime.retry-exhausted.v4",
|
||||
"oracleVersion": "paperclip.chat-catalog.runtime.retry-exhausted.v5",
|
||||
"coverageGaps": [],
|
||||
"expectedRunCount": 5,
|
||||
"expectedHumanMessageCount": 3,
|
||||
@@ -4494,7 +4496,7 @@
|
||||
"../../server/src/vendor/paperclip-runner/index.ts",
|
||||
"chat-scenarios/recovery.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.runtime.infrastructure-recovery.v4",
|
||||
"oracleVersion": "paperclip.chat-catalog.runtime.infrastructure-recovery.v5",
|
||||
"coverageGaps": [],
|
||||
"expectedRunCount": 2,
|
||||
"turnTimeoutMs": 120000,
|
||||
@@ -4716,7 +4718,7 @@
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/routines.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.integrator.routine-invocation.v3",
|
||||
"oracleVersion": "paperclip.chat-catalog.integrator.routine-invocation.v4",
|
||||
"expectedRunCount": 3,
|
||||
"expectedHumanMessageCount": 2,
|
||||
"turnTimeoutMs": 120000,
|
||||
@@ -4763,7 +4765,7 @@
|
||||
"../../ui/src/components/task-chat/TaskChatThreadView.tsx",
|
||||
"../../ui/src/pages/AgentDetail.tsx"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.runtime.inspect-failed-run.v4",
|
||||
"oracleVersion": "paperclip.chat-catalog.runtime.inspect-failed-run.v5",
|
||||
"coverageGaps": [],
|
||||
"expectedRunCount": 3,
|
||||
"turnTimeoutMs": 120000,
|
||||
@@ -5337,7 +5339,7 @@
|
||||
"run-observations.ts",
|
||||
"chat-scenarios/status-cards.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.status-card.explain.v4",
|
||||
"oracleVersion": "paperclip.chat-catalog.status-card.explain.v5",
|
||||
"expectedRunCount": 5,
|
||||
"minimumRunCount": 3,
|
||||
"expectedHumanMessageCount": 2,
|
||||
@@ -5384,7 +5386,7 @@
|
||||
"chat-catalog-domains/integrator-support.ts",
|
||||
"chat-scenarios/status-cards.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.status-card.required-work-failure.v6",
|
||||
"oracleVersion": "paperclip.chat-catalog.status-card.required-work-failure.v7",
|
||||
"expectedRunCount": 7,
|
||||
"expectedHumanMessageCount": 4,
|
||||
"turnTimeoutMs": 120000,
|
||||
@@ -5641,7 +5643,7 @@
|
||||
"run-observations.ts",
|
||||
"chat-scenarios/task-coordination.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.runtime.block-on-design.v4",
|
||||
"oracleVersion": "paperclip.chat-catalog.runtime.block-on-design.v5",
|
||||
"coverageGaps": [],
|
||||
"expectedRunCount": 3,
|
||||
"expectedHumanMessageCount": 2,
|
||||
|
||||
@@ -1,17 +1,18 @@
|
||||
import { redactRunnerApiEvent } from "../../../server/src/services/native-runtime/runner-api-log-redaction.js";
|
||||
|
||||
/**
|
||||
* One call_api tool use and its result, as the runner emits them, passed through
|
||||
* the same redactor PaperclipControlPlanePort applies before saving run events.
|
||||
* One call_api (or dedicated API-backed tool) use and its result, as the runner
|
||||
* emits them, passed through the same redactor PaperclipControlPlanePort applies
|
||||
* before saving run events.
|
||||
*/
|
||||
export function savedCall(input: Record<string, unknown>, status: number, itemId = "7", runId = "run-1", data: unknown = { id: "response-body" }): Record<string, unknown>[] {
|
||||
export function savedCall(input: Record<string, unknown>, status: number, itemId = "7", runId = "run-1", data: unknown = { id: "response-body" }, tool = "call_api", operationId = input.operationId): Record<string, unknown>[] {
|
||||
const row = (eventType: string, item: Record<string, unknown>) => {
|
||||
const event = redactRunnerApiEvent({ eventType, sourceKind: "runner", payload: { item } });
|
||||
return { runId, eventType, payload: { prpEvent: { runId, sourceKind: "runner", payload: event.payload } } };
|
||||
};
|
||||
return [
|
||||
row("item.started", { id: itemId, name: "call_api", type: "tool_use", input }),
|
||||
row("item.started", { id: itemId, name: tool, type: "tool_use", input }),
|
||||
row("item.completed", { id: itemId, type: "tool_result", tool_use_id: itemId,
|
||||
result: { ok: status >= 200 && status < 300, status, contentType: "application/json; charset=utf-8", apiOperationId: input.operationId, data } }),
|
||||
result: { ok: status >= 200 && status < 300, status, contentType: "application/json; charset=utf-8", apiOperationId: operationId, data } }),
|
||||
];
|
||||
}
|
||||
@@ -29,3 +29,30 @@ it("rejects other records, other operations, failures and unpaired results", ()
|
||||
it("reports the status of a bound read so a 404 can prove an unavailable record", () => {
|
||||
expect(apiReadStatuses(savedCall(activity({ id: "task-uuid" }), 404), apiReadDigests(reads))).toEqual([{ operationId: "GET /api/issues/{id}/activity", ok: false, status: 404 }]);
|
||||
});
|
||||
|
||||
const getTask = (taskId: string, status: number, tool = "get_task", runId = "run-1") => savedCall({ taskId }, status, "2", runId, { id: taskId }, tool, "GET /api/issues/{id}");
|
||||
const issueRead = [{ operationId: "GET /api/issues/{id}", pathParams: recordParams("id", "task-uuid", "COM-1") }];
|
||||
|
||||
it("the redactor keeps a dedicated get_task input in plaintext and its result as a receipt", () => {
|
||||
const [use, result] = getTask("task-uuid", 404);
|
||||
expect(JSON.stringify(use)).toContain("task-uuid");
|
||||
expect(JSON.stringify(result)).not.toContain("task-uuid");
|
||||
expect(JSON.stringify(result)).toContain("apiPayloadSha256");
|
||||
});
|
||||
|
||||
it("binds dedicated task read tools to the record like the call_api input the server builds", () => {
|
||||
expect(apiReadStatuses(getTask("task-uuid", 404), apiReadDigests(issueRead))).toEqual([{ operationId: "GET /api/issues/{id}", ok: false, status: 404 }]);
|
||||
expect(readRecord(getTask("COM-1", 200), issueRead)).toBe(true);
|
||||
for (const tool of ["paperclip_get_task", "mcp__paperclip__get_task"]) expect(readRecord(getTask("task-uuid", 200, tool), issueRead)).toBe(true);
|
||||
const docs = [{ operationId: "GET /api/issues/{id}/documents/{key}", pathParams: [{ id: "task-uuid", key: "plan" }] }];
|
||||
expect(readRecord(savedCall({ taskId: "task-uuid", key: "plan" }, 200, "3", "run-1", {}, "read_task_document", "GET /api/issues/{id}/documents/{key}"), docs)).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects dedicated reads of other records, other tools, mismatched receipts and unpaired results", () => {
|
||||
expect(readRecord(getTask("other-task", 200), issueRead)).toBe(false);
|
||||
expect(readRecord(getTask("task-uuid", 200, "comment_on_task"), issueRead)).toBe(false);
|
||||
expect(readRecord(getTask("task-uuid", 200, "get_task_context"), issueRead)).toBe(false);
|
||||
expect(readRecord(savedCall({ taskId: "task-uuid" }, 200, "2", "run-1", {}, "get_task", "GET /api/issues/{id}/comments"), issueRead)).toBe(false);
|
||||
expect(readRecord(getTask("task-uuid", 200).slice(0, 1), issueRead)).toBe(false);
|
||||
expect(readRecord([...getTask("task-uuid", 200, "get_task", "run-a").slice(0, 1), ...getTask("task-uuid", 200, "get_task", "run-b").slice(1)], issueRead)).toBe(false);
|
||||
});
|
||||
@@ -49,8 +49,32 @@ export function recordParams(name: string, ...values: Array<string | null | unde
|
||||
}
|
||||
|
||||
/**
|
||||
* Statuses of this run's call_api results whose saved input digest is one of
|
||||
* `digests`, paired to their tool_use by run and item id.
|
||||
* Dedicated task read tools and the call_api input the server builds for them
|
||||
* (taskBridgeApiRequest in paperclip-runner-tool-authority.ts). Their saved
|
||||
* tool_use keeps the plaintext taskId, while the result carries the same
|
||||
* redacted receipt (apiOperationId, ok, status) as call_api. Harnesses may
|
||||
* namespace the name (paperclip_get_task, mcp__paperclip__get_task).
|
||||
*/
|
||||
const TASK_READ_TOOLS: Readonly<Record<string, string>> = {
|
||||
get_task: "GET /api/issues/{id}",
|
||||
list_task_documents: "GET /api/issues/{id}/documents",
|
||||
read_task_document: "GET /api/issues/{id}/documents/{key}",
|
||||
};
|
||||
function taskReadTool(name: unknown): string | undefined {
|
||||
const match = typeof name === "string" ? /^(?:[A-Za-z0-9_-]*__)?(?:paperclip_)?([a-z_]+)$/.exec(name) : null;
|
||||
return match && Object.hasOwn(TASK_READ_TOOLS, match[1]!) ? match[1] : undefined;
|
||||
}
|
||||
function taskReadInput(tool: string, input: Record<string, unknown>): Record<string, unknown> | undefined {
|
||||
if (typeof input.taskId !== "string" || !input.taskId) return;
|
||||
if (tool === "read_task_document" && (typeof input.key !== "string" || !input.key)) return;
|
||||
return { operationId: TASK_READ_TOOLS[tool], pathParams: { id: input.taskId, ...(tool === "read_task_document" ? { key: input.key } : {}) } };
|
||||
}
|
||||
|
||||
/**
|
||||
* Statuses of this run's record reads whose input digest is one of `digests`,
|
||||
* paired to their tool_use by run and item id: call_api uses by their saved
|
||||
* digest, and dedicated task read tools by the digest of the call_api input
|
||||
* the server builds from their plaintext input.
|
||||
*/
|
||||
export function apiReadStatuses(events: readonly Record<string, unknown>[], digests: ReadonlySet<string>): Array<{ operationId: string; ok: boolean; status: number }> {
|
||||
const uses = new Map<string, string>();
|
||||
@@ -63,6 +87,12 @@ export function apiReadStatuses(events: readonly Record<string, unknown>[], dige
|
||||
if (input.apiPayloadRedacted === true && typeof input.operationId === "string" && digests.has(String(input.apiPayloadSha256))) uses.set(`${run}:${String(item.id)}`, input.operationId);
|
||||
continue;
|
||||
}
|
||||
const tool = item.type === "tool_use" ? taskReadTool(item.name) : undefined;
|
||||
if (tool) {
|
||||
const request = taskReadInput(tool, objectRecord(item.input));
|
||||
if (request && digests.has(callApiInputDigest(request))) uses.set(`${run}:${String(item.id)}`, String(request.operationId));
|
||||
continue;
|
||||
}
|
||||
if (item.type !== "tool_result") continue;
|
||||
const operationId = uses.get(`${run}:${String(item.tool_use_id)}`), result = objectRecord(item.result);
|
||||
if (!operationId || result.apiOperationId !== operationId || typeof result.status !== "number") continue;
|
||||
|
||||
@@ -4,6 +4,7 @@ import type { ChatCatalogCase, ChatCatalogContext } from "../chat-catalog-contra
|
||||
import { checks } from "./internal-support.js";
|
||||
import { createAuthenticatedPerson, enrollPrerequisitePerson } from "./identity-auth.js";
|
||||
import { identityDefinition, identityObservation, observeIdentity, probeAgentPermission, waitForReply } from "./identity-support.js";
|
||||
import { waitForRenderedReply } from "./internal-comment-reference.js";
|
||||
|
||||
interface Session { user: { id: string; name: string; image?: string | null } }
|
||||
type Person = Awaited<ReturnType<typeof createAuthenticatedPerson>>;
|
||||
@@ -49,10 +50,13 @@ export const profileIdentityCases: readonly ChatCatalogCase[] = [identityDefinit
|
||||
} catch (error) { await observer.dispose(); observers.delete(ctx); throw error; }
|
||||
}, gradeProfile, {
|
||||
authentication: "human-session",
|
||||
oracleVersion: "paperclip.chat-catalog.identity-profile.v2",
|
||||
sourceFiles: ["chat-catalog-domains/identity-profile.ts", "chat-catalog-domains/identity-auth.ts", "chat-catalog-domains/identity-private-browser.ts", "chat-catalog-domains/identity-private-browser-worker.ts", "chat-catalog-domains/identity-wizard.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", "chat-scenarios/navigation-preferences.yaml"],
|
||||
oracleVersion: "paperclip.chat-catalog.identity-profile.v3",
|
||||
sourceFiles: ["chat-catalog-domains/identity-profile.ts", "chat-catalog-domains/internal-comment-reference.ts", "chat-catalog-domains/identity-auth.ts", "chat-catalog-domains/identity-private-browser.ts", "chat-catalog-domains/identity-private-browser-worker.ts", "chat-catalog-domains/identity-wizard.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", "chat-scenarios/navigation-preferences.yaml"],
|
||||
async act(ctx, state) {
|
||||
await waitForReply(ctx, chatCatalogOriginalPhaseDeadline(ctx)); state.beforeHumanName = (await ctx.input.api.get<Session>("/api/auth/get-session")).user.name;
|
||||
await waitForReply(ctx, chatCatalogOriginalPhaseDeadline(ctx));
|
||||
// Run settle precedes the reply's render; look for the rendered link only once the saved reply is on the page.
|
||||
await waitForRenderedReply(ctx, { leadId: ctx.input.fixtures.agent.id, runs: await ctx.allRuns() });
|
||||
state.beforeHumanName = (await ctx.input.api.get<Session>("/api/auth/get-session")).user.name;
|
||||
state.agentWriteStatus = await probeAgentPermission(ctx, "/api/auth/profile", "PATCH", { name: "Alex Chen" });
|
||||
const source = ctx.input.page.url();
|
||||
// The person follows the agent's rendered link when there is one; otherwise the journey still saves the name so the remaining checks report.
|
||||
|
||||
@@ -110,7 +110,7 @@ export const integratorRoutineInvocationCases = [integratorDefinition("routines"
|
||||
if (!o || !task) throw new Error("Missing actual routine result task");
|
||||
await inspectResult(ctx, `/${ctx.input.fixtures.company.issuePrefix}/issues/${task.identifier ?? task.id}`, task.title!, resultHref(o, task.identifier ?? task.id, "issues") ?? resultHref(o, task.id, "issues"));
|
||||
}, {
|
||||
oracleVersion: "paperclip.chat-catalog.integrator.routine-invocation.v3",
|
||||
oracleVersion: "paperclip.chat-catalog.integrator.routine-invocation.v4",
|
||||
sourceFiles: ["chat-catalog-domains/integrator-routine-invocation.ts", "chat-catalog-domains/api-read-receipts.ts", "run-observations.ts", "chat-catalog-domains/integrator-support.ts", "chat-catalog-domains/internal-support.ts", "chat-scenarios/routines.yaml"],
|
||||
expectedRunCount: 3, expectedHumanMessageCount: 2, attemptTimeoutMs: 720_000,
|
||||
async act(ctx, state) {
|
||||
|
||||
@@ -331,7 +331,7 @@ export const integratorStatusCardsCases: readonly ChatCatalogCase[] = [
|
||||
define("status-cards.create-card", "none", { oracleVersion: "paperclip.chat-catalog.status-card.create.v2" }),
|
||||
define("status-cards.archive-card", "ceo", { oracleVersion: "paperclip.chat-catalog.status-card.archive.v3", authentication: "human-session", queryAttachments: factualContextQueryAttachment("selected-status-card.md"), expectedRunCount: 3, attemptTimeoutMs: 720_000 }),
|
||||
define("status-cards.other-authors-card", "morgan", { oracleVersion: "paperclip.chat-catalog.status-card.other-authors.v2", expectedRunCount: 5, minimumRunCount: 2 }),
|
||||
define("status-cards.explain-card", "ceo", { oracleVersion: "paperclip.chat-catalog.status-card.explain.v4", authentication: "human-session", queryAttachments: factualContextQueryAttachment("selected-status-card.md"), expectedRunCount: 5, minimumRunCount: 3, expectedHumanMessageCount: 2, attemptTimeoutMs: 720_000,
|
||||
define("status-cards.explain-card", "ceo", { oracleVersion: "paperclip.chat-catalog.status-card.explain.v5", authentication: "human-session", queryAttachments: factualContextQueryAttachment("selected-status-card.md"), expectedRunCount: 5, minimumRunCount: 3, expectedHumanMessageCount: 2, attemptTimeoutMs: 720_000,
|
||||
sourceFiles: ["chat-catalog-domains/integrator-status-cards.ts", "chat-catalog-domains/api-read-receipts.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-definition.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/integrator-support.ts", "run-observations.ts", "chat-scenarios/status-cards.yaml"],
|
||||
async act(ctx, state) {
|
||||
// The lead may refresh its own stale card while answering; wait for that generation to settle
|
||||
@@ -356,7 +356,7 @@ export const integratorStatusCardsCases: readonly ChatCatalogCase[] = [
|
||||
}),
|
||||
define("status-cards.refresh-card", "ceo", { expectedRunCount: 7, expectedHumanMessageCount: 4, attemptTimeoutMs: 1_200_000,
|
||||
runPolicy: { kind: "terminal-counts", terminalStatusCounts: { succeeded: 7 } },
|
||||
oracleVersion: "paperclip.chat-catalog.status-card.required-work-failure.v6",
|
||||
oracleVersion: "paperclip.chat-catalog.status-card.required-work-failure.v7",
|
||||
sourceFiles: ["chat-catalog-domains/integrator-status-cards.ts", "chat-catalog-domains/api-read-receipts.ts", "chat-catalog-domains/status-card-work-failure.ts", "chat-catalog-domains/external-provider.ts", "chat-catalog-domains/runtime-support.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-definition.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/integrator-support.ts", "chat-scenarios/status-cards.yaml"],
|
||||
|
||||
async act(ctx, state) {
|
||||
|
||||
@@ -90,7 +90,7 @@ function findRuns(): ChatCatalogCase {
|
||||
await ctx.input.evidence("chat-catalog-real-interrupted-continuation.json", state); return state;
|
||||
} catch (error) { await gate.close(); gate = undefined; throw error; }
|
||||
}, gradeRunHistory);
|
||||
return { ...base, oracleVersion: "paperclip.chat-catalog.run-history.v3", expectedRunCount: 3, minimumRunCount: 3, attemptTimeoutMs: 660_000,
|
||||
return { ...base, oracleVersion: "paperclip.chat-catalog.run-history.v4", expectedRunCount: 3, minimumRunCount: 3, attemptTimeoutMs: 660_000,
|
||||
runPolicy: { kind: "terminal-counts", terminalStatusCounts: { cancelled: 1, succeeded: 2 } },
|
||||
sourceFiles: [...base.sourceFiles, "chat-catalog-domains/internal-run-history.ts", "chat-catalog-domains/api-read-receipts.ts", "chat-catalog-domains/internal-run-brief.ts", "../e2e/fetch-allowed-port.ts", "run-observations.ts"],
|
||||
async observe(ctx, state): Promise<RunHistory> {
|
||||
|
||||
@@ -89,7 +89,7 @@ it("admits the selected live timeline metadata without invoking fixture callback
|
||||
const spies = callbacks.map(key => vi.spyOn(implementation, key));
|
||||
try {
|
||||
expect(await validateChatCatalogConsistency([descriptor])).toEqual({ descriptorCount: 1, fixtureCallbacksInvoked: 0 });
|
||||
expect(descriptor.oracleVersion).toBe("paperclip.chat-catalog.work-timeline.v15");
|
||||
expect(descriptor.oracleVersion).toBe("paperclip.chat-catalog.work-timeline.v16");
|
||||
expect(descriptor.sourceFiles).toEqual(implementation.sourceFiles);
|
||||
expect(descriptor).toMatchObject({ expectedRunCount: 3, minimumRunCount: 3, attemptTimeoutMs: 720000 });
|
||||
for (const spy of spies) expect(spy).not.toHaveBeenCalled();
|
||||
|
||||
@@ -105,7 +105,7 @@ function timeline(): ChatCatalogCase {
|
||||
const state = { worker, repairWorker, project: p, task: second.t, prepTask, hiddenTask, firstRun, secondRun, blockedTask: first.t, blockedEventId: blockedEvent.id, handoffEventId: handoff.id, doneEventId: done.id, timezone, today, zoneLabel, personId: session.user.id, personName: session.user.name };
|
||||
await ctx.input.evidence("chat-catalog-real-launch-timeline.json", { ...state, verification: verifier.receipts, events }); return state;
|
||||
}, gradeWorkTimeline);
|
||||
return { ...base, oracleVersion: "paperclip.chat-catalog.work-timeline.v15", authentication: "human-session", expectedRunCount: 3, minimumRunCount: 3, attemptTimeoutMs: 720000,
|
||||
return { ...base, oracleVersion: "paperclip.chat-catalog.work-timeline.v16", authentication: "human-session", expectedRunCount: 3, minimumRunCount: 3, attemptTimeoutMs: 720000,
|
||||
sourceFiles: [...base.sourceFiles, "chat-catalog-domains/internal-work-timeline.ts", "chat-catalog-domains/api-read-receipts.ts", "chat-catalog-domains/internal-timeline-verifier.ts", "chat-catalog-domains/internal-privacy.ts", "chat-catalog-domains/identity-auth.ts", "chat-catalog-domains/internal-routine-audit.ts", "../e2e/fetch-allowed-port.ts", "../../server/src/services/native-runtime/status-decision-committer.ts", "../../server/src/services/work-timeline.ts","../../server/src/services/native-runtime/runner-api-catalog.ts"],
|
||||
async observe(ctx, state): Promise<Timeline> {
|
||||
const o = await observeInternal(ctx, state), taskId = entity(state, "task").id ?? "";
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
import { beforeEach, expect, it, vi } from "vitest";
|
||||
|
||||
// Cases that read links from the rendered reply must wait for the saved reply
|
||||
// to render first: run settle precedes the render, and reading the DOM
|
||||
// immediately captured no links although the reply contained the right one.
|
||||
const calls: string[] = [];
|
||||
const waits: Array<{ leadId: string; runs: Array<{ id: string }> }> = [];
|
||||
vi.mock("./internal-comment-reference.js", async original => ({
|
||||
...await original<typeof import("./internal-comment-reference.js")>(),
|
||||
waitForRenderedReply: vi.fn(async (_ctx: unknown, o: { leadId: string; runs: Array<{ id: string }> }) => {
|
||||
calls.push("wait"); waits.push(o); throw new Error("stopped after the rendered reply wait");
|
||||
}),
|
||||
}));
|
||||
const { runtimeHandoffCases } = await import("./runtime-handoffs.js");
|
||||
const { profileIdentityCases } = await import("./identity-profile.js");
|
||||
|
||||
beforeEach(() => { calls.length = 0; waits.length = 0; });
|
||||
function context(runs: Array<Record<string, unknown>>, comments: Array<Record<string, unknown>> = []) {
|
||||
const locator = () => { calls.push("dom-read"); throw new Error("rendered DOM read before the reply wait"); };
|
||||
return {
|
||||
input: { fixtures: { agent: { id: "ceo" }, company: { id: "company", issuePrefix: "COM" } }, execution: { id: "unit" },
|
||||
page: { locator, url: () => "http://fixture.invalid/COM/issues/COM-1" },
|
||||
api: { get: async (path: string) => { calls.push(`api:${path}`); return path.includes("/comments") ? comments : { user: { name: "Alex Former" } }; } } },
|
||||
refreshIssue: async () => {}, issue: () => ({ id: "source", conversationState: "waiting" }), allRuns: async () => runs,
|
||||
} as never;
|
||||
}
|
||||
|
||||
it("track-delegated-work waits for the query replies to render before reading their links", async () => {
|
||||
const delegated = runtimeHandoffCases.find(row => row.scenarioId === "agent-chat.track-delegated-work")!;
|
||||
const runs = [{ id: "handoff", agentId: "ceo", status: "succeeded" }, { id: "worker", agentId: "morgan", status: "running" }, { id: "query", agentId: "ceo", status: "succeeded" }];
|
||||
const ctx = context(runs, [{ id: "reply", authorAgentId: "ceo", createdByRunId: "query", body: "[Website](/COM/issues/COM-2)" }]);
|
||||
await expect(delegated.act!(ctx, { setupRunIds: ["handoff", "worker"], workerRunId: "worker", sourceId: "source" })).rejects.toThrow("stopped after the rendered reply wait");
|
||||
expect(calls.filter(call => !call.startsWith("api:"))).toEqual(["wait"]);
|
||||
expect(waits[0]).toMatchObject({ leadId: "ceo", runs: [{ id: "query" }] });
|
||||
});
|
||||
|
||||
it("edit-profile waits for the reply to render before looking for the profile link", async () => {
|
||||
const ctx = context([{ id: "reply-run", agentId: "ceo", status: "succeeded" }]);
|
||||
await expect(profileIdentityCases[0]!.act!(ctx, {})).rejects.toThrow("stopped after the rendered reply wait");
|
||||
expect(calls).toEqual(["wait"]);
|
||||
expect(waits[0]).toMatchObject({ leadId: "ceo", runs: [{ id: "reply-run" }] });
|
||||
});
|
||||
@@ -61,7 +61,7 @@ export const runtimeDependencyCases = [runtimeDefinition("task-coordination.bloc
|
||||
description: `Implement the approved Design as a small website plan. Once eligible, read the actual Design task and save a task document with the exact reference ${marker}, accessible labels and a clear primary action, then finish normally. Do not deploy, create other tasks or invent approval.` });
|
||||
return { worker, task, design, marker };
|
||||
}, gradeDesignDependency, {
|
||||
expectedHumanMessageCount: 2, expectedRunCount: 3, attemptTimeoutMs: 720_000, oracleVersion: "paperclip.chat-catalog.runtime.block-on-design.v4",
|
||||
expectedHumanMessageCount: 2, expectedRunCount: 3, attemptTimeoutMs: 720_000, oracleVersion: "paperclip.chat-catalog.runtime.block-on-design.v5",
|
||||
sourceFiles: ["chat-catalog-domains/runtime-dependency.ts", "chat-catalog-domains/api-read-receipts.ts", "chat-catalog-domains/runtime-support.ts", "run-observations.ts", "chat-scenarios/task-coordination.yaml"],
|
||||
async act(ctx, state) {
|
||||
await pollUntil({ label: "Saved dependency and canonical reply before human approval", deadlineAt: chatCatalogNativePhaseDeadline(ctx, Date.now() + 150_000),
|
||||
|
||||
@@ -87,7 +87,7 @@ export const runtimeFailedRunCases = [runtimeDefinition("runs-adapters.inspect-f
|
||||
return { worker, task: done.task, ...failure, successfulRun, successArtifactVerified: true, failureLogAvailable };
|
||||
}, gradeFailedAttempt, {
|
||||
sourceFiles: ["chat-catalog-domains/runtime-failed-runs.ts", "chat-catalog-domains/runtime-native-failure.ts", "chat-catalog-domains/runtime-support.ts", "chat-catalog-domains/internal-run-history.ts", "chat-catalog-domains/api-read-receipts.ts", "chat-scenarios/runs-adapters.yaml", "../../ui/src/components/TaskChatThread.tsx", "../../ui/src/components/task-chat/TaskChatMarker.tsx", "../../ui/src/components/task-chat/TaskChatThreadView.tsx", "../../ui/src/pages/AgentDetail.tsx"],
|
||||
oracleVersion: "paperclip.chat-catalog.runtime.inspect-failed-run.v4", expectedRunCount: 3, attemptTimeoutMs: 720_000,
|
||||
oracleVersion: "paperclip.chat-catalog.runtime.inspect-failed-run.v5", expectedRunCount: 3, attemptTimeoutMs: 720_000,
|
||||
runPolicy: { kind: "terminal-counts", terminalStatusCounts: { failed: 1, succeeded: 2 } },
|
||||
async observe(ctx, state) {
|
||||
state.retainedFailure = await ctx.input.api.get(`/api/heartbeat-runs/${(state.failedRun as { id: string }).id}`);
|
||||
|
||||
@@ -8,6 +8,7 @@ import { completionDelivery } from "../completion-updates.js";
|
||||
import { collectRunEvents } from "../run-observations.js";
|
||||
import { apiReadDigests, apiReadStatuses, recordParams, type ApiRead } from "./api-read-receipts.js";
|
||||
import { checks, hasLink, type Entity } from "./internal-support.js";
|
||||
import { waitForRenderedReply } from "./internal-comment-reference.js";
|
||||
import { openStandingChat } from "./identity-support.js";
|
||||
import { prepareRuntimeWorkerGate, readRuntimeTaskGate, releaseRuntimeWorker } from "./runtime-busy-worker.js";
|
||||
import { runtimeDefinition, runtimeObservation, observeRuntime, type RuntimeAgent } from "./runtime-support.js";
|
||||
@@ -96,8 +97,8 @@ export const runtimeHandoffCases = [runtimeDefinition("agent-chat.track-delegate
|
||||
return { worker: prepared.worker, gate: prepared.gate, task, marker, prompt, sourceBefore: source,
|
||||
sourceId: source.id, setupRunIds: phase.runs.map(run => run.id), workerRunId: workerRun.id, workerWasReallyWaiting: true, createdFromSource: true };
|
||||
}, gradeTrackDelegatedWork, {
|
||||
oracleVersion: "paperclip.chat-catalog.track-delegated-work.v2",
|
||||
sourceFiles: ["chat-catalog-domains/runtime-handoffs.ts", "chat-catalog-domains/api-read-receipts.ts", "chat-catalog-domains/runtime-busy-worker.ts", "chat-catalog-domains/runtime-support.ts", "completion-update-flow.ts", "completion-updates.ts", "run-observations.ts", "chat-scenarios/agent-chat.yaml"],
|
||||
oracleVersion: "paperclip.chat-catalog.track-delegated-work.v3",
|
||||
sourceFiles: ["chat-catalog-domains/runtime-handoffs.ts", "chat-catalog-domains/api-read-receipts.ts", "chat-catalog-domains/internal-comment-reference.ts", "chat-catalog-domains/runtime-busy-worker.ts", "chat-catalog-domains/runtime-support.ts", "completion-update-flow.ts", "completion-updates.ts", "run-observations.ts", "chat-scenarios/agent-chat.yaml"],
|
||||
setupHumanMessageCount: 1, expectedHumanMessageCount: 2, expectedRunCount: 4, attemptTimeoutMs: 900_000,
|
||||
setupPolicy: { readiness: "preserve-scenario", invariantCheckIds: ["lead-ready", "native-handoff-owner", "real-source-provenance"] },
|
||||
async verifySetup(ctx, state) {
|
||||
@@ -123,6 +124,9 @@ export const runtimeHandoffCases = [runtimeDefinition("agent-chat.track-delegate
|
||||
const comments = await ctx.input.api.get<Entity[]>(`/api/issues/${state.sourceId}/comments?order=asc`);
|
||||
const replies = comments.filter(row => row.authorAgentId === ctx.input.fixtures.agent.id && queryIds.includes(row.createdByRunId ?? ""));
|
||||
state.queryReply = replies.map(row => row.body ?? "").join("\n");
|
||||
// Run settle precedes the reply's render in the open conversation; read
|
||||
// rendered links only after each saved reply itself is on the page.
|
||||
await waitForRenderedReply(ctx, { leadId: ctx.input.fixtures.agent.id, runs: phase.runs.filter(run => queryIds.includes(run.id)) });
|
||||
const links: string[] = [];
|
||||
for (const reply of replies) links.push(...await ctx.input.page.locator(`[id=${JSON.stringify(`comment-${reply.id}`)}] a[href]`).evaluateAll(nodes => nodes.map(node => node.getAttribute("href")!)));
|
||||
state.queryLinks = links; state.workerStatusAtQueryReply = phase.runs.find(run => run.id === state.workerRunId)!.status;
|
||||
|
||||
@@ -79,7 +79,7 @@ export function gradeInfrastructureRetry(value: unknown) {
|
||||
return c.result();
|
||||
}
|
||||
export const runtimeInfrastructureRetryCases = [runtimeDefinition("recovery.retry-exhausted", exhaustedNativeInfrastructure, gradeInfrastructureRetry, {
|
||||
oracleVersion: "paperclip.chat-catalog.runtime.retry-exhausted.v4", authentication: "human-session", expectedHumanMessageCount: 3, expectedRunCount: 5, attemptTimeoutMs: 1_800_000,
|
||||
oracleVersion: "paperclip.chat-catalog.runtime.retry-exhausted.v5", authentication: "human-session", expectedHumanMessageCount: 3, expectedRunCount: 5, attemptTimeoutMs: 1_800_000,
|
||||
setupPolicy: runtimeInfrastructureCases[0]!.setupPolicy, verifySetup: runtimeInfrastructureCases[0]!.verifySetup,
|
||||
runPolicy: { kind: "terminal-counts", terminalStatusCounts: { failed: 1, succeeded: 4 } },
|
||||
sourceFiles: ["chat-catalog-domains/runtime-infrastructure-retry.ts", "chat-catalog-domains/runtime-infrastructure.ts", "chat-catalog-domains/api-read-receipts.ts", "chat-catalog-domains/runtime-native-fault.ts", "chat-catalog-domains/runtime-busy-worker.ts", "chat-catalog-domains/runtime-support.ts", "../../server/src/services/hot-restart.ts", "../../server/src/vendor/paperclip-runner/index.ts", "chat-scenarios/recovery.yaml"],
|
||||
|
||||
@@ -9,7 +9,7 @@ import { checks, hasLink, type Entity } from "./internal-support.js";
|
||||
import { busyRuntimeWorker, releaseRuntimeWorker } from "./runtime-busy-worker.js";
|
||||
import { faultOwnedNativeProcess, NATIVE_INFRASTRUCTURE_FAULT_CODES, type NativeFaultRun } from "./runtime-native-fault.js";
|
||||
import { runtimeDefinition, runtimeObservation, observeRuntime, type RuntimeAgent, type RuntimeObservation } from "./runtime-support.js";
|
||||
export const INFRASTRUCTURE_RECOVERY_ORACLE = "paperclip.chat-catalog.runtime.infrastructure-recovery.v4";
|
||||
export const INFRASTRUCTURE_RECOVERY_ORACLE = "paperclip.chat-catalog.runtime.infrastructure-recovery.v5";
|
||||
const objectRecord = (value: unknown): Record<string, unknown> => value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
|
||||
/**
|
||||
* The chat run read the source task's saved recovery record through a product
|
||||
|
||||
@@ -17,6 +17,7 @@ import { findChatCatalogCase, loadChatCatalogImplementation } from "./chat-catal
|
||||
import { chatCatalogInventoryDigest } from "./chat-catalog-inventory.js";
|
||||
import { resolveChatCatalogQuery } from "./chat-query-bindings.js";
|
||||
import { chatRunEvidenceApi } from "./chat-run-evidence-observer.js";
|
||||
import { markReturningBrowser } from "./chat-returning-browser.js";
|
||||
|
||||
/** The case owns product semantics; this flow owns the actual chat entry point and evidence lifecycle. */
|
||||
export async function runChatCatalogFlow(context: ChatCatalogContext) {
|
||||
@@ -117,6 +118,8 @@ export async function runChatCatalogFlow(context: ChatCatalogContext) {
|
||||
return { grade, runs: evidence.runs, issue: evidence.issue };
|
||||
}
|
||||
try {
|
||||
// Setup may finish five or more tasks; a returning person has already dismissed the star prompt.
|
||||
await markReturningBrowser(input.page, input.api?.baseURL);
|
||||
await input.evidence("chat-catalog-definition.json", { scenarioId: scenario.id, ...fingerprint,
|
||||
suiteDefinitionHash: input.execution.suiteDefinitionHash, journeyKind: implementation.journeyKind ?? "agent-chat" });
|
||||
// Setup turns (prerequisite chat messages, worker runs) spend before the query. Fund the company and
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
import { readFileSync } from "node:fs";
|
||||
import { expect, it } from "vitest";
|
||||
import { markReturningBrowser, RETURNING_BROWSER_SEEN_KEYS } from "./chat-returning-browser.js";
|
||||
|
||||
it("uses the product's own star prompt seen key", () => {
|
||||
const source = readFileSync(new URL("../../ui/src/components/GitHubStarPrompt.tsx", import.meta.url), "utf8");
|
||||
const key = /export const STAR_PROMPT_SEEN_KEY = "([^"]+)"/.exec(source)?.[1];
|
||||
expect(key).toBeTruthy();
|
||||
expect(RETURNING_BROWSER_SEEN_KEYS).toContain(key);
|
||||
});
|
||||
|
||||
function fakePage(url: string) {
|
||||
const storage = new Map<string, string>([["paperclip.github-star-prompt.seen", "kept"]]);
|
||||
const scripts: Array<{ script: (arg: never) => void; arg: unknown }> = [];
|
||||
const run = (script: (arg: never) => void, arg: unknown, origin: string) => {
|
||||
const previous = (globalThis as { window?: unknown }).window;
|
||||
(globalThis as { window?: unknown }).window = { location: { origin }, localStorage: {
|
||||
getItem: (key: string) => storage.get(key) ?? null, setItem: (key: string, value: string) => { storage.set(key, value); } } };
|
||||
try { script(arg as never); } finally { (globalThis as { window?: unknown }).window = previous; }
|
||||
};
|
||||
const page = {
|
||||
url: () => url,
|
||||
context: () => ({ addInitScript: async (script: (arg: never) => void, arg: unknown) => { scripts.push({ script, arg }); } }),
|
||||
evaluate: async (script: (arg: never) => void, arg: unknown) => run(script, arg, new URL(url).origin),
|
||||
};
|
||||
return { page: page as never, storage, scripts, load: (origin: string) => { for (const row of scripts) run(row.script, row.arg, origin); } };
|
||||
}
|
||||
|
||||
it("marks the prompt seen for later app documents and the open app page, never other origins", async () => {
|
||||
const open = fakePage("http://127.0.0.1:3100/COM/issues/COM-1"); open.storage.clear();
|
||||
await markReturningBrowser(open.page, "http://127.0.0.1:3100");
|
||||
expect(open.storage.get("paperclip.github-star-prompt.seen")).toBeTruthy();
|
||||
expect(open.scripts).toHaveLength(1);
|
||||
|
||||
const blank = fakePage("about:blank"); blank.storage.clear();
|
||||
await markReturningBrowser(blank.page, "http://127.0.0.1:3100");
|
||||
expect(blank.storage.size).toBe(0);
|
||||
blank.load("https://github.com"); expect(blank.storage.size).toBe(0);
|
||||
blank.load("http://127.0.0.1:3100"); expect(blank.storage.get("paperclip.github-star-prompt.seen")).toBeTruthy();
|
||||
});
|
||||
|
||||
it("keeps a choice the browser already saved", async () => {
|
||||
const open = fakePage("http://127.0.0.1:3100/");
|
||||
await markReturningBrowser(open.page, "http://127.0.0.1:3100");
|
||||
expect(open.storage.get("paperclip.github-star-prompt.seen")).toBe("kept");
|
||||
});
|
||||
|
||||
it("does nothing for a page stand-in without a browser context", async () => {
|
||||
await expect(markReturningBrowser({ url: () => "http://127.0.0.1:3100/" } as never, "http://127.0.0.1:3100")).resolves.toBeUndefined();
|
||||
await expect(markReturningBrowser(undefined as never, "http://127.0.0.1:3100")).resolves.toBeUndefined();
|
||||
});
|
||||
@@ -0,0 +1,32 @@
|
||||
import type { Page } from "@playwright/test";
|
||||
|
||||
/**
|
||||
* Once-per-browser prompts a returning person has already dismissed. Chat
|
||||
* journeys test chat, not first-visit nudges: the GitHub star prompt (#20453)
|
||||
* opens after five done tasks and covers the composer. The product is
|
||||
* unchanged; the eval browser carries the same saved choice a returning user's
|
||||
* browser would. Keys mirror ui/src/components/GitHubStarPrompt.tsx (checked
|
||||
* by chat-returning-browser.test.ts).
|
||||
*/
|
||||
export const RETURNING_BROWSER_SEEN_KEYS = ["paperclip.github-star-prompt.seen"] as const;
|
||||
const SEEN_AT = "2026-01-01T00:00:00.000Z";
|
||||
|
||||
function markSeen({ origin, keys, seenAt }: { origin: string; keys: readonly string[]; seenAt: string }) {
|
||||
try {
|
||||
if (window.location.origin !== origin) return;
|
||||
for (const key of keys) if (window.localStorage.getItem(key) === null) window.localStorage.setItem(key, seenAt);
|
||||
} catch { /* opaque or storage-less document */ }
|
||||
}
|
||||
|
||||
/** Marks the prompts seen for every later document of this app origin, and for the one already open. */
|
||||
export async function markReturningBrowser(page: Page, appOrigin: string) {
|
||||
// Flow unit tests drive the flow with page stand-ins that have no browser context; there is nothing to mark.
|
||||
const context = typeof page?.context === "function" ? page.context() : undefined;
|
||||
if (!context || typeof appOrigin !== "string") return;
|
||||
const args = { origin: new URL(appOrigin).origin, keys: [...RETURNING_BROWSER_SEEN_KEYS], seenAt: SEEN_AT };
|
||||
await context.addInitScript(markSeen, args);
|
||||
let current: string | undefined;
|
||||
try { current = new URL(page.url()).origin; } catch { current = undefined; }
|
||||
// The prompt rereads storage before it opens, so an already-open page needs no reload.
|
||||
if (current === args.origin) await page.evaluate(markSeen, args);
|
||||
}
|
||||
@@ -63,6 +63,24 @@ export async function synchronizeSdkPublicInputs(root, revision, { write = false
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Cheap drift guard: the source pins in the registry file must equal the blobs
|
||||
* this checkout committed. Compares Git trees only (no working-tree reads), so
|
||||
* a stale registry fails with the inputs that changed instead of the opaque
|
||||
* "SDK public source proof unavailable" raised later by the source proof.
|
||||
*/
|
||||
export async function staleSdkPublicInputs(root, revision = "HEAD") {
|
||||
const pinned = JSON.parse(await readFile(path.join(root, registryPath), "utf8")).inputs ?? {};
|
||||
const committed = {};
|
||||
for (const entry of git(root, ["ls-tree", "-rz", revision, "--", ...sourceRoots, ...buildInputs]).split("\0").filter(Boolean)) {
|
||||
const match = /^\d+ blob ([a-f0-9]{40})\t(.+)$/.exec(entry);
|
||||
if (match) committed[match[2]] = match[1];
|
||||
}
|
||||
return [...new Set([...Object.keys(pinned), ...Object.keys(committed)])].sort()
|
||||
.filter(file => pinned[file] !== committed[file])
|
||||
.map(file => ({ file, change: !pinned[file] ? "unpinned" : !committed[file] ? "removed" : "changed" }));
|
||||
}
|
||||
|
||||
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
||||
const [revision, mode] = process.argv.slice(2);
|
||||
if (process.argv.length !== 4 || !["--check", "--write"].includes(mode)) throw new Error("Usage: node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD-40hex> --check|--write");
|
||||
|
||||
@@ -5,7 +5,8 @@ import { mkdir, mkdtemp, readFile, realpath, rm, symlink, writeFile } from "node
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, test } from "vitest";
|
||||
import { synchronizeSdkPublicInputs } from "./source-sdk-public-inputs.mjs";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { staleSdkPublicInputs, synchronizeSdkPublicInputs } from "./source-sdk-public-inputs.mjs";
|
||||
|
||||
const registry = "tests/runner-e2e/source-sdk-public-inputs.json";
|
||||
const roots = [];
|
||||
@@ -70,3 +71,23 @@ test("admits only the separately pinned restored lock overlay without changing i
|
||||
await writeFile(path.join(root, "pnpm-lock.yaml"), "unknown lock overlay");
|
||||
await assert.rejects(synchronizeSdkPublicInputs(root, revision));
|
||||
});
|
||||
|
||||
// The pins only prove the SDK public types while they match the committed
|
||||
// source. A PR that edits packages/shared/src (or another pinned input) without
|
||||
// refreshing them breaks every SDK-dependent Runner cell before its server
|
||||
// starts, so fail here, cheaply, with the inputs and the fix command.
|
||||
test("the committed SDK public source pins match this checkout's committed tree", async () => {
|
||||
const root = fileURLToPath(new URL("../..", import.meta.url));
|
||||
const stale = await staleSdkPublicInputs(root);
|
||||
assert.deepEqual(stale, [], `${registry} is stale for ${stale.length} pinned input(s): ${stale.slice(0, 10).map(row => `${row.file} (${row.change})`).join(", ")}${stale.length > 10 ? ", ..." : ""}. `
|
||||
+ "Commit your change, run `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --write`, and commit the updated registry.");
|
||||
});
|
||||
test("the drift guard names changed, unpinned and removed inputs and clears after a resync", async () => {
|
||||
const { root, revision } = await fixture();
|
||||
assert.deepEqual((await staleSdkPublicInputs(root)).map(row => row.change), Array(11).fill("changed"));
|
||||
await synchronizeSdkPublicInputs(root, revision, { write: true });
|
||||
assert.deepEqual(await staleSdkPublicInputs(root), []);
|
||||
await put(root, "packages/shared/src/added.ts", "synthetic added source\n"); await rm(path.join(root, "packages/plugins/sdk/src/types.ts"));
|
||||
git(root, "add", "-A"); git(root, "-c", "user.name=Synthetic unit", "-c", "user.email=synthetic@example.invalid", "commit", "-qm", "drift");
|
||||
assert.deepEqual(await staleSdkPublicInputs(root), [{ file: "packages/plugins/sdk/src/types.ts", change: "removed" }, { file: "packages/shared/src/added.ts", change: "unpinned" }]);
|
||||
});
|
||||
Reference in new issue
Block a user