mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(evals): grade the personal-secret setup reply by its rendered link (#21344)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals check that a chat request reaches the correct saved outcome > - The personal-secret case checks that chat sends the person to their own secure value entry, and never asks for the value in chat > - The case judged the reply's wording with regular expressions, and it found the settings route by splitting the reply's markdown into tokens > - A valid reply failed because its link was followed by a colon, so the tokenizer read the route as `/COM/company/settings/secrets):` > - This pull request grades the rendered link instead: it finds the visible link, clicks it, and requires the person's own-value entry to appear > - The benefit is that the case measures what a person can actually do, and the reply's wording is left to human review ## Linked Issues or Issue Description **What happened?** `chat-features.runner-opencode.local.secrets.personal-secret` failed with "Actual reply must direct the supported own-value settings path for this definition". The retained screenshot and DOM snapshot show a correct reply. The turn rendered as two messages, and both linked **My secrets** to `/COM/company/settings/secrets` and said **Set my value**. The wording check accepted the reply. The link check returned nothing. It extracted routes from the reply's markdown with a regular expression and removed only `.,;!?])}` from the end of each token. In the final message the link was followed by a colon (`[My secrets](/COM/company/settings/secrets): choose`), so the token was `/COM/company/settings/secrets):`, which does not match the settings path. An earlier passing run had `,` after the link, which the check removed. **Expected behavior** A reply passes when its rendered message has a visible link that a person can click to reach their own value entry for the required definition. Wording is not graded by keywords. **Steps to reproduce** Grade a setup reply whose markdown link to the secrets settings is followed by `:`. The old `personalSecretSetupURL` returns `undefined` for the retained reply, although its wording check returns `true`. ## What Changed - Add `tests/runner-e2e/chat-catalog-domains/external-personal-secret-link.ts`. It finds a visible link in the reply's rendered message (`#comment-<id>`) whose target is this company's `/company/settings/secrets`. An optional `definition` query must name the required definition. Other origins, companies, paths, credentials, fragments and query keys are refused. It clicks the link and requires the person's own-value entry to be visible. A link that names the definition opens that definition's detail sheet, so the check requires **Set my value** in the sheet, which must show `PERSONAL_PUBLISHING_KEY`. Otherwise it selects **My secrets** if that tab is not active and requires a visible **Set value** button on the row bound to the required definition (`my-secret-<definitionId>`). It then returns to the conversation and records the proof as `personal-secret-reply-link.json`. - In `external-personal-secret.ts`, the setup reply is no longer checked against wording patterns. It must still be the agent's own comment from the bound run. The grade uses the recorded link proof, which must belong to a reply of the initial run (a turn can post several messages). The reply link is now followed before the human-entry phase. - The confirmation and question paths are unchanged. When a question or confirmation card already carries the handoff, the reply link is not checked, so an optional link cannot fail those paths. - No scenario YAML changed, so the TestOps manifest needs no entry. `chat-catalog-descriptors.json` lists the new source file for this case. ## Verification - `external-personal-secret-link.test.ts`: - An ungated unit test covers the link-target rules. - A Chromium test (gated like the other secret-route browser tests by `PAPERCLIP_CHAT_EVAL_SECRET_ROUTE_CANARY=1`, run locally) uses the retained two-message reply. These cases pass: the retained reply; a link carrying `?definition=` that opens the definition's detail sheet; a page where the **My secrets** tab is selected after the click. - These cases fail: no link (wording only); a plain-text path; a wrong-company link; a foreign link; a wrong-page link; a hidden link; a link that reaches another definition's row; a definition link whose sheet shows another key; a link that reaches no own-value entry. - Red check: the old `personalSecretSetupURL` returns `undefined` for the retained reply, while `personalSecretGuidance` returns `true`. - The personal-secret test files, catalog consistency and contract tests pass (210 tests). `pnpm test:e2e:runner:typecheck` passes. - A narrow paid run of the case on this branch is linked in a comment. ## Risks - Low risk. The change is in eval code only. - A reply that points to the settings page without a clickable link (for example, text menu directions only) no longer passes this path. That matches the stated contract: the person reaches their entry through a link in the reply. Question and confirmation cards are unchanged. - The company-secret case uses the same markdown tokenizer and could hit the same trailing-colon problem. It is not changed here. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
70be453c38
commit
2b69cdfcd6
6 files changed
+214
-22
No files matched your search
@@ -5051,6 +5051,7 @@
|
||||
"../../packages/paperclip-runner/src/protocol-actions/request-human-input.ts",
|
||||
"chat-catalog-domains/external-personal-secret.ts",
|
||||
"chat-catalog-domains/external-personal-secret-menu.ts",
|
||||
"chat-catalog-domains/external-personal-secret-link.ts",
|
||||
"../../ui/src/pages/secrets/MyUserSecretsTab.tsx",
|
||||
"chat-catalog-domains/external-personal-secret-confirmation.ts",
|
||||
"chat-catalog-contract.ts",
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
import { chromium } from "@playwright/test";
|
||||
import { expect, it } from "vitest";
|
||||
import { followPersonalSecretReplyLink, supportedSecretSettingsLink } from "./external-personal-secret-link.js";
|
||||
|
||||
const origin = "https://paperclip.example", chat = `${origin}/COM/chat`, secrets = "/COM/company/settings/secrets";
|
||||
const definition = "b22dfafd-1bae-4d33-a072-7dfb2e5358f8", key = "PERSONAL_PUBLISHING_KEY";
|
||||
|
||||
it("accepts only this company's secrets settings as a link target", () => {
|
||||
expect(supportedSecretSettingsLink(secrets, origin, "COM", definition)).toBe(origin + secrets);
|
||||
expect(supportedSecretSettingsLink(`${secrets}?definition=${definition}`, origin, "COM", definition)).toBe(`${origin}${secrets}?definition=${definition}`);
|
||||
for (const href of ["/OTHER/company/settings/secrets", "/COM/company/settings/other", `https://foreign.example${secrets}`,
|
||||
`${secrets}?definition=other`, `${secrets}?tab=x`, `${secrets}#mine`, null]) {
|
||||
expect(supportedSecretSettingsLink(href, origin, "COM", definition), String(href)).toBeUndefined();
|
||||
}
|
||||
});
|
||||
|
||||
// The reply retained from the failed run: one turn rendered as two messages, with
|
||||
// the settings link followed by punctuation (":" and ",") in rendered markdown.
|
||||
const retained = `<div class="first"><p>Only you can set your own personal credential value — it's stored through your secure entry, not by an agent, and it must never be pasted into chat.</p>
|
||||
<p>Open <a href="${secrets}">My secrets</a>, find the personal publishing credential required by <a href="/COM/issues/COM-1">COM-1</a>, and choose <strong>Set my value</strong> for that definition.</p></div>
|
||||
<div class="final"><p>Set your personal publishing credential through <a href="${secrets}">My secrets</a>: choose <strong>Set my value</strong> for the personal publishing credential required by <a href="/COM/issues/COM-1">COM-1</a>.</p></div>`;
|
||||
|
||||
it.skipIf(process.env.PAPERCLIP_CHAT_EVAL_SECRET_ROUTE_CANARY !== "1")("grades the rendered setup link by clicking it to the own-value entry, not by the reply's wording", async () => {
|
||||
const browser = await chromium.launch({ headless: true, ...process.env.PAPERCLIP_RUNNER_E2E_CHROMIUM_EXECUTABLE ? { executablePath: process.env.PAPERCLIP_RUNNER_E2E_CHROMIUM_EXECUTABLE } : {} });
|
||||
const page = await browser.newPage();
|
||||
const ownEntry = (id: string, button = "Set value") => `<ul><li data-testid="my-secret-${id}"><span>Personal publishing credential</span><code>${key}</code><button>${button}</button></li></ul>`;
|
||||
const scenarios: Array<{ name: string; reply: string; accepted?: true; entry?: string; secretsPage?: string; sheetKey?: string }> = [
|
||||
{ name: "retained valid reply", reply: retained, accepted: true },
|
||||
{ name: "link opening the definition's detail sheet", reply: `<p>Use <a href="${secrets}?definition=${definition}">this page</a>.</p>`, accepted: true },
|
||||
{ name: "definition link whose sheet shows another key", reply: `<p>Use <a href="${secrets}?definition=${definition}">this page</a>.</p>`, sheetKey: "OTHER_KEY" },
|
||||
{ name: "tab selected after the click", reply: retained, accepted: true,
|
||||
secretsPage: `<button role="tab" aria-selected="false" onclick="document.querySelector('#own').hidden=false">My secrets</button><div id="own" hidden>${ownEntry(definition)}</div>` },
|
||||
// Like the real single-page app: the settings route renders its tabs after navigation, on the company tab.
|
||||
{ name: "tabs rendered after navigation", reply: retained, accepted: true,
|
||||
secretsPage: `<div id="late"></div><template id="tabs"><button role="tab" aria-selected="false" onclick="document.querySelector('#own').hidden=false">My secrets</button><div id="own" hidden>${ownEntry(definition)}</div></template><script>setTimeout(()=>document.querySelector('#late').append(document.querySelector('#tabs').content.cloneNode(true)),400)</script>` },
|
||||
{ name: "no link, only wording", reply: "<p>Open My secrets in Settings → Secrets and choose Set my value.</p>" },
|
||||
{ name: "plain-text path, not a link", reply: `<p>Go to <code>${secrets}</code>.</p>` },
|
||||
{ name: "wrong company link", reply: '<p>Open <a href="/OTHER/company/settings/secrets">My secrets</a>.</p>' },
|
||||
{ name: "foreign link", reply: `<p>Open <a href="https://foreign.example${secrets}">My secrets</a>.</p>` },
|
||||
{ name: "wrong page link", reply: '<p>Open <a href="/COM/company/settings/other">My secrets</a>.</p>' },
|
||||
{ name: "hidden link", reply: `<p>Setup details:</p><p hidden>Open <a href="${secrets}">My secrets</a>.</p>` },
|
||||
{ name: "link reaches another definition", reply: retained, entry: ownEntry("other-definition") },
|
||||
{ name: "link reaches no own-value entry", reply: retained, entry: "<p>Nothing to set.</p>" },
|
||||
];
|
||||
try {
|
||||
for (const scenario of scenarios) {
|
||||
await page.unroute("**/*");
|
||||
await page.route("**/*", async route => {
|
||||
const url = new URL(route.request().url()), path = url.pathname;
|
||||
// Like the real page: a definition link opens that definition's detail sheet over the list.
|
||||
const sheet = url.searchParams.get("definition") === definition
|
||||
? `<div role="dialog" data-testid="secret-details"><code>${scenario.sheetKey ?? key}</code><button>Set my value</button></div>` : "";
|
||||
const body = path === secrets
|
||||
? sheet + (scenario.secretsPage ?? `<button role="tab" aria-selected="true">My secrets</button>${scenario.entry ?? ownEntry(definition)}`)
|
||||
: `<main><article id="comment-final-reply">${scenario.reply}</article></main>`;
|
||||
await route.fulfill({ contentType: "text/html; charset=utf-8", body });
|
||||
});
|
||||
await page.goto(chat);
|
||||
const proof = await followPersonalSecretReplyLink(page, { commentId: "final-reply", origin, prefix: "COM", definitionId: definition, definitionKey: key, timeout: 3_000 });
|
||||
if (scenario.accepted) expect(proof, scenario.name).toMatchObject({ commentId: "final-reply", reachedOwnValueEntry: true, definitionKey: key });
|
||||
else expect(proof, scenario.name).toBeUndefined();
|
||||
expect(page.url(), scenario.name).toBe(chat);
|
||||
}
|
||||
} finally {
|
||||
await browser.close();
|
||||
}
|
||||
}, 60_000);
|
||||
@@ -0,0 +1,99 @@
|
||||
import type { Locator, Page } from "@playwright/test";
|
||||
|
||||
/**
|
||||
* Grade a personal-secret setup reply by its rendered link, never by its
|
||||
* wording. The reply passes when its rendered message holds a visible link to
|
||||
* this company's secrets settings and clicking that link, as a person would,
|
||||
* reaches the current person's own-value entry for the required definition.
|
||||
* The reply's wording is left to human review.
|
||||
*/
|
||||
export interface PersonalSecretReplyLinkProof {
|
||||
commentId: string;
|
||||
/** The rendered link's resolved target, as the person's click followed it. */
|
||||
route: string;
|
||||
linkText: string;
|
||||
reachedOwnValueEntry: true;
|
||||
definitionKey: string;
|
||||
}
|
||||
|
||||
/** A rendered link target is supported only for this origin and company's secrets settings. */
|
||||
export function supportedSecretSettingsLink(href: string | null, origin: string, prefix: string, definitionId: string): string | undefined {
|
||||
try {
|
||||
const url = new URL(href ?? "", origin);
|
||||
const query = [...url.searchParams.keys()];
|
||||
if (url.origin !== origin || url.username || url.password || url.hash) return;
|
||||
if (url.pathname !== `/${prefix}/company/settings/secrets`) return;
|
||||
if (query.some(key => key !== "definition") || url.searchParams.getAll("definition").length > 1) return;
|
||||
if (url.searchParams.has("definition") && url.searchParams.get("definition") !== definitionId) return;
|
||||
return url.href;
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
async function visible(node: Locator, timeout: number): Promise<boolean> {
|
||||
if (await node.count() !== 1 || !await node.isVisible()) return false;
|
||||
await node.scrollIntoViewIfNeeded({ timeout });
|
||||
return node.evaluate(element => {
|
||||
const rect = element.getBoundingClientRect();
|
||||
if (!element.checkVisibility({ checkOpacity: true, checkVisibilityCSS: true }) || rect.width <= 0 || rect.height <= 0
|
||||
|| rect.bottom <= 0 || rect.top >= innerHeight || rect.right <= 0 || rect.left >= innerWidth) return false;
|
||||
for (let parent = element.parentElement; parent; parent = parent.parentElement) {
|
||||
const style = getComputedStyle(parent), box = parent.getBoundingClientRect();
|
||||
if (/hidden|clip/.test(style.overflowX) && (rect.right <= box.left || rect.left >= box.right)) return false;
|
||||
if (/hidden|clip/.test(style.overflowY) && (rect.bottom <= box.top || rect.top >= box.bottom)) return false;
|
||||
}
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Find the visible supported link in the rendered reply, click it, and prove
|
||||
* the click reached the own-value entry ("Set value" on the definition's row in
|
||||
* My secrets). Returns to the conversation afterwards. Undefined when the reply
|
||||
* has no such link or the click does not reach the entry.
|
||||
*/
|
||||
export async function followPersonalSecretReplyLink(page: Page, input: {
|
||||
commentId: string; origin: string; prefix: string; definitionId: string; definitionKey: string; timeout: number;
|
||||
}): Promise<PersonalSecretReplyLinkProof | undefined> {
|
||||
const deadline = Date.now() + input.timeout;
|
||||
const remaining = () => {
|
||||
const ms = deadline - Date.now();
|
||||
if (ms <= 0) throw new Error("Personal secret reply link deadline exceeded");
|
||||
return ms;
|
||||
};
|
||||
const comment = page.locator(`[id=${JSON.stringify(`comment-${input.commentId}`)}]`);
|
||||
await comment.waitFor({ state: "visible", timeout: remaining() }).catch(() => undefined);
|
||||
if (!await comment.isVisible()) return;
|
||||
let chosen: { anchor: Locator; route: string; text: string } | undefined;
|
||||
for (const anchor of await comment.locator("a[href]").all()) {
|
||||
const route = supportedSecretSettingsLink(await anchor.getAttribute("href"), input.origin, input.prefix, input.definitionId);
|
||||
if (route && await visible(anchor, remaining())) { chosen = { anchor, route, text: (await anchor.innerText()).trim() }; break; }
|
||||
}
|
||||
if (!chosen) return;
|
||||
const original = page.url();
|
||||
try {
|
||||
await chosen.anchor.click({ timeout: remaining() });
|
||||
await page.waitForURL(url => url.origin === input.origin && url.pathname === new URL(chosen!.route).pathname, { timeout: remaining() });
|
||||
let entry: Locator;
|
||||
if (new URL(chosen.route).searchParams.get("definition") === input.definitionId) {
|
||||
// A link naming the definition opens its detail sheet, whose own-value action is "Set my value".
|
||||
const sheet = page.locator('[data-testid="secret-details"]').filter({ has: page.getByText(input.definitionKey, { exact: true }) });
|
||||
entry = sheet.getByRole("button", { name: "Set my value", exact: true });
|
||||
} else {
|
||||
const tab = page.getByRole("tab", { name: "My secrets", exact: true });
|
||||
await tab.waitFor({ state: "visible", timeout: remaining() }).catch(() => undefined);
|
||||
if (await tab.count() === 1 && await tab.getAttribute("aria-selected") !== "true") await tab.click({ timeout: remaining() });
|
||||
// The row is bound to the required definition by its identity, and shows its key.
|
||||
const row = page.locator(`[data-testid=${JSON.stringify(`my-secret-${input.definitionId}`)}]`)
|
||||
.filter({ has: page.getByText(input.definitionKey, { exact: true }) });
|
||||
entry = row.getByRole("button", { name: "Set value", exact: true });
|
||||
}
|
||||
await entry.waitFor({ state: "visible", timeout: remaining() }).catch(() => undefined);
|
||||
if (Date.now() >= deadline || !await visible(entry, remaining())) return;
|
||||
return { commentId: input.commentId, route: chosen.route, linkText: chosen.text, reachedOwnValueEntry: true, definitionKey: input.definitionKey };
|
||||
} finally {
|
||||
// Returning to the conversation has its own bound so a failed check still restores the page.
|
||||
if (page.url() !== original) await page.goto(original, { waitUntil: "domcontentloaded", timeout: 15_000 });
|
||||
}
|
||||
}
|
||||
+9
-5
@@ -1,17 +1,20 @@
|
||||
import { afterEach, expect, it, vi } from "vitest";
|
||||
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
|
||||
const port = vi.hoisted(() => ({ observation: {} as Record<string, unknown> }));
|
||||
const port = vi.hoisted(() => ({ observation: {} as Record<string, unknown>, linked: true }));
|
||||
// The rendered-link follower is a browser step with its own test; here it only reports whether the reply had a link.
|
||||
vi.mock("./external-personal-secret-link.js", () => ({ followPersonalSecretReplyLink: async (_page: unknown, input: { commentId: string }) => port.linked
|
||||
? { commentId: input.commentId, route: "http://paperclip.test/ACORN/company/settings/secrets", linkText: "My secrets", reachedOwnValueEntry: true, definitionKey: "PERSONAL_PUBLISHING_KEY" } : undefined }));
|
||||
vi.mock("../codex-subscription-fixture.js", () => ({ codexSubscriptionAgentConfig: (value: unknown) => value }));
|
||||
vi.mock("./external-support.js", async original => ({ ...await original<typeof import("./external-support.js")>(), observeExternal: async () => port.observation }));
|
||||
import { personalSecretExternalCases } from "./external-personal-secret.js";
|
||||
afterEach(() => { vi.useRealTimers(); vi.restoreAllMocks(); });
|
||||
const start = 10_000, deadline = start + 630_000;
|
||||
type Fault = "failed" | "cancelled" | "foreign actor" | "foreign company" | "foreign source" | "foreign task context" | "foreign comment" | "foreign person" | "foreign reply" | "no-secret-question" | "no work" | "late read";
|
||||
type Fault = "failed" | "cancelled" | "foreign actor" | "foreign company" | "foreign source" | "foreign task context" | "foreign comment" | "foreign person" | "foreign reply" | "no-settings-link" | "no work" | "late read";
|
||||
function fixture(fault?: Fault, budget: number | "missing" = deadline) {
|
||||
vi.useFakeTimers(); vi.setSystemTime(start);
|
||||
let reachedHumanPhase = false;
|
||||
const query = "Set up my personal publishing credential for this task.";
|
||||
const body = fault === "no-secret-question" ? "Which project should I work on?" : "Open secure My secrets settings and set your own personal publishing value. Never paste your key here.";
|
||||
const body = fault === "no-settings-link" ? "Which project should I work on?" : "Open secure My secrets settings and set your own personal publishing value. Never paste your key here.";
|
||||
const run = { id: "initial", companyId: fault === "foreign company" ? "foreign" : "company", agentId: fault === "foreign actor" ? "morgan" : "ceo", runtimeMode: "native",
|
||||
status: "succeeded", contextSnapshot: { issueId: fault === "foreign source" ? "foreign" : "source", ...(fault === "foreign task context" ? { taskId: "foreign" } : {}), commentId: fault === "foreign comment" ? "foreign" : "human" } };
|
||||
const comments = [{ id: "human", companyId: "company", issueId: "source", authorUserId: fault === "foreign person" ? "riley" : "person", body: query },
|
||||
@@ -19,12 +22,13 @@ function fixture(fault?: Fault, budget: number | "missing" = deadline) {
|
||||
const source = { id: "source", companyId: "company", conversationState: "waiting" };
|
||||
const allRuns = async () => fault === "no work" ? [] : [{ ...run, status: fault === "failed" || fault === "cancelled" ? fault : Date.now() < (fault ? start : start + 190_000) ? "running" : "succeeded" }];
|
||||
const ctx = { ...(budget === "missing" ? {} : { observationDeadlineAt: budget }), scenario: { id: "secrets.personal-secret", query },
|
||||
input: { fixtures: { company: { id: "company", issuePrefix: "ACORN" }, agent: { id: "ceo" } }, api: { get: async (path: string) => {
|
||||
input: { evidence: async () => {}, fixtures: { company: { id: "company", issuePrefix: "ACORN" }, agent: { id: "ceo" } }, api: { get: async (path: string) => {
|
||||
if (path === "/api/issues/source/comments?order=asc") { if (fault === "late read") vi.setSystemTime(deadline + 1); return comments; }
|
||||
if (path === "/api/companies/company/me/user-secrets") { reachedHumanPhase = true; throw Error("NEXT_REAL_HUMAN_PHASE"); }
|
||||
throw Error("Unexpected read");
|
||||
} } }, issue: () => source, refreshIssue: async () => {}, allRuns,
|
||||
} as unknown as ChatCatalogContext;
|
||||
port.linked = fault !== "no-settings-link";
|
||||
port.observation = { companyId: "company", interactions: [], phaseReplies: [{ runId: fault === "foreign reply" ? "foreign" : "initial", commentId: "reply", body }], links: [] };
|
||||
const state = { sourceIssueId: "source", currentPersonId: "person", definitionId: "definition" };
|
||||
async function act(ms = 191_000) {
|
||||
@@ -38,7 +42,7 @@ it("actual personal-secret act admits the canonical native response after190s wi
|
||||
expect(error?.message).toBe("NEXT_REAL_HUMAN_PHASE"); expect(f.reachedHumanPhase()).toBe(true);
|
||||
expect(f.ctx.observationDeadlineAt).toBe(deadline); expect(personalSecretExternalCases[0]!.attemptTimeoutMs).toBe(660_000);
|
||||
});
|
||||
it.each(["failed", "cancelled", "foreign actor", "foreign company", "foreign source", "foreign task context", "foreign comment", "foreign person", "foreign reply", "no-secret-question", "late read"] as const)("rejects %s before private human entry", async fault => {
|
||||
it.each(["failed", "cancelled", "foreign actor", "foreign company", "foreign source", "foreign task context", "foreign comment", "foreign person", "foreign reply", "no-settings-link", "late read"] as const)("rejects %s before private human entry", async fault => {
|
||||
const f = fixture(fault); expect(await f.act()).toBeInstanceOf(Error); expect(f.reachedHumanPhase()).toBe(false);
|
||||
});
|
||||
it.each(["missing", NaN, Infinity, start, start - 1] as const)("rejects invalid or exhausted original deadline %s", async budget => {
|
||||
|
||||
@@ -58,6 +58,8 @@ function valid() {
|
||||
],
|
||||
state: {
|
||||
initialRunId: "initial",
|
||||
replyLinkProof: { commentId: "initial-reply", route: "http://paperclip.test/COM/company/settings/secrets", linkText: "My secrets",
|
||||
reachedOwnValueEntry: true as const, definitionKey: "PERSONAL_PUBLISHING_KEY" } as Record<string, unknown> | undefined,
|
||||
currentMissingBefore: true,
|
||||
requiredReferenceBefore: true,
|
||||
requiredReferenceAfter: true,
|
||||
@@ -153,9 +155,15 @@ it("rejects wrong owner/reference/consumer, premature success and missing/leakin
|
||||
(o: ReturnType<typeof valid>) => {
|
||||
o.state.secretLeaked = true;
|
||||
},
|
||||
// The setup reply is graded by its followed link, not its wording.
|
||||
(o: ReturnType<typeof valid>) => {
|
||||
o.phaseReplies[0]!.body =
|
||||
"Your personal publishing credential is now ready. Use secure settings.";
|
||||
o.state.replyLinkProof = undefined;
|
||||
},
|
||||
(o: ReturnType<typeof valid>) => {
|
||||
o.state.replyLinkProof = { ...o.state.replyLinkProof, commentId: "confirmed-reply" };
|
||||
},
|
||||
(o: ReturnType<typeof valid>) => {
|
||||
o.state.replyLinkProof = { ...o.state.replyLinkProof, reachedOwnValueEntry: false };
|
||||
},
|
||||
(o: ReturnType<typeof valid>) => {
|
||||
o.runs[2]!.startedAt = "2026-11-04T00:00:05.000Z";
|
||||
|
||||
@@ -2,6 +2,7 @@ import {personalSecretMenuNavigation,type PersonalSecretMenuNavigation} from "./
|
||||
import { personalSecretConfirmationHandoff, acceptPersonalSecretConfirmation, confirmThenStartPersonalConsumer, readVisiblePersonalSecretConfirmation } from "./external-personal-secret-confirmation.js";
|
||||
import type {Page} from "@playwright/test";
|
||||
import {visibleSecretSettingsURL} from "./external-secret-route.js";
|
||||
import { followPersonalSecretReplyLink, type PersonalSecretReplyLinkProof } from "./external-personal-secret-link.js";
|
||||
import { createHash, randomBytes } from "node:crypto";
|
||||
import { expect } from "@playwright/test";
|
||||
import type { AskUserQuestionsInteraction } from "../../../packages/shared/src/types/issue.js";
|
||||
@@ -245,10 +246,7 @@ export function gradePersonalSecret(value: unknown) {
|
||||
"A required definition, real separate owners, secure UI entry and independently authenticated admitted worker check are required.",
|
||||
);
|
||||
if (!o) return c.result();
|
||||
const initial = o.phaseReplies.find(
|
||||
(row) => row.runId === o.state.initialRunId,
|
||||
),
|
||||
questionRow = o.interactions.find(row => row.id === o.state.initialQuestionId),
|
||||
const questionRow =o.interactions.find(row => row.id === o.state.initialQuestionId),
|
||||
question = personalSecretQuestionHandoff(questionRow, { companyId: o.companyId, issueId: String(o.state.sourceIssueId),
|
||||
agentId: o.leadId, userId: String(o.state.currentPersonId), runId: String(o.state.initialRunId), definitionId: String(o.state.definitionId) }, "answered"),
|
||||
questionAnswered = !!question && question.payloadHash === o.state.initialQuestionPayloadHash
|
||||
@@ -264,6 +262,11 @@ export function gradePersonalSecret(value: unknown) {
|
||||
const confirmation = personalSecretConfirmationHandoff(confirmationRow, { companyId: o.companyId, issueId: String(o.state.sourceIssueId), agentId: o.leadId, userId: String(o.state.currentPersonId), runId: String(o.state.initialRunId), definitionId: String(o.state.definitionId), canonicalCommentId: String(o.state.canonicalHumanCommentId) }, personalSecretGuidance, "accepted");
|
||||
const confirmationAccepted = o.state.confirmationVisibleBeforeEntry === true && !!confirmation && confirmation.payloadHash === o.state.initialConfirmationPayloadHash && confirmation.continuationPolicy === o.state.initialConfirmationContinuationPolicy
|
||||
&& o.activity.some(event => event.action === "issue.thread_interaction_accepted" && event.companyId === o.companyId && event.entityId === o.state.sourceIssueId && event.actorType === "user" && event.actorId === o.state.currentPersonId && event.details?.interactionId === confirmation.interactionId && event.details.interactionStatus === "accepted");
|
||||
// The reply is graded by its rendered link: clicking it reached the own-value entry for this definition.
|
||||
const replyLink = o.state.replyLinkProof as PersonalSecretReplyLinkProof | undefined;
|
||||
// A turn may post several messages; the graded link may be in any reply of the initial run.
|
||||
const replyLinkReached = replyLink?.reachedOwnValueEntry === true
|
||||
&& o.phaseReplies.some(row => row.runId === o.state.initialRunId && row.commentId === replyLink.commentId);
|
||||
const secret = o.state.currentSecret as Partial<CompanySecret> | undefined;
|
||||
c.add(
|
||||
"missing-own-required-credential-and-truthful-handoff",
|
||||
@@ -271,7 +274,7 @@ export function gradePersonalSecret(value: unknown) {
|
||||
o.state.requiredReferenceBefore === true &&
|
||||
o.state.otherConfiguredBefore === true &&
|
||||
o.state.currentPersonId !== o.state.otherPersonId &&
|
||||
((!!initial && personalSecretGuidance(initial.body)) || questionAnswered || confirmationAccepted) &&
|
||||
(replyLinkReached || questionAnswered || confirmationAccepted) &&
|
||||
o.state.agentValueCreateStatus === 403,
|
||||
"Another person's real stored value does not satisfy the current person's required publishing definition, and chat directs secure own-value entry before claiming completion.",
|
||||
);
|
||||
@@ -499,6 +502,7 @@ export const personalSecretExternalCases = [
|
||||
"../../packages/paperclip-runner/src/protocol-actions/request-human-input.ts",
|
||||
"chat-catalog-domains/external-personal-secret.ts",
|
||||
"chat-catalog-domains/external-personal-secret-menu.ts",
|
||||
"chat-catalog-domains/external-personal-secret-link.ts",
|
||||
"../../ui/src/pages/secrets/MyUserSecretsTab.tsx",
|
||||
"chat-catalog-domains/external-personal-secret-confirmation.ts",
|
||||
"chat-catalog-contract.ts",
|
||||
@@ -631,7 +635,7 @@ export const personalSecretExternalCases = [
|
||||
const commentHandoff = !!reply && reply.runId === run.id
|
||||
&& comments.some(comment => comment.id === reply.commentId && comment.companyId === source.companyId
|
||||
&& comment.issueId === source.id && comment.authorAgentId === run.agentId && comment.createdByRunId === run.id
|
||||
&& comment.body?.trim() === reply.body.trim()) && personalSecretGuidance(reply.body);
|
||||
&& comment.body?.trim() === reply.body.trim());
|
||||
if (initial.companyId !== source.companyId || (pendingQuestions.length ? !question && !confirmation : !commentHandoff))
|
||||
throw new Error("Personal setup must direct the current person's secure publishing value before human action");
|
||||
state.initialRunId = run.id;
|
||||
@@ -643,6 +647,22 @@ export const personalSecretExternalCases = [
|
||||
}
|
||||
if (question) Object.assign(state, { initialQuestionId: question.interactionId,
|
||||
initialQuestionOptionId: question.optionId, initialQuestionPayloadHash: question.payloadHash });
|
||||
const path = `/${ctx.input.fixtures.company.issuePrefix}/company/settings/secrets`,
|
||||
// Graded by its rendered link, followed as a person would; wording is left to human review.
|
||||
// A question or confirmation card already carries the handoff, so a reply link is then optional.
|
||||
replyLink = !question && !confirmation && commentHandoff && reply?.commentId ? await followPersonalSecretReplyLink(ctx.input.page, {
|
||||
commentId: reply.commentId, origin: initial.origin, prefix: String(ctx.input.fixtures.company.issuePrefix),
|
||||
definitionId: String(state.definitionId), definitionKey: "PERSONAL_PUBLISHING_KEY", timeout: remaining(),
|
||||
}) : undefined,
|
||||
link = replyLink?.route;
|
||||
if (replyLink) {
|
||||
state.replyLinkProof = replyLink;
|
||||
await ctx.input.evidence("personal-secret-reply-link.json", replyLink);
|
||||
}
|
||||
if (!link && !question && !confirmation)
|
||||
throw new Error(
|
||||
"Actual reply must link to the current person's own-value entry for this definition",
|
||||
);
|
||||
if (
|
||||
(
|
||||
await ctx.input.api.get<OwnEntry[]>(`${root(ctx)}/me/user-secrets`)
|
||||
@@ -651,22 +671,15 @@ export const personalSecretExternalCases = [
|
||||
throw new Error(
|
||||
"Current person's value was created before secure human entry",
|
||||
);
|
||||
const path = `/${ctx.input.fixtures.company.issuePrefix}/company/settings/secrets`,
|
||||
link = commentHandoff ? await personalSecretSetupURL(ctx.input.page,reply!,initial.origin,String(ctx.input.fixtures.company.issuePrefix),String(state.definitionId),remaining(),proof=>{state.personalMenuNavigation=proof;}) : undefined;
|
||||
if (!link && !question && !confirmation)
|
||||
throw new Error(
|
||||
"Actual reply must direct the supported own-value settings path for this definition",
|
||||
);
|
||||
if(state.personalMenuNavigation)await ctx.input.evidence("personal-secret-menu-navigation.json",state.personalMenuNavigation);
|
||||
const url = new URL(question || confirmation ? path : link!, initial.origin);
|
||||
if (!question && !confirmation && !state.personalMenuNavigation) url.searchParams.set("definition", String(state.definitionId));
|
||||
if (!question && !confirmation) url.searchParams.set("definition", String(state.definitionId));
|
||||
const receipt = await privateBrowserEntry({
|
||||
mode: "personal-secret-create",
|
||||
companyId: ctx.input.fixtures.company.id,
|
||||
personal: {
|
||||
definitionId: String(state.definitionId),
|
||||
userId: String(state.currentPersonId),
|
||||
viaMySecrets: !!question || !!confirmation || !!state.personalMenuNavigation,
|
||||
viaMySecrets: !!question || !!confirmation,
|
||||
},
|
||||
baseURL: initial.origin,
|
||||
setupURL: url.href,
|
||||
|
||||
Reference in new issue
Block a user