mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
fix(evals): compare browser zones by clock, wait for the reloaded profile name, bind accounting reads by returned run (#21760)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature eval catalog runs real agent conversations and grades the saved outcome > - Chat-eval cohort 16 (OpenCode with a DeepSeek model) had three failures that were not product failures of the chat agent > - One setup check compared time zone names, one observer read a form before it filled, and one grader accepted only exact query digests > - An eval that fails valid behavior hides real regressions behind noise, so each check must grade the outcome, not an incidental detail > - This pull request fixes the three checks, and it fixes a first-frame empty display name on the Profile page > - The benefit is that these three cases now fail only for the behavior they test ## Linked Issues or Issue Description **What happened?** Three chat-eval cases failed in cohort 16 (OpenCode, `openrouter/deepseek/deepseek-v4.1-flash-1017`) on correct behavior. (1) `activity.routine-audit`: setup threw "Browser timezone override did not apply (America/Buenos_Aires)". At 11:00-11:59 UTC the picker selects `America/Argentina/Buenos_Aires`, and Chrome reports that zone by its alias `America/Buenos_Aires`. The check compared names. (2) `navigation-preferences.edit-profile`: the name saved correctly, but the observer read the display-name field once, right after reload, and got an empty value. (3) `budgets-costs.spend-breakdown`: the answer was correct, but the grader accepted company cost reports only for `period=all` or `period=month`. The agent read `costs/by-agent` and `costs/summary` for the requested week, and that range covers the unpriced run. **Expected behavior** Each case passes when the product and the agent do the requested work. Each case fails when they do not. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.activity.routine-audit` between 11:00 and 11:59 UTC. 2. Run `chat-features.runner-opencode.local.navigation-preferences.edit-profile` and look at `profileAfterReload`. 3. Grade the retained cohort 16 observation of `budgets-costs.spend-breakdown` with the old grader. **Additional context** The run tool inputs are saved only as SHA-256 digests, so a grader binds a read by recomputing the digest of inputs it can check. For the cohort 16 spend-breakdown reads, the inputs were recovered this way: `from=2026-11-09T00:00:00.000Z`, `to=2026-11-16T00:00:00.000Z`. ## What Changed - `internal-routine-audit.ts`: the browser override check compares the UTC offset of the requested and reported zones, now and 182 days later, instead of the zone names. An unknown zone or a different clock still fails setup. - `identity-profile.ts`: after reload, `settledFieldValue` waits up to 15 s for the display name to reach the saved value, then records what the person sees. The grade still fails a name that does not arrive. - `ProfileSettings.tsx`: unedited fields show the session's saved name and image from the first frame. Before, a session restored from the saved read cache rendered an empty name until an effect filled it. - `identity-accounting.ts`: `readSavedAccounting` also accepts a company cost report whose checked `from`/`to` covers the unpriced run's cost event. The run log keeps only the input digest, so the grader recomputes digests for covering ranges: UTC day boundaries (with or without milliseconds or seconds, date-only, end of day), local midnights at each whole-hour offset, and the run's own `startedAt` and cost-event instants. A pending run (no cost event) counts only through `costs/summary`, `costs/data-quality` or its own run record, because the other reports sum cost events. A link to the run does not count, because the run id is in the task description. The run-log redaction is not changed. - Oracle versions bumped: `routine-audit.v7`, `spend-breakdown.structural.v6`, `identity-profile.v4`. ## Verification - Red/green: each new test fails on the old code and passes on the new code. - `internal-remaining.test.ts`: every candidate zone, reported by its own name, its ICU name and known browser aliases, passes at every hour in January and July. A different clock, a zone with daylight saving time and an unknown name fail. - `identity-personal.test.ts`: the settled read returns the saved name after empty reads, and returns a wrong name when the saved name never arrives. - `ProfileSettings.test.tsx`: the first rendered frame with a cached session shows "Alex Chen", not `""`. - `identity-accounting.test.ts`: the cohort 16 week-range reads pass, as do other covering ranges. Negative controls fail: no reads, run-list reads with unchecked inputs, ranges that exclude the run, another company, a denied read, a read by another run, and a read of a different run record with a link to the target run. - The retained cohort 16 spend-breakdown observation passes the new grader, and fails with its reads removed. - `pnpm test:e2e:runner:unit`: 686 files passed. `pnpm test:e2e:runner:typecheck` passed. `ui` ProfileSettings tests passed. `scripts/check-token-gates.mjs` is clean. - One narrow paid run of the three cases on this branch (result in a comment below). ## Risks - Low risk. The changes are in eval code, plus a small Profile page state change. - A covering range in a form outside the recomputed set (for example 08:00 at -03:00) is not recognized. That is a false negative, not a false pass. - The Profile page now keeps an unedited field in step with the session. A refetch still resets drafts, as before. ## Model Used - Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
c5218f5ec4
commit
f7a3497e9d
9 files changed
+305
-24
No files matched your search
@@ -235,7 +235,7 @@
|
||||
"chat-catalog-domains/internal-routine-audit.ts",
|
||||
"run-observations.ts"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.internal.routine-audit.v6",
|
||||
"oracleVersion": "paperclip.chat-catalog.internal.routine-audit.v7",
|
||||
"coverageGaps": [],
|
||||
"expectedRunCount": 2,
|
||||
"minimumRunCount": 2,
|
||||
@@ -1259,7 +1259,7 @@
|
||||
"chat-catalog-domains/api-read-receipts.ts",
|
||||
"run-observations.ts"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.spend-breakdown.structural.v5",
|
||||
"oracleVersion": "paperclip.chat-catalog.spend-breakdown.structural.v6",
|
||||
"expectedRunCount": 2,
|
||||
"turnTimeoutMs": 120000,
|
||||
"attemptTimeoutMs": 600000,
|
||||
@@ -3668,7 +3668,7 @@
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/navigation-preferences.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.identity-profile.v3",
|
||||
"oracleVersion": "paperclip.chat-catalog.identity-profile.v4",
|
||||
"expectedRunCount": 1,
|
||||
"turnTimeoutMs": 120000,
|
||||
"attemptTimeoutMs": 300000,
|
||||
|
||||
@@ -15,7 +15,8 @@ function valid(scenarioId: string): IdentityObservation {
|
||||
activity: repair ? [{ id: "retry", action: "accounting.retry_requested", entityId: "original", actorType: "user" }] : [],
|
||||
state: { worker: { id: "morgan" }, project: { id: "launch" }, preflightTask: { id: "preflight", identifier: "ACO-5" }, morganReceiptTask: { id: "morgan-receipt" }, rileyReceiptTask: { id: "riley-receipt" }, originalRunId: "original",
|
||||
originalRun: { id: "original", status: "succeeded", usageJson: { inputTokens: 10 } }, afterOriginalRun: { id: "original", status: "succeeded", usageJson: { inputTokens: 10 } },
|
||||
originalUsagePreserved: true, originalLogPreserved: true, originalTaskPreserved: true, unresolvedBefore: { canRetry: true, runs: [{ runId: "original", reason: "retryable" }], events: [] },
|
||||
originalUsagePreserved: true, originalLogPreserved: true, originalTaskPreserved: true, unresolvedBefore: repair ? { canRetry: true, runs: [{ runId: "original", reason: "retryable" }], events: [] }
|
||||
: { canRetry: false, runs: [], events: [{ runId: "original", reason: "missing_price", occurredAt: "2026-10-28T10:00:00.000Z" }] },
|
||||
receiptOrigin: "public-imported-synthetic-fixture", weekFrom: "2026-10-26T00:00:00Z", weekTo: "2026-11-02T00:00:00Z",
|
||||
issueCosts: [{ costCents: 137 }, { costCents: 263 }], projectCosts: [{ projectId: "launch", costCents: 400 }], agentRetryStatus: 403, humanRetryHttpStatus: 200, retryAccounted: false, retryReceiptVisible: true, retryQuality: { runs: [{ runId: "original", reason: "retryable" }], events: [] }, retryLedger: [],
|
||||
initialReplyCommentIds: ["c1"], leadApiCalls: repair ? [costRead("final", "GET /api/companies/{companyId}/costs/by-agent-model", 200, callApiInputDigest({ operationId: "GET /api/companies/{companyId}/costs/by-agent-model", pathParams: { companyId: "company" } }))] : [preflightSummaryRead("answer")], finalReply: "The saved cost still remains unresolved; it is not a confirmed charge.", finalReplyRunId: "final", retryHumanFollowup: "Actual retry receipt", confirmedPricedRepair: false, afterStillUnresolved: true } };
|
||||
@@ -50,6 +51,90 @@ describe("actual saved native accounting contexts", () => {
|
||||
const workerRead = valid("budgets-costs.spend-breakdown"); workerRead.state.leadApiCalls = [preflightSummaryRead("original")];
|
||||
expect(grade(workerRead).passed).toBe(false);
|
||||
});
|
||||
it("spend breakdown accepts a cost report whose checked date range covers the unpriced run (cohort 16)", () => {
|
||||
const grade = accountingIdentityCases[0]!.grade;
|
||||
const byAgent = "GET /api/companies/{companyId}/costs/by-agent";
|
||||
// The run-log receipt keeps only the input digest; the grader recomputes digests of covering ranges.
|
||||
const ranged = (query: { from?: string; to?: string; period?: string }, runId = "answer", status = 200, operationId = byAgent, pathParams: Record<string, string> = { companyId: "company" }) =>
|
||||
costRead(runId, operationId, status, callApiInputDigest({ operationId, pathParams, query }));
|
||||
const list = (digest: string) => costRead("answer", "GET /api/companies/{companyId}/heartbeat-runs", 200, digest);
|
||||
// Cohort 16: costs/by-agent and costs/summary for the requested week (copied from the task), plus run-list pages.
|
||||
const cohort16 = (calls: LeadApiCall[]) => { const o = valid("budgets-costs.spend-breakdown");
|
||||
Object.assign(o.state.originalRun as object, { startedAt: "2026-11-15T11:41:02.000Z", finishedAt: "2026-11-15T11:43:40.000Z" });
|
||||
o.state.unresolvedBefore = { canRetry: false, runs: [], events: [{ runId: "original", reason: "missing_price", occurredAt: "2026-11-15T11:43:40.000Z" }] };
|
||||
o.state.weekFrom = "2026-11-09T00:00:00.000Z"; o.state.weekTo = "2026-11-16T00:00:00.000Z";
|
||||
o.reply = "| Agent | Launch charges |\n|---|---|\n| Riley | $2.63 |\n| Morgan | $1.37 |\n| **Total** | **$4.00** |\n\nMorgan also finished a session on [ACO-5](/ACO/issues/ACO-5) ([the run](/ACO/agents/morgan/runs/original)) whose usage is saved but not yet priced.";
|
||||
o.links.push("/ACO/issues/ACO-5", "/ACO/agents/morgan/runs/original"); o.state.leadApiCalls = calls; return o; };
|
||||
const week = { from: "2026-11-09T00:00:00.000Z", to: "2026-11-16T00:00:00.000Z" };
|
||||
expect(grade(cohort16([ranged(week), list("page-one"), list("page-two")])).passed).toBe(true);
|
||||
// Covering ranges in the forms the grader recognizes: date-only, seconds-less, open-ended, a local-midnight offset
|
||||
// form, a from equal to the run's startedAt, and an all-time read. The cost event is at 11:43:40Z.
|
||||
for (const query of [{ from: "2026-11-09", to: "2026-11-15T23:59:59Z" }, { from: "2026-11-15T00:00Z" }, { to: "2026-11-16" },
|
||||
{ from: "2026-11-15T00:00:00-03:00", to: "2026-11-16T00:00:00-03:00" }, { from: "2026-11-15T11:41:02.000Z", to: "2026-11-15T11:43:40.000Z" },
|
||||
{ from: "2026-11-15T11:41:02Z" }, { period: "all" }, { from: "2026-11-15T00:00:00+00:00" }, { from: "2026-11-09T00:00:00-00:00", to: "2026-11-16T00:00:00-00:00" },
|
||||
{ period: "month", from: "2026-11-09T00:00:00.000Z", to: "2026-11-16T00:00:00.000Z" }, { from: "2026-11-01" }, { from: "2026-11-01T00:00:00Z", to: "2026-12-01T00:00:00Z" }])
|
||||
expect(grade(cohort16([ranged(query)])).passed, JSON.stringify(query)).toBe(true);
|
||||
expect(grade(cohort16([ranged(week, "answer", 200, "GET /api/companies/{companyId}/costs/summary", {})])).passed).toBe(true);
|
||||
// Negative controls: no read, run-list pages whose inputs prove nothing, a range that excludes the run, another
|
||||
// company, a denied read, and a read by another run. A link to the run (its id is in the task) proves no read.
|
||||
expect(grade(cohort16([])).passed).toBe(false);
|
||||
expect(grade(cohort16([list("page-one"), list("page-two")])).passed).toBe(false);
|
||||
// Ranges that exclude the cost event: an earlier week, one ending before the event (also in an offset form), one
|
||||
// starting after it.
|
||||
// An explicit range wins over period=month: last week's bounds with period=month exclude the event too.
|
||||
for (const query of [{ from: "2026-11-02", to: "2026-11-09" }, { to: "2026-11-15T11:43:00Z" }, { from: "2026-11-14T00:00:00-03:00", to: "2026-11-15T00:00:00-03:00" },
|
||||
{ from: "2026-11-15T11:44:00Z" }, { from: "2026-11-16" }, { period: "month", from: "2026-11-02", to: "2026-11-09" }])
|
||||
expect(grade(cohort16([ranged(query)])).passed, JSON.stringify(query)).toBe(false);
|
||||
// The digest names the company: another company's report does not count.
|
||||
expect(grade(cohort16([ranged(week, "answer", 200, byAgent, { companyId: "other-company" })])).passed).toBe(false);
|
||||
expect(grade(cohort16([ranged(week, "answer", 403)])).passed).toBe(false);
|
||||
expect(grade(cohort16([ranged(week, "original")])).passed).toBe(false);
|
||||
// The target's cost event sets the range: a range covering the run but ending before its late cost event fails.
|
||||
const lateEvent = cohort16([ranged({ from: "2026-11-15T00:00:00Z", to: "2026-11-15T11:50:00Z" })]);
|
||||
(lateEvent.state.unresolvedBefore as { events: unknown[] }).events = [{ runId: "original", reason: "missing_price", occurredAt: "2026-11-15T11:52:00.000Z" }];
|
||||
expect(grade(lateEvent).passed).toBe(false);
|
||||
// A pending target (saved usage, no cost event) never appears in by-agent or other ledger groupings: a by-agent read
|
||||
// with a correct-looking reply fails, while summary, data-quality or the run record itself still count.
|
||||
const pendingTarget = (calls: LeadApiCall[]) => { const o = cohort16(calls); (o.state.unresolvedBefore as { runs: unknown[]; events: unknown[] }).runs = [{ runId: "original", reason: "waiting_for_receipt" }];
|
||||
(o.state.unresolvedBefore as { events: unknown[] }).events = []; return o; };
|
||||
expect(grade(pendingTarget([ranged(week)])).passed).toBe(false);
|
||||
expect(grade(pendingTarget([ranged({ period: "all" })])).passed).toBe(false);
|
||||
expect(grade(pendingTarget([preflightSummaryRead("answer")])).passed).toBe(false);
|
||||
for (const operationId of ["GET /api/companies/{companyId}/costs/summary", "GET /api/companies/{companyId}/costs/data-quality"])
|
||||
expect(grade(pendingTarget([ranged(week, "answer", 200, operationId)])).passed, operationId).toBe(true);
|
||||
const targetRecord = costRead("answer", "GET /api/heartbeat-runs/{runId}", 200, callApiInputDigest({ operationId: "GET /api/heartbeat-runs/{runId}", pathParams: { runId: "original" } }));
|
||||
expect(grade(pendingTarget([targetRecord])).passed).toBe(true);
|
||||
// A late-month event (Nov 29) read with a month-start from (28 days earlier) or an end-of-month to counts.
|
||||
for (const query of [{ from: "2026-11-01" }, { from: "2026-11-01T00:00:00Z", to: "2026-11-30T23:59:59.999Z" }, { from: "2026-11-01T00:00:00+00:00", to: "2026-12-01T00:00:00+00:00" }]) {
|
||||
const late = cohort16([ranged(query)]);
|
||||
Object.assign(late.state.originalRun as object, { startedAt: "2026-11-29T10:00:00.000Z", finishedAt: "2026-11-29T10:02:00.000Z" });
|
||||
late.state.unresolvedBefore = { canRetry: false, runs: [], events: [{ runId: "original", reason: "missing_price", occurredAt: "2026-11-29T10:02:00.000Z" }] };
|
||||
expect(grade(late).passed, JSON.stringify(query)).toBe(true);
|
||||
}
|
||||
// A report ending at 11:43:40Z excludes an event at 11:43:40.598Z (bounds are compared as parsed instants).
|
||||
const subSecond = cohort16([ranged({ from: "2026-11-15T00:00:00Z", to: "2026-11-15T11:43:40Z" })]);
|
||||
subSecond.state.unresolvedBefore = { canRetry: false, runs: [], events: [{ runId: "original", reason: "missing_price", occurredAt: "2026-11-15T11:43:40.598Z" }] };
|
||||
expect(grade(subSecond).passed).toBe(false);
|
||||
const subSecondExact = cohort16([ranged({ from: "2026-11-15T00:00:00Z", to: "2026-11-15T11:43:40.598Z" })]);
|
||||
subSecondExact.state.unresolvedBefore = subSecond.state.unresolvedBefore;
|
||||
expect(grade(subSecondExact).passed).toBe(true);
|
||||
// Unranged and period=month reads cover the UTC month when the read ran. In-month they count; across a month
|
||||
// rollover (event on Oct 31, read on Nov 1) they do not, while an all-time read still does.
|
||||
const monthRead = (query: { period?: string } | undefined, eventAt: string, readStart: string, readEnd: string) => {
|
||||
const o = cohort16([costRead("answer", byAgent, 200, callApiInputDigest({ operationId: byAgent, pathParams: { companyId: "company" }, ...(query ? { query } : {}) }))]);
|
||||
Object.assign(o.state.originalRun as object, { startedAt: eventAt, finishedAt: eventAt });
|
||||
o.state.unresolvedBefore = { canRetry: false, runs: [], events: [{ runId: "original", reason: "missing_price", occurredAt: eventAt }] };
|
||||
Object.assign(o.runs.find(run => run.id === "answer")!, { startedAt: readStart, finishedAt: readEnd }); return o; };
|
||||
for (const query of [{ period: "month" }, undefined]) {
|
||||
expect(grade(monthRead(query, "2026-11-15T11:43:40.000Z", "2026-11-15T11:50:00.000Z", "2026-11-15T11:52:00.000Z")).passed, JSON.stringify(query)).toBe(true);
|
||||
expect(grade(monthRead(query, "2026-10-31T23:58:00.000Z", "2026-11-01T00:01:00.000Z", "2026-11-01T00:03:00.000Z")).passed, JSON.stringify(query)).toBe(false);
|
||||
expect(grade(monthRead(query, "2026-10-31T23:58:00.000Z", "2026-10-31T23:59:00.000Z", "2026-11-01T00:01:00.000Z")).passed, JSON.stringify(query)).toBe(false);
|
||||
}
|
||||
expect(grade(monthRead({ period: "all" }, "2026-10-31T23:58:00.000Z", "2026-11-01T00:01:00.000Z", "2026-11-01T00:03:00.000Z")).passed).toBe(true);
|
||||
// The answering run reads a different run's record but links the target run's id: still no read of its accounting.
|
||||
const otherRecord = costRead("answer", "GET /api/heartbeat-runs/{runId}", 200, callApiInputDigest({ operationId: "GET /api/heartbeat-runs/{runId}", pathParams: { runId: "answer" } }));
|
||||
expect(grade(cohort16([otherRecord, list("page-one")])).passed).toBe(false);
|
||||
});
|
||||
it("a HTTP200 retry alone does not establish repaired accounting", () => { const o = valid("budgets-costs.repair-cost-recording"); o.state.afterStillUnresolved = false; expect(accountingIdentityCases[1]!.grade(o).passed).toBe(false); });
|
||||
it("a genuine confirmed priced repair is distinct from still-unresolved", () => { const o = valid("budgets-costs.repair-cost-recording"); o.state.confirmedPricedRepair = true; o.state.afterStillUnresolved = false; o.state.retryAccounted = true; o.state.retryQuality = { runs: [], events: [] }; o.state.retryLedger = [{ agentId: "morgan", eventCount: 1, estimatedEventCount: 0 }]; o.state.finalReply = "The saved usage cost is now recorded. The original work was not rerun."; expect(accountingIdentityCases[1]!.grade(o).passed).toBe(true); });
|
||||
it("grades no reply wording: reworded guidance and explanations pass on the same saved evidence", () => {
|
||||
|
||||
@@ -91,12 +91,24 @@ export function leadApiCalls(events: readonly Record<string, unknown>[]): LeadAp
|
||||
* date range (a range could exclude the run), the prerequisite task's cost summary including the task itself,
|
||||
* and the original run's record (which carries its saved accounting state and usage cost status). */
|
||||
const COMPANY_COST_SEGMENTS = ["summary", "by-agent", "by-agent-model", "by-provider", "by-biller", "by-project", "data-quality"];
|
||||
/** The company reports that also count runs with saved usage but no cost event yet (pendingRunCount, unresolved runs).
|
||||
* The ledger groupings (by-agent, by-project, ...) only sum cost events, so they never show such a run. */
|
||||
const PENDING_RUN_SEGMENTS = ["summary", "data-quality"];
|
||||
export interface SavedAccountingTarget { companyId?: string; task?: Pick<Entity, "id" | "identifier">; originalRunId?: string;
|
||||
/** Run-record reads the reading run itself cites in its saved result, as `GET /api/heartbeat-runs/<runId>`. */
|
||||
citedRunReads?: readonly string[];
|
||||
/** Run records this reading run saved with `saveResponse`, read back from the product's saved asset. */
|
||||
savedRunRecords?: readonly SavedRunRecord[];
|
||||
runListPosition?: RunListPosition }
|
||||
runListPosition?: RunListPosition;
|
||||
/** When the original run started and finished. A company cost report whose `from`/`to` range covers it returns
|
||||
* the run's saved accounting, so that ranged read counts as well as an unranged one. */
|
||||
originalRunWindow?: { startedAt: string; finishedAt: string };
|
||||
/** When the original run's unresolved cost events occurred. An empty list means the run is pending with no cost event,
|
||||
* so only reports that include pending runs (summary, data-quality) or the run record can show it. Left out, every
|
||||
* company report counts (the post-retry check, which reads after the person recorded the cost). */
|
||||
costEventTimes?: readonly string[];
|
||||
/** When the reading run started and finished; month-relative reads are compared with it. */
|
||||
readingRunWindow?: { startedAt: string; finishedAt: string } }
|
||||
export interface RunListPosition { agentId: string; companyAhead: number; agentAhead: number }
|
||||
const RUN_LIST_OPERATION = "GET /api/companies/{companyId}/heartbeat-runs";
|
||||
const runListCache = new Map<string, ReadonlySet<string>>();
|
||||
@@ -125,12 +137,106 @@ export function runListPosition(runs: ReadonlyArray<{ id: string; agentId: strin
|
||||
/** A `call_api` read saved with `saveResponse` becomes a company asset that the product logs as
|
||||
* `asset.created` by the reading run. Its saved content names the record actually read. */
|
||||
export interface SavedRunRecord { runId: string; assetId: string; recordId: string; companyId: string }
|
||||
const coveringCache = new Map<string, { digests: Set<string>; pending: Generator<string> }>();
|
||||
/** Calendar spans (in days) of the ranges people ask for: a day or a few, a week, two weeks and a month. */
|
||||
const RANGE_DAYS = [1, 2, 3, 7, 8, 14, 15, 28, 29, 30, 31, 32];
|
||||
/** Local-midnight ranges at an offset: a day, a week, two weeks or a calendar month. */
|
||||
const LOCAL_RANGE_DAYS = [1, 7, 14, 28, 29, 30, 31];
|
||||
/**
|
||||
* Whether one of the run's company cost-report reads (by input digest) has a `from`/`to` covering the window. Run-log receipts keep only the input
|
||||
* digest, so the grader regenerates the inputs people write and re-parses each bound as the costs route does (both
|
||||
* bounds inclusive, as in resolveCostDateRange). Bounds are UTC day boundaries (with or without milliseconds or seconds,
|
||||
* date-only, end of day) or local midnights at a whole-hour offset (UTC also as +00:00 and -00:00), paired within one
|
||||
* form, open-ended or spanning one of RANGE_DAYS, plus the window's own instants. Every range is also generated with
|
||||
* period=month, since explicit bounds take precedence. A covering range outside these forms is not recognized: a false
|
||||
* negative, never a false pass.
|
||||
*/
|
||||
export function coveringCostReportRead(companyId: string, window: { startedAt: string; finishedAt: string }, segments: readonly string[],
|
||||
extraInstants: readonly string[], readDigests: ReadonlySet<string>): boolean {
|
||||
if (!readDigests.size) return false;
|
||||
// One lazily filled digest set per company, window and report; hashing stops at the first matching read and resumes
|
||||
// from there for the next check, so each covering input is hashed at most once.
|
||||
return segments.some(segment => {
|
||||
const key = JSON.stringify([companyId, window, segment, extraInstants]);
|
||||
let state = coveringCache.get(key);
|
||||
if (!state) coveringCache.set(key, state = { digests: new Set(), pending: coveringDigests(companyId, window, segment, extraInstants) });
|
||||
for (const digest of readDigests) if (state.digests.has(digest)) return true;
|
||||
for (let next = state.pending.next(); !next.done; next = state.pending.next()) { state.digests.add(next.value); if (readDigests.has(next.value)) return true; }
|
||||
return false;
|
||||
});
|
||||
}
|
||||
function* coveringDigests(companyId: string, window: { startedAt: string; finishedAt: string }, segment: string, extraInstants: readonly string[]): Generator<string> {
|
||||
const started = Date.parse(window.startedAt), finished = Date.parse(window.finishedAt);
|
||||
if (!Number.isFinite(started) || !Number.isFinite(finished)) return;
|
||||
const day = 86_400_000, parse = (text: string) => new Date(text).getTime();
|
||||
const iso = (at: number) => new Date(at).toISOString();
|
||||
// Bound forms: each maps a UTC date to the string a person writes for that day's midnight in the form's zone.
|
||||
const ms = (at: number) => iso(at), seconds = (at: number) => iso(at).replace(".000Z", "Z"), minutes = (at: number) => iso(at).replace(":00.000Z", "Z");
|
||||
const date = (at: number) => iso(at).slice(0, 10), endMs = (at: number) => iso(at - 1), endSeconds = (at: number) => iso(at - 1000).replace(".000Z", "Z");
|
||||
const starts = [ms, seconds, minutes, date], ends = [ms, seconds, minutes, date, endMs, endSeconds];
|
||||
// Pairs are written in one style: a precise start with a precise or end-of-day end, a date with a date or end of day.
|
||||
const forms: Array<{ from: Array<(midnight: number) => string>; to: Array<(midnight: number) => string>; month?: boolean; spans?: readonly number[] }> = [
|
||||
{ from: [ms, seconds], to: [ms, seconds, endMs, endSeconds], month: true }, { from: [minutes], to: [minutes], month: true }, { from: [date], to: [date, endMs, endSeconds], month: true }];
|
||||
for (const offset of Array.from({ length: 27 }, (_, index) => index - 12)) for (const zone of offset === 0 ? ["+00:00", "-00:00"] : [`${offset < 0 ? "-" : "+"}${String(Math.abs(offset)).padStart(2, "0")}:00`]) {
|
||||
const write = (midnight: number) => `${iso(midnight).slice(0, 10)}T00:00:00${zone}`;
|
||||
forms.push({ from: [write], to: [write], spans: LOCAL_RANGE_DAYS });
|
||||
}
|
||||
// UTC-form ranges are also generated with period=month (explicit bounds take precedence); offset forms are not.
|
||||
const queries: Array<Record<string, string>> = [], monthQueries: Array<Record<string, string>> = [];
|
||||
const firstDay = Math.floor(started / day) * day, lastDay = Math.floor(finished / day) * day;
|
||||
for (const { from, to, month, spans = RANGE_DAYS } of forms) {
|
||||
const before = queries.length;
|
||||
// Start days reach back as far as the longest span (32 days, so a month-start `from` is included for any event).
|
||||
for (let midnight = firstDay - 32 * day; midnight <= firstDay + day; midnight += day) for (const writeFrom of from) {
|
||||
const fromText = writeFrom(midnight); if (parse(fromText) > started) continue;
|
||||
queries.push({ from: fromText });
|
||||
for (const span of spans) for (const writeTo of to) {
|
||||
const toText = writeTo(midnight + span * day); if (parse(toText) >= finished) queries.push({ from: fromText, to: toText });
|
||||
}
|
||||
}
|
||||
for (let midnight = lastDay; midnight <= lastDay + 32 * day; midnight += day) for (const writeTo of to) { const toText = writeTo(midnight); if (parse(toText) >= finished) queries.push({ to: toText }); }
|
||||
if (month) monthQueries.push(...queries.slice(before));
|
||||
}
|
||||
const instantStart = queries.length;
|
||||
const instants = [window.startedAt, window.finishedAt, ...extraInstants].map(parse).filter(Number.isFinite).flatMap(at => [iso(at), iso(at).replace(/\.\d{3}Z$/, "Z")]);
|
||||
for (const fromText of instants) if (parse(fromText) <= started) { queries.push({ from: fromText }); for (const toText of instants) if (parse(toText) >= finished) queries.push({ from: fromText, to: toText }); }
|
||||
for (const toText of instants) if (parse(toText) >= finished) queries.push({ to: toText });
|
||||
// A UTC day-boundary bound paired with one of the window's own instants (e.g. from today's midnight to the event).
|
||||
const dayStarts = Array.from({ length: 33 }, (_, index) => firstDay - (32 - index) * day + day).flatMap(at => starts.map(write => write(at))).filter(text => parse(text) <= started);
|
||||
const dayEnds = Array.from({ length: 33 }, (_, index) => lastDay + index * day).flatMap(at => ends.map(write => write(at))).filter(text => parse(text) >= finished);
|
||||
for (const instant of instants) {
|
||||
if (parse(instant) >= finished) for (const fromText of dayStarts) queries.push({ from: fromText, to: instant });
|
||||
if (parse(instant) <= started) for (const toText of dayEnds) queries.push({ from: instant, to: toText });
|
||||
}
|
||||
for (const query of [...monthQueries, ...queries.slice(instantStart)]) queries.push({ ...query, period: "month" });
|
||||
for (const query of queries) for (const pathParams of [{ companyId }, {}, undefined])
|
||||
yield callApiInputDigest({ operationId: `GET /api/companies/{companyId}/costs/${segment}`, ...(pathParams ? { pathParams } : {}), query });
|
||||
}
|
||||
export function readSavedAccounting(calls: readonly LeadApiCall[], runId: string, target: SavedAccountingTarget) {
|
||||
const { companyId, task, originalRunId } = target;
|
||||
const companyDigests = companyId ? apiReadDigests(COMPANY_COST_SEGMENTS.map(segment => ({ operationId: `GET /api/companies/{companyId}/costs/${segment}`,
|
||||
// runnerApiUrl fills companyId from the run's company when the call omits it, so that form reads the same report.
|
||||
pathParams: [...recordParams("companyId", companyId), undefined], queries: [{ period: "all" }, { period: "month" }] }))) : new Set<string>();
|
||||
const taskDigests = task ? apiReadDigests([{ operationId: "GET /api/issues/{id}/cost-summary", pathParams: recordParams("id", task.id, task.identifier), queries: [{ excludeRoot: "false" }] }]) : new Set<string>();
|
||||
// A run with cost events shows in every report whose range covers those events; a pending run only in the reports
|
||||
// that count pending runs, over a range covering the run itself.
|
||||
const eventTimes = (target.costEventTimes ?? []).map(Date.parse).filter(Number.isFinite);
|
||||
const pending = !!target.costEventTimes && !eventTimes.length;
|
||||
const segments = pending ? PENDING_RUN_SEGMENTS : COMPANY_COST_SEGMENTS;
|
||||
const window = !target.costEventTimes ? target.originalRunWindow : pending ? target.originalRunWindow
|
||||
: { startedAt: new Date(Math.min(...eventTimes)).toISOString(), finishedAt: new Date(Math.max(...eventTimes)).toISOString() };
|
||||
// runnerApiUrl fills companyId from the run's company when the call omits it, so that form reads the same report.
|
||||
const reportReads = (queries: Array<Record<string, unknown>>) => companyId ? apiReadDigests(segments.map(segment => ({ operationId: `GET /api/companies/{companyId}/costs/${segment}`,
|
||||
pathParams: [...recordParams("companyId", companyId), undefined], queries }))) : new Set<string>();
|
||||
// An unranged read and period=month cover the current UTC month when the read ran. They count only when the target's
|
||||
// cost event (or the pending run) falls in that month for the whole reading run, so a month rollover cannot pass.
|
||||
const monthDigests = reportReads([{ period: "month" }]);
|
||||
const monthCovers = (() => {
|
||||
if (!("readingRunWindow" in target)) return true;
|
||||
if (!target.readingRunWindow || !window) return false;
|
||||
const month = (text: string) => { const at = new Date(text); return Number.isNaN(at.getTime()) ? "" : `${at.getUTCFullYear()}-${at.getUTCMonth()}`; };
|
||||
const months = new Set([window.startedAt, window.finishedAt, target.readingRunWindow.startedAt, target.readingRunWindow.finishedAt].map(month));
|
||||
return months.size === 1 && !months.has("");
|
||||
})();
|
||||
const companyDigests = new Set([...[...reportReads([{ period: "all" }])].filter(digest => !monthDigests.has(digest)), ...(monthCovers ? monthDigests : [])]);
|
||||
// A task's cost summary sums cost events, so it cannot show a pending run either.
|
||||
const taskDigests = task && !pending ? apiReadDigests([{ operationId: "GET /api/issues/{id}/cost-summary", pathParams: recordParams("id", task.id, task.identifier), queries: [{ excludeRoot: "false" }] }]) : new Set<string>();
|
||||
const runDigests = originalRunId ? apiReadDigests([{ operationId: "GET /api/heartbeat-runs/{runId}", pathParams: recordParams("runId", originalRunId) }]) : new Set<string>();
|
||||
// A re-read of the run record can carry extra input fields that change its digest. It still counts when the reading
|
||||
// run's own saved result names exactly that read of the original run; a read of any other run does not.
|
||||
@@ -139,11 +245,18 @@ export function readSavedAccounting(calls: readonly LeadApiCall[], runId: string
|
||||
// unlisted digest. A response the run saved is bound by its content instead: the saved run record is the original run.
|
||||
const originalRunSaved = !!originalRunId && (target.savedRunRecords ?? []).some(saved => saved.runId === runId
|
||||
&& saved.recordId === originalRunId && (!companyId || saved.companyId === companyId));
|
||||
// A from equal to the run's own startedAt (copied from its record) is a natural covering bound too.
|
||||
// Generated per report actually read by this run, since the covering set is large.
|
||||
const readSegments = segments.filter(segment => calls.some(call => call.runId === runId && call.operationId === `GET /api/companies/{companyId}/costs/${segment}`));
|
||||
const ok = (call: LeadApiCall) => call.runId === runId && call.ok && call.status >= 200 && call.status < 300;
|
||||
const rangedReads = new Set(calls.filter(call => ok(call) && readSegments.some(segment => call.operationId === `GET /api/companies/{companyId}/costs/${segment}`)
|
||||
&& !companyDigests.has(call.digest)).map(call => call.digest));
|
||||
const listDigests = companyId && target.runListPosition ? runListDigests(companyId, target.runListPosition) : new Set<string>();
|
||||
return calls.some(call => call.runId === runId && call.ok && call.status >= 200 && call.status < 300
|
||||
&& (companyDigests.has(call.digest) || taskDigests.has(call.digest)
|
||||
|| call.operationId === RUN_LIST_OPERATION && listDigests.has(call.digest)
|
||||
|| call.operationId === "GET /api/heartbeat-runs/{runId}" && (runDigests.has(call.digest) || originalRunCited || originalRunSaved)));
|
||||
|| call.operationId === "GET /api/heartbeat-runs/{runId}" && (runDigests.has(call.digest) || originalRunCited || originalRunSaved)))
|
||||
|| !!companyId && !!window && coveringCostReportRead(companyId, window, readSegments, target.originalRunWindow ? [target.originalRunWindow.startedAt] : [], rangedReads);
|
||||
}
|
||||
/** Run records the given run saved through `call_api` `saveResponse`, from the product's own activity and asset. */
|
||||
export async function savedRunRecordReads(ctx: ChatCatalogContext, runId: string, agentId: string): Promise<SavedRunRecord[]> {
|
||||
@@ -219,7 +332,9 @@ export function gradeAccounting(value: unknown) {
|
||||
const savedRunRecords = Array.isArray(o.state.savedRunRecords) ? o.state.savedRunRecords as SavedRunRecord[] : [];
|
||||
c.add("unresolved-record-read-from-saved-accounting", !!calls && o.runs.some(run => run.agentId === o.leadId && run.status === "succeeded"
|
||||
&& readSavedAccounting(calls, run.id, { companyId: o.companyId, task: o.state.preflightTask as Entity | undefined, originalRunId: String(o.state.originalRunId ?? ""),
|
||||
citedRunReads: citedRunReads(run), savedRunRecords, runListPosition: runListPosition(o.runs, String(o.state.originalRunId ?? ""), run) })),
|
||||
citedRunReads: citedRunReads(run), savedRunRecords, runListPosition: runListPosition(o.runs, String(o.state.originalRunId ?? ""), run),
|
||||
originalRunWindow: runWindow(o.state.originalRun as HeartbeatRun | undefined), readingRunWindow: runWindow(run as unknown as HeartbeatRun),
|
||||
costEventTimes: (quality?.events ?? []).filter(row => row.runId === o.state.originalRunId).map(row => row.occurredAt) })),
|
||||
"A chat run read the unresolved record's saved accounting through a product cost projection instead of inferring or inventing its cost.");
|
||||
} else if (o.scenarioId === "budgets-costs.repair-cost-recording") {
|
||||
c.add("real-human-saved-cost-retry", quality?.runs.some(row => row.runId === before?.id && row.reason === "retryable") === true && o.state.agentRetryStatus === 403
|
||||
@@ -322,7 +437,7 @@ export const accountingIdentityCases: readonly ChatCatalogCase[] = [
|
||||
}
|
||||
await ctx.input.api.patch(`/api/issues/${ctx.issue()!.id}`, { description: `The requested reporting week is ${weekFrom} through ${weekTo} (UTC). Launch's API charges this week are recorded on tasks ${morganReceiptTask.identifier} (Morgan) and ${rileyReceiptTask.identifier} (Riley). Morgan also finished one work session on Launch (run ${state.originalRunId}) whose usage is saved but not yet priced. Leave out charges from other projects and earlier weeks. Plan subscription fees are separate from API charges.` });
|
||||
return { ...state, riley, morganReceiptTask, rileyReceiptTask, weekFrom, weekTo, importedReceiptIds: receipts, receiptOrigin: "public-imported-synthetic-fixture" };
|
||||
}, gradeAccounting, { authentication: "human-session", oracleVersion: "paperclip.chat-catalog.spend-breakdown.structural.v5", sourceFiles: [...sources, "chat-catalog-domains/api-read-receipts.ts", "run-observations.ts"], expectedRunCount: 2, attemptTimeoutMs: 600_000, observe: observeAccounting, verifyResult: verifyAccountingLink, verifySetup: verifyUnresolvedSetup,
|
||||
}, gradeAccounting, { authentication: "human-session", oracleVersion: "paperclip.chat-catalog.spend-breakdown.structural.v6", sourceFiles: [...sources, "chat-catalog-domains/api-read-receipts.ts", "run-observations.ts"], expectedRunCount: 2, attemptTimeoutMs: 600_000, observe: observeAccounting, verifyResult: verifyAccountingLink, verifySetup: verifyUnresolvedSetup,
|
||||
setupPolicy: { funding: "standard", readiness: "all-ready", invariantCheckIds: ["genuine-unresolved-saved-native-usage"] } }),
|
||||
identityDefinition("budgets-costs.repair-cost-recording", async ctx => {
|
||||
try {
|
||||
@@ -376,6 +491,11 @@ export const accountingIdentityCases: readonly ChatCatalogCase[] = [
|
||||
/** The reply identifies the record whose usage has no settled cost (its task by link or identifier,
|
||||
* or its run id), and no line or sentence that names it states a zero amount for it. */
|
||||
/** The reply points at the unresolved record: a rendered link to its task or run, or its identifier or run id written out. */
|
||||
/** When a saved run started (or was created) and finished, for binding ranged cost reports to it. */
|
||||
export function runWindow(run: Pick<HeartbeatRun, "createdAt" | "startedAt" | "finishedAt"> | undefined) {
|
||||
const startedAt = run?.startedAt ?? run?.createdAt, finishedAt = run?.finishedAt;
|
||||
return startedAt && finishedAt ? { startedAt: new Date(startedAt).toISOString(), finishedAt: new Date(finishedAt).toISOString() } : undefined;
|
||||
}
|
||||
export function unresolvedRecordIdentified(o: { reply: string; links: string[]; origin: string }, record: Entity | undefined, runId: string): boolean {
|
||||
const keys = [record?.identifier, record?.id, runId || undefined, runId ? runId.slice(0, 8) : undefined].filter((key): key is string => typeof key === "string" && key.length > 0);
|
||||
if (!keys.length) return false;
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { profileIdentityCases } from "./identity-profile.js";
|
||||
import { profileIdentityCases, settledFieldValue } from "./identity-profile.js";
|
||||
import { companyLifecycleIdentityCases } from "./identity-company-lifecycle.js";
|
||||
import type { IdentityObservation } from "./identity-support.js";
|
||||
function valid(id: string): IdentityObservation {
|
||||
@@ -72,4 +72,12 @@ describe("personal identity and company lifecycle oracles", () => {
|
||||
const o = valid("navigation-preferences.edit-profile"); o.agents[0]!.name = "Alex Chen";
|
||||
expect(profileIdentityCases[0]!.grade(o).passed).toBe(false);
|
||||
});
|
||||
it("reads the reloaded display name once the form fills, not the empty first read (cohort 16)", async () => {
|
||||
// Cohort 16 saved the name correctly, but a one-shot read right after reload saw "" before the form filled.
|
||||
const reads = ["", "", "Alex Chen"]; let calls = 0;
|
||||
const field = { inputValue: async () => reads[Math.min(calls++, reads.length - 1)]! };
|
||||
await expect(settledFieldValue(field, "Alex Chen", 1_000, 1)).resolves.toBe("Alex Chen");
|
||||
// A name that never arrives is reported as what the person sees, so the grade still fails it.
|
||||
await expect(settledFieldValue({ inputValue: async () => "Alex Former" }, "Alex Chen", 20, 1)).resolves.toBe("Alex Former");
|
||||
});
|
||||
});
|
||||
@@ -12,6 +12,20 @@ const observers = new WeakMap<ChatCatalogContext, Person>();
|
||||
/** Unauthenticated (non-session credential) and forbidden are both denials of the human-only profile route. */
|
||||
const AGENT_DENIED: readonly number[] = [401, 403];
|
||||
const PROFILE_EDITOR_PATH = "/company/settings/instance/profile";
|
||||
/**
|
||||
* A reloaded page fills its form after the first frame. Wait until the field reaches the expected value
|
||||
* (or the deadline passes) and return what the person finally sees, so the grade reads the settled value
|
||||
* instead of a one-shot read taken before hydration.
|
||||
*/
|
||||
export async function settledFieldValue(field: { inputValue(): Promise<string> }, expected: string, timeoutMs = 15_000, intervalMs = 250) {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
for (;;) {
|
||||
const value = await field.inputValue();
|
||||
if (value === expected || Date.now() >= deadline) return value;
|
||||
await new Promise(resolve => setTimeout(resolve, intervalMs));
|
||||
}
|
||||
}
|
||||
|
||||
export function isProfileEditorLink(href: string, origin: string, prefix: string): boolean {
|
||||
try {
|
||||
const url = new URL(href, origin);
|
||||
@@ -50,7 +64,7 @@ export const profileIdentityCases: readonly ChatCatalogCase[] = [identityDefinit
|
||||
} catch (error) { await observer.dispose(); observers.delete(ctx); throw error; }
|
||||
}, gradeProfile, {
|
||||
authentication: "human-session",
|
||||
oracleVersion: "paperclip.chat-catalog.identity-profile.v3",
|
||||
oracleVersion: "paperclip.chat-catalog.identity-profile.v4",
|
||||
sourceFiles: ["chat-catalog-domains/identity-profile.ts", "chat-catalog-domains/internal-comment-reference.ts", "chat-catalog-domains/identity-auth.ts", "chat-catalog-domains/identity-private-browser.ts", "chat-catalog-domains/identity-private-browser-worker.ts", "chat-catalog-domains/identity-wizard.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", "chat-scenarios/navigation-preferences.yaml"],
|
||||
async act(ctx, state) {
|
||||
await waitForReply(ctx, chatCatalogOriginalPhaseDeadline(ctx));
|
||||
@@ -70,7 +84,7 @@ export const profileIdentityCases: readonly ChatCatalogCase[] = [identityDefinit
|
||||
await ctx.input.page.getByRole("button", { name: "Save profile", exact: true }).click();
|
||||
await expect.poll(async () => (await ctx.input.api.get<Session>("/api/auth/get-session")).user.name).toBe("Alex Chen");
|
||||
await ctx.input.page.reload({ waitUntil: "domcontentloaded" });
|
||||
state.profileAfterReload = await ctx.input.page.getByLabel("Display name", { exact: true }).inputValue();
|
||||
state.profileAfterReload = await settledFieldValue(ctx.input.page.getByLabel("Display name", { exact: true }), "Alex Chen");
|
||||
const account = ctx.input.page.getByRole("button", { name: "Open account menu", exact: true });
|
||||
await expect(account).toBeVisible();
|
||||
state.accountFooterVisible = await account.getByText("Alex Chen", { exact: true }).count() === 1;
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { expect, it } from "vitest";
|
||||
import { createHash } from "node:crypto";
|
||||
import { gradeProjectDefaults, gradeProjectDefaultsSetup, projectReadCalls } from "./internal-project-defaults.js";
|
||||
import { callApiInputDigest, gradeRoutineAudit, localClock, morningTimezone } from "./internal-routine-audit.js";
|
||||
import { callApiInputDigest, gradeRoutineAudit, localClock, morningTimezone, MORNING_CANDIDATE_TIMEZONES, sameZoneClock } from "./internal-routine-audit.js";
|
||||
import { gradeWorkTimeline } from "./internal-work-timeline.js";
|
||||
import { gradeArchivePartial } from "./internal-archive-partial.js";
|
||||
function base(scenarioId: string) { return { scenarioId, origin: "https://fixture.test", companyId: "company", leadId: "ceo", state: {} as Record<string, unknown>, tasks: [] as any[], projects: [] as any[], goals: [] as any[], comments: [] as any[], activity: [] as any[], documents: [], revisions: [], links: [] as string[], reply: "", humanQueries: [], runs: [] as any[] }; }
|
||||
@@ -101,6 +101,28 @@ it("picks a real timezone where the evaluation clock reads 08:xx in every season
|
||||
const now = new Date(Date.UTC(year, month, day, hour, 17)); expect(localClock(now, morningTimezone(now)).hour).toBe(8);
|
||||
}
|
||||
});
|
||||
// Browsers report a zone by their own canonical name. Chrome reports America/Argentina/Buenos_Aires as
|
||||
// America/Buenos_Aires, which failed every 11:xx UTC dispatch in cohort 16 under a name comparison.
|
||||
const BROWSER_ALIASES: Readonly<Record<string, readonly string[]>> = { "America/Argentina/Buenos_Aires": ["America/Buenos_Aires"], "Pacific/Pago_Pago": ["Pacific/Samoa", "US/Samoa"],
|
||||
"Pacific/Honolulu": ["US/Hawaii"], "America/Phoenix": ["US/Arizona"], "Asia/Dhaka": ["Asia/Dacca"], "Asia/Tokyo": ["Japan"], "Asia/Singapore": ["Singapore"] };
|
||||
it("accepts the browser's report of every candidate zone, at every hour, by its clock rather than its name", () => {
|
||||
for (const zone of MORNING_CANDIDATE_TIMEZONES) {
|
||||
const reported = [zone, new Intl.DateTimeFormat("en-US", { timeZone: zone }).resolvedOptions().timeZone, ...(BROWSER_ALIASES[zone] ?? [])];
|
||||
for (let hour = 0; hour < 24; hour++) for (const month of [0, 6]) {
|
||||
const now = new Date(Date.UTC(2026, month, 15, hour, 52));
|
||||
for (const name of reported) expect(sameZoneClock(name, zone, now), `${name} for ${zone} at ${now.toISOString()}`).toBe(true);
|
||||
}
|
||||
}
|
||||
for (let hour = 0; hour < 24; hour++) {
|
||||
const now = new Date(Date.UTC(2026, 10, 15, hour, 52)); const zone = morningTimezone(now);
|
||||
expect(sameZoneClock(zone === "America/Argentina/Buenos_Aires" ? "America/Buenos_Aires" : zone, zone, now)).toBe(true);
|
||||
}
|
||||
// A different clock, a zone with daylight saving time, or an unknown name is still an override that did not apply.
|
||||
const now = new Date(Date.UTC(2026, 10, 15, 11, 52));
|
||||
expect(sameZoneClock("UTC", "America/Argentina/Buenos_Aires", now)).toBe(false);
|
||||
expect(sameZoneClock("America/Santiago", "America/Argentina/Buenos_Aires", now)).toBe(false);
|
||||
expect(sameZoneClock("Not/AZone", "America/Argentina/Buenos_Aires", now)).toBe(false);
|
||||
});
|
||||
it("accepts a real edit before a failed scheduled morning run that the chat run read", () => expect(gradeRoutineAudit(routine()).passed).toBe(true));
|
||||
it("does not grade the reply wording", () => { for (const reply of ["Nothing useful happened today — the scheduled 8:42 attempt stalled when Morgan couldn't find the readiness doc.", "Short answer: no. Details are on the routine page."]) { const o = routine(); o.reply = reply; expect(gradeRoutineAudit(o).passed).toBe(true); } });
|
||||
it("accepts a redacted call_api read of the routine runs and a link to the blocked execution task", () => {
|
||||
|
||||
@@ -30,6 +30,22 @@ export function morningTimezone(now: Date) {
|
||||
if (!zone) throw new Error("No candidate timezone currently reads 08:xx");
|
||||
return zone;
|
||||
}
|
||||
/** Minutes the zone's wall clock is ahead of UTC at the instant. */
|
||||
export function utcOffsetMinutes(instant: Date, timezone: string) {
|
||||
const parts = Object.fromEntries(new Intl.DateTimeFormat("en-US", { timeZone: timezone, year: "numeric", month: "2-digit", day: "2-digit", hour: "2-digit", minute: "2-digit", hourCycle: "h23" })
|
||||
.formatToParts(instant).map(part => [part.type, part.value]));
|
||||
const wall = Date.UTC(Number(parts.year), Number(parts.month) - 1, Number(parts.day), Number(parts.hour), Number(parts.minute));
|
||||
return Math.round((wall - Math.floor(instant.getTime() / 60_000) * 60_000) / 60_000);
|
||||
}
|
||||
/**
|
||||
* Whether the browser's reported zone keeps the same clock as the requested one. Browsers report a zone by
|
||||
* their own canonical name, which may be an alias (Chrome reports America/Argentina/Buenos_Aires as
|
||||
* America/Buenos_Aires), so names are not compared: the UTC offsets must agree now and half a year away.
|
||||
*/
|
||||
export function sameZoneClock(reported: string, requested: string, now: Date) {
|
||||
const later = new Date(now.getTime() + 182 * 86_400_000);
|
||||
try { return [now, later].every(instant => utcOffsetMinutes(instant, reported) === utcOffsetMinutes(instant, requested)); } catch { return false; }
|
||||
}
|
||||
function morning(instant: string | undefined, timezone: string, observedAt: string) {
|
||||
return !!instant && localDay(instant, timezone) === localDay(observedAt, timezone) && localClock(new Date(instant), timezone).hour < 12;
|
||||
}
|
||||
@@ -123,7 +139,7 @@ function routineAudit(): ChatCatalogCase {
|
||||
// The person's browser lives in the routine's timezone, where it is now morning.
|
||||
const cdp = await page.context().newCDPSession(page); await cdp.send("Emulation.setTimezoneOverride", { timezoneId: timezone });
|
||||
const browserTimezone = await page.evaluate(() => Intl.DateTimeFormat().resolvedOptions().timeZone);
|
||||
if (browserTimezone !== timezone) throw new Error(`Browser timezone override did not apply (${browserTimezone})`);
|
||||
if (!sameZoneClock(browserTimezone, timezone, new Date())) throw new Error(`Browser timezone override did not apply (requested ${timezone}, browser reports ${browserTimezone})`);
|
||||
const p = await project(ctx, "Launch"); const worker = await fixtureWorker(ctx);
|
||||
// The harness funds the company and lead before setup; fund the routine's worker the same way so its
|
||||
// unpriced setup turn does not budget-pause it before the query.
|
||||
@@ -154,7 +170,7 @@ function routineAudit(): ChatCatalogCase {
|
||||
await ctx.input.evidence("chat-catalog-routine-audit-setup.json", { companyId: ctx.input.fixtures.company.id, timezone, trigger, edit, runs: outcome.runs, task: outcome.task, now });
|
||||
return { project: p, worker: { id: worker.id }, routine: saved, timezone, editEventId: edit.id, failedRunId: failed.id, executionTaskId: failed.linkedIssueId, routineRunCount: outcome.runs.length, now };
|
||||
}, gradeRoutineAudit);
|
||||
return { ...base, oracleVersion: "paperclip.chat-catalog.internal.routine-audit.v6", authentication: "human-session", expectedRunCount: 2, minimumRunCount: 2, attemptTimeoutMs: 720_000,
|
||||
return { ...base, oracleVersion: "paperclip.chat-catalog.internal.routine-audit.v7", authentication: "human-session", expectedRunCount: 2, minimumRunCount: 2, attemptTimeoutMs: 720_000,
|
||||
sourceFiles: [...base.sourceFiles, "chat-catalog-domains/internal-routine-audit.ts", "run-observations.ts"],
|
||||
async observe(ctx, state): Promise<RoutineAudit> {
|
||||
const id = entity(state, "routine").id; const o = await observeInternal(ctx, state); const leadEvents: RunEvent[] = [];
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
import { act } from "react";
|
||||
import { createRoot } from "react-dom/client";
|
||||
import { renderToStaticMarkup } from "react-dom/server";
|
||||
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { sanitizeAssetNamespace } from "@paperclipai/shared";
|
||||
@@ -94,6 +95,19 @@ describe("ProfileSettings", () => {
|
||||
vi.clearAllMocks();
|
||||
});
|
||||
|
||||
it("shows the saved display name in the first frame when the session is already cached", () => {
|
||||
// A reload restores the session from the saved read cache before any response. The first
|
||||
// committed frame must show the saved name, not an empty field that an effect fills later.
|
||||
const queryClient = new QueryClient({ defaultOptions: { queries: { retry: false, staleTime: Infinity } } });
|
||||
queryClient.setQueryData(queryKeys.auth.session, {
|
||||
session: { id: "session-1", userId: "user-1" },
|
||||
user: { id: "user-1", name: "Alex Chen", email: "alex@example.com", image: null },
|
||||
});
|
||||
const html = renderToStaticMarkup(<QueryClientProvider client={queryClient}><ProfileSettings /></QueryClientProvider>);
|
||||
const field = new DOMParser().parseFromString(html, "text/html").getElementById("profile-name") as HTMLInputElement | null;
|
||||
expect(field?.getAttribute("value")).toBe("Alex Chen");
|
||||
});
|
||||
|
||||
it("does not render a keyboard shortcuts toggle because shortcuts are always enabled", async () => {
|
||||
const root = createRoot(container);
|
||||
const queryClient = new QueryClient({ defaultOptions: { queries: { retry: false } } });
|
||||
|
||||
@@ -29,14 +29,18 @@ export function ProfileSettings() {
|
||||
const queryClient = useQueryClient();
|
||||
const avatarInputId = useId();
|
||||
const avatarInputRef = useRef<HTMLInputElement | null>(null);
|
||||
const [name, setName] = useState("");
|
||||
const [image, setImage] = useState("");
|
||||
// Unedited fields show the session's saved values from the first frame; a
|
||||
// restored session must not render an empty name until an effect fills it.
|
||||
const [nameDraft, setName] = useState<string | null>(null);
|
||||
const [imageDraft, setImage] = useState<string | null>(null);
|
||||
const [actionError, setActionError] = useState<string | null>(null);
|
||||
const sessionQuery = useQuery({
|
||||
queryKey: queryKeys.auth.session,
|
||||
queryFn: () => authApi.getSession(),
|
||||
});
|
||||
const sessionView = useQueryView(sessionQuery);
|
||||
const name = nameDraft ?? sessionQuery.data?.user.name ?? "";
|
||||
const image = imageDraft ?? sessionQuery.data?.user.image ?? "";
|
||||
|
||||
useEffect(() => {
|
||||
setBreadcrumbs([
|
||||
@@ -46,10 +50,8 @@ export function ProfileSettings() {
|
||||
}, [setBreadcrumbs]);
|
||||
|
||||
useEffect(() => {
|
||||
const session = sessionQuery.data;
|
||||
if (!session) return;
|
||||
setName(session.user.name ?? "");
|
||||
setImage(session.user.image ?? "");
|
||||
setName(null);
|
||||
setImage(null);
|
||||
}, [sessionQuery.data]);
|
||||
|
||||
function syncSessionProfile(profile: CurrentUserProfile) {
|
||||
|
||||
Reference in new issue
Block a user