Commit Graph
7716 Commits
Author SHA1 Message Date
Dotta a37a2e4fd5 test(chat): check run route and issue-run field without source-layout literals
Match the registered run route path and the runsForIssue select block's
runId field instead of a full JSX line and a fixed character window, so
harmless refactors do not break the guidance regression test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit a490180801)
2026-11-07 16:10:08 -06:00
DottaandClaude Opus 5.5 ca973e3b13 fix(native): do not also report a classified model rejection as unclassified
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 16:09:54 -06:00
DottaandClaude Opus 5.5 f04cbfd134 fix(evals): refusal naming never throws; derive module coverage from transfer imports
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 14:45:19 -06:00
DottaandClaude Opus 5.5 c80393e966 fix(evals): name refusals from dispatch authority and chat admission too
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 14:32:36 -06:00
Dotta 1892ba16ad fix(skills): keep the overlay promise to Cmd/Ctrl+K and say where operators apply
The sidebar Search item links to the full search page, so it does not keep
the person's place. The palette sends only free text, so operators filter
only after the Search all handoff. Correct both statements and test them
against the sidebar and palette source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit ff83378b26)
2026-11-07 14:32:03 -06:00
Dotta 05193427cb fix(chat): use runId from issue run listings in run link guidance
GET /api/issues/{id}/runs returns the run ID as runId, not id. Name the
field each response returns so chat can build the run link, and pin the
listing's field name in the regression test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 123d7fb940)
2026-11-07 14:31:42 -06:00
DottaandClaude Opus 5.5 63613c80dc fix(evals): name why a private evidence transfer refused, without its text
The trusted transfer printed one fixed line for every refusal, so a refused
attempt kept no evidence and no reason. Report a closed refusal code: a slug of a
fixed literal message from the trusted evidence modules, a Node error code, an
error type, or 'dynamic'. Message text is never printed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 14:26:58 -06:00
DottaandClaude Opus 5.5 8cc11b7593 test(evals): identify an opened agent profile by its header portrait
Since #19535 the agent profile h1 is the editable title ("Frontend
engineer"), and the name is secondary text. The runtime result check
waited for a heading named "Morgan" and failed on the correct page in
run 42406653921. Identify the profile by the header portrait, which
the page labels "<name> avatar".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 14:12:16 -06:00
DottaandClaude Opus 5.5 741ca2e94c test(evals): grade resume-agent from saved state and attribution, not reply phrases
The resume-agent grader required phrase lists in both replies. A valid
reply such as "Morgan hasn't started the work yet" failed the
no-invented-execution regex. Grade the first resume by its attribution to
the first request's own run, keep the no-worker-run check, and require
a bound reply for each phase.

Add no-agent-budget-override: in run 42404423330 the CEO lifted the
board's budget hard stop through PATCH /api/agents/{id} after the budget
route refused it. That outcome must fail even when the reply explains it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 14:12:16 -06:00
Dotta ba4ac2d81a fix(chat): link the profile of an agent chat acted on
Chat guidance defined only the task-link template. Asked to resume an
agent, the chat agent resumed it and reported the result but never
linked the agent, leaving the person no direct route to check its
saved availability.

Tell chat to link an affected agent's profile at
/<prefix>/agents/<urlKey or id>, and keep that link distinct from an
agent:// mention, which notifies the agent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 5324cfc4df)
2026-11-07 14:12:16 -06:00
DottaandClaude Opus 5.5 d1954bbbe9 test: give the budget-authority unit a DB that accepts work hints
Configuration preparation on current master publishes passive work
hints through db.execute, so the empty object mock from #17857 throws
before the budget assertions run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 14:12:16 -06:00
Dotta ab0116963b fix: keep provisioning run attribution in supported activity audit
Co-Authored-By: Paperclip <noreply@paperclip.ing>
(cherry picked from commit b6cd49d4b6)
2026-11-07 14:12:16 -06:00
Dotta 2819586b5b fix: retain approving budget authority and guard built-in provisioning
Co-Authored-By: Paperclip <noreply@paperclip.ing>
(cherry picked from commit e5682d22fa)
2026-11-07 14:12:16 -06:00
Dotta 7a36d50497 fix: save initial hire budgets within lifecycle transaction
Co-Authored-By: Paperclip <noreply@paperclip.ing>
(cherry picked from commit 8c06aba41c)
2026-11-07 14:12:16 -06:00
Dotta 0113b10ce5 fix: require board authority for agent budget policies
Co-Authored-By: Paperclip <noreply@paperclip.ing>
(cherry picked from commit f970f6b01f)
2026-11-07 14:12:15 -06:00
Dotta 40f9d71214 fix(chat): give chat replies the approval route
Chat agents get a route pattern for task links but none for approvals.
Approval tools return no href, so an agent that lists a pending approval
cannot link it without guessing. Add the company-prefixed approval route
that the board UI serves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 6af054a22e)
2026-11-07 14:11:18 -06:00
DottaandClaude Opus 5.5 f54c4761b7 test(evals): grade decision listing by source links and section structure
The show-decisions grader read only the line that named each item and
required words such as "you" or "answered" there. A correct reply that
put "waiting on you" in its list heading failed. Read each item with its
heading, require the exact approval and task links, require Riley to be
named beyond the setup titles, and check that Riley's question and the
answered request are listed apart from the person's own decision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 14:11:00 -06:00
DottaandClaude Opus 5.5 57b559cde7 test(evals): accept production cancellation intent ids; name each post-seal step
The native Stop boundary required a bare UUID intent id, but production writes
native-cancellation:<uuid>. Label the health-detail and source-SDK attach steps and
code the remaining diagnostic and SDK-proof refusals.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 14:06:46 -06:00
DottaandClaude Opus 5.5 68a9654520 test(evals): grade ambiguous-yes disambiguation without a keyword list
The reply path required one of "which/choose/select", so a valid
"Do you want the launch poster or the welcome note?" failed. Now a reply
must name both setup-chosen proposal titles and pose a question, or the
evaluated run must have created a pending question naming both. Bumps the
case oracle to paperclip.chat-catalog.identity.ambiguous-yes.v2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 14:01:56 -06:00
DottaandClaude Opus 5.5 9c2c3811f5 test(evals): identify created goals by structure, not title phrasing
goals.create-goal-tree matched model-chosen goal titles with narrow phrase
regexes, so valid titles such as "Ship the website" or "Reach 1000
customer signups" failed, and a truthful "0% complete" note counted as
invented progress. Identify the goals by level and parent and require only
the requested subject (1,000 + signups; website) in saved title/description.
Bump the oracle to goal-hierarchy.v3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 14:01:38 -06:00
DottaandClaude Opus 5.5 b968c6ae87 test(evals): grade routines.run-now on saved state and tool reads, not reply wording
The later-result check required keyword combinations in the follow-up reply and
rejected any mention of 'failed', so a correct answer saying 'nothing failed'
failed, while an acceptance-only reply could pass through the
'summary ... is' alternative. The routine output check likewise required
specific words in the routine's own comment.

Grade instead that the routine's native run read the actual Launch project
and saved its summary, and that the follow-up answer came from a new chat run
which read the completed result task after the invocation finished, with the
task linked. The first-reply guard is now sentence-scoped and ignores
future/conditional framing. Bump the case oracle to
integrator.routine-invocation.v2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:54:50 -06:00
Dotta 95c618e078 fix(chat): tell chat how to link execution runs and name their agents
Chat guidance defined only the task-link template. Asked to show a
task's runs, the chat agent had GET /api/issues/{id}/runs data with
agentId and run id but no run route or agent name, so it listed raw
UUIDs instead of linking each run and naming its agent.

Add the registered /<prefix>/agents/<agentId>/runs/<runId> route and
ask chat to read and name the agent rather than show a raw ID.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 5c326e2fb7)
2026-11-07 13:47:11 -06:00
DottaandClaude Opus 5.5 657c629679 test(evals): grade find-runs on its run-history read and exact run links, not reply wording
The interrupted-and-continuation-states-explained check matched state
words on reply lines that contained a raw run UUID plus the literal
"Morgan". Correct replies that link runs on a separate line, or use other
state words, failed; it also pushed the agent toward pasting raw IDs.

Replace it with a check that the chat run read the website task's run
history (GET /api/issues/{id}/runs or the exact run detail) through a
product tool. Exact native run links per agent, and opening them to the
real cancelled/succeeded status, remain required. Bump the oracle to
paperclip.chat-catalog.run-history.v2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:47:11 -06:00
DottaandClaude Opus 5.5 5730403309 test(evals): name Node error codes and types; read memfd-launched opencode serve
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:47:09 -06:00
Dotta c7a13acc6c fix(skills): teach the board skill how people find things in the web UI
Asked how to quickly find tasks without leaving their page, the chat agent
had no source describing quick search: it searched for docs, said it did not
know the UI, and told the person to ask in chat instead of naming Cmd/Ctrl+K.
Document the command palette, its search operators, the Search all handoff to
full search, and that the agent cannot drive the person's browser. Add a test
that ties the documented shortcuts to the palette's actual key bindings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 0595904175)
2026-11-07 13:44:35 -06:00
DottaandClaude Opus 5.5 b90f2f2bf7 test(evals): wait for the app before opening quick search; grade the actual palette shortcut
The independent observer pressed Cmd/Ctrl+K right after domcontentloaded,
before the app mounted the palette listener, so the palette never opened
(run 42404426034). Wait for the shell and retry the shortcut.

The reply check accepted any 'command ... k' or a bare 'command palette'
mention. Require the shortcut itself (a modifier followed by K, any common
spelling), which is the fact the person needs; bump the oracle version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:44:34 -06:00
DottaandClaude Opus 5.5 00b406425e test(evals): grade write-plan by brief coverage instead of a phrase matcher
The write-plan oracle required nine concept regexes plus negation parsing
over the saved plan. A correct plan flipped to failing when reworded with a
single synonym (phone for mobile, tab-key for keyboard, landing page for
single page). Measure instead how much of the task's own brief the saved
plan covers, sharing the brief between setup and grader, so paraphrases pass
and generic or foreign plans still fail. Bump the case oracle version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:44:34 -06:00
DottaandClaude Opus 5.5 56b6f25d3d test(evals): agent-feedback wrong outcome relays another agent's feedback text
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:44:26 -06:00
DottaandClaude Opus 5.5 4f5a663233 test(evals): grade enable-for-agent by saved assignment, not reply wording
Run 42404415872 enabled the skill correctly: the CEO synced only Morgan's
skills, policy and installed revision were unchanged, and Morgan's next
task received and read the exact revision. It failed only the reply
regex, which read "Beta skill versioning isn't enabled" as a denial that
Morgan could use the skill.

Replace the phrase matcher with a state check: the chat run configured
no agent other than Morgan, and it replied. Saved assignment, unchanged
policy and future delivery remain graded. Bump oracle to v3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:40:56 -06:00
DottaandClaude Opus 5.5 aef08f17eb test(evals): name post-seal refusals; read fd-launched provider executables
Post-seal attachment steps kept the seal-original stage and their messages had no
closed codes; give them an attach-original stage and codes. The post-stop fixture
could not read opencode serve launched from an inherited descriptor; fall back to
the kernel's /proc/<pid>/exe link.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:25:14 -06:00
DottaandClaude Opus 5.5 fd2765b030 test(evals): recognize opencode serve launched through node
The runner's deepest provider process is node running the extensionless opencode
launcher with serve. Qualify that shape and record absolute-argument basenames and
whether serve is present.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:08:29 -06:00
DottaandClaude Opus 5.5 4751eaf4a8 test(evals): name refusals from sealing's dependencies and filesystem errors
The seal still reported 'unclassified': its refusal came from a module it calls
or from a filesystem error. Add closed codes for framework-report, redaction and
retention messages, and report other filesystem errors by their error code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:05:19 -06:00
DottaandClaude Opus 5.5 df57973e38 test(evals): accept node flags before the proxy script; record closed process facts
The runner's provider chain is three processes deep and nothing qualified. Allow
node runtime flags before the absolute proxy script (still last, still digest
checked), and record each descendant's command name, argument count and script
basenames so the next refusal shows why.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 12:53:48 -06:00
DottaandClaude Opus 5.5 e0c81d25e2 test(evals): give every fixed evidence refusal a closed diagnostic code
Packaging, sealing and report-asset refusals that were not in the closed table
surfaced as 'unclassified', so a refused attempt said nothing about why. Add a
code for each fixed message and a test that keeps the table complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 12:42:15 -06:00
DottaandClaude Opus 5.5 b264e3a77c test(evals): search the runner's process tree for the post-stop provider
The provider launch can be wrapped, so the provider need not be a direct runner
child. Accept exactly one qualifying descendant (depth <= 4) under its actual
parent, and record a closed summary of the descendants seen.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 12:37:15 -06:00
DottaandClaude Opus 5.5 e9bf8cc0ab test(evals): find the post-stop provider as the verified runner's sole child
The local session runs inside the verified runnerd, which launches the OpenCode
proxy (or Codex app-server) as its direct child without a checkpoint record.
Choose the one qualifying child; all existing ownership checks still apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 12:22:46 -06:00
Dotta cc88d1031f Revert "fix(runner): name the spawned opencode serve in OpenCode session start"
This reverts commit af78a9c57f.
2026-11-07 12:21:45 -06:00
Dotta beb834cc51 Revert "fix(runner): name the local app-server process in Codex session start"
This reverts commit da14022655.
2026-11-07 12:21:45 -06:00
DottaandClaude Opus 5.5 da14022655 fix(runner): name the local app-server process in Codex session start
Local Codex and OpenCode (through the app-server proxy, which keeps opencode serve
in its process group) run behind the Codex app-server driver. Its session.started
carried no process identity, so the control plane's stop proof never checked that
process. Emit its pid, process group and OS start time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 12:06:43 -06:00
DottaandClaude Opus 5.5 22fb7a67e2 test(evals): qualify the recorded OpenCode provider as proxy or direct serve
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 12:06:41 -06:00
DottaandClaude Opus 5.5 af78a9c57f fix(runner): name the spawned opencode serve in OpenCode session start
The OpenCode server driver spawns `opencode serve` in its own process group but
its session.started/resumed carried no process identity. The server's stop proof
checks processId from these events, so it never verified that this provider
process had exited. Emit its pid, process group and OS start time, and share the
runner's local process start reader.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 11:48:28 -06:00
DottaandClaude Opus 5.5 51dc541273 test(evals): find the direct opencode serve from its durable session receipt
Local OpenCode now runs opencode serve directly from the runner instead of the
app-server proxy behind the runnerd transport, so the checkpoint has no provider
process. Read the provider PID and start time from the run's session receipt, and
qualify the runner's own spawned opencode serve by parent, uid, start and command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 11:48:27 -06:00
DottaandClaude Opus 5.5 3178ef875a fix(native): let a run blocked for a rejected model be retried
The status arbiter already blocks the task for a model choice. The release path
then opened a native reconciliation incident because the task was blocked, which
put the run in recovery_needed and hid its Retry. A rejected model or incomplete
configuration has no execution to reconcile; release without the incident.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 11:02:16 -06:00
DottaandClaude Opus 5.5 d22e4962f9 fix(native): record OpenCode's rejected model as a model rejection
A driver-classified model_unavailable failure that requires action now maps to
native_provider_model_rejected for any provider, so the task is blocked for a
model choice and chat shows Model unavailable instead of a generic adapter
failure. The shared blocker text no longer names a ChatGPT connection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 10:42:00 -06:00
DottaandClaude Opus 5.5 7eaf28031e test(evals): accept a lead's honest refresh of its own stale card in explain-card
The rerun's lead answered correctly (card generated before RUN-1 completed, so it
is stale), refreshed the card it authored, and said so. The harness waited for
exactly two runs, so the refresh run made it time out. Bound runs at 3..5, wait for
any refresh to settle before deleting the source, compare the access-change probe
with the card as answered, and require the original summary to stay in history,
changing only through a lead-requested refresh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 10:21:22 -06:00
DottaandClaude Opus 5.5 2b6909f755 test(runner): correct merged runner catalog sizes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 10:19:44 -06:00
DottaandClaude Opus 5.5 2af6c0feb1 Merge master into chat-eval candidate
Resolved: kept candidate OpenCode finish diagnostics alongside master's signal-aware
controller calls; kept the shared human-input description constant with master's
batched-questions wording; kept both core-contract tests; unioned run-completion
test imports; regenerated semantic tool contracts and protocol manifest; updated
runner catalog sizes (first-task 56, chat-setup-health 1, chat-features 247).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 10:19:28 -06:00
DottaandClaude Opus 5.5 d75c7c548f test(evals): grade explain-card by the agent's source reads, not reply wording
The explain-card grader required regex matches on the model's explanation and a
link back to the status card the person had attached. A correct, grounded reply
failed. Grade what the agent actually did instead: it read the live source task
and linked it before explaining, and after the source was deleted its follow-up
turn attempted a fresh read that returned 404 with no successful read.

Explanation wording is no longer graded; a reply that misstates a correctly read
live status is not detected by this oracle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 09:44:35 -06:00
Dotta 096a7c0891 fix(commentary): give a denied agent a ready Markdown link to the Commentary page
Agents relayed the bare path as inline code, which the person cannot click.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 2e3a19c2aa)
2026-11-07 09:31:00 -06:00
Dotta 5cbbc12f17 feat(commentary): agents read only their own commentary unless granted commentary:view
Product rule: an agent cannot read other agents' commentary unless it holds
commentary:view, which is not a default agent grant. An agent may read its
own entries by asking for them explicitly. Without the grant, listing all
commentary or another agent's entries is a 403 that names the human page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 0fc6a7a635)
2026-11-07 09:31:00 -06:00