The baseline attempt failed although the agent only previewed: the
before-choice check compared the existing agent's full configuration
response, whose lifecycle status and updatedAt change after setup
(preparation and harness budget funding). Compare the configuration
without those fields (configurationRevision, the product's own content
hash, stays). Replace the conflict/choice keyword regexes over the reply
with structure: the first turn ended waiting on the person with no import
activity, and the reply names the setup-chosen conflicting employee.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The revision request attaches its source task, so the shared result-navigation
check's requirement for a rendered task link rejected a correct reply that
returned both version downloads. The case still opens the returned download and
both versions' library stacks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The revise-delivery grader rejected a correct revision for four eval reasons:
the fixture footer marker ended in the nonce's attempt suffix (-1), which the
agent reasonably bumped as a version label on website-v1.html; it required a
link to the source task the person had attached; it required the revision on
the source task although the chat run's register_deliverable files it on the
requesting conversation; and it graded revision context with keyword regexes.
Hash the marker, accept the source task or the conversation, and grade
revision context structurally: a separately titled registered version plus a
returned route to the retained original.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The concurrent phase required the reply to say Casey, offline and
prototype, and the approval check scanned reply words. The read and
final phases now accept a revision identified by id or number, the final
phase accepts naming the intervening edit by revision or editor, and
saved state decides restore versus preserve. Approval is checked from
activity only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An agent asked to read the current plan fetched the revision it had just
restored by id and reported it as the latest, missing a newer edit by
another person. The exact-revision read now carries the document's
latestRevisionId, latestRevisionNumber and isLatest, and its API
description points to the current-document read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The grader required either a copied attachment or an attachment URL in
the delegated task, so it rejected the agent copying the complete brief
verbatim into the task description, which the chat guidance itself
recommends ("copy the relevant existing content into the handoff") and
which gave Morgan everything needed to build the site.
- Accept the complete exact brief text (setup-chosen literal, tolerant
only of line endings, indentation, blank lines and quote markers) in
the full saved task description or its documents, and require Morgan's
own bearer to read it there.
- Read the saved task in full; company issue listings shorten
descriptions to 1200 characters, which could also hide a reference.
- Drop the website/landing-page keyword check on the model-written title.
- Bump the case oracle version.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Chat agents asked to change the person's own display name had no route to
offer, so they searched the API, tried the human-only profile endpoint and
ended with vague "profile/account settings" directions. Tell chat where the
personal profile editor lives and that agent credentials cannot change it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The account popover check failed in strict mode because the profile page now
has its own display-name heading; scope it to the popover. Accept 401 as well
as 403 for the agent's denied profile write (a non-session credential is
unauthenticated on the human-only route). Replace the reply keyword and
negative wording regexes with a required rendered link to the person's
profile editor, which the person follows to save the name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A chat request such as "Mark the landing-page task as high priority" with
two accessible tasks of that title left the agent deliberating between
picking one, changing both and asking, until the run was cancelled. Chat
guidance now says to ask which candidate is meant, with links and the
distinguishing detail, before changing any of them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The clarify-before-write check required a question mark or one of
which/choose/select/clarify in the free-form reply. A valid question
phrased as "Tell me the one to raise" failed it, and the keywords proved
nothing the saved state and links do not. The check now relies on no
early mutation of either task and links to both candidates; the later
checks still reject picking the first match, changing both, or the wrong
task after the selection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The edit-instructions oracle matched the saved rule and the reply against
phrase regexes. It now requires the agent's own audited save to add guidance
on top of the prior and concurrent human revisions, and a link to Morgan or
the saved revision. An unsaved rule fails whatever the reply says.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Completion feedback forbids guessing routes when no href was returned, and
the instruction tools returned none, so chat replies after an instruction
edit could not link the edited agent. Save and restore receipts now carry a
resourceLinks entry with the agent's real profile href and state that runs
starting after the save use the new instructions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /exports and /exports/preview registered no request body, so agents that
discover operations through the API catalog sent none. The defaults export
only the company and agents, and an agent then told a user that the company
had no projects or skills to export. Register companyPortabilityExportSchema
and describe the include defaults, as the legacy /export route already does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit e7b5673439)
The export-company oracle could never pass: an inline import preview never fills
manifest.source, the product exports secret_ref inputs as valueless optional
inputs by design, and the reply was matched against keyword regexes. Grade
provenance from the package's own COMPANY.md company entry, foreign scope from
manifest agents/projects belonging to the source company, and require the reply
to name the setup-chosen omitted credential (key or secret name).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A failed private chat cell lists its HTML report's trace viewer program
files (codicon .ttf, playwright-logo .svg, manifest.webmanifest) in its
evidence manifest. The campaign report copied them into
normalized/evidence/<id>/attempt-N/html-report/trace/, where the trusted
generated-report seal refuses those carrier types, so the whole merged
report was lost with "Unsupported packaged evidence carrier".
The report no longer stages a cell file whose type that seal refuses; the
merged report keeps its own authenticated trace viewer and the cell archive
keeps the files. Before sealing, the transfer now names the first refused
carrier with a closed category (font, svg, webmanifest, image, source map,
no extension, other, trace viewer asset or bytes), never its path. The seal
itself, its limits and its scans are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Playwright's HTML reporter copies every attachment, including the failure
trace and video, into html-report/data/<sha1>.<ext>, while the same bytes
stay in playwright-output. Private chat cells seal both into the original
and the packaged trees, so a large failed attempt carried the trace four
times outside the blob report and lost all private evidence once the
bundle crossed the 256 MB allowance.
Before packaging, drop a playwright-output file only when html-report/data
holds an independently verified, byte-identical regular file under its
content-derived name, and record what was dropped in a snapshot receipt.
The HTML report, its trace viewer and the blob report are unchanged.
Framework-JSON cells, which register playwright-output attachments by path,
are left untouched. No trusted transfer file, limit or scan changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Match the registered run route path and the runsForIssue select block's
runId field instead of a full JSX line and a fixed character window, so
harmless refactors do not break the guidance regression test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit a490180801)
The sidebar Search item links to the full search page, so it does not keep
the person's place. The palette sends only free text, so operators filter
only after the Search all handoff. Correct both statements and test them
against the sidebar and palette source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit ff83378b26)
GET /api/issues/{id}/runs returns the run ID as runId, not id. Name the
field each response returns so chat can build the run link, and pin the
listing's field name in the regression test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 123d7fb940)
The trusted transfer printed one fixed line for every refusal, so a refused
attempt kept no evidence and no reason. Report a closed refusal code: a slug of a
fixed literal message from the trusted evidence modules, a Node error code, an
error type, or 'dynamic'. Message text is never printed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since #19535 the agent profile h1 is the editable title ("Frontend
engineer"), and the name is secondary text. The runtime result check
waited for a heading named "Morgan" and failed on the correct page in
run 42406653921. Identify the profile by the header portrait, which
the page labels "<name> avatar".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The resume-agent grader required phrase lists in both replies. A valid
reply such as "Morgan hasn't started the work yet" failed the
no-invented-execution regex. Grade the first resume by its attribution to
the first request's own run, keep the no-worker-run check, and require
a bound reply for each phase.
Add no-agent-budget-override: in run 42404423330 the CEO lifted the
board's budget hard stop through PATCH /api/agents/{id} after the budget
route refused it. That outcome must fail even when the reply explains it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Chat guidance defined only the task-link template. Asked to resume an
agent, the chat agent resumed it and reported the result but never
linked the agent, leaving the person no direct route to check its
saved availability.
Tell chat to link an affected agent's profile at
/<prefix>/agents/<urlKey or id>, and keep that link distinct from an
agent:// mention, which notifies the agent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 5324cfc4df)
Configuration preparation on current master publishes passive work
hints through db.execute, so the empty object mock from #17857 throws
before the budget assertions run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Chat agents get a route pattern for task links but none for approvals.
Approval tools return no href, so an agent that lists a pending approval
cannot link it without guessing. Add the company-prefixed approval route
that the board UI serves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 6af054a22e)
The show-decisions grader read only the line that named each item and
required words such as "you" or "answered" there. A correct reply that
put "waiting on you" in its list heading failed. Read each item with its
heading, require the exact approval and task links, require Riley to be
named beyond the setup titles, and check that Riley's question and the
answered request are listed apart from the person's own decision.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The native Stop boundary required a bare UUID intent id, but production writes
native-cancellation:<uuid>. Label the health-detail and source-SDK attach steps and
code the remaining diagnostic and SDK-proof refusals.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reply path required one of "which/choose/select", so a valid
"Do you want the launch poster or the welcome note?" failed. Now a reply
must name both setup-chosen proposal titles and pose a question, or the
evaluated run must have created a pending question naming both. Bumps the
case oracle to paperclip.chat-catalog.identity.ambiguous-yes.v2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
goals.create-goal-tree matched model-chosen goal titles with narrow phrase
regexes, so valid titles such as "Ship the website" or "Reach 1000
customer signups" failed, and a truthful "0% complete" note counted as
invented progress. Identify the goals by level and parent and require only
the requested subject (1,000 + signups; website) in saved title/description.
Bump the oracle to goal-hierarchy.v3.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The later-result check required keyword combinations in the follow-up reply and
rejected any mention of 'failed', so a correct answer saying 'nothing failed'
failed, while an acceptance-only reply could pass through the
'summary ... is' alternative. The routine output check likewise required
specific words in the routine's own comment.
Grade instead that the routine's native run read the actual Launch project
and saved its summary, and that the follow-up answer came from a new chat run
which read the completed result task after the invocation finished, with the
task linked. The first-reply guard is now sentence-scoped and ignores
future/conditional framing. Bump the case oracle to
integrator.routine-invocation.v2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Chat guidance defined only the task-link template. Asked to show a
task's runs, the chat agent had GET /api/issues/{id}/runs data with
agentId and run id but no run route or agent name, so it listed raw
UUIDs instead of linking each run and naming its agent.
Add the registered /<prefix>/agents/<agentId>/runs/<runId> route and
ask chat to read and name the agent rather than show a raw ID.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 5c326e2fb7)
The interrupted-and-continuation-states-explained check matched state
words on reply lines that contained a raw run UUID plus the literal
"Morgan". Correct replies that link runs on a separate line, or use other
state words, failed; it also pushed the agent toward pasting raw IDs.
Replace it with a check that the chat run read the website task's run
history (GET /api/issues/{id}/runs or the exact run detail) through a
product tool. Exact native run links per agent, and opening them to the
real cancelled/succeeded status, remain required. Bump the oracle to
paperclip.chat-catalog.run-history.v2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Asked how to quickly find tasks without leaving their page, the chat agent
had no source describing quick search: it searched for docs, said it did not
know the UI, and told the person to ask in chat instead of naming Cmd/Ctrl+K.
Document the command palette, its search operators, the Search all handoff to
full search, and that the agent cannot drive the person's browser. Add a test
that ties the documented shortcuts to the palette's actual key bindings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 0595904175)
The independent observer pressed Cmd/Ctrl+K right after domcontentloaded,
before the app mounted the palette listener, so the palette never opened
(run 42404426034). Wait for the shell and retry the shortcut.
The reply check accepted any 'command ... k' or a bare 'command palette'
mention. Require the shortcut itself (a modifier followed by K, any common
spelling), which is the fact the person needs; bump the oracle version.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The write-plan oracle required nine concept regexes plus negation parsing
over the saved plan. A correct plan flipped to failing when reworded with a
single synonym (phone for mobile, tab-key for keyboard, landing page for
single page). Measure instead how much of the task's own brief the saved
plan covers, sharing the brief between setup and grader, so paraphrases pass
and generic or foreign plans still fail. Bump the case oracle version.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run 42404415872 enabled the skill correctly: the CEO synced only Morgan's
skills, policy and installed revision were unchanged, and Morgan's next
task received and read the exact revision. It failed only the reply
regex, which read "Beta skill versioning isn't enabled" as a denial that
Morgan could use the skill.
Replace the phrase matcher with a state check: the chat run configured
no agent other than Morgan, and it replied. Saved assignment, unchanged
policy and future delivery remain graded. Bump oracle to v3.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Post-seal attachment steps kept the seal-original stage and their messages had no
closed codes; give them an attach-original stage and codes. The post-stop fixture
could not read opencode serve launched from an inherited descriptor; fall back to
the kernel's /proc/<pid>/exe link.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The runner's deepest provider process is node running the extensionless opencode
launcher with serve. Qualify that shape and record absolute-argument basenames and
whether serve is present.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The seal still reported 'unclassified': its refusal came from a module it calls
or from a filesystem error. Add closed codes for framework-report, redaction and
retention messages, and report other filesystem errors by their error code.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
