fix(native): read "as the file reads" as a comparison, not a file request (#21983)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run reports its result with `paperclip_finish`, and the
server checks the cited evidence before it accepts a "done" result
> - The check uses `explicitlyRequestsFileOutput` to decide if the task
asked for a downloadable file
> - The check read "save the document ... exactly as the file reads" as
"save ... as the file", so it decided that the task asked for a file
> - Because of this, the server rejected each correct completion, and
the agent had to upload a scratch log with `register_deliverable` and
report again
> - Each extra report is one more model request, so the task takes
longer, and a slow provider can make it miss its deadline
> - This pull request makes the check treat "as the file reads / is /
shows ..." as a comparison, not as an output destination
> - The benefit is that the agent finishes on its first correct report,
without an unnecessary upload and a second model round trip

## Linked Issues or Issue Description

**What happened?**
A task asked the agent to run seven rounds that append to
`ledger-<id>.log`, then to "Save this issue's document with key
`ledger`: the full contents of `ledger-<id>.log` exactly as the file
reads". The agent saved the document and cited it. The server rejected
the completion with "The requested file has no accessible delivery
evidence" (or "Completion cites a workspace-only file"). The agent then
uploaded the scratch log as a deliverable and reported again. This
happened in 7 of 7 restart-chaos twin waves for this task (seed s7106)
on three master revisions. The detour (first report to accepted report)
took 9-23 s with fast providers and 107-316 s when the provider was
slow. In one wave the task missed its 40.6-minute budget.

**Expected behavior**
"Exactly as the file reads" describes the contents of the requested
document. It is not a request for a file. The first correct report that
cites the published document must be accepted.

**Steps to reproduce**
1. Call `explicitlyRequestsFileOutput("Save this issue's document with
key `ledger` (title \"Ledger\"): the full contents of
`ledger-c8d89f8d.log` exactly as the file reads, then every line the
board asked for.")`.
2. On master it returns `true`. It must return `false`.

**Paperclip version or commit**
master 54f8068df4

## What Changed

- `server/src/services/native-runtime/native-deliverable-feedback.ts`:
in `explicitlyRequestsFileOutputBySentence`, a comparison to an existing
file is not a file request. A comparison needs all of these: the object
is a generic container noun ("file", "files"); it comes right after "as
the"; and it is followed by an optional adverb (actually, really, still,
previously, currently, now, already) and a verb ("reads", "read", "is",
"are", "was", "were", "appears", "looks", "stands", "says", "shows",
"exists"). A format or file name (CSV, PDF, JSON, output.json) is never
a comparison. Past-participle "read" followed by "by", "from" or "into"
describes the requested output and is not a comparison. "Is / was
[adverb] named, called or titled" after the object still marks a
destination; both patterns share one adverb list.
-
`server/src/services/native-runtime/native-deliverable-feedback.test.ts`:
comparison sentences that must not require a file: the exact chaos-task
sentence, "as the file reads", "as the files are", "as the file actually
/ really reads", "as the file read before the last update", "as the file
reads from the log". Destination and description sentences that must
still require a file: "as a file", "as the file named ledger.csv", "as
the file is named / called / currently titled / actually named
ledger.csv", "as a CSV that is compatible with Excel", "as a PDF that is
printable", "as output.json that is valid JSON", "as a file read from
the log", "as the CSV read by the dashboard", "as the file read by the
dashboard", "as the CSV is described above".

## Verification

- Red: on master the new comparison cases fail
(`explicitlyRequestsFileOutput` returns `true`).
- Green: `cd server && npx vitest run
src/services/native-runtime/native-deliverable-feedback.test.ts` passes
(223 tests). The four test files that use the classifier pass (281
tests).
- `pnpm verify` passes for the affected selection.
- Evidence from restart-chaos twin waves (seed s7106), per wave, time
from the task's first `paperclip_finish` to the accepted report: 9, 17,
23, 13, 126, 107 and 316 s. Each detour contained a rejected report and
a `register_deliverable` upload of the scratch log.

## Risks

- Low risk. The change only narrows when the classifier sees a file
request. It affects a sentence where "as the <file>" is followed by a
verb.
- A task that says "save it as the file is named X" could now not
require publication. This phrasing is unusual. A task can still name a
file with "named", "called" or a file name, and those paths are
unchanged.
- No schema, API or UI change.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edit, test execution) and
extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-17 03:59:22 -06:00
1 parent 41b0f76d75
commit ddb2b25055
2 files changed
+31

No files matched your search

@@ -55,6 +55,23 @@ describe("explicit file output requirements", () => {
"Write report.txt and return it. This is an internal verification file, not a deliverable.",
"Write report.txt and send it as an attachment in your response. This is an internal verification file, not a deliverable.",
"Write report.txt. This is an internal verification file, not a deliverable. Send it as a download link in your response.",
"Save the results with the totals as a file.",
"Export the ledger with its hashes as the file named ledger.csv.",
"Export the results as a CSV that is compatible with Excel.",
"Export the results as a file that is easy to open.",
"Save it as a file that is named out.csv",
"Export the results as the CSV that is compatible with Excel.",
"Save the report as a PDF that is printable.",
"Write the data as output.json that is valid JSON.",
"Save the output as a file read from the log.",
"Export the results as the CSV read by the dashboard.",
"Export the results as the CSV read from the warehouse.",
"Export the results as the file read by the dashboard.",
"Export the results as the CSV is described above.",
"Save the summary of the log as the file is named ledger.csv.",
"Save the summary of the log as the file is called ledger.csv.",
"Save the summary of the log as the file is currently titled ledger.csv.",
"Save the summary of the log as the file is actually named ledger.csv.",
])("recognizes an explicit output request: %s", objective => {
expect(explicitlyRequestsFileOutput(objective)).toBe(true);
});
@@ -125,6 +142,15 @@ describe("explicit file output requirements", () => {
"Write internal-proof.txt and send it as a code block in your response. This is an internal verification file, not a deliverable.",
"Write internal-proof.txt and return it in my reply. This is an internal verification file, not a deliverable.",
"Write internal-proof.txt and provide it as plain text. This is an internal verification file, not a deliverable.",
"Save this issue's document with key `ledger` (title \"Ledger\"): the full contents of `ledger-c8d89f8d.log` exactly as the file reads, then every line the board asked for.",
"Write the document with the log exactly as the file currently reads.",
"Write the summary from the log as the file shows it.",
"Save this issue's document exactly as the file reads.",
"Save this issue's document exactly as the file read before the last update.",
"Save this issue's document exactly as the file reads from the log.",
"Save this issue's document exactly as the files are.",
"Save this issue's document exactly as the file actually reads.",
"Save this issue's document exactly as the file really reads.",
])("does not require a file for a text or source-review request: %s", objective => {
expect(explicitlyRequestsFileOutput(objective)).toBe(false);
});
@@ -201,6 +201,11 @@ function explicitlyRequestsFileOutputBySentence(objective: string): boolean {
? /\b(as|into|to|named|called)\s+(?:(?:a|an|the|new|separate|markdown|word|excel)\s+)*["'`*]*(?:[^\s]*\/)?$/iu
: /\b(?:as|into|to)\s+(?:(?:a|an|the|new|separate|markdown|word|excel)\s+)*$/iu).exec(prefix);
const beforeDestination = prefix.slice(0, destinationMatch?.index ?? 0);
const adverb = "(?:(?:actually|really|still|previously|currently|now|already)\\s+)?";
const namedDestination = new RegExp(`^\\s+(?:is|was)\\s+${adverb}(?:named|called|titled)\\b`, "iu").test(suffix);
const comparison = !namedDestination && /^(?:files?|documents?)$/iu.test(match[0]) && /\bas\s+the\s+$/iu.test(prefix)
&& new RegExp(`^\\s+${adverb}(?:reads|read(?!\\s+(?:by|from|into)\\b)|is|are|was|were|appears|looks|stands|says|shows|exists)\\b`, "iu").test(suffix);
if (comparison) return false;
const destination = destinationMatch !== null && (!json || !/^(?:to|named|called)$/iu.test(destinationMatch[1])
|| !inputReference.test(beforeDestination));
if (json) {