From ddb2b250551ecfe8a8c348379d71c4a52fcbd6a6 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 17 Nov 2026 03:59:22 -0600 Subject: [PATCH] fix(native): read "as the file reads" as a comparison, not a file request (#21983) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run reports its result with `paperclip_finish`, and the server checks the cited evidence before it accepts a "done" result > - The check uses `explicitlyRequestsFileOutput` to decide if the task asked for a downloadable file > - The check read "save the document ... exactly as the file reads" as "save ... as the file", so it decided that the task asked for a file > - Because of this, the server rejected each correct completion, and the agent had to upload a scratch log with `register_deliverable` and report again > - Each extra report is one more model request, so the task takes longer, and a slow provider can make it miss its deadline > - This pull request makes the check treat "as the file reads / is / shows ..." as a comparison, not as an output destination > - The benefit is that the agent finishes on its first correct report, without an unnecessary upload and a second model round trip ## Linked Issues or Issue Description **What happened?** A task asked the agent to run seven rounds that append to `ledger-.log`, then to "Save this issue's document with key `ledger`: the full contents of `ledger-.log` exactly as the file reads". The agent saved the document and cited it. The server rejected the completion with "The requested file has no accessible delivery evidence" (or "Completion cites a workspace-only file"). The agent then uploaded the scratch log as a deliverable and reported again. This happened in 7 of 7 restart-chaos twin waves for this task (seed s7106) on three master revisions. The detour (first report to accepted report) took 9-23 s with fast providers and 107-316 s when the provider was slow. In one wave the task missed its 40.6-minute budget. **Expected behavior** "Exactly as the file reads" describes the contents of the requested document. It is not a request for a file. The first correct report that cites the published document must be accepted. **Steps to reproduce** 1. Call `explicitlyRequestsFileOutput("Save this issue's document with key `ledger` (title \"Ledger\"): the full contents of `ledger-c8d89f8d.log` exactly as the file reads, then every line the board asked for.")`. 2. On master it returns `true`. It must return `false`. **Paperclip version or commit** master 54f8068df4 ## What Changed - `server/src/services/native-runtime/native-deliverable-feedback.ts`: in `explicitlyRequestsFileOutputBySentence`, a comparison to an existing file is not a file request. A comparison needs all of these: the object is a generic container noun ("file", "files"); it comes right after "as the"; and it is followed by an optional adverb (actually, really, still, previously, currently, now, already) and a verb ("reads", "read", "is", "are", "was", "were", "appears", "looks", "stands", "says", "shows", "exists"). A format or file name (CSV, PDF, JSON, output.json) is never a comparison. Past-participle "read" followed by "by", "from" or "into" describes the requested output and is not a comparison. "Is / was [adverb] named, called or titled" after the object still marks a destination; both patterns share one adverb list. - `server/src/services/native-runtime/native-deliverable-feedback.test.ts`: comparison sentences that must not require a file: the exact chaos-task sentence, "as the file reads", "as the files are", "as the file actually / really reads", "as the file read before the last update", "as the file reads from the log". Destination and description sentences that must still require a file: "as a file", "as the file named ledger.csv", "as the file is named / called / currently titled / actually named ledger.csv", "as a CSV that is compatible with Excel", "as a PDF that is printable", "as output.json that is valid JSON", "as a file read from the log", "as the CSV read by the dashboard", "as the file read by the dashboard", "as the CSV is described above". ## Verification - Red: on master the new comparison cases fail (`explicitlyRequestsFileOutput` returns `true`). - Green: `cd server && npx vitest run src/services/native-runtime/native-deliverable-feedback.test.ts` passes (223 tests). The four test files that use the classifier pass (281 tests). - `pnpm verify` passes for the affected selection. - Evidence from restart-chaos twin waves (seed s7106), per wave, time from the task's first `paperclip_finish` to the accepted report: 9, 17, 23, 13, 126, 107 and 316 s. Each detour contained a rejected report and a `register_deliverable` upload of the scratch log. ## Risks - Low risk. The change only narrows when the classifier sees a file request. It affects a sentence where "as the " is followed by a verb. - A task that says "save it as the file is named X" could now not require publication. This phrasing is unusual. A task can still name a file with "named", "called" or a file name, and those paths are unchanged. - No schema, API or UI change. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, test execution) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../native-deliverable-feedback.test.ts | 26 +++++++++++++++++++ .../native-deliverable-feedback.ts | 5 ++++ 2 files changed, 31 insertions(+) diff --git a/server/src/services/native-runtime/native-deliverable-feedback.test.ts b/server/src/services/native-runtime/native-deliverable-feedback.test.ts index e73cf2d7983..5804a0c5e10 100644 --- a/server/src/services/native-runtime/native-deliverable-feedback.test.ts +++ b/server/src/services/native-runtime/native-deliverable-feedback.test.ts @@ -55,6 +55,23 @@ describe("explicit file output requirements", () => { "Write report.txt and return it. This is an internal verification file, not a deliverable.", "Write report.txt and send it as an attachment in your response. This is an internal verification file, not a deliverable.", "Write report.txt. This is an internal verification file, not a deliverable. Send it as a download link in your response.", + "Save the results with the totals as a file.", + "Export the ledger with its hashes as the file named ledger.csv.", + "Export the results as a CSV that is compatible with Excel.", + "Export the results as a file that is easy to open.", + "Save it as a file that is named out.csv", + "Export the results as the CSV that is compatible with Excel.", + "Save the report as a PDF that is printable.", + "Write the data as output.json that is valid JSON.", + "Save the output as a file read from the log.", + "Export the results as the CSV read by the dashboard.", + "Export the results as the CSV read from the warehouse.", + "Export the results as the file read by the dashboard.", + "Export the results as the CSV is described above.", + "Save the summary of the log as the file is named ledger.csv.", + "Save the summary of the log as the file is called ledger.csv.", + "Save the summary of the log as the file is currently titled ledger.csv.", + "Save the summary of the log as the file is actually named ledger.csv.", ])("recognizes an explicit output request: %s", objective => { expect(explicitlyRequestsFileOutput(objective)).toBe(true); }); @@ -125,6 +142,15 @@ describe("explicit file output requirements", () => { "Write internal-proof.txt and send it as a code block in your response. This is an internal verification file, not a deliverable.", "Write internal-proof.txt and return it in my reply. This is an internal verification file, not a deliverable.", "Write internal-proof.txt and provide it as plain text. This is an internal verification file, not a deliverable.", + "Save this issue's document with key `ledger` (title \"Ledger\"): the full contents of `ledger-c8d89f8d.log` exactly as the file reads, then every line the board asked for.", + "Write the document with the log exactly as the file currently reads.", + "Write the summary from the log as the file shows it.", + "Save this issue's document exactly as the file reads.", + "Save this issue's document exactly as the file read before the last update.", + "Save this issue's document exactly as the file reads from the log.", + "Save this issue's document exactly as the files are.", + "Save this issue's document exactly as the file actually reads.", + "Save this issue's document exactly as the file really reads.", ])("does not require a file for a text or source-review request: %s", objective => { expect(explicitlyRequestsFileOutput(objective)).toBe(false); }); diff --git a/server/src/services/native-runtime/native-deliverable-feedback.ts b/server/src/services/native-runtime/native-deliverable-feedback.ts index ba36256ccd5..606c4ee2443 100644 --- a/server/src/services/native-runtime/native-deliverable-feedback.ts +++ b/server/src/services/native-runtime/native-deliverable-feedback.ts @@ -201,6 +201,11 @@ function explicitlyRequestsFileOutputBySentence(objective: string): boolean { ? /\b(as|into|to|named|called)\s+(?:(?:a|an|the|new|separate|markdown|word|excel)\s+)*["'`*]*(?:[^\s]*\/)?$/iu : /\b(?:as|into|to)\s+(?:(?:a|an|the|new|separate|markdown|word|excel)\s+)*$/iu).exec(prefix); const beforeDestination = prefix.slice(0, destinationMatch?.index ?? 0); + const adverb = "(?:(?:actually|really|still|previously|currently|now|already)\\s+)?"; + const namedDestination = new RegExp(`^\\s+(?:is|was)\\s+${adverb}(?:named|called|titled)\\b`, "iu").test(suffix); + const comparison = !namedDestination && /^(?:files?|documents?)$/iu.test(match[0]) && /\bas\s+the\s+$/iu.test(prefix) + && new RegExp(`^\\s+${adverb}(?:reads|read(?!\\s+(?:by|from|into)\\b)|is|are|was|were|appears|looks|stands|says|shows|exists)\\b`, "iu").test(suffix); + if (comparison) return false; const destination = destinationMatch !== null && (!json || !/^(?:to|named|called)$/iu.test(destinationMatch[1]) || !inputReference.test(beforeDestination)); if (json) {