Files
PaperClipAI/server/src/services/native-runtime/native-deliverable-feedback.test.ts
T
DottaandClaude Opus 5.5 ddb2b25055 fix(native): read "as the file reads" as a comparison, not a file request (#21983)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run reports its result with `paperclip_finish`, and the
server checks the cited evidence before it accepts a "done" result
> - The check uses `explicitlyRequestsFileOutput` to decide if the task
asked for a downloadable file
> - The check read "save the document ... exactly as the file reads" as
"save ... as the file", so it decided that the task asked for a file
> - Because of this, the server rejected each correct completion, and
the agent had to upload a scratch log with `register_deliverable` and
report again
> - Each extra report is one more model request, so the task takes
longer, and a slow provider can make it miss its deadline
> - This pull request makes the check treat "as the file reads / is /
shows ..." as a comparison, not as an output destination
> - The benefit is that the agent finishes on its first correct report,
without an unnecessary upload and a second model round trip

## Linked Issues or Issue Description

**What happened?**
A task asked the agent to run seven rounds that append to
`ledger-<id>.log`, then to "Save this issue's document with key
`ledger`: the full contents of `ledger-<id>.log` exactly as the file
reads". The agent saved the document and cited it. The server rejected
the completion with "The requested file has no accessible delivery
evidence" (or "Completion cites a workspace-only file"). The agent then
uploaded the scratch log as a deliverable and reported again. This
happened in 7 of 7 restart-chaos twin waves for this task (seed s7106)
on three master revisions. The detour (first report to accepted report)
took 9-23 s with fast providers and 107-316 s when the provider was
slow. In one wave the task missed its 40.6-minute budget.

**Expected behavior**
"Exactly as the file reads" describes the contents of the requested
document. It is not a request for a file. The first correct report that
cites the published document must be accepted.

**Steps to reproduce**
1. Call `explicitlyRequestsFileOutput("Save this issue's document with
key `ledger` (title \"Ledger\"): the full contents of
`ledger-c8d89f8d.log` exactly as the file reads, then every line the
board asked for.")`.
2. On master it returns `true`. It must return `false`.

**Paperclip version or commit**
master 54f8068df4

## What Changed

- `server/src/services/native-runtime/native-deliverable-feedback.ts`:
in `explicitlyRequestsFileOutputBySentence`, a comparison to an existing
file is not a file request. A comparison needs all of these: the object
is a generic container noun ("file", "files"); it comes right after "as
the"; and it is followed by an optional adverb (actually, really, still,
previously, currently, now, already) and a verb ("reads", "read", "is",
"are", "was", "were", "appears", "looks", "stands", "says", "shows",
"exists"). A format or file name (CSV, PDF, JSON, output.json) is never
a comparison. Past-participle "read" followed by "by", "from" or "into"
describes the requested output and is not a comparison. "Is / was
[adverb] named, called or titled" after the object still marks a
destination; both patterns share one adverb list.
-
`server/src/services/native-runtime/native-deliverable-feedback.test.ts`:
comparison sentences that must not require a file: the exact chaos-task
sentence, "as the file reads", "as the files are", "as the file actually
/ really reads", "as the file read before the last update", "as the file
reads from the log". Destination and description sentences that must
still require a file: "as a file", "as the file named ledger.csv", "as
the file is named / called / currently titled / actually named
ledger.csv", "as a CSV that is compatible with Excel", "as a PDF that is
printable", "as output.json that is valid JSON", "as a file read from
the log", "as the CSV read by the dashboard", "as the file read by the
dashboard", "as the CSV is described above".

## Verification

- Red: on master the new comparison cases fail
(`explicitlyRequestsFileOutput` returns `true`).
- Green: `cd server && npx vitest run
src/services/native-runtime/native-deliverable-feedback.test.ts` passes
(223 tests). The four test files that use the classifier pass (281
tests).
- `pnpm verify` passes for the affected selection.
- Evidence from restart-chaos twin waves (seed s7106), per wave, time
from the task's first `paperclip_finish` to the accepted report: 9, 17,
23, 13, 126, 107 and 316 s. Each detour contained a rejected report and
a `register_deliverable` upload of the scratch log.

## Risks

- Low risk. The change only narrows when the classifier sees a file
request. It affects a sentence where "as the <file>" is followed by a
verb.
- A task that says "save it as the file is named X" could now not
require publication. This phrasing is unusual. A task can still name a
file with "named", "called" or a file name, and those paths are
unchanged.
- No schema, API or UI change.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edit, test execution) and
extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 03:59:22 -06:00

270 lines
17 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { describe, expect, it } from "vitest";
import { explicitlyRequestsFileOutput, explicitlyRequestsTaskDocumentOutput } from "./native-deliverable-feedback.js";
describe("explicit file output requirements", () => {
it.each([
"Prepare a requested file",
"Make a Markdown file named checklist.md with three items.",
"Write connection-proof.json with the computed result.",
"Write a new result.json.",
"Write a detailed result.json.",
"Write a summary of input.json as output.json.",
"Write a summary of input.json to a file named output.json.",
"Write result.json.md inline in your response.",
"Write the verified report named result.json.",
"Write the result to reports/output.json.",
"Write result.json with the contents of the verified report.",
"Create report.pdf and include the JSON inline in chat.",
"Create a PDF report and include the JSON inline in chat.",
"Write result.json inline and attach it as a download.",
"Write result.json inline and include a download link.",
"Write result.json and describe it inline in chat.",
"Write notes-and-results.json and describe it inline in chat.",
"Write result.json and explain the result inline in chat.",
"Write result.json and summarize it inline in your response.",
"Write the result to `output.json`.",
"Write result.json inline and publish it.",
"Write result.json inline. Publish it on this task.",
"Write result.json and update package.json.",
"Publish connection-proof.json on this child task with register_deliverable using the workspace-relative path, correct byte size, and correct SHA-256.",
"Write internal-proof.json. This is an internal verification file, not a deliverable. Publish it on this task.",
"Export the results as a CSV.",
"Do not modify repository files, but create an attachment.",
"Do not create attachments; export the results as CSV.",
"Do not create attachments and instead export a CSV.",
"Give me a downloadable report.",
"Please create out/answer.pdf and attach it.",
"Do not use external services. Create a file with the results.",
"Make a file but do not send it to anyone else.",
"Export a summary of this PDF as CSV.",
"Create no temporary files; export the results as CSV.",
"Write report.pdf. This is an internal verification file, not a deliverable. Also export the results as CSV.",
"Write a downloadable report.pdf. This is personal memory, not a task deliverable.",
"Attempt native write to report.txt with content requested.",
"Write report.pdf and checklist.md. This is an internal assertion file, not a deliverable.",
"Write report.pdf; write internal-proof.txt. This is an internal verification file, not a deliverable.",
"Write report.pdf and attempt native write to /outside/probe.txt; this negative test must be denied.",
"Attempt native write to /outside/probe.json and write report.pdf; this negative test must be denied.",
"Write report.txt and attach it. This is an internal verification file, not a deliverable.",
"Attach it.",
"Write report.txt and return it as an attachment. This is an internal verification file, not a deliverable.",
"Write report.txt and send it to me. This is an internal verification file, not a deliverable.",
"Write report.txt. This is an internal verification file, not a deliverable. Send it to me.",
"Write report.txt and provide it to me. This is an internal verification file, not a deliverable.",
"Write report.txt and give me it. This is an internal verification file, not a deliverable.",
"Write report.txt and return it. This is an internal verification file, not a deliverable.",
"Write report.txt and send it as an attachment in your response. This is an internal verification file, not a deliverable.",
"Write report.txt. This is an internal verification file, not a deliverable. Send it as a download link in your response.",
"Save the results with the totals as a file.",
"Export the ledger with its hashes as the file named ledger.csv.",
"Export the results as a CSV that is compatible with Excel.",
"Export the results as a file that is easy to open.",
"Save it as a file that is named out.csv",
"Export the results as the CSV that is compatible with Excel.",
"Save the report as a PDF that is printable.",
"Write the data as output.json that is valid JSON.",
"Save the output as a file read from the log.",
"Export the results as the CSV read by the dashboard.",
"Export the results as the CSV read from the warehouse.",
"Export the results as the file read by the dashboard.",
"Export the results as the CSV is described above.",
"Save the summary of the log as the file is named ledger.csv.",
"Save the summary of the log as the file is called ledger.csv.",
"Save the summary of the log as the file is currently titled ledger.csv.",
"Save the summary of the log as the file is actually named ledger.csv.",
])("recognizes an explicit output request: %s", objective => {
expect(explicitlyRequestsFileOutput(objective)).toBe(true);
});
it.each([
"Explain how a newsletter works",
"Read the file and explain what it does.",
"Review the PDF and answer in the chat.",
"Do not create a file; answer inline.",
"Run hostname, id, and pwd on your assigned computer. Report their output in a short completion comment, followed by CONTROL_PLANE_CHECK_OK. Do not modify repository files or create attachments. Mark the task done when finished.",
"Do not modify repository files or create attachments.",
"Do not edit files and do not create attachments.",
"Do not create files; instead, don't create attachments either.",
"Do not create files and attach them.",
"Don’t edit files or create attachments.",
"Never edit files nor create attachments.",
"Don't attach a file. Reply with three bullets.",
"Fix a crash in parser.ts.",
"Update config.json to fix the parser.",
"Write the fix and update package.json.",
"Write the migration then edit package-lock.json.",
"Write a summary and read result.json.",
"Write a function that reads config.json.",
"Write code to parse tsconfig.json.",
"Write the route and wire it in routes.json.",
"Write a function reading config.json.",
"Write a short summary after reading input.json.",
"Write a short summary explaining config.json.",
"Write a short summary of changes to config.json.",
"Write a summary of the file named input.json.",
"Write about config.json.",
"Write concerning config.json.",
"Write a comparison of input.json to baseline.json.",
"Write a summary of input.json to output.json.",
"Write a wrapper around config.json.",
"Write tests covering package.json.",
"Write a migration touching package-lock.json.",
"Return the answer inline with no download link.",
"Return the results inline without a download link.",
"Return the answer without a download link.",
"Write result.json inline in your response as a code block; do not create any files.",
"Write notes-and-results.json inline in your response.",
"Return result.json as a code block in the reply.",
"Read connection-input.json and explain its contents in chat.",
"Write a summary of connection-input.json inline in your response.",
"Write the contents of connection-proof.json inline in your response.",
"Do not publish connection-proof.json; reply inline.",
"Write internal-proof.json. This is an internal verification file, not a deliverable. Do not publish it.",
"Read the file and write a short explanation inline.",
"No downloadable file is needed.",
"Write a summary of this PDF in chat.",
"Create a review of README.md; reply inline.",
"Give me advice on file permissions.",
"Post exactly one durable progress comment whose entire body is TRACKED, then finish this child task. Create no files and do not delegate or create any further tasks.",
"Create no files.",
"Generate no attachments and answer in chat.",
"Write a reply without any files.",
"Use native write/read file tools for this task, not bash or the instructions API.",
"Write a memory entry to memory/pi-native.txt inside the registered AGENT_HOME. This is personal memory, not a task deliverable.",
"Use native write to copy those exact bytes into pi-agent-memory-proof.txt in the task workspace. This is an internal assertion file, not a deliverable.",
"Before finishing, attempt native write exactly once to /outside/pi-unassigned.txt with content forbidden. This intentionally unassigned root must be denied.",
"Write internal-proof.txt; then check it exists. This is an internal verification file, not a deliverable.",
"Attempt native write to /outside/probe.txt and create no files. This negative test must be denied.",
"Write internal-proof.txt; do not attach it. This is an internal verification file, not a deliverable.",
"Write a reply and return it in chat.",
"Write internal-proof.txt and return it in chat. This is an internal verification file, not a deliverable.",
"Write internal-proof.txt and send it in chat. This is an internal verification file, not a deliverable.",
"Write internal-proof.txt. This is an internal verification file, not a deliverable. Return it inline.",
"Write internal-proof.txt and send it as a code block in your response. This is an internal verification file, not a deliverable.",
"Write internal-proof.txt and return it in my reply. This is an internal verification file, not a deliverable.",
"Write internal-proof.txt and provide it as plain text. This is an internal verification file, not a deliverable.",
"Save this issue's document with key `ledger` (title \"Ledger\"): the full contents of `ledger-c8d89f8d.log` exactly as the file reads, then every line the board asked for.",
"Write the document with the log exactly as the file currently reads.",
"Write the summary from the log as the file shows it.",
"Save this issue's document exactly as the file reads.",
"Save this issue's document exactly as the file read before the last update.",
"Save this issue's document exactly as the file reads from the log.",
"Save this issue's document exactly as the files are.",
"Save this issue's document exactly as the file actually reads.",
"Save this issue's document exactly as the file really reads.",
])("does not require a file for a text or source-review request: %s", objective => {
expect(explicitlyRequestsFileOutput(objective)).toBe(false);
});
});
describe("explicit task-document output", () => {
it.each([
"Use the connected page service to find recent pages and create a short Markdown briefing document on this task. Include the titles and verification code returned by the service.",
"Save a document on this task.",
"Write a report document attached to the issue.",
"Do not call HubSpot, but create a document on this task.",
"You may skip HubSpot, but create a document on this task.",
"Explain the lookup error, but save a document on this task.",
])("requires a published task document: %s", objective => {
expect(explicitlyRequestsTaskDocumentOutput(objective)).toBe(true);
});
it.each([
"Explain the document on this task.",
"Write a summary of the document on this task in chat.",
"Do not create a document on this task; reply inline.",
"Create no document on this task.",
"Write a response without a document on this task.",
"Create a document about this task in the repository.",
"Explain how to create a document on this task.",
"Create briefing.md in the workspace.",
"If the lookup succeeds, create a document on this task.",
"Create a document on this task if the lookup succeeds.",
"Only create a document on this task when the lookup succeeds.",
"Create a document on this task, but only if the lookup succeeds.",
"If the lookup succeeds, do not call HubSpot, but create a document on this task.",
"Read the task, but do not yet create a document on this task.",
"Unless I decline, create a document on this task.",
"Once approved, create a document on this task.",
"Do not yet create a document on this task.",
"Don’t ever create a document on this task.",
"You may optionally create a document on this task.",
"Create a document on this task only if useful; otherwise answer inline.",
"Connect HubSpot so you can read my recent contacts. If the contacts are unavailable, a brief explanation is enough instead of the contact list.",
])("preserves other output scopes: %s", objective => {
expect(explicitlyRequestsTaskDocumentOutput(objective)).toBe(false);
});
});
describe("deliverable labels that name files", () => {
const delegatedBrief = [
"Build a simple, polished static marketing website that introduces the business, Acorn Notes.",
"",
"Deliverable:",
"- A self-contained static website in this task's workspace: index.html plus a stylesheet (for example styles.css). Plain HTML/CSS, no build step.",
"- Sections: a hero with the value proposition and a contact call to action.",
"",
"Constraints:",
"- Do not publish or deploy externally; no hosting, DNS, analytics, or third-party services.",
].join("\n");
it.each([
delegatedBrief,
"Deliverables: index.html and styles.css",
"## Deliverables\n- `index.html`\n- `styles.css`",
"**Deliverable:** report.pdf",
"Requested deliverable:\n1. summary.csv with one row per team",
"Deliverable: dist/index.html",
"Deliverable: reports/summary.csv",
"Deliverables:\n- src/styles.css\n- ./site/index.html",
"## Deliverables\n\n- index.html\n- styles.css",
"Deliverables:\n\n- A landing page\n\n- index.html with the hero section",
"Deliverable: branch-report.pdf",
"Deliverable: commit-history.csv covering the last release",
"Deliverable: a comparison of input.csv and baseline.csv, saved as comparison.csv",
"Deliverable: report.pdf, with a short summary in the reply",
"Deliverable: index.html and a short summary in the reply",
"Deliverable: index.html plus a summary inline in chat",
"Deliverable: index.html; do not upload it externally",
"Deliverable: an updated report.pdf",
"Deliverable: update report.pdf and publish it",
"Deliverable: update report.pdf, then attach the updated report.pdf",
"Deliverable: index.html with a short summary in the reply",
"Deliverable: a summary in the reply and index.html",
"Deliverable: a short explanation; do not create notes.md, but attach report.pdf",
"Deliverables:\n- report.pdf, do not attach it to an email",
])("requires publication for a labelled file output: %s", objective => {
expect(explicitlyRequestsFileOutput(objective)).toBe(true);
});
it.each([
"Build a simple website introducing the business.",
"A self-contained static website in this task's workspace: index.html plus a stylesheet.",
"Deliverable: a short answer in chat.",
"Deliverables:\n- A summary of input.csv in the reply",
"Deliverable: a pull request that updates index.html",
"Deliverables:\n- Repository change to styles.css",
"Deliverable:\n\nindex.html is mentioned after the list ended.",
"Deliverable quality matters more than speed for index.html.",
"Deliverable: Write result.json inline in your response as a code block; do not create any files.",
"Deliverable: result.json as a code block in the reply",
"Deliverables:\n- notes.md content inline in the chat",
"Deliverable: a comparison of input.csv and baseline.csv",
"Deliverable: a summary of a.csv, b.csv and c.csv",
"Deliverable: differences between old.json and new.json",
"Deliverable: a pull request on the feature branch that updates dist/index.html",
"Deliverables:\n\n- A summary in chat\n\nNotes:\n- index.html is the current page",
"Deliverable: a review of https://example.com/index.html",
"Deliverable: summary.md in the reply; do not create any new files",
"Deliverable: notes.txt without creating any files",
"Deliverable: update styles.css so the header is blue",
"Deliverables:\n- Edit the existing `index.html` to add a footer",
"Deliverable: a short explanation; do not create report.pdf",
"Deliverable: never write a separate summary.md",
"Deliverable: index.html with its contents inline in the reply",
"Deliverables: index.html and styles.css inline in the reply",
"Deliverable: update styles.css; do not publish it",
"Deliverable: update styles.css and send a short summary in the reply",
"Deliverable: index.html with the full markup inline in the reply",
])("does not treat this as a labelled file output: %s", objective => {
expect(explicitlyRequestsFileOutput(objective)).toBe(false);
});
});