mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-18 06:01:49 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run reports its result with `paperclip_finish`, and the
server checks the cited evidence before it accepts a "done" result
> - The check uses `explicitlyRequestsFileOutput` to decide if the task
asked for a downloadable file
> - The check read "save the document ... exactly as the file reads" as
"save ... as the file", so it decided that the task asked for a file
> - Because of this, the server rejected each correct completion, and
the agent had to upload a scratch log with `register_deliverable` and
report again
> - Each extra report is one more model request, so the task takes
longer, and a slow provider can make it miss its deadline
> - This pull request makes the check treat "as the file reads / is /
shows ..." as a comparison, not as an output destination
> - The benefit is that the agent finishes on its first correct report,
without an unnecessary upload and a second model round trip
## Linked Issues or Issue Description
**What happened?**
A task asked the agent to run seven rounds that append to
`ledger-<id>.log`, then to "Save this issue's document with key
`ledger`: the full contents of `ledger-<id>.log` exactly as the file
reads". The agent saved the document and cited it. The server rejected
the completion with "The requested file has no accessible delivery
evidence" (or "Completion cites a workspace-only file"). The agent then
uploaded the scratch log as a deliverable and reported again. This
happened in 7 of 7 restart-chaos twin waves for this task (seed s7106)
on three master revisions. The detour (first report to accepted report)
took 9-23 s with fast providers and 107-316 s when the provider was
slow. In one wave the task missed its 40.6-minute budget.
**Expected behavior**
"Exactly as the file reads" describes the contents of the requested
document. It is not a request for a file. The first correct report that
cites the published document must be accepted.
**Steps to reproduce**
1. Call `explicitlyRequestsFileOutput("Save this issue's document with
key `ledger` (title \"Ledger\"): the full contents of
`ledger-c8d89f8d.log` exactly as the file reads, then every line the
board asked for.")`.
2. On master it returns `true`. It must return `false`.
**Paperclip version or commit**
master 54f8068df4
## What Changed
- `server/src/services/native-runtime/native-deliverable-feedback.ts`:
in `explicitlyRequestsFileOutputBySentence`, a comparison to an existing
file is not a file request. A comparison needs all of these: the object
is a generic container noun ("file", "files"); it comes right after "as
the"; and it is followed by an optional adverb (actually, really, still,
previously, currently, now, already) and a verb ("reads", "read", "is",
"are", "was", "were", "appears", "looks", "stands", "says", "shows",
"exists"). A format or file name (CSV, PDF, JSON, output.json) is never
a comparison. Past-participle "read" followed by "by", "from" or "into"
describes the requested output and is not a comparison. "Is / was
[adverb] named, called or titled" after the object still marks a
destination; both patterns share one adverb list.
-
`server/src/services/native-runtime/native-deliverable-feedback.test.ts`:
comparison sentences that must not require a file: the exact chaos-task
sentence, "as the file reads", "as the files are", "as the file actually
/ really reads", "as the file read before the last update", "as the file
reads from the log". Destination and description sentences that must
still require a file: "as a file", "as the file named ledger.csv", "as
the file is named / called / currently titled / actually named
ledger.csv", "as a CSV that is compatible with Excel", "as a PDF that is
printable", "as output.json that is valid JSON", "as a file read from
the log", "as the CSV read by the dashboard", "as the file read by the
dashboard", "as the CSV is described above".
## Verification
- Red: on master the new comparison cases fail
(`explicitlyRequestsFileOutput` returns `true`).
- Green: `cd server && npx vitest run
src/services/native-runtime/native-deliverable-feedback.test.ts` passes
(223 tests). The four test files that use the classifier pass (281
tests).
- `pnpm verify` passes for the affected selection.
- Evidence from restart-chaos twin waves (seed s7106), per wave, time
from the task's first `paperclip_finish` to the accepted report: 9, 17,
23, 13, 126, 107 and 316 s. Each detour contained a rejected report and
a `register_deliverable` upload of the scratch log.
## Risks
- Low risk. The change only narrows when the classifier sees a file
request. It affects a sentence where "as the <file>" is followed by a
verb.
- A task that says "save it as the file is named X" could now not
require publication. This phrasing is unusual. A task can still name a
file with "named", "called" or a file name, and those paths are
unchanged.
- No schema, API or UI change.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edit, test execution) and
extended reasoning.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
270 lines
17 KiB
TypeScript
270 lines
17 KiB
TypeScript
import { describe, expect, it } from "vitest";
|
||
import { explicitlyRequestsFileOutput, explicitlyRequestsTaskDocumentOutput } from "./native-deliverable-feedback.js";
|
||
|
||
describe("explicit file output requirements", () => {
|
||
it.each([
|
||
"Prepare a requested file",
|
||
"Make a Markdown file named checklist.md with three items.",
|
||
"Write connection-proof.json with the computed result.",
|
||
"Write a new result.json.",
|
||
"Write a detailed result.json.",
|
||
"Write a summary of input.json as output.json.",
|
||
"Write a summary of input.json to a file named output.json.",
|
||
"Write result.json.md inline in your response.",
|
||
"Write the verified report named result.json.",
|
||
"Write the result to reports/output.json.",
|
||
"Write result.json with the contents of the verified report.",
|
||
"Create report.pdf and include the JSON inline in chat.",
|
||
"Create a PDF report and include the JSON inline in chat.",
|
||
"Write result.json inline and attach it as a download.",
|
||
"Write result.json inline and include a download link.",
|
||
"Write result.json and describe it inline in chat.",
|
||
"Write notes-and-results.json and describe it inline in chat.",
|
||
"Write result.json and explain the result inline in chat.",
|
||
"Write result.json and summarize it inline in your response.",
|
||
"Write the result to `output.json`.",
|
||
"Write result.json inline and publish it.",
|
||
"Write result.json inline. Publish it on this task.",
|
||
"Write result.json and update package.json.",
|
||
"Publish connection-proof.json on this child task with register_deliverable using the workspace-relative path, correct byte size, and correct SHA-256.",
|
||
"Write internal-proof.json. This is an internal verification file, not a deliverable. Publish it on this task.",
|
||
"Export the results as a CSV.",
|
||
"Do not modify repository files, but create an attachment.",
|
||
"Do not create attachments; export the results as CSV.",
|
||
"Do not create attachments and instead export a CSV.",
|
||
"Give me a downloadable report.",
|
||
"Please create out/answer.pdf and attach it.",
|
||
"Do not use external services. Create a file with the results.",
|
||
"Make a file but do not send it to anyone else.",
|
||
"Export a summary of this PDF as CSV.",
|
||
"Create no temporary files; export the results as CSV.",
|
||
"Write report.pdf. This is an internal verification file, not a deliverable. Also export the results as CSV.",
|
||
"Write a downloadable report.pdf. This is personal memory, not a task deliverable.",
|
||
"Attempt native write to report.txt with content requested.",
|
||
"Write report.pdf and checklist.md. This is an internal assertion file, not a deliverable.",
|
||
"Write report.pdf; write internal-proof.txt. This is an internal verification file, not a deliverable.",
|
||
"Write report.pdf and attempt native write to /outside/probe.txt; this negative test must be denied.",
|
||
"Attempt native write to /outside/probe.json and write report.pdf; this negative test must be denied.",
|
||
"Write report.txt and attach it. This is an internal verification file, not a deliverable.",
|
||
"Attach it.",
|
||
"Write report.txt and return it as an attachment. This is an internal verification file, not a deliverable.",
|
||
"Write report.txt and send it to me. This is an internal verification file, not a deliverable.",
|
||
"Write report.txt. This is an internal verification file, not a deliverable. Send it to me.",
|
||
"Write report.txt and provide it to me. This is an internal verification file, not a deliverable.",
|
||
"Write report.txt and give me it. This is an internal verification file, not a deliverable.",
|
||
"Write report.txt and return it. This is an internal verification file, not a deliverable.",
|
||
"Write report.txt and send it as an attachment in your response. This is an internal verification file, not a deliverable.",
|
||
"Write report.txt. This is an internal verification file, not a deliverable. Send it as a download link in your response.",
|
||
"Save the results with the totals as a file.",
|
||
"Export the ledger with its hashes as the file named ledger.csv.",
|
||
"Export the results as a CSV that is compatible with Excel.",
|
||
"Export the results as a file that is easy to open.",
|
||
"Save it as a file that is named out.csv",
|
||
"Export the results as the CSV that is compatible with Excel.",
|
||
"Save the report as a PDF that is printable.",
|
||
"Write the data as output.json that is valid JSON.",
|
||
"Save the output as a file read from the log.",
|
||
"Export the results as the CSV read by the dashboard.",
|
||
"Export the results as the CSV read from the warehouse.",
|
||
"Export the results as the file read by the dashboard.",
|
||
"Export the results as the CSV is described above.",
|
||
"Save the summary of the log as the file is named ledger.csv.",
|
||
"Save the summary of the log as the file is called ledger.csv.",
|
||
"Save the summary of the log as the file is currently titled ledger.csv.",
|
||
"Save the summary of the log as the file is actually named ledger.csv.",
|
||
])("recognizes an explicit output request: %s", objective => {
|
||
expect(explicitlyRequestsFileOutput(objective)).toBe(true);
|
||
});
|
||
it.each([
|
||
"Explain how a newsletter works",
|
||
"Read the file and explain what it does.",
|
||
"Review the PDF and answer in the chat.",
|
||
"Do not create a file; answer inline.",
|
||
"Run hostname, id, and pwd on your assigned computer. Report their output in a short completion comment, followed by CONTROL_PLANE_CHECK_OK. Do not modify repository files or create attachments. Mark the task done when finished.",
|
||
"Do not modify repository files or create attachments.",
|
||
"Do not edit files and do not create attachments.",
|
||
"Do not create files; instead, don't create attachments either.",
|
||
"Do not create files and attach them.",
|
||
"Don’t edit files or create attachments.",
|
||
"Never edit files nor create attachments.",
|
||
"Don't attach a file. Reply with three bullets.",
|
||
"Fix a crash in parser.ts.",
|
||
"Update config.json to fix the parser.",
|
||
"Write the fix and update package.json.",
|
||
"Write the migration then edit package-lock.json.",
|
||
"Write a summary and read result.json.",
|
||
"Write a function that reads config.json.",
|
||
"Write code to parse tsconfig.json.",
|
||
"Write the route and wire it in routes.json.",
|
||
"Write a function reading config.json.",
|
||
"Write a short summary after reading input.json.",
|
||
"Write a short summary explaining config.json.",
|
||
"Write a short summary of changes to config.json.",
|
||
"Write a summary of the file named input.json.",
|
||
"Write about config.json.",
|
||
"Write concerning config.json.",
|
||
"Write a comparison of input.json to baseline.json.",
|
||
"Write a summary of input.json to output.json.",
|
||
"Write a wrapper around config.json.",
|
||
"Write tests covering package.json.",
|
||
"Write a migration touching package-lock.json.",
|
||
"Return the answer inline with no download link.",
|
||
"Return the results inline without a download link.",
|
||
"Return the answer without a download link.",
|
||
"Write result.json inline in your response as a code block; do not create any files.",
|
||
"Write notes-and-results.json inline in your response.",
|
||
"Return result.json as a code block in the reply.",
|
||
"Read connection-input.json and explain its contents in chat.",
|
||
"Write a summary of connection-input.json inline in your response.",
|
||
"Write the contents of connection-proof.json inline in your response.",
|
||
"Do not publish connection-proof.json; reply inline.",
|
||
"Write internal-proof.json. This is an internal verification file, not a deliverable. Do not publish it.",
|
||
"Read the file and write a short explanation inline.",
|
||
"No downloadable file is needed.",
|
||
"Write a summary of this PDF in chat.",
|
||
"Create a review of README.md; reply inline.",
|
||
"Give me advice on file permissions.",
|
||
"Post exactly one durable progress comment whose entire body is TRACKED, then finish this child task. Create no files and do not delegate or create any further tasks.",
|
||
"Create no files.",
|
||
"Generate no attachments and answer in chat.",
|
||
"Write a reply without any files.",
|
||
"Use native write/read file tools for this task, not bash or the instructions API.",
|
||
"Write a memory entry to memory/pi-native.txt inside the registered AGENT_HOME. This is personal memory, not a task deliverable.",
|
||
"Use native write to copy those exact bytes into pi-agent-memory-proof.txt in the task workspace. This is an internal assertion file, not a deliverable.",
|
||
"Before finishing, attempt native write exactly once to /outside/pi-unassigned.txt with content forbidden. This intentionally unassigned root must be denied.",
|
||
"Write internal-proof.txt; then check it exists. This is an internal verification file, not a deliverable.",
|
||
"Attempt native write to /outside/probe.txt and create no files. This negative test must be denied.",
|
||
"Write internal-proof.txt; do not attach it. This is an internal verification file, not a deliverable.",
|
||
"Write a reply and return it in chat.",
|
||
"Write internal-proof.txt and return it in chat. This is an internal verification file, not a deliverable.",
|
||
"Write internal-proof.txt and send it in chat. This is an internal verification file, not a deliverable.",
|
||
"Write internal-proof.txt. This is an internal verification file, not a deliverable. Return it inline.",
|
||
"Write internal-proof.txt and send it as a code block in your response. This is an internal verification file, not a deliverable.",
|
||
"Write internal-proof.txt and return it in my reply. This is an internal verification file, not a deliverable.",
|
||
"Write internal-proof.txt and provide it as plain text. This is an internal verification file, not a deliverable.",
|
||
"Save this issue's document with key `ledger` (title \"Ledger\"): the full contents of `ledger-c8d89f8d.log` exactly as the file reads, then every line the board asked for.",
|
||
"Write the document with the log exactly as the file currently reads.",
|
||
"Write the summary from the log as the file shows it.",
|
||
"Save this issue's document exactly as the file reads.",
|
||
"Save this issue's document exactly as the file read before the last update.",
|
||
"Save this issue's document exactly as the file reads from the log.",
|
||
"Save this issue's document exactly as the files are.",
|
||
"Save this issue's document exactly as the file actually reads.",
|
||
"Save this issue's document exactly as the file really reads.",
|
||
])("does not require a file for a text or source-review request: %s", objective => {
|
||
expect(explicitlyRequestsFileOutput(objective)).toBe(false);
|
||
});
|
||
});
|
||
|
||
|
||
describe("explicit task-document output", () => {
|
||
it.each([
|
||
"Use the connected page service to find recent pages and create a short Markdown briefing document on this task. Include the titles and verification code returned by the service.",
|
||
"Save a document on this task.",
|
||
"Write a report document attached to the issue.",
|
||
"Do not call HubSpot, but create a document on this task.",
|
||
"You may skip HubSpot, but create a document on this task.",
|
||
"Explain the lookup error, but save a document on this task.",
|
||
])("requires a published task document: %s", objective => {
|
||
expect(explicitlyRequestsTaskDocumentOutput(objective)).toBe(true);
|
||
});
|
||
it.each([
|
||
"Explain the document on this task.",
|
||
"Write a summary of the document on this task in chat.",
|
||
"Do not create a document on this task; reply inline.",
|
||
"Create no document on this task.",
|
||
"Write a response without a document on this task.",
|
||
"Create a document about this task in the repository.",
|
||
"Explain how to create a document on this task.",
|
||
"Create briefing.md in the workspace.",
|
||
"If the lookup succeeds, create a document on this task.",
|
||
"Create a document on this task if the lookup succeeds.",
|
||
"Only create a document on this task when the lookup succeeds.",
|
||
"Create a document on this task, but only if the lookup succeeds.",
|
||
"If the lookup succeeds, do not call HubSpot, but create a document on this task.",
|
||
"Read the task, but do not yet create a document on this task.",
|
||
"Unless I decline, create a document on this task.",
|
||
"Once approved, create a document on this task.",
|
||
"Do not yet create a document on this task.",
|
||
"Don’t ever create a document on this task.",
|
||
"You may optionally create a document on this task.",
|
||
"Create a document on this task only if useful; otherwise answer inline.",
|
||
"Connect HubSpot so you can read my recent contacts. If the contacts are unavailable, a brief explanation is enough instead of the contact list.",
|
||
])("preserves other output scopes: %s", objective => {
|
||
expect(explicitlyRequestsTaskDocumentOutput(objective)).toBe(false);
|
||
});
|
||
});
|
||
|
||
describe("deliverable labels that name files", () => {
|
||
const delegatedBrief = [
|
||
"Build a simple, polished static marketing website that introduces the business, Acorn Notes.",
|
||
"",
|
||
"Deliverable:",
|
||
"- A self-contained static website in this task's workspace: index.html plus a stylesheet (for example styles.css). Plain HTML/CSS, no build step.",
|
||
"- Sections: a hero with the value proposition and a contact call to action.",
|
||
"",
|
||
"Constraints:",
|
||
"- Do not publish or deploy externally; no hosting, DNS, analytics, or third-party services.",
|
||
].join("\n");
|
||
it.each([
|
||
delegatedBrief,
|
||
"Deliverables: index.html and styles.css",
|
||
"## Deliverables\n- `index.html`\n- `styles.css`",
|
||
"**Deliverable:** report.pdf",
|
||
"Requested deliverable:\n1. summary.csv with one row per team",
|
||
"Deliverable: dist/index.html",
|
||
"Deliverable: reports/summary.csv",
|
||
"Deliverables:\n- src/styles.css\n- ./site/index.html",
|
||
"## Deliverables\n\n- index.html\n- styles.css",
|
||
"Deliverables:\n\n- A landing page\n\n- index.html with the hero section",
|
||
"Deliverable: branch-report.pdf",
|
||
"Deliverable: commit-history.csv covering the last release",
|
||
"Deliverable: a comparison of input.csv and baseline.csv, saved as comparison.csv",
|
||
"Deliverable: report.pdf, with a short summary in the reply",
|
||
"Deliverable: index.html and a short summary in the reply",
|
||
"Deliverable: index.html plus a summary inline in chat",
|
||
"Deliverable: index.html; do not upload it externally",
|
||
"Deliverable: an updated report.pdf",
|
||
"Deliverable: update report.pdf and publish it",
|
||
"Deliverable: update report.pdf, then attach the updated report.pdf",
|
||
"Deliverable: index.html with a short summary in the reply",
|
||
"Deliverable: a summary in the reply and index.html",
|
||
"Deliverable: a short explanation; do not create notes.md, but attach report.pdf",
|
||
"Deliverables:\n- report.pdf, do not attach it to an email",
|
||
])("requires publication for a labelled file output: %s", objective => {
|
||
expect(explicitlyRequestsFileOutput(objective)).toBe(true);
|
||
});
|
||
it.each([
|
||
"Build a simple website introducing the business.",
|
||
"A self-contained static website in this task's workspace: index.html plus a stylesheet.",
|
||
"Deliverable: a short answer in chat.",
|
||
"Deliverables:\n- A summary of input.csv in the reply",
|
||
"Deliverable: a pull request that updates index.html",
|
||
"Deliverables:\n- Repository change to styles.css",
|
||
"Deliverable:\n\nindex.html is mentioned after the list ended.",
|
||
"Deliverable quality matters more than speed for index.html.",
|
||
"Deliverable: Write result.json inline in your response as a code block; do not create any files.",
|
||
"Deliverable: result.json as a code block in the reply",
|
||
"Deliverables:\n- notes.md content inline in the chat",
|
||
"Deliverable: a comparison of input.csv and baseline.csv",
|
||
"Deliverable: a summary of a.csv, b.csv and c.csv",
|
||
"Deliverable: differences between old.json and new.json",
|
||
"Deliverable: a pull request on the feature branch that updates dist/index.html",
|
||
"Deliverables:\n\n- A summary in chat\n\nNotes:\n- index.html is the current page",
|
||
"Deliverable: a review of https://example.com/index.html",
|
||
"Deliverable: summary.md in the reply; do not create any new files",
|
||
"Deliverable: notes.txt without creating any files",
|
||
"Deliverable: update styles.css so the header is blue",
|
||
"Deliverables:\n- Edit the existing `index.html` to add a footer",
|
||
"Deliverable: a short explanation; do not create report.pdf",
|
||
"Deliverable: never write a separate summary.md",
|
||
"Deliverable: index.html with its contents inline in the reply",
|
||
"Deliverables: index.html and styles.css inline in the reply",
|
||
"Deliverable: update styles.css; do not publish it",
|
||
"Deliverable: update styles.css and send a short summary in the reply",
|
||
"Deliverable: index.html with the full markup inline in the reply",
|
||
])("does not treat this as a labelled file output: %s", objective => {
|
||
expect(explicitlyRequestsFileOutput(objective)).toBe(false);
|
||
});
|
||
});
|