From 556b249897a7f76110ef3d27b2520dd76f925c1e Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 11 Nov 2026 06:38:32 -0600 Subject: [PATCH] fix: preserve exact Markdown in native document writes (#20605) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents save Markdown documents through native tools and REST routes. > - The native write_document handler trims the supplied body. > - This removes final newlines and can change Markdown whitespace. > - This pull request preserves the original body after required-content validation. > - Agents can now write and read the same Markdown string across these paths. ## Linked Issues or Issue Description **What happened?** The native write_document tool removes leading and trailing whitespace. A body with one final newline loses that newline on read-back. The REST document PUT and write_task_document preserve it. **Expected behavior** The native write_document tool must preserve the supplied Markdown body exactly. Required-content checks and key and title normalization must stay in place. REST-backed paths retain surrounding whitespace and keep their existing escaped-line-break conversion for single-line payloads. **Steps to reproduce** 1. Write a disposable document through write_document with a final newline. 2. Read the document and compare the full body with the supplied string. 3. Repeat through write_task_document and REST. These paths preserve the string. Related work: Refs #18824. That change aligns document keys. This change preserves body strings. A search of open and closed PRs found no duplicate fix. ## What Changed - Use a separate untrimmed required-string helper for the document body. Keep the existing requiredString callback signature and all other field normalization. - Add real tool-handler, database, and authenticated HTTP tests for create, update, read-back, and replay. Cover leading spaces, hard breaks, blank lines, final newlines, no final newline, Unicode, and mixed line endings. - Keep blank and invalid bodies rejected. Document whitespace preservation and the existing REST escaped-line-break exception. - Fix the Runner support archive-scanner shutdown defect exposed by CI: rejected ZIP scanners now receive SIGKILL so an ignored SIGTERM cannot leave publication waiting for process closure. Keep all secret and archive limits unchanged. A real child-process regression fails before the change and passes afterward. - Address all three initial review findings: callback type compatibility, contribution comment policy, and the REST documentation limit. ## Verification Final tested head: `c9ff8cf3f83aa19e5d8313c65816a3a2e6562368`. - All applicable [CI checks](https://github.com/paperclipai/paperclip/actions/runs/43019885807) passed. This includes typecheck, build, Runner support, Rust, native Runner, general tests, serialized suites, all eight E2E shards, and the required aggregate gates. - [Greptile review](https://github.com/paperclipai/paperclip/runs/130520170309) passed on this head with confidence 5/5. All three review threads are resolved. The independent Paperclip review also reports 5/5 with no findings. - Red at `7368a0b15c2f02107778a961144d86e2eb59a952`: `pnpm --dir server exec vitest run src/services/native-runtime/task-document-body.integration.test.ts` failed because write_document removed the final newline. Cross-task and REST cases passed. - Green on the final source: 176 tests passed across the document body, document key, native tool authority, task scope, document service, and document route suites. The tests cover create, update, exact read-back, replay, blank rejection, revision guards, and company scope. - The ZIP shutdown regression failed before repair with “scanner did not stop”. After repair, 54 tests passed across zip-inspection-shutdown, private-reporter-artifacts, manual-private-cell-report-assets, and private-evidence-budget. The regression uses a real process that ignores SIGTERM. - Earlier CI runs had two archive-related timeouts. The source repair above replaces blind reruns. The Runner support suite and aggregate verify gate now pass. - A disposable live native document reproduced the original whitespace loss. REST preserved the same string. The fixed handler passed local integration tests. The running service has not been deployed. - Local environment: Linux x64, Node 25.9.0, Vitest 5.0.3, and embedded PostgreSQL. Local Rust preparation was unavailable because cargo is absent. Final-head CI passed Rust verification. `pnpm verify --explain` requested broad CI because evidence.ts is a global input. Required CI supplied that coverage. ## Risks Low risk. The change retains required-content validation. The product change affects document body whitespace handling. The CI repair forcefully stops only already-rejected archive scanners; it retains fail-closed publication and all existing scan limits. It does not change schema, authorization, revision guards, receipts, or persistence. ## Model Used CI repair continuation: GPT-6.1, using repository tools and code execution. OpenAI Codex, GPT-6 (the exact serving model ID and context window were not exposed in this session). Used reasoning, repository tools, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/DEVELOPING.md | 9 +++ .../paperclip-runner-tool-authority.ts | 8 +- .../task-document-body.integration.test.ts | 73 +++++++++++++++++++ tests/runner-e2e/evidence.ts | 6 +- .../zip-inspection-shutdown.test.ts | 42 +++++++++++ 5 files changed, 133 insertions(+), 5 deletions(-) create mode 100644 server/src/services/native-runtime/task-document-body.integration.test.ts create mode 100644 tests/runner-e2e/zip-inspection-shutdown.test.ts diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 816536af5ca..1023f2f1414 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1141,6 +1141,15 @@ orphan recovery. pnpm exec vitest run server/src/services/heartbeat/agent-status.test.ts server/src/__tests__/heartbeat-agent-status-recovery.test.ts ``` +## Task document bodies + +The native `write_document` tool preserves the supplied Markdown body, including +leading whitespace, hard-break spaces, line endings, and final newlines. It still +requires a nonblank body and normalizes key/title fields. `write_task_document` +and REST document PUT also retain surrounding whitespace. Those REST-backed +paths keep their existing escaped-line-break normalization: a body with no real +line breaks can have literal `\n` sequences converted to newlines. + ## Test Commands The [feature map](../maps/feature-map/README.md) is an optional reference for user diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts index 7a718df59be..11fe6b14140 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts @@ -1907,7 +1907,7 @@ export class PaperclipRunnerToolAuthority { key: requiredString(input.key), title: requiredString(input.title), format: "markdown", - body: requiredString(input.body), + body: requiredUntrimmedString(input.body), baseRevisionId: nullableProviderId(input.baseRevisionId), changeSummary: input.changeSummary === null || input.changeSummary === undefined ? null @@ -3324,8 +3324,12 @@ export class PaperclipRunnerToolAuthority { } function requiredString(value: unknown): string { + return requiredUntrimmedString(value).trim(); +} + +function requiredUntrimmedString(value: unknown): string { if (typeof value !== "string" || value.trim() === "") throw new Error("paperclip_runner_tool_input_invalid"); - return value.trim(); + return value; } /** diff --git a/server/src/services/native-runtime/task-document-body.integration.test.ts b/server/src/services/native-runtime/task-document-body.integration.test.ts new file mode 100644 index 00000000000..fe4cc41da74 --- /dev/null +++ b/server/src/services/native-runtime/task-document-body.integration.test.ts @@ -0,0 +1,73 @@ +import { randomUUID } from "node:crypto"; +import { afterAll, beforeAll, expect, it } from "vitest"; +import { startRunnerApiTestServer } from "../../__tests__/helpers/runner-api-server.js"; +import { createLocalAgentJwt } from "../../agent-auth-jwt.js"; +import { documentService } from "../documents.js"; + +let server: Awaited>; +const previousSecret = process.env.PAPERCLIP_AGENT_JWT_SECRET; +beforeAll(async () => { + process.env.PAPERCLIP_AGENT_JWT_SECRET = randomUUID(); + server = await startRunnerApiTestServer(); +}, 60_000); +afterAll(async () => { + await server?.close(); + if (previousSecret === undefined) delete process.env.PAPERCLIP_AGENT_JWT_SECRET; + else process.env.PAPERCLIP_AGENT_JWT_SECRET = previousSecret; +}); + +const bodies = [ + "# Final newline\n", + " Leading whitespace\n\nA Markdown hard break \nnext line\n\n", + "No final newline", + "\t日本語 café 🧭\r\n\r\nSecond line \r\n\n", +]; + +it.each(["write_document", "write_task_document", "rest"] as const)( + "%s preserves exact Markdown on create, update, and read-back", async surface => { + const fixture = await server.fixture({ disableWakeOnDemand: true }); + const issueId = surface === "write_task_document" ? fixture.blockerId : fixture.issueId; + const service = documentService(server.db); + const token = createLocalAgentJwt(fixture.agentId, fixture.companyId, "paperclip_runner", fixture.runId); + const headers = { Authorization: `Bearer ${token}`, "Content-Type": "application/json" }; + for (const [index, body] of bodies.entries()) { + const key = `exact-body-${index}`; + let baseRevisionId: string | null = null; + for (const phase of ["create", "update"] as const) { + const supplied = phase === "create" ? body : `Updated\n${body}`; + const args = { key, title: " Exact body ", body: supplied, baseRevisionId }; + const call = { tool: surface, callId: randomUUID(), arguments: surface === "write_document" + ? { ...args, idempotencyKey: randomUUID() } : { ...args, taskId: issueId } }; + if (surface === "rest") { + const response = await fetch(`${server.apiUrl}/api/issues/${issueId}/documents/${key}`, { + method: "PUT", headers, body: JSON.stringify({ ...args, format: "markdown" }), + }); + expect(response.status).toBe(phase === "create" ? 201 : 200); + expect((await response.json()).body).toBe(supplied); + } else { + const result = await fixture.authority.execute(call); + expect(result).toMatchObject(surface === "write_document" + ? { document: { body: supplied, title: "Exact body" } } : { ok: true, status: phase === "create" ? 201 : 200 }); + expect(await fixture.authority.execute({ ...call, callId: surface === "write_document" ? randomUUID() : call.callId })).toEqual(result); + } + const saved = await service.getIssueDocumentByKey(issueId, key); + expect(saved?.body).toBe(supplied); + expect(saved?.latestRevisionNumber).toBe(phase === "create" ? 1 : 2); + const read = await fixture.authority.execute({ tool: surface === "write_task_document" ? "read_task_document" : "read_document", + callId: randomUUID(), arguments: surface === "write_task_document" ? { taskId: issueId, key } : { key } }) as { document?: { body: string }; data?: { body: string } }; + expect(surface === "write_task_document" ? read.data?.body : read.document?.body).toBe(supplied); + baseRevisionId = saved!.latestRevisionId; + } + } + }, 30_000, +); + +it("retains required body validation for write_document", async () => { + const fixture = await server.fixture({ disableWakeOnDemand: true }); + for (const body of [undefined, null, 42, "", " \n\t"]) { + await expect(fixture.authority.execute({ tool: "write_document", callId: randomUUID(), arguments: { + key: "invalid-body", title: "Invalid body", body, baseRevisionId: null, idempotencyKey: randomUUID(), + } })).rejects.toThrow("paperclip_runner_tool_input_invalid"); + } + expect(await documentService(server.db).getIssueDocumentByKey(fixture.issueId, "invalid-body")).toBeNull(); +}); diff --git a/tests/runner-e2e/evidence.ts b/tests/runner-e2e/evidence.ts index 3e138d5d69d..8b6ea803550 100644 --- a/tests/runner-e2e/evidence.ts +++ b/tests/runner-e2e/evidence.ts @@ -145,16 +145,16 @@ export async function inspectZip(source: string, secrets: readonly string[]) { let leak: string | null = null; let spawnError: Error | null = null; let expanded = 0; - const timer = setTimeout(() => { leak = "zip inspection exceeded its time bound"; unzip.kill(); }, 30_000); + const timer = setTimeout(() => { leak = "zip inspection exceeded its time bound"; unzip.kill("SIGKILL"); }, 30_000); unzip.stdout.on("data", (chunk: Buffer) => { if (leak) return; expanded += chunk.length; - if (expanded > 256 * 1024 * 1024) { leak = "zip inspection exceeded its expanded-byte bound"; unzip.kill(); return; } + if (expanded > 256 * 1024 * 1024) { leak = "zip inspection exceeded its expanded-byte bound"; unzip.kill("SIGKILL"); return; } const data = Buffer.concat([carry, chunk]); leak = findSecretLeak(data, secrets, { includeShapes: false }); carry = data.subarray(Math.max(0, data.length - overlap)); - if (leak) unzip.kill(); + if (leak) unzip.kill("SIGKILL"); }); // Drain diagnostics so a noisy unzip cannot block. Error text is withheld // because archive paths and contents belong to private attempt evidence. diff --git a/tests/runner-e2e/zip-inspection-shutdown.test.ts b/tests/runner-e2e/zip-inspection-shutdown.test.ts new file mode 100644 index 00000000000..720b86a07fc --- /dev/null +++ b/tests/runner-e2e/zip-inspection-shutdown.test.ts @@ -0,0 +1,42 @@ +import { chmod, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { expect, it, vi } from "vitest"; +import { inspectZip } from "./evidence.js"; + +it.skipIf(process.platform === "win32")("finishes a rejected ZIP scan even when the scanner ignores SIGTERM", async () => { + const root = await mkdtemp(path.join(os.tmpdir(), "zip-shutdown-")); + const source = path.join(root, "report.zip"); + const executable = path.join(root, "unzip"); + const secret = "synthetic-zip-secret-928472"; + await writeFile(executable, `#!${process.execPath} +const fs = require('node:fs'); +process.on('SIGTERM', () => {}); +fs.writeFileSync(process.argv[3] + '.pid', String(process.pid)); +process.stdout.write(${JSON.stringify(secret)}); +setInterval(() => {}, 1000); +`); + await chmod(executable, 0o700); + vi.stubEnv("PATH", `${root}${path.delimiter}${process.env.PATH ?? ""}`); + let timer: ReturnType | undefined; + let stopped = false; + const scan = inspectZip(source, [secret]).then(result => { stopped = true; return result; }); + try { + const result = await Promise.race([ + scan, + new Promise(resolve => { timer = setTimeout(() => resolve("scanner did not stop"), 2000); }), + ]); + expect(result).toBe("exact secret value"); + } finally { + clearTimeout(timer); + vi.unstubAllEnvs(); + const pid = Number(await readFile(`${source}.pid`, "utf8").catch(() => "0")); + if (!stopped && pid > 0) { + try { process.kill(pid, "SIGKILL"); } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ESRCH") throw error; + } + } + await scan; + await rm(root, { recursive: true, force: true }); + } +});