From 0b8578055ac1bd446ec1a095ff2e3cb92372624d Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 09:06:25 -0600 Subject: [PATCH] fix: restore trailing spaces in queued steering messages (#19456) ## Thinking Path > - Paperclip lets people manage AI agents and give them more instructions during work. > - The task composer sends those instructions through the shared Markdown editor. > - The editor writes a final text space as the HTML entity ` `. > - The composer trims spaces before it submits a message. An entity survives this step. > - Queued messages can therefore contain extra text that the person did not type. > - This pull request restores final prose spaces at the editor output boundary. > - The queue and the agent receive clean message text. Code and escaped examples keep their source. ## Linked Issues or Issue Description **What happened?** A message that ends with a space can contain a final ` ` when it enters the steering queue. The shared editor exports this entity, and normal message trimming does not remove it. **Expected behavior** The editor must return final prose spaces as spaces. Code samples and escaped literal entities must keep their exact text. **Steps to reproduce** 1. Open a task while its agent works. 2. Type a steering message with a space at the end. 3. Submit the message to the queue. 4. Inspect its stored Markdown. Before this fix, it can end with ` `. **Paperclip version or commit** Reproduced from `70d23612dd` with the installed Markdown serializer. The serializer writes `Steer the current turn ` as `Steer the current turn ` plus a newline. Searches for related trailing-space and steering-entity PRs found no duplicate. ## What Changed - Restore trailing space entities only in the final Markdown text node. - Apply this step before the editor removes its other transport escapes. - Keep code, raw HTML, escaped entities, links, internal entities, and hard breaks unchanged. - Limit the fix to an entity at the end of the message. Keep entities before internal newlines or closing emphasis markers. Replacing those entities with spaces can create hard breaks or remove bold and italic formatting. Add tests for these limits. - Add editor regression cases and focused Markdown cases. Keep the fix in the shared editor; no queue API change is needed. ## Verification - TDD: the new editor test failed in 3 cases before the fix. It received literal entities instead of spaces. All 6 cases pass after the fix. - All 353 focused editor, Markdown, composer, and queue tests pass. This includes 10 new cases for internal newlines, emphasis, and a final space after these cases. - `pnpm check:token-gates` and `git diff --check` pass. - UI typecheck and UI production build pass. - Repository typecheck and build could not complete locally because Cargo is not installed. The full test command was started, then stopped after confirming its native Runner cases also need Cargo. No complete local full-suite verdict is claimed. Current-head [CI](https://github.com/paperclipai/paperclip/actions/runs/42229743296) passes for `350af775850664226d6630072cb8e08732e2a218`. This includes typecheck, build, tests, and E2E checks. - [Greptile](https://github.com/paperclipai/paperclip/pull/19456#issuecomment-6686109030) and [Paperclip review](https://github.com/paperclipai/paperclip/pull/19456#issuecomment-6686063906) each score the current commit 5/5. No review threads remain open. - Checked the design-review dependency claim against the current commit. [ui/package.json already declares the parser](https://github.com/paperclipai/paperclip/blob/350af775850664226d6630072cb8e08732e2a218/ui/package.json#L71). The existing underscore helper uses the same parser. Six direct checks also pass for GFM strikethrough, task lists, URLs, and tables. - No visual style changed. No database or API contract changed. The helper comment documents the storage rule. ## Risks - Low risk. The change only restores encoded spaces at the end of the document. Internal line endings and spaces inside emphasis are out of scope. The tests keep these entities unchanged to preserve Markdown formatting. Parser source positions protect code and HTML. Escape pairs protect literal entity examples. - Existing stored messages are not rewritten. They use the corrected output after an editor change. ## Model Used - OpenAI Codex, GPT-6. GPT-6.1 Sol (`gpt-6.1-sol`) assisted with the review follow-up. The runtime does not expose the context-window size. Used reasoning, repository tools, code execution, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- ui/src/components/MarkdownEditor.test.tsx | 28 +++++++++++++++ ui/src/components/MarkdownEditor.tsx | 4 ++- ui/src/lib/trailing-markdown-spaces.test.ts | 38 +++++++++++++++++++++ ui/src/lib/trailing-markdown-spaces.ts | 32 +++++++++++++++++ 4 files changed, 101 insertions(+), 1 deletion(-) create mode 100644 ui/src/lib/trailing-markdown-spaces.test.ts create mode 100644 ui/src/lib/trailing-markdown-spaces.ts diff --git a/ui/src/components/MarkdownEditor.test.tsx b/ui/src/components/MarkdownEditor.test.tsx index 82f87566e86..9831b3a61f4 100644 --- a/ui/src/components/MarkdownEditor.test.tsx +++ b/ui/src/components/MarkdownEditor.test.tsx @@ -980,6 +980,34 @@ describe("MarkdownEditor", () => { }); }); + it.each([ + ["Steer the current turn ", "Steer the current turn "], + ["Steer the current turn \n", "Steer the current turn \n"], + ["Use ` ` ", "Use ` ` "], + ["First line \nsecond \n", "First line \nsecond \n"], + ["**bold ** \n", "**bold ** \n"], + ["First line \nsecond", "First line \nsecond"], + ["**bold **\n", "**bold **\n"], + ["Use \\ ", "Use \\ "], + ["```text\n ", "```text\n "], + [" ", " "], + ])("stores trailing editor spaces without changing literal entities: %s", async (exported, stored) => { + mdxEditorMockState.emitMountChange = exported; + const handleChange = vi.fn(); + const root = createRoot(container); + + await act(async () => { + root.render(); + }); + await flush(); + + expect(handleChange).toHaveBeenCalledWith(stored); + + await act(async () => { + root.unmount(); + }); + }); + it("stores intraword underscores as typed instead of the exporter's escaped form", async () => { const value = "Your final answer must be exactly E2E_ASK_12_journey_1793913565974_507b01b8."; // The real exporter escapes every literal underscore, including inside words. diff --git a/ui/src/components/MarkdownEditor.tsx b/ui/src/components/MarkdownEditor.tsx index fe991f82ead..fc69ef6ff85 100644 --- a/ui/src/components/MarkdownEditor.tsx +++ b/ui/src/components/MarkdownEditor.tsx @@ -59,6 +59,7 @@ import { } from "../lib/angle-bracket-markdown"; import { unescapeBlockquoteMarkers } from "../lib/blockquote-markdown"; import { unescapeIntrawordUnderscores } from "../lib/underscore-markdown"; +import { restoreTrailingMarkdownSpaces } from "../lib/trailing-markdown-spaces"; import { pasteNormalizationPlugin } from "../lib/paste-normalization"; import { cn } from "../lib/utils"; import { useEditorAutocomplete, type SlashCommandOption } from "../context/EditorAutocompleteContext"; @@ -189,9 +190,10 @@ function prepareMarkdownForEditor(value: string): string { * feed agent prompts and must stay in the clean, human-authored form. */ function toStoredMarkdown(markdown: string): string { + const withTrailingSpaces = restoreTrailingMarkdownSpaces(markdown); // Unescape underscores first, on the exporter's exact text: removing `\<` or // `\>` can turn following lines into HTML or blockquotes and move code ranges. - return unescapeAngleBracketEscapes(unescapeBlockquoteMarkers(unescapeIntrawordUnderscores(markdown))); + return unescapeAngleBracketEscapes(unescapeBlockquoteMarkers(unescapeIntrawordUnderscores(withTrailingSpaces))); } function escapeRegExp(value: string): string { diff --git a/ui/src/lib/trailing-markdown-spaces.test.ts b/ui/src/lib/trailing-markdown-spaces.test.ts new file mode 100644 index 00000000000..cb388d356e4 --- /dev/null +++ b/ui/src/lib/trailing-markdown-spaces.test.ts @@ -0,0 +1,38 @@ +import { describe, expect, it } from "vitest"; +import { restoreTrailingMarkdownSpaces } from "./trailing-markdown-spaces"; + +describe("restoreTrailingMarkdownSpaces", () => { + it.each([ + ["Send this ", "Send this "], + ["Send this \n", "Send this \n"], + ["Send this \n\n", "Send this \n\n"], + ["> - Send this ", "> - Send this "], + ["Keep this ", "Keep this "], + ["First line \nsecond \n", "First line \nsecond \n"], + ["**bold ** \n", "**bold ** \n"], + ["Use ` ` ", "Use ` ` "], + [String.raw`Use \\ `, String.raw`Use \\` + " "], + ])("restores trailing prose spaces: %s", (input, expected) => { + expect(restoreTrailingMarkdownSpaces(input)).toBe(expected); + }); + + it.each([ + "", + "Keep two spaces \nfor a hard break", + "Keep this", + "First line \nsecond", + "First line \nsecond", + "**bold **\n", + "*italic *\n", + String.raw`Keep \ `, + String.raw`Keep \\\ `, + "Use ` `", + "```text\n ", + "> ```text\n> ", + " ", + "
\n ", + "[link](https://example.com/ )", + ])("preserves literal entities and other Markdown: %s", (input) => { + expect(restoreTrailingMarkdownSpaces(input)).toBe(input); + }); +}); diff --git a/ui/src/lib/trailing-markdown-spaces.ts b/ui/src/lib/trailing-markdown-spaces.ts new file mode 100644 index 00000000000..e86b44ec821 --- /dev/null +++ b/ui/src/lib/trailing-markdown-spaces.ts @@ -0,0 +1,32 @@ +import { fromMarkdown } from "mdast-util-from-markdown"; + +interface PositionedNode { + type: string; + position?: { start: { offset?: number }; end: { offset?: number } }; + children?: PositionedNode[]; +} + +/** + * The rich editor exports a final prose space as ` `. Restore that space + * before storing the draft so message submission can trim it normally. + * Only touch a document-final prose suffix. Entities before internal newlines or + * closing emphasis markers stay encoded: literal spaces there can create a + * hard break or invalidate emphasis. Code, HTML, and escaped literal entities + * must keep their source. Run on the exact export, before other unescaping. + */ +export function restoreTrailingMarkdownSpaces(markdown: string): string { + const suffix = /(?: [ \t\r\n]*)+$/.exec(markdown); + if (!suffix) return markdown; + + let node: PositionedNode = fromMarkdown(markdown); + while (node.children?.length) node = node.children[node.children.length - 1]; + const start = node.position?.start.offset; + const end = node.position?.end.offset; + if (node.type !== "text" || start === undefined || end === undefined || end <= suffix.index) { + return markdown; + } + + const text = markdown.slice(start, end).replace(/\\[\s\S]| /g, (value, offset: number) => + value === " " && start + offset >= suffix.index ? " " : value); + return markdown.slice(0, start) + text + markdown.slice(end); +}