0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-03 17:51:58 +01:00

Files

T

AlexFrontegg 8424a51018 fix(redaction): scan long letter/digit runs in linear time (#16999 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip removes secrets from run logs, tool results and
diagnostics with `redactSensitiveText` before it stores or shows them.
> - The JSON field-name patterns in that function start with an
unanchored `[A-Za-z0-9_-]*`. Each start position inside a long run of
letters and digits scans to the end of the run, so the cost grows with
the square of the run length.
> - Base64 of text is one long run of letters and digits. A 40 KB base64
blob of source code in a shell command takes about 5 s to redact, and a
large Gmail RAW tool result does not finish.
> - The redaction runs on the main event loop, so one such chunk blocks
every request and every run on the instance.
> - This pull request makes these scans start only at the start of a
run. The scans become linear, and the redacted output does not change.
> - The benefit is that large base64 payloads no longer freeze the
server, and every secret that was redacted before is still redacted.

## Linked Issues or Issue Description

Fixes #15393. The tool content guard runs `redactSensitiveText` on tool
results. A Gmail RAW body is base64url of a MIME message, which is one
long letter/digit run. `validateToolContent` on a Gmail-shaped result
with a 68 KB `raw` field takes 59 s on master and 19 ms with this
change; a 5.4 MB `raw` field takes 0.65 s.

Thanks @ts-beat for the clear report in #15393.

Refs #12309 (false positives in the same patterns; this pull request
does not change which text matches).

I searched issues and pull requests for "redaction",
"redactSensitiveText", "quadratic", "backtracking", "hangs the server"
and "100% CPU". #15393 reports the hang. No open pull request fixes it.

**Subsystem affected:** secret redaction (`server/src/redaction.ts`,
`packages/adapter-utils/src/command-redaction.ts`), used by run logs,
tool content guards and run diagnostics.

**Problem:** `compactRunLogChunk` redacts each run log chunk before it
truncates the chunk. A shell command such as `printf '%s' '<base64>' |
base64 -d > file.ts` with base64 of source text makes one long
letter/digit run. Base64 of ASCII text contains `+` or `/` only where
every third byte is `>`, `?`, `~` or DEL, so random base64 does not show
the problem. One `redactSensitiveText` call on such a tool-call line
takes (local measurement):

| Blob | Before | After |
| --- | --- | --- |
| 4 KB | 66 ms | 1.1 ms |
| 8 KB | 272 ms | 0.5 ms |
| 16 KB | 1.1 s | 1.0 ms |
| 40 KB | 5.4 s | 2.5 ms |
| 1 MB | not measured | 36 ms |

The `\b`-anchored command patterns (`KEY=value`, JWT, `Authorization`)
and the CLI option pattern have the same quadratic cost on
dash-separated runs, for example base64url data.

**Proposed solution:** start the unanchored field-name scans only where
no name character precedes them (`(?<![A-Za-z0-9_-])`). A prefixed name
such as `myApiKey`, `x_auth_token` or `GITHUB_TOKEN` still matches,
because the scan from the start of the run absorbs the prefix.

**Alternatives considered:**

- Truncate or cap the input before redaction. Rejected: a cut can split
a secret and leave part of it unredacted, and the scan stays quadratic
below the cap.
- Run the scan in a worker thread with a deadline, as #15393 suggests.
That limits the damage, but the CPU cost stays, and a timeout must then
drop or block the content. It is still useful as an additional defense.

## What Changed

- `server/src/redaction.ts`: `JSON_SECRET_FIELD_TEXT_RE` and
`ESCAPED_JSON_SECRET_FIELD_TEXT_RE` start a field name only where no
name character precedes it. The lookbehind comes after the optional
opening quote, so quoted keys behave as before. The `Authorization` word
scan uses the same start instead of `\b`.
- `packages/adapter-utils/src/command-redaction.ts`: the `KEY=value`
pattern uses the same start. The JWT pattern starts only at the first
word character of a run. A CLI option name can no longer span `--`, so
each `--` starts its own short scan.
- Redaction still runs before truncation. `compactRunLogChunk` does not
change.

Why the output does not change: when an old pattern could match from the
middle of a run, it could also match from the start of that run, because
the `[A-Za-z0-9_-]*` prefix absorbs the extra characters. The engine
tries the start of the run first. So the old patterns never matched from
the middle of a run, and the new start condition only skips positions
that cannot produce a match. The old JWT pattern also always matched
from the first word character of a run. A CLI option name always ends at
the end of its run, so the match from the last `--` in the run ends at
the same place.

## Screenshots

No UI change.

## Verification

- New tests:
- prefixed secret field names in plain, single-quoted and escaped JSON
are still redacted, also when the name is glued to the end of a long
run;
- a 48 KB base64-of-source tool call through `compactRunLogChunk` stays
intact, the secret in the same command is redacted, and the call
finishes within 1 s;
- `redactSensitiveText` and `redactCommandText` handle 48-120 KB
letter/digit, dash-separated, `--`-separated and base64url runs within 1
s, and redact the secret that follows each run;
- `x--token`, `--x--token`, `a-b--api-key=`, `prefix-MY_API_KEY=` and
`--<jwt>` are still redacted, and `my-token is here` is not changed.
- Without the change, the timing tests fail: 9.3 s, 9.8 s, 32 s and 60 s
against the 1 s budget. The correctness tests pass with and without the
change.
- Mutation check: I reverted each of the six pattern changes one at a
time. Each revert makes at least one test fail.
- Differential fuzzing: 1.1 million random inputs built from secret
names, quotes, escaped quotes, dashes, dots, colons and JWT parts gave
the same output from the old and new `redactSensitiveText`,
`redactCommandText` and `redactDiagnosticText`.
- `cd server && npx vitest run <files>`: all 33 server test files that
use the redaction helpers, with embedded Postgres, 1730 passed.
- `npx vitest run --project @paperclipai/adapter-utils
packages/adapter-utils/src/command-redaction.test.ts`: 25 passed.
- `npx vitest run --project @paperclipai/adapter-utils`, complete suite:
80 of 82 files passed in the parallel run. The 2 other files
(`github-launcher-environment`, `sandbox-managed-runtime`) had 5 s
timeouts in git-based tests and passed on rerun.

## Risks

- Lookbehind assertions need ES2018. `redactCommandText` also runs in
the UI bundle. Node and current browsers support lookbehind, and the
server (`server/src/log-redaction.ts`) and the UI
(`ui/src/components/task-chat/task-chat-attachments.ts`) already use it.
- A change to a redaction pattern could leak a secret. The reasoning
above, the unchanged existing tests and the differential fuzzing show
the same output as before.
- Not changed here, found while checking: `redactDiagnosticText`
backtracks exponentially on an escaped JSON secret value made of many
escaped backslashes with no closing quote, and `redactSensitiveText`
redacts a value only up to its first quote of either kind
(`{"apiKey":"abc\"tail"}` keeps `tail"`). Both behave the same before
and after this pull request.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code, with
tool use (shell, code search, and test runs) and extended thinking. A
human directed the design choices.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

2026-11-02 10:49:08 -06:00

src

fix(redaction): scan long letter/digit runs in linear time (#16999 )

2026-11-02 10:49:08 -06:00

CHANGELOG.md

fix: bound workspace Git scans (#11572 )

2026-08-17 22:11:30 -05:00

package.json

feat: configure company agent observability (#16874 )

2026-10-19 15:36:04 -05:00

README.md

fix: bound sandbox bridge waits and flag silent runs sooner (#14979 )

2026-10-02 15:29:41 -07:00

tsconfig.json

…

README.md

@paperclipai/adapter-utils

Shared utilities for Paperclip adapters: process spawning, environment injection, sandbox/SSH transport, workspace sync, and the round-trip helpers that move code between the local execution-workspace cwd and wherever the agent actually runs.

For the adapter-author guide see docs/adapters/creating-an-adapter.md and the in-repo notes at packages/adapters/AUTHORING.md.

Sandbox bridge deadlines

Command-managed bridge control operations (queue reads, input delivery, setup, and cleanup) have a host-enforced deadline of at most 30 seconds per shell command. A shorter configured timeout still applies. These operations cannot inherit the agent run's hours-long lifetime or wait forever on a provider that ignores its timeout. Long-lived agent session commands keep their own limits.

A control timeout reports a transport failure. It does not prove that the remote command stopped and does not replay an uncertain write. The process session closes failed input delivery and uses its existing shutdown path; normal execution settlement must still verify termination.

No-remote-git contract

The local execution-workspace cwd is the only persistence boundary across runs. No adapter may depend on a git remote for cross-run state.

Adapters that run the agent on a different host should use the SSH round-trip helpers in src/ssh.ts:

prepareWorkspaceForSshExecution({ spec, localDir, remoteDir }) — bundles the local cwd (tracked files, dirty edits, untracked additions, and the git history needed to reconstruct it) to remoteDir before the run starts. Runs with no git remote configured.
restoreWorkspaceFromSshExecution({ spec, localDir, remoteDir, ... }) — syncs the remote cwd back into localDir after the run, including any new commits the agent created. Also runs with no git remote configured.

prepareRemoteManagedRuntime in src/remote-managed-runtime.ts wraps both calls for adapters that want a per-run remote workspace and an automatic restoreWorkspace() finally hook.

The invariant is pinned by the no-remote-git contract case in src/ssh-fixture.test.ts, which asserts that a remote-only commit propagates to the local worktree through the prepare → restore round-trip with no git remote configured at any point. Do not regress that test.