Files
PaperClipAI/packages/adapter-utils/src/command-redaction.ts
T
AlexFrontegg 8424a51018 fix(redaction): scan long letter/digit runs in linear time (#16999)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip removes secrets from run logs, tool results and
diagnostics with `redactSensitiveText` before it stores or shows them.
> - The JSON field-name patterns in that function start with an
unanchored `[A-Za-z0-9_-]*`. Each start position inside a long run of
letters and digits scans to the end of the run, so the cost grows with
the square of the run length.
> - Base64 of text is one long run of letters and digits. A 40 KB base64
blob of source code in a shell command takes about 5 s to redact, and a
large Gmail RAW tool result does not finish.
> - The redaction runs on the main event loop, so one such chunk blocks
every request and every run on the instance.
> - This pull request makes these scans start only at the start of a
run. The scans become linear, and the redacted output does not change.
> - The benefit is that large base64 payloads no longer freeze the
server, and every secret that was redacted before is still redacted.

## Linked Issues or Issue Description

Fixes #15393. The tool content guard runs `redactSensitiveText` on tool
results. A Gmail RAW body is base64url of a MIME message, which is one
long letter/digit run. `validateToolContent` on a Gmail-shaped result
with a 68 KB `raw` field takes 59 s on master and 19 ms with this
change; a 5.4 MB `raw` field takes 0.65 s.

Thanks @ts-beat for the clear report in #15393.

Refs #12309 (false positives in the same patterns; this pull request
does not change which text matches).

I searched issues and pull requests for "redaction",
"redactSensitiveText", "quadratic", "backtracking", "hangs the server"
and "100% CPU". #15393 reports the hang. No open pull request fixes it.

**Subsystem affected:** secret redaction (`server/src/redaction.ts`,
`packages/adapter-utils/src/command-redaction.ts`), used by run logs,
tool content guards and run diagnostics.

**Problem:** `compactRunLogChunk` redacts each run log chunk before it
truncates the chunk. A shell command such as `printf '%s' '<base64>' |
base64 -d > file.ts` with base64 of source text makes one long
letter/digit run. Base64 of ASCII text contains `+` or `/` only where
every third byte is `>`, `?`, `~` or DEL, so random base64 does not show
the problem. One `redactSensitiveText` call on such a tool-call line
takes (local measurement):

| Blob | Before | After |
| --- | --- | --- |
| 4 KB | 66 ms | 1.1 ms |
| 8 KB | 272 ms | 0.5 ms |
| 16 KB | 1.1 s | 1.0 ms |
| 40 KB | 5.4 s | 2.5 ms |
| 1 MB | not measured | 36 ms |

The `\b`-anchored command patterns (`KEY=value`, JWT, `Authorization`)
and the CLI option pattern have the same quadratic cost on
dash-separated runs, for example base64url data.

**Proposed solution:** start the unanchored field-name scans only where
no name character precedes them (`(?<![A-Za-z0-9_-])`). A prefixed name
such as `myApiKey`, `x_auth_token` or `GITHUB_TOKEN` still matches,
because the scan from the start of the run absorbs the prefix.

**Alternatives considered:**

- Truncate or cap the input before redaction. Rejected: a cut can split
a secret and leave part of it unredacted, and the scan stays quadratic
below the cap.
- Run the scan in a worker thread with a deadline, as #15393 suggests.
That limits the damage, but the CPU cost stays, and a timeout must then
drop or block the content. It is still useful as an additional defense.

## What Changed

- `server/src/redaction.ts`: `JSON_SECRET_FIELD_TEXT_RE` and
`ESCAPED_JSON_SECRET_FIELD_TEXT_RE` start a field name only where no
name character precedes it. The lookbehind comes after the optional
opening quote, so quoted keys behave as before. The `Authorization` word
scan uses the same start instead of `\b`.
- `packages/adapter-utils/src/command-redaction.ts`: the `KEY=value`
pattern uses the same start. The JWT pattern starts only at the first
word character of a run. A CLI option name can no longer span `--`, so
each `--` starts its own short scan.
- Redaction still runs before truncation. `compactRunLogChunk` does not
change.

Why the output does not change: when an old pattern could match from the
middle of a run, it could also match from the start of that run, because
the `[A-Za-z0-9_-]*` prefix absorbs the extra characters. The engine
tries the start of the run first. So the old patterns never matched from
the middle of a run, and the new start condition only skips positions
that cannot produce a match. The old JWT pattern also always matched
from the first word character of a run. A CLI option name always ends at
the end of its run, so the match from the last `--` in the run ends at
the same place.

## Screenshots

No UI change.

## Verification

- New tests:
- prefixed secret field names in plain, single-quoted and escaped JSON
are still redacted, also when the name is glued to the end of a long
run;
- a 48 KB base64-of-source tool call through `compactRunLogChunk` stays
intact, the secret in the same command is redacted, and the call
finishes within 1 s;
- `redactSensitiveText` and `redactCommandText` handle 48-120 KB
letter/digit, dash-separated, `--`-separated and base64url runs within 1
s, and redact the secret that follows each run;
- `x--token`, `--x--token`, `a-b--api-key=`, `prefix-MY_API_KEY=` and
`--<jwt>` are still redacted, and `my-token is here` is not changed.
- Without the change, the timing tests fail: 9.3 s, 9.8 s, 32 s and 60 s
against the 1 s budget. The correctness tests pass with and without the
change.
- Mutation check: I reverted each of the six pattern changes one at a
time. Each revert makes at least one test fail.
- Differential fuzzing: 1.1 million random inputs built from secret
names, quotes, escaped quotes, dashes, dots, colons and JWT parts gave
the same output from the old and new `redactSensitiveText`,
`redactCommandText` and `redactDiagnosticText`.
- `cd server && npx vitest run <files>`: all 33 server test files that
use the redaction helpers, with embedded Postgres, 1730 passed.
- `npx vitest run --project @paperclipai/adapter-utils
packages/adapter-utils/src/command-redaction.test.ts`: 25 passed.
- `npx vitest run --project @paperclipai/adapter-utils`, complete suite:
80 of 82 files passed in the parallel run. The 2 other files
(`github-launcher-environment`, `sandbox-managed-runtime`) had 5 s
timeouts in git-based tests and passed on rerun.

## Risks

- Lookbehind assertions need ES2018. `redactCommandText` also runs in
the UI bundle. Node and current browsers support lookbehind, and the
server (`server/src/log-redaction.ts`) and the UI
(`ui/src/components/task-chat/task-chat-attachments.ts`) already use it.
- A change to a redaction pattern could leak a secret. The reasoning
above, the unchanged existing tests and the differential fuzzing show
the same output as before.
- Not changed here, found while checking: `redactDiagnosticText`
backtracks exponentially on an escaped JSON secret value made of many
escaped backslashes with no closing quote, and `redactSensitiveText`
redacts a value only up to its first quote of either kind
(`{"apiKey":"abc\"tail"}` keeps `tail"`). Both behave the same before
and after this pull request.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code, with
tool use (shell, code search, and test runs) and extended thinking. A
human directed the design choices.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-02 10:49:08 -06:00

150 lines
5.6 KiB
TypeScript

export const REDACTED_COMMAND_TEXT_VALUE = "***REDACTED***";
// Public Executor helper addresses retained for callers of this predicate.
// Dotted addresses alone do not meet the JWT credential heuristic.
const PUBLIC_EXECUTOR_TOOL_SELECTORS = new Set([
"executor.coreTools.integrations.list",
"executor.coreTools.connections.list",
"executor.coreTools.policies.list",
]);
export function isPublicExecutorToolSelector(value: string): boolean {
return PUBLIC_EXECUTOR_TOOL_SELECTORS.has(value);
}
const SECRET_NAME_WORDS = String.raw`(?:api[-_]?key|(?:access[-_]?|auth[-_]?)?token|token|authorization(?:[-_]?code)?|bearer|secrets?|passwd|passwords?|credentials?|jwt|private[-_]?key|cookie|connectionstring)(?:[-_]?(?:value|header|prod(?:uction)?|dev(?:elopment)?|test|staging|primary|secondary))*`;
const SECRET_NAME_PATTERN = String.raw`[A-Za-z0-9_-]*${SECRET_NAME_WORDS}`;
const SECRET_NAME_START = String.raw`(?<![A-Za-z0-9_-])`;
const SECRET_OPTION_NAME_PATTERN = String.raw`(?:[A-Za-z0-9_]|-(?!-))*${SECRET_NAME_WORDS}`;
const COMMAND_CLI_SECRET_OPTION_RE = new RegExp(
String.raw`(\B-{1,2}${SECRET_OPTION_NAME_PATTERN}(?:\s+|=)(["']?))[^\s"'` +
"`" +
String.raw`]+(\2)`,
"gi",
);
const COMMAND_ENV_SECRET_ASSIGNMENT_RE = new RegExp(
String.raw`(${SECRET_NAME_START}${SECRET_NAME_PATTERN}\s*=\s*)(?:(\\["'])([\s\S]*?)\2|(["'])([^"'` +
"`" +
String.raw`\r\n]*)\4|([^\s"'` +
"`" +
String.raw`]+))`,
"gi",
);
const COMMAND_AUTHORIZATION_BEARER_RE =
/(\bAuthorization\s*:\s*Bearer\s+)[^\s"'`]+/gi;
const COMMAND_OPENAI_KEY_RE = /\bsk-[A-Za-z0-9_-]{12,}\b/g;
const COMMAND_GITHUB_TOKEN_RE = /\bgh[pousr]_[A-Za-z0-9_]{20,}\b/g;
const COMMAND_JWT_RE =
/\b(?<![A-Za-z0-9_]-*)[A-Za-z0-9_-]{8,}(?:\.[A-Za-z0-9_-]{8,}){2}(?:\.[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,})?\b/g;
/** Recognize encoded JSON headers, without treating dotted identifiers as tokens. */
export function looksLikeCredentialJwt(value: string): boolean {
const segments = value.split(".");
if (![3, 5].includes(segments.length)) return false;
if (segments.some((part, index) => {
// Compact JWE direct encryption has no encrypted-key value.
if (segments.length === 5 && index === 1 && part === "") return false;
return !/^[A-Za-z0-9_-]+$/.test(part);
})) return false;
if (segments[0].startsWith("eyJ")) return true;
try {
const header = JSON.parse(atob(segments[0].replace(/-/g, "+").replace(/_/g, "/")));
return typeof header === "object" && header !== null && typeof header.alg === "string";
} catch {
return false;
}
}
const COMMAND_SECRET_HINTS = [
"api",
"key",
"token",
"auth",
"bearer",
"secret",
"pass",
"credential",
"jwt",
"private",
"cookie",
"connectionstring",
"sk-",
"ghp_",
"gho_",
"ghu_",
"ghs_",
"ghr_",
] as const;
function maybeContainsSecretText(command: string) {
const lower = command.toLowerCase();
return (
COMMAND_SECRET_HINTS.some((hint) => lower.includes(hint)) ||
command.includes(".")
);
}
export function redactCommandText(
command: string,
redactedValue = REDACTED_COMMAND_TEXT_VALUE,
): string {
if (!maybeContainsSecretText(command)) return command;
return command
.replace(COMMAND_AUTHORIZATION_BEARER_RE, `$1${redactedValue}`)
.replace(COMMAND_CLI_SECRET_OPTION_RE, `$1${redactedValue}$3`)
.replace(
COMMAND_ENV_SECRET_ASSIGNMENT_RE,
(
_match,
prefix: string,
escapedQuote: string | undefined,
_escapedValue: string | undefined,
rawQuote: string | undefined,
) => {
const quote = escapedQuote ?? rawQuote;
return quote
? `${prefix}${quote}${redactedValue}${quote}`
: `${prefix}${redactedValue}`;
},
)
.replace(COMMAND_OPENAI_KEY_RE, redactedValue)
.replace(COMMAND_GITHUB_TOKEN_RE, redactedValue)
.replace(COMMAND_JWT_RE, (match) => looksLikeCredentialJwt(match) ? redactedValue : match);
}
// A JSON secret field is a key/value pair such as `"token":"opaque-value"`. The
// command redaction handles shell `KEY=value` syntax only. A sandbox diagnostic
// can also carry a serialized JSON error, so the sanitizer must redact the JSON
// form too. The value body consumes JSON escape sequences. An escaped quote
// (`\"`) inside the value does not end the match early.
const JSON_SECRET_FIELD_RE = new RegExp(
String.raw`("(?:${SECRET_NAME_PATTERN})"\s*:\s*")(?:\\[\s\S]|[^"\\])*(")`,
"gi",
);
// An escaped JSON secret field is the same pair inside a JSON string. The double
// quote appears as `\"` and a backslash appears as `\\`. The value body
// consumes the doubled escape sequences. An escaped quote inside the value does
// not end the match early. The value ends at the next unescaped `\"`.
const JSON_ESCAPED_SECRET_FIELD_RE = new RegExp(
String.raw`(\\"(?:${SECRET_NAME_PATTERN})\\"\s*:\s*\\")(?:\\\\\\\\|\\\\\\"|\\\\[\s\S]|[^\\"])*(\\")`,
"gi",
);
/**
* Redact secrets from an untrusted diagnostic string.
*
* The function first runs the command redaction. The command redaction handles
* shell `KEY=value` assignments, CLI secret options, bearer headers, and common
* token shapes. The function then redacts JSON and escaped-JSON secret fields,
* because a sandbox diagnostic can carry a serialized JSON error such as
* `{"token":"opaque-value"}`. The caller must still bound the length after this
* step.
*/
export function redactDiagnosticText(
text: string,
redactedValue = REDACTED_COMMAND_TEXT_VALUE,
): string {
return redactCommandText(text, redactedValue)
.replace(JSON_ESCAPED_SECRET_FIELD_RE, `$1${redactedValue}$2`)
.replace(JSON_SECRET_FIELD_RE, `$1${redactedValue}$2`);
}