Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-25 02:26:12 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
daea92b647ce7a36a21542f140901754604f7c68
PaperClipAI/server/src/redaction.ts
T
DottaandPaperclip 889947c238 feat: add experimental native chat connectors (#13038) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People also ask agents for work in their existing chat tools.
> - Each external conversation needs one task and a current authorized
source.
> - Retries, Stop, and provider failures must not duplicate work or
expose private data.
> - The first chat PR establishes the opt-in provider and data
contracts.
> - This PR adds experimental channel integration and its durable
control plane.
> - Users can request work from connected channels and inspect delivery
in Paperclip.

## Linked Issues or Issue Description

Refs #13100 and #13092. This is the second of exactly two chat PRs.
Foundation #13100 is merged and changed 143 files. Runner prerequisite
#13092 is also merged. This PR changes 400 files against master, below
the 500-file review limit. It contains no wireframe images or HTML
galleries.

## What Changed

- Add native Slack, GitHub, Microsoft Teams, Telegram, and Discord chat
connections. Keep chat disabled unless the operator enables experimental
chat connectors. Preserve the production GitHub tool connection and its
normal setup path.
- Bind each provider bot identity to one immutable Paperclip agent. Bind
each admitted external conversation to one task. Paperclip owns tasks,
runs, permissions, and audit records.
- Add durable admission, per-conversation queues, questions, task
controls, progress, final replies, images, files, and delivery receipts.
Board comments remain internal unless explicitly sent to the channel.
- Check current identity, provider reach, resource access, credentials,
runtime generation, and exact source before provider effects. Keep
private responses private. Never send raw reasoning, private logs,
credentials, or tool arguments.
- Hold uncertain sends for explicit audited resolution. Make Board
Send-to-channel atomic and idempotent. Keep reconnect and setup
credentials in Paperclip secret storage.
- Preserve current native-runner authority across retries, lost
acknowledgements, and recovery. Keep immutable input and completion
contracts separate from newer user input. Receipt reconciliation cannot
launch a provider.
- Reconcile chat close/new ordering and provider-effect lock order.
Audit resource access changes in the same transaction. Submit only the
selected resource from each UI toggle so stale pages cannot undo
unrelated access changes.
- Drain Codex stdout before certifying process exit. Bound the drain
with the existing shutdown grace. Preserve observed terminal authority
without treating an undrained process as successful or reusable.
- Incorporate master `018ca5da` with its ACP Stop, mobile task layout,
runner packaging, and official lock changes. Preserve dedicated
chat-answer continuations in both directions when ordinary queued
comments are adopted after Stop.
- Fence late adapter readiness behind an earlier Stop for the same run.
Preserve verified cleanup for registered adapters. Handle single Stop,
agent pause, duplicate Stops, and failure release without creating a
false cancellation receipt.
- Incorporate master's `6dd48cad4` wake-queue extraction. Preserve exact
failed-chat retry authorization and lineage, retired question-source
suppression, and the block on generic recovery that would discard the
admitted source. Fresh deferred input retains its separate promotion
path.
- Incorporate master `2a05b5ed3` and its queue-admission extraction,
simplified transaction ports, and separate runner CI job. Preserve exact
durable receipts, actor separation, and dedicated-answer isolation
through the new module. A failed receipt insert rolls back the
accompanying deferred-wake merge.

## Verification

Current head: `afe19299d06253cb628eb398e91d1200ea9f412a`, incorporating
master `2a05b5ed3457ea33efd6895520447d1d97fe98d8`. The conflicts are
resolved. This successor fixes two test-harness boundaries exposed by
CI: per-case route-module preparation and actual durable-save completion
before intentional runner termination. Production code and all existing
test/turn deadlines are unchanged. [Exact-head Greptile
review](https://github.com/paperclipai/paperclip/pull/13038#issuecomment-5587250594)
is **5/5**, completed September 10 at 13:20:55 UTC, with no actionable
findings or open review threads. [Fresh exact-head
CI](https://github.com/paperclipai/paperclip/actions/runs/34481724341)
passes **all 24 jobs**, including Build and both required aggregates.
Normal exact-head guarded merge was attempted and rejected by the
remaining branch approval policy: CODEOWNER review is required and no
human approval is present. Normal **squash auto-merge is enabled** as of
September 10 at 13:36:26 UTC. Requested CODEOWNERS have been notified;
no approval bypass or self-approval was used. Earlier-head results below
remain historical evidence, not qualification of this successor.

- Final exact-head Linux evidence: 995/995 chat integration cases; 36/36
agent-skills routes; 35/35 runner live-session cases, including real
process kill/resume; 1948 runner Vitest cases with three existing
benchmark/platform guards; 870/870 API-authority cases; and 104 browser
cases with four existing optional skips. Rust, conformance/replay, full
repository build, typecheck, canary, all server/workspace shards, and
both required aggregates pass with normal CI concurrency. Earlier failed
attempts remain recorded below.

- Latest test-only qualification: 141/141
route/permissions/authentication cases pass in separate cold forks, with
plain server types and independent review clear. The real-runner suite
passes 35/35, with plain runner types and independent review clear. A
controlled premature-save acknowledgement fails as expected; matching
ownership/effect/process evidence, rejected saves, real turn outcome,
test abort, and pre-kill liveness are covered. No local reproduction of
the original CI scheduling failure is claimed. The preceding [CI
run](https://github.com/paperclipai/paperclip/actions/runs/34479680858)
passes 21/24 jobs, including all 995 Linux chat cases and browser
aggregate (104 passed, four existing optional skips); only Build, the
skills serialized shard, and the required verification aggregate fail.
Its exact-head Greptile review was 5/5. Both failed job logs are
retained.

- Final fixture qualification: all eight focused Discord cases and all
995 chat integration cases pass. The exact modal statement/PID is
observed before taking the real connection lock; the test then proves
its actual blocking relationship before mutation. Original SQL
execution, provider behavior, negative assertions, and 1s/15s timeouts
remain unchanged. Independent review is clear and test/production hashes
remain frozen. The preceding [CI
attempt](https://github.com/paperclipai/paperclip/actions/runs/34477184777)
passed 22 jobs, including Build/runner, typecheck, canary, all other
test shards, and browser aggregate (104 passed, four existing optional
skips); the two fixture failures and failed verification aggregate
remain recorded, not relabeled as a pass.

- Current queue-module composition: 308/308 recovery/batching/queue/Stop
tests; 995/995 full chat integration; 89/89 module tests, including real
PostgreSQL receipt-insert rollback; 24/24 workflow/module-boundary
tests; plain server and UI types. All four actual local process/ACP
browser paths pass in 1.4 minutes. Fresh databases, no skips or retries,
stable reviewed source hashes. The initial boundary failure is retained;
its no-op service wrapper was removed without changing recovery context
or weakening the check. An exploratory standalone test-directory
typecheck fails because its new upstream transformation config is not a
standalone typechecking project; standard CI/build does not invoke it,
and no configuration was weakened to suppress those diagnostics.

- The preceding head `e02a63d462ce5d47433b0aeb632bb6fd20aab1ba` passed
[all 24 CI
jobs](https://github.com/paperclipai/paperclip/actions/runs/34436462958)
and exact-head Greptile review at 5/5. Required CODEOWNER review
prevented its normal merge before master advanced again.

- Final extracted-module composition: 307/307 recovery, batching, queue
and Stop-control tests; 995/995 full chat integration; 49/49 module
tests including eight PostgreSQL adapter cases; and 19/19 issue-update
tests. Plain server types pass. All four actual local process/ACP
browser paths pass in 1.3 minutes. Fresh databases, no skips or retries
in these cohorts, frozen source hashes, and independent review clear.

- The preceding head `3e4e1c1c` passes [all PR CI
jobs](https://github.com/paperclipai/paperclip/actions/runs/34415826820),
including Build and required `ci / verify` and `ci / e2e`. Both the
original Rust failure and the previously load-sensitive lineage fixture
pass with unchanged Linux concurrency. Master advanced afterward and
required this reconciliation.
- Final master composition: 448/448 focused UI tests, 186/186 adapter
tests, 24/24 queue/control tests, and 11/11 packaging tests. Plain UI,
server, shared, and adapter types pass. Token gates and diff checks
pass. Independent server and UI reviews are clear.
- Stop-registration regression: both real-service cases fail against
exact `a95` source and pass with the fix. The full corrected
recovery/control suite passes 265/265. Duplicate-owner and failed-Stop
controls also pass. Plain server types pass. The readiness barrier
prevents provider startup without adding an acknowledgment to an already
terminal run.
- Final qualification strengthens terminal-field equality and repeats
both affected cases successfully on a fresh database. All four actual
local process/ACP browser paths pass again in 1.3 minutes, without skips
or retries. The final screenshot shows Cancelled, a paused subtree,
retained input, and no error toast.
- Two new actual-service regressions fail before the merge fix. They
prove that queued-comment adoption could consume a dedicated chat answer
or add unrelated input to that answer. The fixed four-case cohort
passes, including ordinary upstream continuation and adapter Stop
controls. Full recovery passes 257/257. All four actual local
process/ACP Stop browser flows pass in 1.4 minutes, without skips or
retries, on a fresh database.
- The unchanged runner artifact was qualified with 171/171 transport
tests, 870/870 API-authority tests, conformance 1/1, and replay 11/11.
Six controlled reader tests prove the exit/drain repair. Its local
serial Rust workspace passed 546 top-level cases plus two invoked
helpers; the later passing Linux CI supplies default-concurrency
evidence.
- Prior exact-source full chat integration passes 995/995. Settings
regressions cover concurrent stale pages, 501 destinations, pending
state, rejected updates, and explicit retry. These deterministic tests
do not prove live provider behavior.
- Retained failed attempts and their causes are in the [qualification
log](https://github.com/paperclipai/paperclip/blob/afe19299d06253cb628eb398e91d1200ea9f412a/doc/plans/chat-adapters/2026-09-08-chat-queue-and-webhook-repair.md).
The first merge adapter run timed out while macOS slept for 290 seconds.
Its unchanged repeat passed with a temporary sleep guard. No assertion,
deadline, or CI gate was weakened.

Review commands include `pnpm --filter @paperclipai/server exec vitest
run src/__tests__/heartbeat-process-recovery.test.ts
src/__tests__/issue-queued-comments-routes.test.ts` and `pnpm exec
playwright test --config tests/e2e/playwright.config.ts
tests/e2e/acp-stop-continuation.spec.ts`. Database suites require fresh
disposable databases. See the [browser
runbook](https://github.com/paperclipai/paperclip/blob/afe19299d06253cb628eb398e91d1200ea9f412a/doc/plans/chat-adapters/2026-09-04-chat-adapters-browser-e2e-runbook.md)
for provider setup and separate live acceptance steps.

## Risks

- This remains experimental. Deterministic tests and bounded live
evidence do not establish every provider feature, tenant, permission
layout, or media shape. Teams work-tenant qualification is still open.
- Failed and uncertain provider effects remain visible and can require
operator action. A transport receipt does not prove recipient
visibility.
- Native controller and runner artifacts must remain compatible.
Preserve lease ownership, terminal authority, source binding, and
quarantine during future changes.
- Access and audit rows commit together, but activity notifications
remain best-effort. This is not a new durable event outbox.
- The PR operation does not deploy a live server, replace its runner, or
change provider permissions. Remaining live qualification is documented
in the [temporary
handoff](https://github.com/paperclipai/paperclip/blob/afe19299d06253cb628eb398e91d1200ea9f412a/doc/plans/chat-adapters/2026-09-08-open-qualification-followups.md).

## Model Used

OpenAI Codex assisted with implementation, tool execution, testing, and
review. The work records `gpt-6-astra` assistance. The environment does
not report a context-window size. No private reasoning traces are
included.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-10 10:06:45 -05:00

988 lines
33 KiB
TypeScript
Raw Blame History

import { redactCommandText } from "@paperclipai/adapter-utils";
const SECRET_FIELD_NAME_PATTERN = String.raw`[A-Za-z0-9_-]*(?:api[-_]?key|access[-_]?token|auth(?:_?token)?|token|authorization|bearer|secret|passwd|password|credential|jwt|private[-_]?key|cookie|connectionstring|browser[-_]?code|login[-_]?url)[A-Za-z0-9_-]*`;
const SECRET_PAYLOAD_KEY_RE = new RegExp(SECRET_FIELD_NAME_PATTERN, "i");
// Authorization reasons are policy decision codes, not credentials. They must
// remain visible in audit receipts even though the field name contains
// "authorization". JWT-shaped values are still caught by the value guard below.
const AUDIT_REASON_PAYLOAD_KEY_RE = /^authorizationReason$/;
const AUDIT_SURFACE_PAYLOAD_KEY_RE = /^surface$/;
/**
* Cleanup counts on a connection-removal receipt (PAP-17119). Their names name
* the thing they counted — secrets, bindings, tokens — so the key guard above
* would blank the whole receipt and leave the operator unable to see what a
* revocation actually tore down. They pass only while the value really is a
* finite number, so nothing that could carry material rides through on the
* strength of a familiar key name.
*/
const AUDIT_COUNT_PAYLOAD_KEYS = new Set([
"secretsRevoked",
"secretsRetainedShared",
"credentialRefsCleared",
"secretBindingsRemoved",
"tokenIssuanceHashesCleared",
"gatewayTokensRevoked",
// PRP usage.reported metrics count model tokens; they never carry token
// credential material. Keep the exemption closed to finite numbers.
"inputTokens",
"outputTokens",
"cacheReadTokens",
"cacheWriteTokens",
"preTokens",
"postTokens",
]);
function isAuditCountField(key: string, value: unknown): boolean {
return (
AUDIT_COUNT_PAYLOAD_KEYS.has(key) &&
typeof value === "number" &&
Number.isFinite(value)
);
}
const COMMAND_PAYLOAD_KEY_RE =
/(^command$|^cmd$|command[-_]?line|resolved[-_]?command|PAPERCLIP_RESOLVED_COMMAND)/i;
const COMMAND_ARGS_PAYLOAD_KEY_RE = /^(commandArgs|command_?args|argv)$/i;
const JWT_VALUE_RE =
/^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+(?:\.[A-Za-z0-9_-]+)?$/;
// Durable protocol schema identifiers share JWT's broad dotted shape but are
// public discriminators, not credentials. Exempt the Paperclip schema
// namespace only in fields that actually declare a schema; the same value in
// arbitrary provider data remains subject to the fail-closed JWT guard.
const PAPERCLIP_SCHEMA_FIELDS = new Set(["schema", "runtimeSchema"]);
export const PAPERCLIP_PUBLIC_SCHEMA_IDS = new Set([
"paperclip.artifact.generated.v1",
"paperclip.artifact.viewed.v1",
"paperclip.capability-discovery.v1",
"paperclip.capability.authorization-record.v1",
"paperclip.capability.control-plane.v1",
"paperclip.capability.devtools.v1",
"paperclip.capability.eval-parity-report.v1",
"paperclip.capability.exposure.v1",
"paperclip.capability.issue-thread-view.v1",
"paperclip.capability.live-session-checkpoint.v1",
"paperclip.capability.live-session.v1",
"paperclip.capability.live-smoke.v1",
"paperclip.capability.mock-export.v1",
"paperclip.capability.mock-state.v1",
"paperclip.capability.model-tool-result.v1",
"paperclip.capability.run-artifact.v1",
"paperclip.capability.run-context.v1",
"paperclip.capability.scenario-index.v1",
"paperclip.capability.scenario-parity.v1",
"paperclip.capability.semantic-tool-runtime.v1",
"paperclip.capability.state-diff.v1",
"paperclip.capability.tool-result.v1",
"paperclip.capability.turn-stream.v1",
"paperclip.capability.visible-tools.v1",
"paperclip.completion-contract.v1",
"paperclip.context.compacted.v1",
"paperclip.delegation.v1",
"paperclip.fake_harness.script.v1",
"paperclip.hook.v1",
"paperclip.interaction_request.v1",
"paperclip.local-provider-boundary-golden.v1",
"paperclip.memory.citation.v1",
"paperclip.model.provider_message.v1",
"paperclip.model.route_changed.v1",
"paperclip.model.verification.v1",
"paperclip.native-cancellation.v1",
"paperclip.native-execution-input.v1",
"paperclip.native-execution-input.v2",
"paperclip.native-execution-input.v3",
"paperclip.native-execution-input.v4",
"paperclip.native-finalization.v1",
"paperclip.native-finalization.v2",
"paperclip.native-harness-backup-stamp.v1",
"paperclip.native-harness-backup-stamp.v2",
"paperclip.native-harness-backup.v1",
"paperclip.native-model-envelope.v1",
"paperclip.native-model-envelope.v2",
"paperclip.native-session-scope.v2",
"paperclip.native-session-supervisor.v1",
"paperclip.plan.updated.v1",
"paperclip.provider.native.v1",
"paperclip.provider.notice.v1",
"paperclip.provider_trace_frame.v1",
"paperclip.provider_trace_interpretation.v1",
"paperclip.provider_trace_metadata.v1",
"paperclip.prp.capabilities.v1",
"paperclip.prp.command.v1",
"paperclip.prp.contract_manifest.v1",
"paperclip.prp.event.v1",
"paperclip.prp.fixture.v1",
"paperclip.prp.identity.v1",
"paperclip.prp.semantic_tool.v1",
"paperclip.prp.semantic_tool.v2",
"paperclip.prp.semantic_tools.v1",
"paperclip.prp.session-snapshot.v1",
"paperclip.prp.stop_reason.v1",
"paperclip.prp.terminal.v1",
"paperclip.question_adapter_fixture.v1",
"paperclip.question_response.v1",
"paperclip.question_response_delivery.v1",
"paperclip.question_set.v1",
"paperclip.research.v1",
"paperclip.review.mode_changed.v1",
"paperclip.run-performance-span.v1",
"paperclip.run-result.v1",
"paperclip.run_presentation_decision.v1",
"paperclip.run_result.v1",
"paperclip.runner.acpx-identity.v1",
"paperclip.runner.acpx-identity.v2",
"paperclip.runner.authorized-tools.v1",
"paperclip.runner.codex.metadata.v1",
"paperclip.runner.codex.trace.v1",
"paperclip.runner.compatibility.v1",
"paperclip.runner.conformance.fixture.v1",
"paperclip.runner.conformance.output.v1",
"paperclip.runner.durable.control-plane-state.v1",
"paperclip.runner.durable.diagnostics.v1",
"paperclip.runner.durable.state.v1",
"paperclip.runner.durable.trace.v1",
"paperclip.runner.eval-bundle-evidence.v1",
"paperclip.runner.eval-bundle.v1",
"paperclip.runner.eval-scorecard.v1",
"paperclip.runner.eval-scorecard.v2",
"paperclip.runner.eval-slice-report.v1",
"paperclip.runner.live-console.conformance.v1",
"paperclip.runner.live-eval-candidate.v1",
"paperclip.runner.live-eval-schedule.v1",
"paperclip.runner.local-acpx-authority.v1",
"paperclip.runner.local-runner.metadata.v1",
"paperclip.runner.local-runner.summary.v1",
"paperclip.runner.local-runner.trace.v1",
"paperclip.runner.profile.v1",
"paperclip.runner.retained_cleanup_failure.v1",
"paperclip.runner.sanitized-provider-fixture.v1",
"paperclip.runner.secure-frame.v1",
"paperclip.runner.standalone.standalone-demo.v1",
"paperclip.runner.stream.v1",
"paperclip.runner.stress-eval-traceability.v1",
"paperclip.runner.workflow-eval-case.v1",
"paperclip.runner.workflow-eval-report.v1",
"paperclip.runner.workflow-observation.v1",
"paperclip.runtime-asset-manifest.v1",
"paperclip.runtime-asset.v1",
"paperclip.runtime_request.v1",
"paperclip.runtime_request.v2",
"paperclip.safety.review.v1",
"paperclip.semantic-action.v1",
"paperclip.semantic-authorization-record.v1",
"paperclip.semantic-binding.v1",
"paperclip.semantic-conformance-report.v1",
"paperclip.semantic-denial.v1",
"paperclip.semantic-discovery.v1",
"paperclip.semantic-interaction-result.v1",
"paperclip.semantic-tool.v1",
"paperclip.semantic_tool_result.v1",
"paperclip.semantic_tool_result_chunks.v1",
"paperclip.skillless_task.v1",
"paperclip.status-authority-conformance.v1",
"paperclip.stop_reason.v1",
"paperclip.tagged_graph.v1",
"paperclip.task_envelope.v1",
"paperclip.terminal.input_sent.v1",
"paperclip.tool.execution.v1",
"paperclip.wait.v1",
"paperclip.workspace.diff.v1",
"paperclip.workspace.file_reference.v1",
"paperclip.workspace_create_target.v1",
"paperclip.workspace_entry.v1",
"paperclip.workspace_relative_display.v2",
]);
// Keep this closed catalog aligned with PRP v1's event.schema.json. These
// values are public protocol discriminators, but their dotted shape overlaps
// the deliberately broad JWT heuristic. They are exempt only in the
// discriminator field of a PRP v1 event envelope; the same string anywhere
// else remains subject to redaction.
export const PRP_V1_EVENT_TYPES = new Set([
"runner.connected",
"runner.reconnected",
"runner.reconciled",
"runner.disconnected",
"runner.draining",
"runner.backpressure",
"runner.suspending",
"runner.suspended",
"runner.stopped",
"runner.diagnostic",
"runtime.phase.changed",
"sandbox.metric",
"workspace.ready",
"workspace.change.updated",
"workspace.diff.recorded",
"workspace.file.referenced",
"harness.starting",
"harness.ready",
"harness.exited",
"harness.diagnostic",
"plan.updated",
"tool.execution.started",
"tool.execution.progressed",
"tool.execution.completed",
"research.started",
"research.progressed",
"research.completed",
"delegation.started",
"delegation.updated",
"delegation.completed",
"model.route.changed",
"model.verification.updated",
"context.compacted",
"artifact.viewed",
"artifact.generated",
"review.mode.changed",
"hook.started",
"hook.completed",
"memory.citation.referenced",
"safety.review.started",
"safety.review.completed",
"terminal.input.sent",
"wait.started",
"wait.completed",
"provider.notice.recorded",
"session.starting",
"session.started",
"session.resuming",
"session.resumed",
"session.reconciled",
"session.updated",
"session.closed",
"session.failed",
"turn.submitted",
"turn.accepted",
"turn.started",
"turn.completed",
"turn.failed",
"turn.interrupted",
"turn.cancelled",
"item.started",
"item.delta",
"item.completed",
"item.failed",
"usage.reported",
"semantic_tool.input",
"semantic_tool.result",
"semantic_tool.reconciled",
"mcp_app.discovered",
"mcp_app.resource.resolved",
"mcp_app.initializing",
"mcp_app.ready",
"mcp_app.tool_input",
"mcp_app.tool_result",
"mcp_app.action.requested",
"mcp_app.action.resolved",
"mcp_app.host_context.changed",
"mcp_app.failed",
"mcp_app.teardown",
"runtime_request.created",
"runtime_request.resolved",
"runtime_request.expired",
"runtime_request.cancelled",
"interaction.request.proposed",
"interaction.request.materialized",
"interaction.request.rejected",
"interaction.response.progressed",
"interaction.response.resolved",
"interaction.response.delivered",
"run.attached",
"run.detached",
"run.result.proposed",
"run.result.accepted",
"run.result.rejected",
"attention.request.proposed",
"attention.request.routed",
"attention.request.resolved",
"attention.request.expired",
"attention.request.superseded",
"work.assessment.recorded",
"issue.status.decision.recorded",
"issue.status.decision.applied",
"issue.status.decision.rejected",
"issue.status.decision.superseded",
"run.terminal",
]);
const NATIVE_RUN_SPAN_SCHEMA = "paperclip.run-performance-span.v1";
const NATIVE_RUN_SPAN_FIELDS = ["span", "parentSpan"] as const;
const NATIVE_RUN_SPAN_NAMES = new Set([
"agent.turn",
"environment.acquire",
"environment.startup",
"environment.workspace.realize",
"native.coordinator.claim",
"native.result.finalize",
"native.session.execute",
"provider.session.continuity_break",
"provider.session.resume",
"provider.time_to_first_agent_event",
"provider.turn.queue",
"question_response.to_run_created",
"runner.artifact.discover",
"runner.artifact.prepare",
"runner.prp.authenticate",
"runner.prp.route.register",
"runner.runtime.stage",
"runner.session.bootstrap",
"runner.session.resume",
"runner.session.startup",
"runner.transport.connect",
"runner.transport.selected",
"runner.turn.submit",
"task.prepare",
"task.run",
"task.run.measured",
"task.settle",
]);
const CLI_SECRET_FLAG_RE = new RegExp(
String.raw`^-{1,2}${SECRET_FIELD_NAME_PATTERN}$`,
"i",
);
const JSON_SECRET_FIELD_TEXT_RE = new RegExp(
String.raw`((?:"|')?${SECRET_FIELD_NAME_PATTERN}(?:"|')?\s*:\s*(?:"|'))[^"'` +
"`" +
String.raw`\r\n]+((?:"|'))`,
"gi",
);
const ESCAPED_JSON_SECRET_FIELD_TEXT_RE = new RegExp(
String.raw`((?:\\")?${SECRET_FIELD_NAME_PATTERN}(?:\\")?\s*:\s*(?:\\"))[^\\\r\n]+((?:\\"))`,
"gi",
);
const SECRET_TEXT_HINTS = [
"api",
"key",
"token",
"auth",
"bearer",
"secret",
"pass",
"credential",
"jwt",
"private",
"cookie",
"connectionstring",
"sk-",
"ghp_",
"gho_",
"ghu_",
"ghs_",
"ghr_",
] as const;
export const REDACTED_EVENT_VALUE = "***REDACTED***";
function maybeContainsSecretText(input: string) {
const lower = input.toLowerCase();
return (
SECRET_TEXT_HINTS.some((hint) => lower.includes(hint)) ||
input.includes(".")
);
}
function inlineWhitespaceEnd(input: string, start: number): number {
let index = start;
while (
index < input.length &&
(input[index] === " " || input[index] === "\t")
) {
index += 1;
}
return index;
}
function quotedValueBoundary(
input: string,
index: number,
): "hard" | "line" | "provisional" | "unsafe" {
const boundary = input[index];
if (boundary === undefined || /[,;&)}\]]/.test(boundary)) return "hard";
if (boundary === "\r" || boundary === "\n") return "line";
if (/\s/.test(boundary)) return "provisional";
return "unsafe";
}
function isTrustedQuotedValueBoundary(input: string, index: number) {
const boundary = quotedValueBoundary(input, index);
return boundary === "hard" || boundary === "provisional";
}
function startsIndependentCredentialLine(input: string, index: number) {
if (input[index] !== "\r" && input[index] !== "\n") return false;
let lineStart = index + 1;
if (input[index] === "\r" && input[lineStart] === "\n") lineStart += 1;
const lineEndCandidates = [
input.indexOf("\r", lineStart),
input.indexOf("\n", lineStart),
].filter((candidate) => candidate >= 0);
const lineEnd =
lineEndCandidates.length > 0
? Math.min(...lineEndCandidates)
: input.length;
const words = input
.slice(lineStart, lineEnd)
.trimStart()
.toLowerCase()
.split(/[ \t]+/);
return (
(words[0] === "bearer" && words.length >= 2) ||
(words[0] === "request" &&
words[1] === "failed" &&
words[2] === "with" &&
words[3] === "bearer" &&
words.length >= 5)
);
}
function rawQuotedValueEnd(
input: string,
start: number,
quote: '"' | "'",
): number {
const end = input.length;
let provisionalEnd: number | null = null;
let unsafeAfterProvisional = false;
for (let index = start + 1; index < end; index += 1) {
if (input[index] === "\\") {
index += 1;
continue;
}
if (input[index] === quote) {
const candidateEnd = index + 1;
const boundary = quotedValueBoundary(input, candidateEnd);
if (boundary === "hard") return candidateEnd;
if (boundary === "provisional") {
provisionalEnd = candidateEnd;
unsafeAfterProvisional = false;
} else if (
boundary === "line" &&
startsIndependentCredentialLine(input, candidateEnd)
) {
return candidateEnd;
} else {
if (boundary === "line") provisionalEnd = null;
unsafeAfterProvisional = true;
}
}
}
// Whitespace normally separates safe trailing context, so retain a final
// provisional delimiter when no later quote contradicts it. Literal newlines
// can occur inside provider-controlled credentials, so an unterminated
// malformed value fails closed through the complete bounded diagnostic.
return provisionalEnd !== null && !unsafeAfterProvisional
? provisionalEnd
: end;
}
function escapedQuotedValueEnd(
input: string,
start: number,
quote: '"' | "'",
): number {
const end = input.length;
let provisionalEnd: number | null = null;
let unsafeAfterProvisional = false;
for (let index = start + 2; index < end; index += 1) {
if (input[index] !== "\\") continue;
let afterSlashes = index + 1;
while (afterSlashes < end && input[afterSlashes] === "\\") {
afterSlashes += 1;
}
if (input[afterSlashes] !== quote) {
index = afterSlashes - 1;
continue;
}
// A serialized outer quote has one slash. Three or more slashes encode a
// quote nested inside that value, so keep scanning for the outer delimiter.
if (afterSlashes - index === 1) {
const candidateEnd = afterSlashes + 1;
const boundary = quotedValueBoundary(input, candidateEnd);
if (boundary === "hard") return candidateEnd;
if (boundary === "provisional") {
provisionalEnd = candidateEnd;
unsafeAfterProvisional = false;
} else if (
boundary === "line" &&
startsIndependentCredentialLine(input, candidateEnd)
) {
return candidateEnd;
} else {
if (boundary === "line") provisionalEnd = null;
unsafeAfterProvisional = true;
}
}
index = afterSlashes;
}
return provisionalEnd !== null && !unsafeAfterProvisional
? provisionalEnd
: end;
}
function escapedQuoteAt(input: string, index: number): '"' | "'" | null {
if (input[index] !== "\\") return null;
const quote = input[index + 1];
return quote === '"' || quote === "'" ? quote : null;
}
function credentialEndAfterQuotedDelimiter(input: string, quotedEnd: number) {
if (isTrustedQuotedValueBoundary(input, quotedEnd)) return quotedEnd;
if (startsIndependentCredentialLine(input, quotedEnd)) return quotedEnd;
// A closing delimiter followed immediately by more token bytes is not a
// trustworthy credential boundary (for example `"abc"defg`). Once a
// provider diagnostic is malformed this way, whitespace is not a safe
// boundary either (`"a"b c"`). Fail closed through the rest of the
// diagnostic so no later credential fragment survives.
return input.length;
}
interface AuthorizationCredentialRange {
start: number;
end: number;
replacement: string;
}
function authorizationCredentialRange(
input: string,
wordEnd: number,
): AuthorizationCredentialRange | null {
let valueStart = wordEnd;
// JSON and serialized JSON close the quoted key before the colon. Treat a
// quote as part of the key only when a colon follows it; otherwise it is the
// opening delimiter of a header value such as Authorization "Bearer ...".
const rawKeyQuote = input[valueStart];
const escapedKeyQuote = escapedQuoteAt(input, valueStart);
const keyQuoteWidth =
rawKeyQuote === '"' || rawKeyQuote === "'" ? 1 : escapedKeyQuote ? 2 : 0;
if (keyQuoteWidth > 0) {
const possibleColon = inlineWhitespaceEnd(
input,
valueStart + keyQuoteWidth,
);
if (input[possibleColon] === ":") valueStart = possibleColon;
}
valueStart = inlineWhitespaceEnd(input, valueStart);
if (input[valueStart] === ":" || input[valueStart] === "=") {
valueStart = inlineWhitespaceEnd(input, valueStart + 1);
}
const rawValueQuote = input[valueStart];
if (rawValueQuote === '"' || rawValueQuote === "'") {
const quotedEnd = rawQuotedValueEnd(input, valueStart, rawValueQuote);
const content = input.slice(valueStart + 1, quotedEnd - 1).trimStart();
if (!/^(?:Bearer|Basic)\b/i.test(content)) return null;
return {
start: valueStart,
end: credentialEndAfterQuotedDelimiter(input, quotedEnd),
replacement: `${rawValueQuote}${REDACTED_EVENT_VALUE}${rawValueQuote}`,
};
}
const escapedValueQuote = escapedQuoteAt(input, valueStart);
if (escapedValueQuote) {
const quotedEnd = escapedQuotedValueEnd(
input,
valueStart,
escapedValueQuote,
);
const content = input.slice(valueStart + 2, quotedEnd - 2).trimStart();
if (!/^(?:Bearer|Basic)\b/i.test(content)) return null;
const delimiter = `\\${escapedValueQuote}`;
return {
start: valueStart,
end: credentialEndAfterQuotedDelimiter(input, quotedEnd),
replacement: `${delimiter}${REDACTED_EVENT_VALUE}${delimiter}`,
};
}
const scheme = /^(?:Bearer|Basic)\b/i.exec(input.slice(valueStart));
if (!scheme) return null;
let credentialStart = inlineWhitespaceEnd(
input,
valueStart + scheme[0].length,
);
if (credentialStart === valueStart + scheme[0].length) return null;
if (
credentialStart >= input.length ||
/[\r\n]/.test(input[credentialStart])
) {
return null;
}
let end: number;
const rawCredentialQuote = input[credentialStart];
if (rawCredentialQuote === '"' || rawCredentialQuote === "'") {
end = credentialEndAfterQuotedDelimiter(
input,
rawQuotedValueEnd(input, credentialStart, rawCredentialQuote),
);
} else {
const escapedCredentialQuote = escapedQuoteAt(input, credentialStart);
if (escapedCredentialQuote) {
end = credentialEndAfterQuotedDelimiter(
input,
escapedQuotedValueEnd(input, credentialStart, escapedCredentialQuote),
);
} else {
end = credentialStart;
// Embedded quote/backtick bytes do not make an unquoted credential safe;
// consume them through the next structural or whitespace boundary.
while (end < input.length && !/[\s,;}\]]/.test(input[end])) end += 1;
}
}
return { start: valueStart, end, replacement: REDACTED_EVENT_VALUE };
}
function redactAuthorizationCredentials(input: string): string {
// Include compound diagnostic labels such as proxyAuthorization while the
// value parser still requires a Basic/Bearer scheme before redacting.
const authorizationWord = /\b[A-Za-z0-9_-]*Authorization[A-Za-z0-9_-]*\b/gi;
const parts: string[] = [];
let copiedThrough = 0;
let match: RegExpExecArray | null;
while ((match = authorizationWord.exec(input)) !== null) {
const range = authorizationCredentialRange(
input,
match.index + match[0].length,
);
if (!range || range.start < copiedThrough) continue;
parts.push(input.slice(copiedThrough, range.start), range.replacement);
copiedThrough = range.end;
authorizationWord.lastIndex = range.end;
}
if (copiedThrough === 0) return input;
parts.push(input.slice(copiedThrough));
return parts.join("");
}
function redactStandaloneBearerCredentials(input: string): string {
const bearerWord = /\bBearer\b/gi;
const parts: string[] = [];
let copiedThrough = 0;
let match: RegExpExecArray | null;
while ((match = bearerWord.exec(input)) !== null) {
const wordEnd = match.index + match[0].length;
const credentialStart = inlineWhitespaceEnd(input, wordEnd);
if (
credentialStart === wordEnd ||
credentialStart >= input.length ||
/[\r\n]/.test(input[credentialStart])
) {
continue;
}
let end: number;
let replacement = REDACTED_EVENT_VALUE;
const rawQuote = input[credentialStart];
if (rawQuote === '"' || rawQuote === "'") {
end = credentialEndAfterQuotedDelimiter(
input,
rawQuotedValueEnd(input, credentialStart, rawQuote),
);
replacement = `${rawQuote}${REDACTED_EVENT_VALUE}${rawQuote}`;
} else {
const escapedQuote = escapedQuoteAt(input, credentialStart);
if (escapedQuote) {
end = credentialEndAfterQuotedDelimiter(
input,
escapedQuotedValueEnd(input, credentialStart, escapedQuote),
);
const delimiter = `\\${escapedQuote}`;
replacement = `${delimiter}${REDACTED_EVENT_VALUE}${delimiter}`;
} else {
end = credentialStart;
while (end < input.length && !/[\s,;}\]]/.test(input[end])) end += 1;
}
}
if (credentialStart < copiedThrough) continue;
parts.push(input.slice(copiedThrough, credentialStart), replacement);
copiedThrough = end;
bearerWord.lastIndex = end;
}
if (copiedThrough === 0) return input;
parts.push(input.slice(copiedThrough));
return parts.join("");
}
function isPlainObject(value: unknown): value is Record<string, unknown> {
if (typeof value !== "object" || value === null || Array.isArray(value))
return false;
const proto = Object.getPrototypeOf(value);
return proto === Object.prototype || proto === null;
}
function sanitizeValue(value: unknown): unknown {
if (value === null || value === undefined) return value;
// Adapter diagnostics are provider-controlled text. Secret-bearing header or
// command fragments can appear under otherwise innocuous keys such as
// `message`, `reason`, or inside arrays, so apply the text scanner at every
// string leaf after validated protocol discriminators have had a chance to
// opt in above in sanitizeRecord.
if (typeof value === "string") {
return JWT_VALUE_RE.test(value)
? REDACTED_EVENT_VALUE
: redactSensitiveText(value);
}
if (Array.isArray(value)) return value.map(sanitizeValue);
if (isSecretRefBinding(value)) {
const version = safeSecretVersion(value.version);
return {
type: "secret_ref",
secretId: value.secretId,
...(version === undefined ? {} : { version }),
...(value.projectionClass === "unclassified" ||
value.projectionClass === "class_3_static_lease"
? { projectionClass: value.projectionClass }
: {}),
...(value.projectionAllowlistKey === null
? { projectionAllowlistKey: null }
: typeof value.projectionAllowlistKey === "string" &&
PROJECTION_ALLOWLIST_KEY_RE.test(value.projectionAllowlistKey) &&
redactSensitiveText(value.projectionAllowlistKey) ===
value.projectionAllowlistKey
? { projectionAllowlistKey: value.projectionAllowlistKey }
: {}),
};
}
if (isUserSecretRefBinding(value)) {
const version = safeSecretVersion(value.version);
return {
type: "user_secret_ref",
key: value.key,
...(version === undefined ? {} : { version }),
...(typeof value.required === "boolean"
? { required: value.required }
: {}),
...(typeof value.allowMissingOverride === "boolean"
? { allowMissingOverride: value.allowMissingOverride }
: {}),
};
}
if (isPlainBinding(value))
return { type: "plain", value: sanitizeValue(value.value) };
if (!isPlainObject(value)) return value;
return sanitizeRecord(value);
}
const SECRET_REFERENCE_ID_RE =
/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
const USER_SECRET_KEY_RE = /^[A-Za-z0-9_.-]{1,120}$/;
const PROJECTION_ALLOWLIST_KEY_RE =
/^[A-Za-z0-9_-]{1,80}\.[A-Za-z0-9_-]{1,80}$/;
function safeSecretVersion(value: unknown): number | "latest" | undefined {
if (value === "latest") return value;
return typeof value === "number" && Number.isInteger(value) && value > 0
? value
: undefined;
}
function isSecretRefBinding(value: unknown): value is {
type: "secret_ref";
secretId: string;
version?: unknown;
projectionClass?: unknown;
projectionAllowlistKey?: unknown;
} {
if (!isPlainObject(value)) return false;
return (
value.type === "secret_ref" &&
typeof value.secretId === "string" &&
SECRET_REFERENCE_ID_RE.test(value.secretId)
);
}
function isUserSecretRefBinding(value: unknown): value is {
type: "user_secret_ref";
key: string;
version?: unknown;
required?: unknown;
allowMissingOverride?: unknown;
} {
if (!isPlainObject(value)) return false;
return (
value.type === "user_secret_ref" &&
typeof value.key === "string" &&
USER_SECRET_KEY_RE.test(value.key) &&
redactSensitiveText(value.key) === value.key
);
}
function isPlainBinding(
value: unknown,
): value is { type: "plain"; value: unknown } {
if (!isPlainObject(value)) return false;
return value.type === "plain" && "value" in value;
}
function sanitizeCommandArgs(args: unknown[]): unknown[] {
let redactNext = false;
return args.map((arg) => {
if (redactNext) {
redactNext = false;
return REDACTED_EVENT_VALUE;
}
if (typeof arg !== "string") return sanitizeValue(arg);
if (CLI_SECRET_FLAG_RE.test(arg.trim())) {
redactNext = true;
return arg;
}
return redactSensitiveText(arg);
});
}
function isKnownPrpEventDiscriminator(
container: Record<string, unknown>,
key: string,
value: unknown,
): value is string {
return (
key === "eventType" &&
container.schema === "paperclip.prp.event.v1" &&
container.schemaVersion === 1 &&
typeof value === "string" &&
PRP_V1_EVENT_TYPES.has(value)
);
}
function isPaperclipSchemaDiscriminator(
key: string,
value: unknown,
): value is string {
return (
PAPERCLIP_SCHEMA_FIELDS.has(key) &&
typeof value === "string" &&
PAPERCLIP_PUBLIC_SCHEMA_IDS.has(value)
);
}
export function sanitizeRecord(
record: Record<string, unknown>,
): Record<string, unknown> {
const redacted: Record<string, unknown> = {};
for (const [key, value] of Object.entries(record)) {
if (COMMAND_ARGS_PAYLOAD_KEY_RE.test(key) && Array.isArray(value)) {
redacted[key] = sanitizeCommandArgs(value);
continue;
}
if (COMMAND_PAYLOAD_KEY_RE.test(key) && typeof value === "string") {
redacted[key] = redactSensitiveText(value);
continue;
}
if (
SECRET_PAYLOAD_KEY_RE.test(key) &&
!AUDIT_REASON_PAYLOAD_KEY_RE.test(key) &&
!isAuditCountField(key, value)
) {
if (isSecretRefBinding(value)) {
redacted[key] = sanitizeValue(value);
continue;
}
if (isUserSecretRefBinding(value)) {
redacted[key] = sanitizeValue(value);
continue;
}
if (isPlainBinding(value)) {
redacted[key] = { type: "plain", value: REDACTED_EVENT_VALUE };
continue;
}
redacted[key] = REDACTED_EVENT_VALUE;
continue;
}
// Interpret a validated schema field before applying generic value-shape
// heuristics. The exemption is deliberately closed to this one PRP v1
// discriminator; the same dotted string in any other field is redacted.
if (isKnownPrpEventDiscriminator(record, key, value)) {
redacted[key] = value;
continue;
}
if (isPaperclipSchemaDiscriminator(key, value)) {
redacted[key] = value;
continue;
}
if (AUDIT_SURFACE_PAYLOAD_KEY_RE.test(key) && typeof value === "string") {
redacted[key] = redactSensitiveText(value);
continue;
}
if (
typeof value === "string" &&
JWT_VALUE_RE.test(value) &&
!isPaperclipSchemaDiscriminator(key, value)
) {
redacted[key] = REDACTED_EVENT_VALUE;
continue;
}
redacted[key] = sanitizeValue(value);
}
return redacted;
}
export function redactEventPayload(
payload: Record<string, unknown> | null,
): Record<string, unknown> | null {
if (!payload) return null;
if (!isPlainObject(payload)) return payload;
const sanitized = sanitizeRecord(payload);
if (payload.schema !== NATIVE_RUN_SPAN_SCHEMA) return sanitized;
// Native run span identities are controlled diagnostics emitted by
// createNativeRunTrace, not provider data. Their dotted names overlap the
// broad JWT-value heuristic, so restore only these two fields on the exact
// run-performance schema. Hostnames and JWT-shaped values on every other
// field and schema still fail closed through sanitizeRecord above.
for (const field of NATIVE_RUN_SPAN_FIELDS) {
const value = payload[field];
if (typeof value === "string" && NATIVE_RUN_SPAN_NAMES.has(value)) {
sanitized[field] = value;
}
}
return sanitized;
}
function redactAgentEnvBinding(value: unknown): unknown {
if (isSecretRefBinding(value) || isUserSecretRefBinding(value)) {
return sanitizeValue(value);
}
if (typeof value === "string" || isPlainBinding(value)) {
return { type: "plain", value: REDACTED_EVENT_VALUE };
}
if (value === null || value === undefined) return value;
return REDACTED_EVENT_VALUE;
}
export function redactAgentAdapterConfig(
adapterConfig: Record<string, unknown>,
): Record<string, unknown> {
if (!isPlainObject(adapterConfig)) return adapterConfig;
if (!isPlainObject(adapterConfig.env)) return redactEventPayload(adapterConfig) ?? {};
// Redact `env` here and sanitize the remaining keys separately, so bindings
// are never processed twice. `redactAgentEnvBinding` is authoritative for
// `env`; keeping it out of `sanitizeRecord` means a future change there
// cannot alter entries this function has already redacted.
const { env, ...rest } = adapterConfig;
const redactedEnv = Object.fromEntries(
Object.entries(env).map(([key, value]) => [key, redactAgentEnvBinding(value)]),
);
return { ...(redactEventPayload(rest) ?? {}), env: redactedEnv };
}
export function redactSensitiveText(input: string): string {
if (!maybeContainsSecretText(input)) return input;
return redactCommandText(
redactStandaloneBearerCredentials(redactAuthorizationCredentials(input))
.replace(JSON_SECRET_FIELD_TEXT_RE, `$1${REDACTED_EVENT_VALUE}$2`)
.replace(
ESCAPED_JSON_SECRET_FIELD_TEXT_RE,
`$1${REDACTED_EVENT_VALUE}$2`,
),
REDACTED_EVENT_VALUE,
);
}
Reference in new issue View Git Blame Copy Permalink