perf(chat): project reply lineage run context (#17932)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A run can continue a task that began in an external chat.
> - The reply router follows durable lineage to find the authorized
conversation.
> - Each lineage hop reads the full run context, although it uses only
nine keys.
> - Wide prompts and workspace data make that read transfer far more
data than the router needs.
> - This pull request selects those keys inside PostgreSQL and preserves
their exact JSON values.
> - The benefit is less database transfer with the same reply
destination and authorization checks.

## Linked Issues or Issue Description

Refs: #17255, #17894. The first owns related chat review delivery. The
second is an independent result-projection optimization. This PR has no
dependency on another open PR. A public issue and PR search found no
duplicate change. The roadmap does not list this query optimization as
planned core work.

**What existing behavior does this improve?**

The server resolves the conversation that may receive a reply after one
or more task continuations.

**Current behavior**

Each hop reads agent identity, responsible user identity, and the
complete context snapshot. The snapshot can contain large prompts and
workspace data that the router never reads.

**Proposed behavior**

Select only the nine context keys that this function consumes. Preserve
missing keys, explicit JSON nulls, arrays, objects, and scalar values.
Keep non-object contexts unchanged. Keep all durable authorization
queries and the cycle-safe traversal.

**Reason and benefit**

A synthetic wide context needs 262,402 DataRow bytes before the change
and 244 after it. A valid 24-continuation lineage needs 6,560,338 bytes
before and 6,388 after it. The reduction occurs before PostgreSQL sends
the result.

**Breaking changes**

None. The public API, stored context, reply bindings, tenant checks, and
admission proofs retain their existing behavior.

## What Changed

- Project an object context through `jsonb_each` and `jsonb_object_agg`.
Retain only the nine consumed keys. Preserve all selected JSON values
and key presence.
- Keep the agent and responsible-user columns. Keep the company and run
predicates.
- Add 22 real PostgreSQL tests for routing, malformed JSON, strict
tool-action receipt comparisons, tenant boundaries, 24 valid
continuation hops, and an authorized cycle.
- Add a synthetic benchmark that loads the pinned baseline service,
captures its generated SQL, compares complete bindings and selected
columns, and measures text-format PostgreSQL DataRow frames.
- Run owned resource cleanup in nested `finally` blocks so fixture
deletion or client close failures cannot skip cluster and scratch
cleanup. Preserve the original benchmark failure alongside a cleanup
failure.

The complete consumer-key audit is:

| Context keys | Consumer |
| --- | --- |
| `issueId`, `taskId` | Task identity fence, with the existing fallback
order |
| `wakeCommentIds`, `wakeCommentId`, `commentId` | Board reply and root
comment provenance |
| `source`, `sourceRunId`, `interactionId` | Durable continuation
branches and tool-action receipt checks |
| `endpointId` | Root destination fence |

Missing `interactionId` stays absent. It does not become JSON null. This
matters because the tool-action branch compares it strictly with the
durable wake payload. The tests cover absent, null, string, number, and
object values. They also remove the delivery receipt to prove that
context alone cannot authorize a reply.

## Verification

Baseline: `3cd3af5d75f200185ed4d9703a3145594a489b13`. Treatment:
`3439ba7b714090d9f251816efec26622b3df223b`. PostgreSQL 17.11, Node
26.4.0. All data is synthetic. The benchmark holds imported helpers and
fixtures identical and loads only the baseline `issues.ts` from history.
It records generated SQL, parameters, plans, source SHA-256, and the
initial worktree state before scratch creation. The committed final
measurement reports a clean worktree and source SHA-256
`cd7e955179118fda12f6b2e981a17efff54137cbf241019785a3f8d16a7dcbb8`.

Reproduce the benchmark:

```sh
node --import ./server/node_modules/tsx/dist/loader.mjs scripts/benchmark-chat-reply-lineage.ts --baseline-ref=3cd3af5d75f200185ed4d9703a3145594a489b13
```

The default creates disposable embedded PostgreSQL.
`PAPERCLIP_CHAT_LINEAGE_BENCHMARK_DATABASE_URL` can name a dedicated
external database that ends in `_test`. The harness applies migrations
and removes its own fixture rows. It never uses an application database
by default.

Each wide run has 192 KiB of random bytes encoded as base64. This
produces a 262,144-byte incompressible body. Two warmups precede seven
timed samples. Baseline and treatment alternate order. Every scenario
returns the same complete bindings. Every retained SQL column matches
its baseline value, and the context matches the exact key projection.
The acceptance check requires removal of every unrelated synthetic body
byte.

| Scenario | Baseline DataRow bytes | Changed DataRow bytes | Baseline
SQL median ms (range) | Changed SQL median ms (range) |
| --- | ---: | ---: | ---: | ---: |
| small | 282 | 244 | 0.927 (0.900–1.455) | 1.027 (0.801–1.063) |
| wide | 262,402 | 244 | 2.618 (2.440–4.831) | 1.490 (1.400–1.815) |
| wide-24-continuations | 6,560,338 | 6,388 | 87.450 (80.103–130.197) |
43.103 (38.549–52.864) |

These bytes are exact text-format DataRow frame sizes. They exclude row
descriptions, command metadata, TLS, and TCP. Timings cover the captured
lineage SQL, including all hops. They exclude the other authorization
queries. The small control is neutral and timing ranges overlap on wider
cases. An earlier loaded run had a small-control median of 1.15 ms
before and 1.33 ms after. No deployed latency or billing improvement is
claimed. The byte reduction is the primary result.

Local checks:

- `pnpm run preflight:workspace-links`: passed.
- `pnpm exec vitest run
server/src/services/chat-reply-lineage-context.test.ts`: 22 passed on
the committed rebased head.
- The existing `chat-interaction-continuation.test.ts` file: all 28
tests passed. These cover confirmation, verdict, connection, and
invalid-authority paths.
- Five selected cases in `chat-channels.integration.test.ts`: passed in
a fresh dedicated database. They cover closed-choice question delivery
and steering, Telegram confirmation, ambiguous close control, and both
fresh-after-close authorization paths.
- `pnpm -r typecheck` and `pnpm build`: passed again after the master
rebase.
- `node scripts/check-module-boundaries.mjs` and `git diff --check`:
passed.
- Cleanup repair at `3439ba7b714090d9f251816efec26622b3df223b`: all 22
focused PostgreSQL tests, the benchmark above, and server `tsc --noEmit`
passed. Six controlled cleanup-failure cases confirmed later finalizers
run; benchmark and cleanup errors remain visible together.

The unchanged baseline failed the wide-result regression: 262,383 value
bytes exceeded its 1 KiB bound. The treatment passes. Initial fixture
setup needed a required conversation label and explicit foreign-key
cleanup. The first full typecheck found a row-type mismatch in the new
capture helper; that helper was corrected. The complete typecheck then
passed. No production logic changed for those setup repairs.

A prior assembled run reused the interaction test database. Its retained
running fixtures caused a global milestone-count assertion to fail.
Later reuse exposed retained bot registration and identity state. An
attempted manual reset named a nonexistent table and did not complete.
These failed attempts are not acceptance evidence. The final five-case
assembled proof used a newly created and migrated database and passed
every assertion. No existing assertion was changed.

The full local `pnpm test:run` was not run. Focused tests used the
coordinated disposable PostgreSQL server to avoid uncontrolled
embedded-cluster concurrency on this host. The full final-head CI matrix
passes on `3439ba7b714090d9f251816efec26622b3df223b`: 55 successful
checks, one neutral check, and two optional Storybook skips. Greptile
completed at 5/5 on that exact head, with zero unresolved findings. The
existing interaction and assembled cases ran before the unrelated master
rebase; the measured service source is unchanged. Focused tests,
benchmark, full typecheck, and build passed on the rebased
production-change head. The current cleanup-only head reran focused
tests, benchmark, and server TypeScript validation; production service
source is unchanged.

## Risks

- Future lineage consumers must add any new context key to the SQL
projection and the correctness oracle.
- Selected values retain their complete JSON content. Non-object
contexts pass through unchanged. Savings depend on unrelated object data
being present.
- PostgreSQL must still read the stored object to make the projection.
The change reduces returned data, rather than removing all server-side
JSON work.
- Question delivery, tool action, connection, monitor, publication,
company, and agent proofs retain their existing owners. No hop limit,
cache, schema change, startup task, or deployment is added.
- Benchmark usage and the result contract are documented here. No public
command or API contract changes.

## Model Used

OpenAI GPT-6 through Codex. The exact runtime model ID and context
window are not exposed by this environment. Capabilities used: source
analysis, code editing, shell tools, real PostgreSQL experiments, and
automated tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

scripts/benchmark-chat-reply-lineage.ts

@@ -0,0 +1,110 @@
+import assert from "node:assert/strict";
+import { execFileSync } from "node:child_process";
+import { createHash } from "node:crypto";
+import { promises as fs, readFileSync } from "node:fs";
+import path from "node:path";
+import { fileURLToPath, pathToFileURL } from "node:url";
+import { applyPendingMigrations, createDb, startEmbeddedPostgresTestDatabase, type Db } from "../packages/db/src/index.js";
+import { resolveChatOriginPublicationBindings } from "../server/src/services/issues.js";
+import { captureLineageReads, chatReplyLineageFixture, cleanupChatReplyLineageFixture, expectedLineageContext, type LineageQuery, type LineageResolver } from "../server/src/__tests__/helpers/chat-reply-lineage.js";
+import { resolveExternalTestDatabaseUrl } from "../server/src/__tests__/helpers/external-test-database.js";
+
+const repoRoot = fileURLToPath(new URL("../", import.meta.url));
+const ref = process.argv.find(arg => arg.startsWith("--baseline-ref="))?.slice(15);
+assert.ok(ref, "Pass a pinned pre-fix --baseline-ref");
+const git = (args: string[]) => execFileSync("git", args, { cwd: repoRoot, encoding: "utf8" }).trim();
+const sourcePath = "server/src/services/issues.ts";
+const baselineRevision = git(["rev-parse", `${ref}^{commit}`]), treatmentRevision = git(["rev-parse", "HEAD"]);
+const uncommittedChanges = git(["status", "--porcelain"]), treatmentDiff = git(["diff", "HEAD", "--", sourcePath]);
+const sourceHash = () => createHash("sha256").update(readFileSync(path.join(repoRoot, sourcePath))).digest("hex");
+const treatmentSourceSha256 = sourceHash();
+const keepAlive = setInterval(() => {}, 1000);
+let db: Db | undefined, temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined, baselineDirectory: string | undefined;
+const companyIds: string[] = [];
+let benchmarkFailure: { error: unknown } | undefined;
+try {
+  const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_CHAT_LINEAGE_BENCHMARK_DATABASE_URL");
+  if (externalUrl) await applyPendingMigrations(externalUrl);
+  else temporary = await startEmbeddedPostgresTestDatabase("chat-lineage-benchmark-");
+  db = createDb(externalUrl ?? temporary!.connectionString);
+  baselineDirectory = await fs.mkdtemp(path.join(repoRoot, "server/scripts/.chat-lineage-baseline-"));
+  const original = pathToFileURL(path.join(repoRoot, sourcePath));
+  const baselineSource = execFileSync("git", ["show", `${baselineRevision}:${sourcePath}`], { cwd: repoRoot, encoding: "utf8" });
+  await fs.writeFile(path.join(baselineDirectory, "issues.ts"), baselineSource.replace(/(from\s+|import\()(["'])(\.{1,2}\/[^"']+)\2/g,
+    (_match, prefix: string, quote: string, specifier: string) => `${prefix}${quote}${fileURLToPath(new URL(specifier, original))}${quote}`));
+  const baseline = (await import(pathToFileURL(path.join(baselineDirectory, "issues.ts")).href)).resolveChatOriginPublicationBindings as LineageResolver;
+  const measurements = [];
+  for (const scenario of [{ name: "small", bytes: 16, hops: 0 }, { name: "wide", bytes: 192 * 1024, hops: 0 }, { name: "wide-24-continuations", bytes: 192 * 1024, hops: 24 }]) {
+    const f = await chatReplyLineageFixture(db, scenario.bytes); companyIds.push(f.companyId);
+    let runId = f.root.id;
+    for (let hop = 0; hop < scenario.hops; hop++) runId = (await f.continueFrom(runId)).run.id;
+    await db.$client.unsafe("ANALYZE");
+    async function capture(resolver: LineageResolver) {
+      const reads: LineageQuery[] = [];
+      const result = await resolver(captureLineageReads(db!, reads), f.companyId, f.issueId, runId);
+      assert.deepEqual(result, f.expected);
+      assert.equal(reads.length, scenario.hops + 1);
+      return { result, reads };
+    }
+    const before = await capture(baseline), after = await capture(resolveChatOriginPublicationBindings);
+    assert.deepEqual(after.result, before.result);
+    for (let index = 0; index < before.reads.length; index++) {
+      const row = before.reads[index].rows[0];
+      assert.deepEqual(after.reads[index].rows, [{ ...row, contextSnapshot: expectedLineageContext(row.contextSnapshot) }]);
+    }
+    const variants = [before, after], samples: number[][] = [[], []], bytes: number[] = [0, 0];
+    for (let iteration = 0; iteration < 9; iteration++) {
+      for (const variant of iteration % 2 ? [1, 0] : [0, 1]) {
+        const started = performance.now(); let returned = 0;
+        for (const query of variants[variant].reads) {
+          const rows = await db.$client.unsafe(query.sql, query.params as never[]).raw();
+          returned += rows.reduce((sum, row) => sum + 7 + row.reduce((n: number, value: Buffer | null) => n + 4 + (value?.length ?? 0), 0), 0);
+        }
+        if (iteration >= 2) samples[variant].push(performance.now() - started);
+        bytes[variant] = returned;
+      }
+    }
+    const minimumSaving = Buffer.byteLength(f.context.prompt) * (scenario.hops + 1);
+    assert.ok(bytes[0] - bytes[1] >= minimumSaving, "All unrelated synthetic body bytes must be removed before transfer");
+    const plans = [];
+    for (const variant of variants) {
+      const query = variant.reads[0];
+      const [plan] = await db.$client.unsafe(`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) ${query.sql}`, query.params as never[]);
+      plans.push(plan["QUERY PLAN"]);
+    }
+    const timing = (values: number[]) => { const sorted = [...values].sort((a, b) => a - b); return { medianMs: sorted[3], minMs: sorted[0], maxMs: sorted[6], samplesMs: values }; };
+    measurements.push({ scenario, bodyBytesPerRun: Buffer.byteLength(f.context.prompt), completeBindingsEqual: true, selectedColumnsEqual: true,
+      baselineDataRowBytes: bytes[0], treatmentDataRowBytes: bytes[1], minimumSaving, baselineTiming: timing(samples[0]), treatmentTiming: timing(samples[1]),
+      baselineQueries: before.reads.map(({ rows: _rows, ...query }) => query), treatmentQueries: after.reads.map(({ rows: _rows, ...query }) => query), plans });
+  }
+  assert.equal(sourceHash(), treatmentSourceSha256, "Measured service source changed during the benchmark");
+  console.log(JSON.stringify({ baselineRevision, treatmentRevision, sourcePath, treatmentSourceSha256, uncommittedChanges, treatmentDiff,
+    byteBoundary: "Exact text-format PostgreSQL DataRow frames; excludes row descriptions, command metadata, TLS and TCP. Timings cover captured lineage SQL only, not the complete authorization service.",
+    dependencyScope: "Only the pinned baseline issues.ts is loaded from history; imported helpers and synthetic fixtures are held identical.",
+    postgres: await db.$client.unsafe("select version()"), settings: await db.$client.unsafe("select name,setting,unit from pg_settings where name in ('shared_buffers','work_mem','jit') order by name"),
+    node: process.version, measurements }, null, 2));
+} catch (error) {
+  benchmarkFailure = { error };
+  throw error;
+} finally {
+  try {
+    try {
+      if (db) for (const companyId of companyIds) await cleanupChatReplyLineageFixture(db, companyId);
+    } finally {
+      try {
+        await db?.$client.end({ timeout: 0 });
+      } finally {
+        try {
+          await temporary?.cleanup();
+        } finally {
+          try {
+            if (baselineDirectory) await fs.rm(baselineDirectory, { recursive: true, force: true });
+          } finally { clearInterval(keepAlive); }
+        }
+      }
+    }
+  } catch (error) {
+    if (benchmarkFailure) throw new AggregateError([benchmarkFailure.error, error], "Benchmark and cleanup failed");
+    throw error;
+  }
+}