Files
PaperClipAI/server/src/services/native-runtime/native-response-presentation-query.ts
T
DottaandPaperclip bf7f77d8cd fix: avoid response recovery self-contention (#18715)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server must present accepted agent responses as comments under
current chat authority.
> - The recovery worker can check four tasks at the same time.
> - Different tasks can share a chat endpoint or external principal.
Their authorization transactions then compete for endpoint or identity
locks.
> - A failed transaction leaves the accepted response pending and
repeats its source queries on a later retry.
> - This pull request groups checks that share a task, endpoint or
principal before dispatch. It keeps every transactional authority check
and lock.
> - The benefit is less failed recovery work and prompt, exactly-once
presentation of valid accepted responses.

## Linked Issues or Issue Description

Refs #17714 for a separate publication-commit lock-order fix. Refs
#18686 for separate run-list JSON work. Neither changes this recovery
admission path.

**What happened?**

A response-presentation worker reported `could not obtain lock on row in
relation "chat_endpoints"`. Four real PostgreSQL recovery fixtures
reproduced three such failures in one pass with no external ingress. The
worker grouped by task, while authorization locks shared endpoints and
principals. Review also exposed two valid Slack endpoints in one
workspace that share one principal and compete for its try-advisory
lock.

**Expected behavior**

Recovery checks that share an endpoint or principal should retain task
order without competing with one another. Unrelated endpoints should
make progress. The server must still reject stale authority and retry
legitimate external contention.

**Steps to reproduce**

Run the shared-endpoint regression or the portable comparison helper
linked below. The fixture creates four accepted historical responses for
distinct tasks and principals on one actual endpoint.

## What Changed

- Read optional endpoint and principal scheduling hints through a
company-scoped join to the canonical wake action. The join does not
change eligibility or grant authority.
- Extend existing bounded check groups to merge intersecting resource
keys. Preserve task order, transitive conflicts, independent-group
progress, shutdown, drain and pending scopes.
- Add real PostgreSQL controls for shared endpoints, a principal shared
across two endpoints, an observed blocked transaction with independent
endpoint progress, external locks, exact selected output, and endpoint
revocation despite a forged hint.
- Add a portable helper that materializes the pinned baseline with `git
show`, runs the existing real fixture, retains results, and restores
temporary files.

## Verification

- Final principal-extension head
`57047441b28765ed1cf49b8924efc69f5c03dcb4`: 34 focused scheduler/chat
controls passed on Node 24.22.2 / pnpm 9.15.4. Independent exact-head
source review is green. Full local `pnpm -r typecheck` and `pnpm build`
pass on this head. Fresh Greptile and Paperclip reviews are both 5/5
with no findings. All 59 final-head contexts are complete: 55 success, 3
skipped, and the neutral Paperclip review at 5/5. There are no failures
or pending checks.
- Earlier endpoint-only head `eea437e7edbe7eb41eb880d4d8fd6d99a9c247da`:
full local typecheck/build and all remote CI passed. The redundant local
full suite was interrupted for this review fix after 956 passing chat
cases; it is not a full local pass.
- Final shared-principal counts-only comparison completed the same two
exact responses: product calls 165 → 147, rows 114 → 100, outer
contention failures 1 → 0. SQL query errors stayed zero; advisory-lock
refusal is not a failed SQL query.
- The earlier endpoint-only study completed five alternating pairs / ten
measured runs: median calls 443 → 293, rows 314 → 200, query errors 6 →
0 for four exact responses. Its instrumented shared-host CPU data
predates the principal extension and is not a final-head or hosted CPU
claim. Retry clocks were accelerated; wall time is not hosted latency.
- [Raw samples, final principal work counts, portable commands and
failure
accounting](https://github.com/paperclipai/paperclip/blob/57047441b28765ed1cf49b8924efc69f5c03dcb4/doc/performance/2026-11-03-response-presentation-contention.md).
- Initial server4 CI reported three provider calls instead of two in the
unchanged Anthropic identity fixture. The exact local case passed; the
full 88-case file passed with observations showing the two expected
identity-route calls. The third CI call's origin remains unknown. One
bounded rerun of the failed job and its dependent CI aggregate passed.
No assertion or source change was made for that failure.
- The causal benchmark baseline worker/query is
`82c033ed127d07862e6765aa4b69a3cf339bde81`. The PR base is `36df8933f0`.
They are not claims about the currently serving revision.

## Risks

- Connected resource groups can reduce parallelism when tasks link
several endpoints. This preserves the existing task ordering contract.
- Missing or stale endpoint/principal hints retain current transactional
checks. Other workers and ingress can still cause legitimate NOWAIT
retries.
- There is no schema, API, permission or provider-transport change.
Endpoint revocation and company isolation remain enforced by their
existing owners.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), with tool use and code execution,
produced and reviewed this change. The runtime does not expose the
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 06:04:25 -06:00

96 lines
8.1 KiB
TypeScript

import { QueryBuilder } from "drizzle-orm/pg-core";
import { and, asc, eq, exists, gt, inArray, isNotNull, isNull, notExists, or, sql } from "drizzle-orm";
import { chatActions, heartbeatRunEvents, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, statusDecisions, statusDecisionEffects } from "@paperclipai/db";
import type { NativeResponsePresentationScope } from "../native-response-presentation-notifications.js";
export function nativeResponsePresentationCandidates(scope?: NativeResponsePresentationScope, after?: string, runId?: string) {
const db = new QueryBuilder();
return db.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId,
issueId: nativeRunFinalizations.issueId, agentId: heartbeatRuns.agentId, nextAttemptAt: nativeRunFinalizations.nextAttemptAt,
endpointId: chatActions.endpointId, principalId: chatActions.principalId,
}).from(nativeRunFinalizations)
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunFinalizations.runId),
eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId), eq(heartbeatRuns.nativeIssueId, nativeRunFinalizations.issueId)))
.leftJoin(chatActions, and(eq(chatActions.id, heartbeatRuns.wakeupRequestId),
eq(chatActions.companyId, heartbeatRuns.companyId)))
.innerJoin(issues, and(eq(issues.id, nativeRunFinalizations.issueId), eq(issues.companyId, nativeRunFinalizations.companyId)))
.innerJoin(nativeRunResults, and(eq(nativeRunResults.id, nativeRunFinalizations.resultId),
eq(nativeRunResults.companyId, nativeRunFinalizations.companyId), eq(nativeRunResults.issueId, nativeRunFinalizations.issueId),
eq(nativeRunResults.runId, heartbeatRuns.id), eq(nativeRunResults.completionContractId, heartbeatRuns.completionContractId)))
.innerJoin(statusDecisions, and(eq(statusDecisions.id, nativeRunFinalizations.decisionId),
eq(statusDecisions.companyId, nativeRunFinalizations.companyId), eq(statusDecisions.issueId, nativeRunFinalizations.issueId),
eq(statusDecisions.runId, heartbeatRuns.id), eq(statusDecisions.assessmentId, nativeRunFinalizations.assessmentId)))
.where(and(
eq(heartbeatRuns.runtimeMode, "native"), inArray(heartbeatRuns.status, ["succeeded", "failed", "timed_out"]),
isNotNull(heartbeatRuns.finishedAt), eq(nativeRunFinalizations.phase, "committed"), isNull(nativeRunFinalizations.leaseOwner),
eq(nativeRunResults.schemaStatus, "accepted"),
sql`jsonb_typeof(${heartbeatRuns.resultJson} #> '{presentationDecision,commentId}') is distinct from 'string'`,
sql`coalesce(${heartbeatRuns.resultJson} #> '{presentationDecision,commentAction}', '"create"'::jsonb) = '"create"'::jsonb`,
sql`${heartbeatRuns.contextSnapshot}->'skipIssueComment' is distinct from 'true'::jsonb`,
sql`coalesce(${statusDecisions.decisionJson}->'externalChatReviewPresentation', 'null'::jsonb) in ('null'::jsonb, 'false'::jsonb, '0'::jsonb, '""'::jsonb)`,
sql`${nativeRunResults.resultJson} #>> '{result,schema}' = 'paperclip.run_result.v1'`,
sql`${nativeRunResults.resultJson} #>> '{terminal,runTerminalState}' = 'succeeded'`,
sql`${nativeRunResults.resultJson} #>> '{terminal,turnTerminalState}' = 'completed'`,
or(
and(eq(heartbeatRuns.status, "succeeded"),
or(sql`(coalesce(${issues.conversationAgentId}::text, '') = '' or coalesce(${issues.conversationUserId}, '') = '')`,
and(eq(statusDecisions.reasonCode, "conversation_turn_finished"),
eq(issues.conversationAgentId, heartbeatRuns.agentId), eq(issues.conversationUserId, heartbeatRuns.responsibleUserId),
eq(issues.status, "in_progress"), eq(issues.conversationState, "active"),
or(isNull(issues.executionRunId), eq(issues.executionRunId, heartbeatRuns.id)),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issues.id}::text`,
sql`jsonb_typeof(${heartbeatRuns.contextSnapshot}->'conversationSessionGeneration') = 'number'`,
sql`${heartbeatRuns.contextSnapshot}->>'conversationSessionGeneration' = ${issues.conversationSessionGeneration}::text`,
exists(db.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents).where(and(
eq(heartbeatRunEvents.companyId, heartbeatRuns.companyId), eq(heartbeatRunEvents.runId, heartbeatRuns.id),
eq(heartbeatRunEvents.agentId, heartbeatRuns.agentId), eq(heartbeatRunEvents.eventType, "item.completed"),
sql`${heartbeatRunEvents.payload} #>> '{prpEvent,runId}' = ${heartbeatRuns.id}::text`,
sql`${heartbeatRunEvents.payload} #>> '{prpEvent,normalizedSessionId}' = ${heartbeatRuns.nativeSessionId}::text`,
sql`${heartbeatRunEvents.payload} #>> '{prpEvent,turnId}' = ${nativeRunResults.turnId}`,
sql`${heartbeatRunEvents.payload} #>> '{prpEvent,payload,kind}' = 'agentMessage'`,
sql`${heartbeatRunEvents.payload} #>> '{prpEvent,payload,channel}' = 'final'`,
sql`${heartbeatRunEvents.payload} #>> '{prpEvent,payload,text}' ~ '[^[:space:]]'`,
))))),
or(
and(sql`${nativeRunResults.resultJson} #>> '{result,reportedWorkDisposition}' = 'done'`,
sql`${nativeRunResults.resultJson} #>> '{terminal,reportedWorkDisposition}' = 'done'`),
and(sql`(coalesce(${issues.conversationAgentId}::text, '') = '' or coalesce(${issues.conversationUserId}, '') = '')`,
sql`${nativeRunResults.resultJson} #>> '{result,reportedWorkDisposition}' = 'needs_review'`,
sql`${nativeRunResults.resultJson} #>> '{terminal,reportedWorkDisposition}' = 'needs_review'`),
)),
and(sql`${nativeRunResults.resultJson} #>> '{result,reportedWorkDisposition}' = 'yielded'`,
sql`${nativeRunResults.resultJson} #>> '{terminal,reportedWorkDisposition}' = 'yielded'`,
sql`jsonb_typeof(${nativeRunResults.resultJson} #> '{result,summary}') = 'string'
and ${nativeRunResults.resultJson} #>> '{result,summary}' ~ '[^[:space:]]'`,
sql`jsonb_typeof(${nativeRunResults.resultJson} #> '{result,continuation,idempotencyKey}') = 'string'
and ${nativeRunResults.resultJson} #>> '{result,continuation,idempotencyKey}' ~ '[^[:space:]]'`,
or(
and(sql`${nativeRunResults.resultJson} #>> '{result,continuation,kind}' = 'response_wake'`,
sql`${nativeRunResults.resultJson} #> '{result,attentionRequests}' = '[]'::jsonb`,
sql`jsonb_typeof(${nativeRunResults.resultJson} #> '{result,continuation,summary}') = 'string'
and ${nativeRunResults.resultJson} #>> '{result,continuation,summary}' ~ '[^[:space:]]'`),
and(eq(heartbeatRuns.status, "succeeded"), or(
and(sql`${nativeRunResults.resultJson} #>> '{result,continuation,kind}' = 'same_agent'`,
eq(statusDecisions.reasonCode, "live_continuation_registered")),
and(sql`${nativeRunResults.resultJson} #>> '{result,continuation,kind}' = 'monitor'`,
inArray(statusDecisions.reasonCode, ["scheduled_monitor_waiting", "scheduled_monitor_dispatched", "scheduled_monitor_preserved"])),
)),
)),
),
notExists(db.select({ id: statusDecisionEffects.id }).from(statusDecisionEffects).where(and(
eq(statusDecisionEffects.companyId, nativeRunFinalizations.companyId),
eq(statusDecisionEffects.issueId, nativeRunFinalizations.issueId),
eq(statusDecisionEffects.decisionId, nativeRunFinalizations.decisionId),
inArray(statusDecisionEffects.deliveryState, ["pending", "failed"]),
))),
scope ? and(eq(heartbeatRuns.companyId, scope.companyId), scope.issueId ? eq(nativeRunFinalizations.issueId, scope.issueId) : undefined) : undefined,
scope?.runId ? eq(heartbeatRuns.id, scope.runId) : undefined,
scope?.agentId ? eq(heartbeatRuns.agentId, scope.agentId) : undefined,
after ? gt(heartbeatRuns.id, after) : undefined, runId ? eq(heartbeatRuns.id, runId) : undefined,
)).orderBy(asc(heartbeatRuns.id)).limit(100);
}
export function nativeResponsePresentationPendingQuery() {
return nativeResponsePresentationCandidates();
}