mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 09:50:48 +01:00
fix(github): end a superseded automatic review with a clear outcome (#22218)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for work
> - The GitHub bot can do an automatic pull request review. An agent run does the review with review tools.
> - Each automatic run has authority for one commit only: the head commit of its pull request event.
> - When a new push moves the head during the review, the review tools refuse the run. This is correct.
> - But the refusal told the agent to "continue with the newest authorized PR event". The agent cannot do that. Agents call the refused tools again, then post the stale review as an ordinary comment.
> - This pull request makes the refusal a clear, terminal "superseded" outcome. It says if a newer review is already queued and tells the agent to stop.
> - The benefit is fewer wasted tool calls and no stale review comments on the pull request. Commit-specific authority does not change.
## Linked Issues or Issue Description
**What happened?**
An automatic pull request review starts for commit A. During the review, the author pushes commit B. Each review tool call (for example `read_pull_request`) then fails with "This automatic review was requested for an older pull request head. Continue with the newest authorized PR event or an authorized manual request." Agents call the tools again several times. Then they post their review as an ordinary pull request comment, because the comment tool does not check the head.
**What did you expect?**
The refusal tells the agent that the review is superseded. It names the old and new commits. It says if a newer automatic review for the new head is already queued on the task. It tells the agent to stop, not to call review tools again, and not to post the review as a comment. The error has a machine-readable reason code and is not retryable.
**Steps to reproduce**
1. Configure the GitHub bot for automatic reviews.
2. Open a pull request. An automatic review run starts.
3. Push a new commit while the run reviews.
4. The run calls a review tool. The tool refuses with the old, unclear message.
## What Changed
- `server/src/services/chat-github-reviews.ts`: the stale-head check now builds the error in a new `supersededAutomaticReview` helper. The helper looks for a newer automatic delivery for the current head of the same pull request. It matches the repository id and pull number that the event saves, because a new delivery has no conversation id until it is processed. It counts only deliveries that are newer than the run's own event, in the states `received`, `processing`, `processed`, or `retry`. The query stays scoped to the company and endpoint. The 409 error message names both commits, says if a newer review is queued, and tells the agent to stop. The error details carry `reasonCode: "github_review_superseded"`, `retryable: false`, `requestedCommit`, `currentCommit`, and `newerReviewQueued`.
- The first sentence of the message does not change, so existing logs and searches still match.
- The run still cannot read or review the newer head. Authorization does not change.
- `server/src/__tests__/chat-channels.integration.test.ts`: the existing stale-head assertion now checks three outcomes: a newer review is queued and processed, a newer review is still waiting with no conversation id, and no newer review is queued (the head moved with no event).
## Verification
- Failing before: `cd server && npx vitest run src/__tests__/chat-channels.integration.test.ts -t "admits signed PR events into an ordinary assigned task"` with only the test change. The test failed: `expected Error: This automatic review was requeste… to match object { status: 409, details: { reasonCode: "github_review_superseded", … } }`, and the received error had `details: undefined`.
- Passing after: the same command with the fix: `Tests 1 passed`.
- Failing before (second case): the test sends a second `synchronize` event and does not process it, so its delivery has no conversation id. Before the identity match, the test failed: expected `newerReviewQueued: true`, received `newerReviewQueued: false`. After the fix, the test passes.
- `npx vitest run src/__tests__/chat-channels.integration.test.ts -t "GitHub agent review workflow"`: 300 passed.
- `pnpm --filter @paperclipai/server typecheck` (layered typecheck): passed.
- `pnpm verify` selected 2,925 of 3,792 test files because many modules import this service. I stopped it before the end. It reported no failures before I stopped it. Required CI gives the full coverage.
## Risks
- Low risk. The change applies only to the error path when an automatic review's head is stale.
- The error path does one more indexed-scope query on `chat_deliveries` (endpoint and conversation). This query runs only when a refusal occurs.
- The message text changes after its first sentence. Code that matches the full old text would need an update. No code in this repository does.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), by Anthropic, in Claude Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context to this change
- [x] I have specified the model used (with version and capability details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before requesting merge
Related: #21290 (completes the check run of a superseded head; a different code path).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
575ef12575
commit
3b978692ee
2 files changed
+73
-10
No files matched your search
@@ -7546,12 +7546,37 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
f.setSupplementalProviderFetch(async (input) => String(input).endsWith("/pulls/81")
|
||||
? Response.json({ ...payload.pull_request, head: { sha: "c".repeat(40) } })
|
||||
: undefined);
|
||||
await expect(githubChatReviewService(db, f.providerFetch).execute({
|
||||
companyId: f.companyId,
|
||||
agentId: f.assignedAgentId,
|
||||
issueId: task.id,
|
||||
runId: run.id,
|
||||
}, "read_pull_request", { section: "metadata" })).rejects.toThrow("older pull request head");
|
||||
const staleSession = { companyId: f.companyId, agentId: f.assignedAgentId, issueId: task.id, runId: run.id };
|
||||
const superseded = githubChatReviewService(db, f.providerFetch).execute(staleSession, "read_pull_request", { section: "metadata" });
|
||||
await expect(superseded).rejects.toMatchObject({
|
||||
status: 409,
|
||||
details: { reasonCode: "github_review_superseded", retryable: false, requestedCommit: "b".repeat(40),
|
||||
currentCommit: "c".repeat(40), newerReviewQueued: true },
|
||||
});
|
||||
await expect(superseded).rejects.toThrow(/older pull request head.*newer automatic review for that commit is already queued.*Stop this review/s);
|
||||
await f.service.handleWebhook(f.endpoint.publicId, "github", signedGitHubWebhookRequest({
|
||||
event: "pull_request", delivery: randomUUID(), webhookSecret: f.webhookSecret,
|
||||
payload: { ...payload, action: "synchronize", before: "c".repeat(40),
|
||||
pull_request: { ...payload.pull_request, head: { sha: "d".repeat(40) } } },
|
||||
}));
|
||||
const waiting = (await readChatDeliveriesByEndpointId(f.endpoint.id)).filter((row) =>
|
||||
(row.normalizedEvent.githubAutomatic as { context?: { headSha?: string } } | undefined)?.context?.headSha === "d".repeat(40));
|
||||
expect(waiting).toHaveLength(1);
|
||||
expect(waiting[0]).toMatchObject({ conversationId: null });
|
||||
f.setSupplementalProviderFetch(async (input) => String(input).endsWith("/pulls/81")
|
||||
? Response.json({ ...payload.pull_request, head: { sha: "d".repeat(40) } })
|
||||
: undefined);
|
||||
await expect(githubChatReviewService(db, f.providerFetch).execute(staleSession, "read_pull_request", { section: "metadata" }))
|
||||
.rejects.toMatchObject({ status: 409, details: { reasonCode: "github_review_superseded", currentCommit: "d".repeat(40), newerReviewQueued: true } });
|
||||
f.setSupplementalProviderFetch(async (input) => String(input).endsWith("/pulls/81")
|
||||
? Response.json({ ...payload.pull_request, head: { sha: "e".repeat(40) } })
|
||||
: undefined);
|
||||
const unqueued = githubChatReviewService(db, f.providerFetch).execute(staleSession, "read_pull_request", { section: "metadata" });
|
||||
await expect(unqueued).rejects.toMatchObject({
|
||||
status: 409,
|
||||
details: { reasonCode: "github_review_superseded", currentCommit: "e".repeat(40), newerReviewQueued: false },
|
||||
});
|
||||
await expect(unqueued).rejects.toThrow(/no newer automatic review is queued.*Stop this review/s);
|
||||
});
|
||||
async function reviewCheckWorkerFixture(status: "running" | "failed" = "failed") {
|
||||
const f = await toolReplyFixture("github:paperclipai/paperclip:418");
|
||||
|
||||
@@ -16,7 +16,7 @@ import { githubReviewSummary } from "./chat-channels/github/review-heading.js";
|
||||
import { githubPublicationRetry, GITHUB_INLINE_RETRY_WINDOW_MS, GITHUB_INLINE_MAX_ATTEMPTS } from "./chat-channels/github/retry.js";
|
||||
import { createHash } from "node:crypto";
|
||||
import { githubResponseCommentService, writeGitHubResponseComment } from "./chat-github-response-comments.js";
|
||||
import { and, desc, eq, gt, isNull, or, sql } from "drizzle-orm";
|
||||
import { and, desc, eq, gt, inArray, isNull, or, sql } from "drizzle-orm";
|
||||
import { z } from "zod";
|
||||
import {
|
||||
toolInvocations,
|
||||
@@ -334,6 +334,46 @@ export function githubChatReviewService(db: Db, fetchImpl = fetch) {
|
||||
github<T>(`${prefix}${path}`, options),
|
||||
};
|
||||
}
|
||||
async function supersededAutomaticReview(
|
||||
source: Awaited<ReturnType<typeof scope>>,
|
||||
requestedCommit: string,
|
||||
currentCommit: string,
|
||||
) {
|
||||
const [newer] = await db
|
||||
.select({ id: chatDeliveries.id })
|
||||
.from(chatDeliveries)
|
||||
.where(
|
||||
and(
|
||||
eq(chatDeliveries.companyId, source.endpoint.companyId),
|
||||
eq(chatDeliveries.endpointId, source.endpoint.id),
|
||||
or(
|
||||
eq(chatDeliveries.conversationId, source.conversation.id),
|
||||
isNull(chatDeliveries.conversationId),
|
||||
),
|
||||
gt(chatDeliveries.createdAt, source.delivery.createdAt),
|
||||
inArray(chatDeliveries.state, ["received", "processing", "processed", "retry"]),
|
||||
sql`${chatDeliveries.normalizedEvent}->'githubAutomatic'->'context'->>'repositoryId' = ${source.repositoryId}`,
|
||||
sql`${chatDeliveries.normalizedEvent}->'githubAutomatic'->'context'->>'pullNumber' = ${String(source.number)}`,
|
||||
sql`${chatDeliveries.normalizedEvent}->'githubAutomatic'->'context'->>'headSha' = ${currentCommit}`,
|
||||
),
|
||||
)
|
||||
.limit(1);
|
||||
const next = newer
|
||||
? "a newer automatic review for that commit is already queued on this task"
|
||||
: "no newer automatic review is queued for it. An authorized person can request a manual review";
|
||||
return conflict(
|
||||
`This automatic review was requested for an older pull request head (${requestedCommit.slice(0, 7)}). ` +
|
||||
`The pull request now points to ${currentCommit.slice(0, 7)}, and ${next}. ` +
|
||||
"Stop this review: do not call review tools again and do not post the review as an ordinary comment.",
|
||||
{
|
||||
reasonCode: "github_review_superseded",
|
||||
retryable: false,
|
||||
requestedCommit,
|
||||
currentCommit,
|
||||
newerReviewQueued: !!newer,
|
||||
},
|
||||
);
|
||||
}
|
||||
async function reviewForHead(
|
||||
source: Awaited<ReturnType<typeof scope>>,
|
||||
pull: Pull,
|
||||
@@ -633,9 +673,7 @@ export function githubChatReviewService(db: Db, fetchImpl = fetch) {
|
||||
const automatic = source.delivery.normalizedEvent.githubAutomatic as
|
||||
{ context: GitHubReviewEventContext } | undefined;
|
||||
if (automatic && automatic.context.headSha !== pull.head.sha)
|
||||
throw conflict(
|
||||
"This automatic review was requested for an older pull request head. Continue with the newest authorized PR event or an authorized manual request.",
|
||||
);
|
||||
throw await supersededAutomaticReview(source, automatic.context.headSha, pull.head.sha);
|
||||
if (name === "read_pull_request") {
|
||||
const parsed = readSchema.parse(input);
|
||||
if (parsed.section === "metadata") {
|
||||
|
||||
Reference in new issue
Block a user