mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-29 12:59:13 +01:00
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/213/changes/paperclip/213 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - GitHub review checks report the state of admitted agent work. > - The chat maintenance loop searches for check work even when none remains. > - Existing action rows already record unfinished publication and retry deadlines. > - Committed changes can wake those rows through the shared coordinator. > - An unresolved check must still detect a remote head change when a webhook is missed. > - This pull request schedules only outstanding checks and leaves settled history quiet. ## Linked Issues or Issue Description **What happened?** The GitHub chat maintenance callback repeatedly queries review-check actions. Empty queues and completed history still incur database reads. Local state changes wait for the maintenance callback. **Expected behavior** Committed check creation, review binding, run changes, and publication-lease release wake the worker. Existing retry and remote-head deadlines apply only to unfinished checks. A settled worker makes no recurring database or provider calls. **Steps to reproduce** Configure a GitHub bot with no outstanding checks, or finish all of its checks. Leave the instance idle. The old maintenance callback continues to call the check service. Refs #17661, the merged terminal-response parent. Related #17656 migrates response acknowledgements. Related #16385 changes branch-policy admission; this PR preserves that separate policy boundary. Late stale-head projection repair remains a separate follow-up slice. ## What Changed - Move the existing check-state reader into the GitHub provider owner. Preserve current authority, run selection, assessment ownership, and check replacement behavior. - Register a scoped worker with the existing coordinator and scheduler. Existing received and retryable-failed actions remain durable truth. No new table is added. - Signal check enqueue, task review receipts, manual review insertion/binding, run lifecycle changes, and publication-lease release through existing transaction ownership. - Preserve the saved failure backoff, eight-attempt policy, and competing lease expiry. Keep five-second remote-head reconciliation only while a received check remains unresolved. - Check admission before each HTTP request. Shutdown leaves durable work recoverable without spending a provider failure attempt. - Remove check dispatch from the periodic GitHub callback. Keep the separate late-projection repair and the manual combined recovery method compatible. - Add an idempotent concurrent partial index for unfinished check selection. Retain the newest five generated snapshots. - Add eleven real-database worker cases and startup wiring coverage. Document producer coverage and the external reconciliation limit. ## Verification Current reviewed source: `2bba08403b195bf646eda48992b46d6f5cebf756`. - Normal merges preserve the complete review-check patch, the landed terminal-response parent, and current master changes. No landed SQL migration or journal entry changes. - At `6995f84b240a14a7b8660e1cf095b3aa1e1e3168`, **331 distinct tests pass**: 264 complete GitHub workflow cases, 30 completion cases, eight issue-tree producer cases, 16 login-cleanup cases, and 13 startup cases. The GitHub command filters out 1,485 unrelated cases. Login cleanup uses its real embedded PostgreSQL fixture. - Combined-master acceptance: **277 distinct tests pass** at `9c538aed1251fd7f7cecb829a0ecbce6110fb1ed`: 264 complete GitHub workflow cases and 13 startup cases. The GitHub command filters out 1,485 unrelated cases. A new PostgreSQL database receives all migrations through 0402.. - Migration `0402_ancient_malice.sql` preserves the original partial index definition and concurrent idempotent creation. All 400 prior migration receipts stay unchanged during upgrade. One new receipt is added. Replay adds no receipt. The index is ready and valid on fresh and upgraded PostgreSQL databases. The five newest snapshots are retained. - A rollback-only query probe adds 10,000 processed actions and one pending action. The actual parameterized candidate predicate selects the one pending row through the partial index, with two shared buffers and 0.007 ms execution time. This measures candidate discovery, not all lease or provenance reads. - Database build, server typecheck and build, module boundaries, migration safety and migration order pass. Audit: [GitHub review-check scheduling](https://github.com/paperclipai/paperclip/blob/2bba08403b195bf646eda48992b46d6f5cebf756/doc/plans/2026-10-25-github-review-check-sleep-audit.md). - Fresh checks and Greptile review are required for this head. Prior scores do not count. These checks do not prove deployed host suspension or fleet savings. ## Risks - Unresolved queued/running checks still make remote-head requests at the existing five-second deadline. GitHub webhook receipt does not prove future pushes cannot be missed. This PR does not claim active reviews become network-idle. - Run and configuration hints rebuild outstanding checks for the affected company. Enqueue, review, and lease hints narrow to the affected endpoint/repository/PR. Startup and scope reconstruction read existing pending rows; they are not constant-cost operations. - Delivery admission failures remain discoverable through the unresolved check's external reconciliation deadline. Not every shared delivery mutation gains a new immediate signal. - Notifications use the existing process-local database ownership contract. Independent SQL writers and other replicas need their own supported integration. - Provider markers, publication leases, and saved actions preserve recovery after uncertainty. This change does not expand Cloud alarm coverage or prove deployed host/database suspension. - Each remaining sleep slice needs its own review and CI checks before merge. ## Model Used OpenAI Codex (GPT-6), with reasoning and code execution. The exact serving model ID and context-window size were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
126 lines
4.2 KiB
TypeScript
126 lines
4.2 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { and, eq, gt, lte, sql } from "drizzle-orm";
|
|
import { chatEndpointLeases, type Db } from "@paperclipai/db";
|
|
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
|
|
|
type Transaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
|
const TTL_SECONDS = 90;
|
|
|
|
export class GitHubPublicationLeaseLost extends Error {
|
|
constructor() {
|
|
super("GitHub publication ownership was lost");
|
|
}
|
|
}
|
|
|
|
/** Serialize provider writes for one PR without holding a database connection
|
|
* during network I/O. A crashed worker leaves the existing outbox retryable.
|
|
* The token fences receipt commits; losing renewal aborts in-flight requests.
|
|
* Provider markers still recover a write whose response was lost. */
|
|
export async function withGitHubPublicationLease<T>(
|
|
db: Db,
|
|
scope: {
|
|
companyId: string;
|
|
endpointId: string;
|
|
repositoryId: string;
|
|
number: number;
|
|
},
|
|
fetchImpl: typeof fetch,
|
|
operation: (lease: {
|
|
fetch: typeof fetch;
|
|
commit<R>(write: (tx: Transaction) => Promise<R>): Promise<R>;
|
|
}) => Promise<T>,
|
|
): Promise<T | undefined> {
|
|
const token = randomUUID();
|
|
const leaseKey = `github-publication:${scope.repositoryId}:${scope.number}`;
|
|
const expiresAt = sql`now() + interval '90 seconds'`;
|
|
const [claimed] = await db
|
|
.insert(chatEndpointLeases)
|
|
.values({
|
|
companyId: scope.companyId,
|
|
endpointId: scope.endpointId,
|
|
leaseKey,
|
|
token,
|
|
expiresAt,
|
|
})
|
|
.onConflictDoUpdate({
|
|
target: [chatEndpointLeases.endpointId, chatEndpointLeases.leaseKey],
|
|
set: { token, expiresAt, updatedAt: sql`now()` },
|
|
setWhere: and(
|
|
eq(chatEndpointLeases.companyId, scope.companyId),
|
|
lte(chatEndpointLeases.expiresAt, sql`now()`),
|
|
),
|
|
})
|
|
.returning({ id: chatEndpointLeases.id });
|
|
if (!claimed) return;
|
|
|
|
const controller = new AbortController();
|
|
let lost = false;
|
|
const ownership = and(
|
|
eq(chatEndpointLeases.id, claimed.id),
|
|
eq(chatEndpointLeases.companyId, scope.companyId),
|
|
eq(chatEndpointLeases.token, token),
|
|
gt(chatEndpointLeases.expiresAt, sql`now()`),
|
|
);
|
|
const renew = async (writer: Db | Transaction = db) => {
|
|
if (lost) throw new GitHubPublicationLeaseLost();
|
|
try {
|
|
const rows = await writer
|
|
.update(chatEndpointLeases)
|
|
.set({ expiresAt, updatedAt: sql`now()` })
|
|
.where(ownership)
|
|
.returning({ id: chatEndpointLeases.id });
|
|
if (!rows.length) throw new GitHubPublicationLeaseLost();
|
|
} catch {
|
|
lost = true;
|
|
controller.abort();
|
|
throw new GitHubPublicationLeaseLost();
|
|
}
|
|
};
|
|
let renewal: Promise<void> | undefined;
|
|
const timer = setInterval(
|
|
() => {
|
|
if (renewal) return;
|
|
renewal = renew()
|
|
.catch(() => {})
|
|
.finally(() => {
|
|
renewal = undefined;
|
|
});
|
|
},
|
|
(TTL_SECONDS * 1000) / 3,
|
|
);
|
|
timer.unref?.();
|
|
try {
|
|
return await operation({
|
|
fetch: (async (input, init) => {
|
|
await renew();
|
|
const signal =
|
|
init?.signal ?? (input instanceof Request ? input.signal : undefined);
|
|
const response = await fetchImpl(input, {
|
|
...init,
|
|
signal: signal
|
|
? AbortSignal.any([signal, controller.signal])
|
|
: controller.signal,
|
|
});
|
|
await renew();
|
|
return response;
|
|
}) as typeof fetch,
|
|
commit: (write) =>
|
|
db.transaction(async (tx) => {
|
|
// Updating the lease locks it until the short receipt transaction ends.
|
|
await renew(tx);
|
|
return write(tx);
|
|
}),
|
|
});
|
|
} finally {
|
|
clearInterval(timer);
|
|
await renewal;
|
|
await db.transaction(async tx => {
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatGitHubReviewChecks, JSON.stringify(scope));
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatGitHubPublications);
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatGitHubTerminalResponses, JSON.stringify(scope));
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatGitHubAcknowledgements, JSON.stringify(scope));
|
|
await tx.delete(chatEndpointLeases).where(and(eq(chatEndpointLeases.id, claimed.id), eq(chatEndpointLeases.token, token)));
|
|
});
|
|
}
|
|
}
|