mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-02 16:58:52 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connected chats show safe notices when a run starts, waits, finishes, or fails. > - A recurring scan projected these notices even when no eligible work existed. > - Eligibility depends on committed run state, conversation lineage, and current authority. > - Existing database rows can reconstruct that work after restart. > - This PR wakes a scoped worker from those writes and removes lifecycle polling. > - Native progress events keep their separate existing owner until the next migration. ## Linked Issues or Issue Description **What happened?** The chat maintenance timer repeatedly scanned run lifecycle state. Empty and settled instances still queried the database. A run-status event alone cannot replace the scan: prompt receipts, continuation receipts, identity changes, and restored authority can make a run eligible later. **What did you expect?** Committed changes wake projection. Startup reconstructs eligible work. A completed pass stops until another relevant change arrives. Failed work retains recovery without scanning settled history forever. **Steps to reproduce** Configure an automatic chat endpoint. Finish its work, or leave it unused. Observe recurring lifecycle projection queries from the chat maintenance callback. Stacked on #17592. Related foundation: #15625. This PR owns lifecycle milestones only. ## What Changed - Preserve upstream coalesced Cloud membership writes together with locked membership restoration and its committed chat notification. - Signal milestone eligibility from the real Slack account-link transaction. - Register the lifecycle worker with the existing delivery coordinator. Remove lifecycle projection from the periodic chat callback. Keep native progress in a separately named lane and preserve the explicit combined API. - Reconstruct company/issue scopes from the existing selector. Drain bounded batches with company fairness. Retry failures, retain writes through drain/resume, and disarm settled work. - Signal from supported run, presentation, interaction, receipt, lineage, conversation, recovery, and authority writers. Preserve their transaction ownership, including nested savepoints and uncertain settlement. No new table is added. - Preserve provider authorization, authored-final suppression, responsible-user continuity, deduplication, the 24-hour selection window, and raw-output restrictions. Keep provider interpretation in its owner. - Preserve PostgreSQL timestamp precision in lifecycle pagination. Rejected submillisecond candidates must not repeat the same page and starve later notices. - Document producer coverage and alternate-writer dispositions in `doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md`. ## Verification Published head: `1d056f68c5b1005933b9e12de8745bf1c4cbeeca`, targeting master after #17592 merged at `7d75ae0569e41786640cf09647ae8849411d2f22`. A normal merge integrates the actual landed parent. The entire 77-path milestone patch is binary-identical to the locally verified preparation across that ancestry reconciliation; both unrelated master paths are preserved. The four earlier producer/test integration conflicts retain scoped milestone and voice notices, current onboarding completion admission, and expanded identity query/read-scope coverage. - **330 distinct selected cases pass on the verified unchanged patch:** 305 across 14 complete suites (all five milestone worker suites, run identity, board claim, Cloud provisioning, auth session routes, Cloud actor authentication, stale-lock recovery, GitHub credentials, project Git authorization, and app admission wiring), plus 25 selected real chat producer cases. The remaining 1,590 chat integration cases were skipped, not passed. - The real producer cases cover saved milestone discovery, GitHub and Slack identity changes, account-link rollback/reconfirmation, ordinary ingress reactivation, post-close action/delivery receipt settlement, all ten authority-restoration paths including Cloud synchronization, and changed native review metadata. - Worker coverage verifies outer commit, rollback, uncertain-write recovery, restart, bounded batching/fairness, timestamp precision, retries, drain/resume, and settled SQL quiet. Authorization and existing publication deduplication remain with their owning services. - Server production typechecking, build, and module-boundary checks pass. The tests above were executed separately. Production typechecking excludes `src/__tests__` and does not replace those runs. Each database owner uses its own newly created and app-migrated PostgreSQL fixture. - [Current-parent acceptance and preserved failure ledger](https://github.com/paperclipai/paperclip/blob/1d056f68c5b1005933b9e12de8745bf1c4cbeeca/doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md#current-parent-integration-and-acceptance-october-28) records commands, counts, producer scope, conflict dispositions, and limits. Earlier combined-stack counts and failures keep their original revision scope; local acceptance does not erase old CI failures. - No new migration, queue table, dependency, workflow, or shared scheduler contract is introduced. Parent migration 0397, bounded consent recovery, voice cleanup/retention repairs, and runner checkpoint restoration remain present. - **Exact-head Greptile is 5/5 and all 55 current checks are clean.** No Ponytail review is present, which is an explicitly authorized nonblocker; no failing review is waived. Older reviews and parent checks are not current passes. Actual failing Ponytail findings must be fixed; an absent/stopped/paused turn without a verdict is an authorized nonblocker. Active unfinished Ponytail uses its original request/start timer, with a 25-minute monitor cutoff respecting the user's 30-minute maximum. - No complete local repository matrix, live provider tenant, external alarm, physical host/database suspension, or fleet savings claim is made. Native progress remains a separate following slice. ## Risks - A future eligibility writer must notify the existing projection boundary. Independent SQL writers or another active server process are outside this process-local signal contract. - Authority and lineage remain sensitive. The worker uses the existing selectors and publication policy; it does not grant execution or disclosure authority. - Native progress and other publication maintenance still have recurring work. This PR removes one reason, not the entire chat maintenance timer. - Worker quiet does not prove database suspension, host sleep, or external alarm coverage. Cloud sleep blockers remain unchanged. - Merge eligibility requires passing current-head CI and Greptile 5/5 after this actual-master reconciliation. Subsequent stack slices retain their own gates. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window size are not exposed. Used repository inspection, reasoning, shell execution, PostgreSQL tests, browser E2Es and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details available in this session) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run the documented focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
270 lines
11 KiB
TypeScript
270 lines
11 KiB
TypeScript
import { and, asc, eq, sql } from "drizzle-orm";
|
|
import { chatConversations, chatEndpointResources, chatEndpoints, type Db } from "@paperclipai/db";
|
|
import { conflict, notFound } from "../../../errors.js";
|
|
import type { CredentialMutationLeaseGuard } from "../../chat-credential-mutation-lease.js";
|
|
import type { ChatProviderResourceInventoryItem } from "../../chat-provider-inventory.js";
|
|
type EndpointRow = typeof chatEndpoints.$inferSelect;
|
|
type DbTransaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
|
|
|
export function githubRepositoryInventoryItemFromPayload(
|
|
payload: unknown,
|
|
): ChatProviderResourceInventoryItem | null {
|
|
if (!payload || typeof payload !== "object" || Array.isArray(payload))
|
|
return null;
|
|
const repository = (payload as { repository?: unknown }).repository;
|
|
if (
|
|
!repository ||
|
|
typeof repository !== "object" ||
|
|
Array.isArray(repository)
|
|
)
|
|
return null;
|
|
const value = repository as {
|
|
id?: unknown;
|
|
full_name?: unknown;
|
|
html_url?: unknown;
|
|
name?: unknown;
|
|
owner?: { id?: unknown; login?: unknown };
|
|
private?: unknown;
|
|
};
|
|
const repositoryId =
|
|
typeof value.id === "number" && Number.isSafeInteger(value.id)
|
|
? String(value.id)
|
|
: typeof value.id === "string" && /^\d+$/.test(value.id)
|
|
? value.id
|
|
: null;
|
|
const owner =
|
|
typeof value.owner?.login === "string" ? value.owner.login.trim() : "";
|
|
const name = typeof value.name === "string" ? value.name.trim() : "";
|
|
const fullName =
|
|
typeof value.full_name === "string" && value.full_name.includes("/")
|
|
? value.full_name.trim()
|
|
: owner && name
|
|
? `${owner}/${name}`
|
|
: "";
|
|
if (!repositoryId || !fullName) return null;
|
|
const ownerId =
|
|
typeof value.owner?.id === "number" && Number.isSafeInteger(value.owner.id)
|
|
? String(value.owner.id)
|
|
: typeof value.owner?.id === "string" && /^\d+$/.test(value.owner.id)
|
|
? value.owner.id
|
|
: undefined;
|
|
return {
|
|
providerResourceId: fullName.toLowerCase(),
|
|
parentProviderResourceId: ownerId,
|
|
type: "repository",
|
|
label: fullName,
|
|
providerUrl:
|
|
typeof value.html_url === "string" && value.html_url.length > 0
|
|
? value.html_url
|
|
: `https://github.com/${fullName}`,
|
|
metadata: {
|
|
providerRepositoryId: repositoryId,
|
|
fullName,
|
|
...(owner ? { owner } : {}),
|
|
private: value.private === true,
|
|
source: "provider_webhook",
|
|
},
|
|
};
|
|
}
|
|
|
|
function githubRepositoryStableId(
|
|
item: ChatProviderResourceInventoryItem,
|
|
): string | null {
|
|
const value = item.metadata?.providerRepositoryId;
|
|
return typeof value === "string" && /^\d+$/.test(value) ? value : null;
|
|
}
|
|
|
|
export async function reconcileGitHubRepositoryCoordinates(tx: DbTransaction, endpoint: EndpointRow, item: ChatProviderResourceInventoryItem) {
|
|
const stableGitHubId =
|
|
githubRepositoryStableId(item);
|
|
if (stableGitHubId) {
|
|
await tx.execute(
|
|
sql`select pg_advisory_xact_lock(hashtextextended(${`chat-github-repository:${endpoint.id}:${stableGitHubId}`}, 0))`,
|
|
);
|
|
const stableResource = await tx
|
|
.select()
|
|
.from(chatEndpointResources)
|
|
.where(
|
|
and(
|
|
eq(chatEndpointResources.companyId, endpoint.companyId),
|
|
eq(chatEndpointResources.endpointId, endpoint.id),
|
|
eq(chatEndpointResources.type, "repository"),
|
|
sql`${chatEndpointResources.metadata}->>'providerRepositoryId' = ${stableGitHubId}`,
|
|
),
|
|
)
|
|
.for("update")
|
|
.then((rows) => rows[0] ?? null);
|
|
if (
|
|
stableResource &&
|
|
stableResource.providerResourceId !== item.providerResourceId
|
|
) {
|
|
const coordinateResource = await tx
|
|
.select()
|
|
.from(chatEndpointResources)
|
|
.where(
|
|
and(
|
|
eq(chatEndpointResources.companyId, endpoint.companyId),
|
|
eq(chatEndpointResources.endpointId, endpoint.id),
|
|
eq(chatEndpointResources.type, "repository"),
|
|
eq(
|
|
chatEndpointResources.providerResourceId,
|
|
item.providerResourceId,
|
|
),
|
|
),
|
|
)
|
|
.for("update")
|
|
.then((rows) => rows[0] ?? null);
|
|
if (coordinateResource && coordinateResource.id !== stableResource.id) {
|
|
const coordinateConversation = await tx
|
|
.select({ id: chatConversations.id })
|
|
.from(chatConversations)
|
|
.where(
|
|
and(
|
|
eq(chatConversations.endpointId, endpoint.id),
|
|
eq(chatConversations.resourceId, coordinateResource.id),
|
|
),
|
|
)
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
if (coordinateConversation) {
|
|
throw conflict(
|
|
"GitHub repository identity is already bound to two coordinates",
|
|
{ code: "chat_github_repository_identity_conflict" },
|
|
);
|
|
}
|
|
await tx
|
|
.delete(chatEndpointResources)
|
|
.where(eq(chatEndpointResources.id, coordinateResource.id));
|
|
}
|
|
|
|
const oldRepository = stableResource.providerResourceId;
|
|
const oldThreadPrefix = `github:${oldRepository}`;
|
|
const newThreadPrefix = `github:${item.providerResourceId}`;
|
|
const oldProviderUrl = `https://github.com/${oldRepository}`;
|
|
const newProviderUrl = `https://github.com/${item.providerResourceId}`;
|
|
await tx
|
|
.update(chatEndpointResources)
|
|
.set({
|
|
providerResourceId: item.providerResourceId,
|
|
parentProviderResourceId: item.parentProviderResourceId ?? null,
|
|
label: item.label,
|
|
providerUrl: item.providerUrl ?? null,
|
|
availability: "available",
|
|
enabled:
|
|
stableResource.enabled || coordinateResource?.enabled === true,
|
|
metadata: item.metadata ?? {},
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(chatEndpointResources.id, stableResource.id));
|
|
await tx
|
|
.update(chatConversations)
|
|
.set({
|
|
externalConversationId: sql<string>`case
|
|
when lower(${chatConversations.externalConversationId}) = ${oldRepository} then ${item.providerResourceId}
|
|
when lower(${chatConversations.externalConversationId}) = ${oldThreadPrefix} then ${newThreadPrefix}
|
|
else ${chatConversations.externalConversationId}
|
|
end`,
|
|
externalThreadId: sql<string>`case
|
|
when lower(${chatConversations.externalThreadId}) = ${oldThreadPrefix} then ${newThreadPrefix}
|
|
when lower(${chatConversations.externalThreadId}) like ${`${oldThreadPrefix}:%`}
|
|
then ${newThreadPrefix} || substring(
|
|
${chatConversations.externalThreadId}
|
|
from char_length(${oldThreadPrefix}) + 1
|
|
)
|
|
else ${chatConversations.externalThreadId}
|
|
end`,
|
|
externalLabel: item.label,
|
|
providerUrl: sql<string | null>`case
|
|
when lower(${chatConversations.providerUrl}) like ${`${oldProviderUrl}/%`}
|
|
then ${newProviderUrl} || substring(
|
|
${chatConversations.providerUrl}
|
|
from char_length(${oldProviderUrl}) + 1
|
|
)
|
|
else ${chatConversations.providerUrl}
|
|
end`,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(
|
|
and(
|
|
eq(chatConversations.companyId, endpoint.companyId),
|
|
eq(chatConversations.endpointId, endpoint.id),
|
|
eq(chatConversations.resourceId, stableResource.id),
|
|
),
|
|
);
|
|
return { id: stableResource.id };
|
|
}
|
|
}
|
|
|
|
return null;
|
|
}
|
|
|
|
type GitHubResourceOptions = {
|
|
endpointRecord: (endpointId: string) => Promise<{ endpoint: EndpointRow } | null>;
|
|
upsertProviderResourceRow: (tx: DbTransaction, endpoint: EndpointRow, item: ChatProviderResourceInventoryItem) => Promise<{ id: string } | null>;
|
|
restoreResourceConversations: (tx: DbTransaction, endpoint: EndpointRow, resourceId: string) => Promise<void>;
|
|
updateResourceSelection: (endpointId: string, updates: Array<{ id: string; enabled: boolean }>, actorUserId?: string | null, options?: { allGitHubRepositoriesEnabled?: boolean }) => Promise<void>;
|
|
};
|
|
|
|
export function createGitHubResources(db: Db, options: GitHubResourceOptions) {
|
|
const { endpointRecord, upsertProviderResourceRow, restoreResourceConversations, updateResourceSelection } = options;
|
|
async function reconcileGitHubWebhookRepository(
|
|
endpoint: EndpointRow,
|
|
payload: unknown,
|
|
credentialLease: CredentialMutationLeaseGuard,
|
|
) {
|
|
const item = githubRepositoryInventoryItemFromPayload(payload);
|
|
if (!item) return;
|
|
await db.transaction(async (tx) => {
|
|
await credentialLease.assertOwned(tx);
|
|
const resource = await upsertProviderResourceRow(tx, endpoint, item);
|
|
if (!resource) return;
|
|
// A correctly signed repository callback is current provider proof. Keep
|
|
// Paperclip's enabled choice, and reopen only conversations previously
|
|
// quarantined for provider availability loss.
|
|
await restoreResourceConversations(tx, endpoint, resource.id);
|
|
await credentialLease.assertOwned(tx);
|
|
});
|
|
}
|
|
|
|
async function listGitHubRepositories(
|
|
endpointId: string,
|
|
query: { limit: number; offset: number; search: string },
|
|
) {
|
|
const record = await endpointRecord(endpointId);
|
|
if (!record || record.endpoint.provider !== "github") throw notFound("GitHub bot not found");
|
|
const scope = and(
|
|
eq(chatEndpointResources.companyId, record.endpoint.companyId),
|
|
eq(chatEndpointResources.endpointId, endpointId),
|
|
eq(chatEndpointResources.type, "repository"),
|
|
);
|
|
// Literal, case-insensitive substring search (%, _ and backslashes are
|
|
// ordinary characters). Counts always describe the entire connection.
|
|
const search = query.search
|
|
? sql`strpos(lower(${chatEndpointResources.label}), lower(${query.search})) > 0`
|
|
: undefined;
|
|
const [rows, [counts]] = await Promise.all([
|
|
db.select().from(chatEndpointResources).where(and(scope, search))
|
|
.orderBy(asc(chatEndpointResources.label), asc(chatEndpointResources.id))
|
|
.offset(query.offset).limit(query.limit + 1),
|
|
db.select({
|
|
totalCount: sql<number>`count(*)::integer`,
|
|
enabledCount: sql<number>`count(*) filter (where ${chatEndpointResources.enabled})::integer`,
|
|
availableCount: sql<number>`count(*) filter (where ${chatEndpointResources.availability} = 'available')::integer`,
|
|
}).from(chatEndpointResources).where(scope),
|
|
]);
|
|
return {
|
|
items: rows.slice(0, query.limit),
|
|
nextOffset: rows.length > query.limit ? query.offset + query.limit : null,
|
|
totalCount: counts!.totalCount,
|
|
enabledCount: counts!.enabledCount,
|
|
availableCount: counts!.availableCount,
|
|
};
|
|
}
|
|
|
|
async function toggleAllGitHubRepositories(endpointId: string, enabled: boolean, actorUserId?: string | null) {
|
|
await updateResourceSelection(endpointId, [], actorUserId, { allGitHubRepositoriesEnabled: enabled });
|
|
return { success: true as const };
|
|
}
|
|
return { reconcileGitHubWebhookRepository, listGitHubRepositories, toggleAllGitHubRepositories };
|
|
}
|