Files
PaperClipAI/server/src/services/project-skill-source-bindings.ts
T
DottaandPaperclip 15176963b1 fix: stop idle skill-source polling (#17008)
## Thinking Path

> - Paperclip manages agent work and the skills that agents use.
> - Project repositories create durable import jobs, and installed
sources can refresh automatically.
> - The server queried these records every five seconds, even when no
work existed.
> - Idle tenant databases need to stop receiving those discovery
queries.
> - This change uses the existing transaction notifications and delivery
coordinator.
> - Saved refresh, retry, and lease deadlines determine when the worker
runs next.
> - Cloud does not yet wake these deadlines, so outstanding skill work
retains the instance.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Skill-source background synchronization. This continues the idle-work
migrations in #15625 and #17000. A search found no overlapping open
polling-removal PR.

**Current behavior**
On master `b11010b0d4`, startup installs a five-second skill-source
interval and also calls it from the external-object scheduler. Each
sweep queries active companies, import capacity, pending imports, and
automatic refreshes.

**Proposed behavior**
Run at startup, after a relevant transaction settles, or at a saved
deadline. Stay SQL-quiet when no eligible work remains. Preserve import
admission, authorization, atomic claims, retry backoff, and the existing
five-minute upstream check.

**Reason and benefit**
Remove recurring database activity from inactive tenant instances
without requiring new tables or a new scheduler.

## What Changed

- Replace both recurring skill-sync calls with one delivery-coordinator
registration.
- Notify from attachment/detachment, source creation/settings/manual
refresh/disconnection, project/company deletion, and company status
changes, including budget resume.
- Restore the earliest eligible `nextSyncAt` or lease-expiry deadline.
Due backlogs and conflicts retry after five seconds; no idle recovery
cadence remains.
- Admit deferred imports again after a batch frees capacity. Preserve
requests through drain, standby, restart, and commits during a sweep.
- Add sleep-safety predicates for imports and eligible automatic sources
until Cloud supports these deadlines.
- Document the lifecycle and process-local writer contract. No schema or
API changes.
- Guard external test database URLs before migrations or cleanup:
require a `_test` database name and reject database/options overrides.

## Verification

- 147 targeted tests pass: 39 server-startup tests, 14 new
real-PostgreSQL lifecycle tests, 20 shared coordinator/scheduler tests,
63 existing import/refresh/retry regression tests, and 11 test-database
guard cases.
- CI exposed a stale startup mock without the new `start` method. The
fixture is updated and all 39 startup tests pass locally.
- Tests exercise real enqueue/settings/company-reactivation producers,
nested commit, rollback, saved retry/lease restart, manual refresh
failure, capacity admission, concurrent commit, shutdown cancellation,
and SQL quiet after startup or pause.
- Sleep-safety tests reject sleep for future imports and automatic
refreshes, and accept inactive retained state.
- `pnpm -r typecheck` and `pnpm build` pass.
- `pnpm test:run` was attempted and stopped after repeated
embedded-PostgreSQL startup failures in unrelated suites. The two
existing import/refresh suites initially skipped for the same host
limitation; after adding explicit disposable-database options, all 63
regression tests passed against PostgreSQL 17. Full CI passed on
`ddb3ee51369baa8c7153bae9fd88efb8b903f2bc`: 55 successful checks, one
neutral review check, and two intentional skips.

## Risks

- Notifications are process-local. Independent SQL writers and separate
live replicas do not wake this worker; startup reconstructs persisted
work.
- Enabled automatic sources still contact GitHub every five minutes.
Their upstream changes have no event ingress here.
- Cloud scheduled wakes currently cover routines and issue monitors, not
skill sync. This change retains instances with outstanding skill
obligations. It does not claim measured host/database suspension or
fleet savings.
- No change to source authorization, installed content on failure, or
existing claim semantics.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, and code
execution. The host does not expose an exact model revision or
context-window size to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 16:21:42 -05:00

70 lines
5.2 KiB
TypeScript

import { DELIVERY_QUEUES, notifyDeliveryWork } from './delivery-work-notifications.js';
import { lockSkillImportQueue, MAX_COMPANY_PENDING_SKILL_IMPORTS, MAX_PROJECT_PENDING_SKILL_IMPORTS, SKILL_IMPORT_CAPACITY_WAIT } from './skill-source-import-queue.js';
import { and, eq, ne } from 'drizzle-orm';
import { projectSkillSources, type Db, projectWorkspaces } from '@paperclipai/db';
import { parseGitHubSkillRepositoryUrl } from '@paperclipai/shared';
import { currentSkillActor } from './skill-access.js';
import { captureRunIdentity } from './run-identity.js';
import { forbidden } from '../errors.js';
export async function projectSkillSetupIdentity(db: Db, companyId: string, actor = currentSkillActor()) {
let userId = actor?.type === 'board' ? actor.userId ?? null : null;
let connectionId: string | null = null, refreshGrantId: string | null = null;
if (actor?.type === 'agent' && actor.source === 'agent_jwt') {
if (!actor.runId || !actor.agentId || actor.companyId !== companyId) throw forbidden('Project skill setup requires an authenticated agent run.');
const identity = await captureRunIdentity(db, { companyId, agentId: actor.agentId, runId: actor.runId });
userId = identity.run.responsibleUserId;
// Only reuse the connection already authorized by managed GitHub delivery
// for this active run. Responsibility alone cannot discover another grant.
if (identity.context?.github?.status === 'available') {
connectionId = identity.context.github.connectionId ?? null;
refreshGrantId = identity.context.github.grantId ?? null;
}
}
// An API key can attach repositories, but cannot supply a setup person.
// Keep the durable job so an authorized person can adopt it from Sources.
return { refreshUserId: userId, refreshAgentId: actor?.type === 'agent' ? actor.agentId ?? null : null,
refreshLocalTrusted: actor?.source === 'local_implicit' || userId === 'local-board', connectionId, refreshGrantId };
}
export function projectSkillRepository(workspace: Pick<typeof projectWorkspaces.$inferSelect, 'repoUrl' | 'defaultRef' | 'repoRef'>) {
const url = workspace.repoUrl?.replace(/^git@github\.com:/i, 'https://github.com/');
const parsed = url ? parseGitHubSkillRepositoryUrl(url) : null;
if (!parsed) return null;
return { repositoryUrl: parsed.repositoryUrl.toLowerCase(), trackingRef: workspace.defaultRef ?? workspace.repoRef ?? parsed.trackingRef ?? 'HEAD' };
}
/** Called inside the workspace mutation transaction; no provider calls here. */
export async function enqueueProjectSkillSource(db: Db, workspace: typeof projectWorkspaces.$inferSelect, setupIdentity?: Awaited<ReturnType<typeof projectSkillSetupIdentity>>) {
return db.transaction(async tx => {
const db = tx as unknown as Db;
const repository = projectSkillRepository(workspace);
const setup = repository ? setupIdentity ?? await projectSkillSetupIdentity(db, workspace.companyId) : undefined;
await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync);
const pending = await lockSkillImportQueue(db, workspace.companyId);
await db.update(projectSkillSources).set({ active: false, lastError: null })
.where(and(eq(projectSkillSources.companyId, workspace.companyId), eq(projectSkillSources.workspaceId, workspace.id),
...(repository ? [ne(projectSkillSources.trackingRef, repository.trackingRef)] : [])));
if (!repository) return null;
await db.update(projectSkillSources).set({ active: false, lastError: null })
.where(and(eq(projectSkillSources.companyId, workspace.companyId), eq(projectSkillSources.workspaceId, workspace.id), ne(projectSkillSources.repositoryUrl, repository.repositoryUrl)));
const [existing] = await db.select().from(projectSkillSources).where(and(eq(projectSkillSources.workspaceId, workspace.id),
eq(projectSkillSources.repositoryUrl, repository.repositoryUrl), eq(projectSkillSources.trackingRef, repository.trackingRef)));
const available = Boolean(existing?.sourceId || pending.some(row => row.id === existing?.id))
|| (pending.length < MAX_COMPANY_PENDING_SKILL_IMPORTS && pending.filter(row => row.projectId === workspace.projectId).length < MAX_PROJECT_PENDING_SKILL_IMPORTS);
const [binding] = await db.insert(projectSkillSources).values({
companyId: workspace.companyId, projectId: workspace.projectId, workspaceId: workspace.id, ...repository, nextSyncAt: new Date(),
...setup, active: available, lastError: available ? null : SKILL_IMPORT_CAPACITY_WAIT,
}).onConflictDoUpdate({ target: [projectSkillSources.workspaceId, projectSkillSources.repositoryUrl, projectSkillSources.trackingRef],
set: { active: available, lastError: available ? existing?.lastError === SKILL_IMPORT_CAPACITY_WAIT ? null : existing?.lastError : SKILL_IMPORT_CAPACITY_WAIT } }).returning();
return binding!;
});
}
export async function detachProjectSkillSource(db: Db, companyId: string, workspaceId: string) {
await db.transaction(async tx => {
await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync);
await tx.update(projectSkillSources).set({ active: false, lastError: null }).where(and(eq(projectSkillSources.companyId, companyId), eq(projectSkillSources.workspaceId, workspaceId)));
});
}