Files
PaperClipAI/server/src/services/work-products.ts
T
DottaandPaperclip 7c016c049b perf(workspaces): wake terminal cleanup from committed changes (#16282)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use execution workspaces to retain their files and Git
branches.
> - A reaper closes eligible workspaces after their task tree is
terminal.
> - The reaper scanned every heartbeat tick and read Git before cheap
eligibility checks.
> - Permanent scans also query idle databases when no cleanup work
exists.
> - This pull request wakes cleanup after transaction settlement and
restores bounded work at startup.
> - The benefit is fewer scans while fresh cleanup guards continue to
protect retained work.

## Linked Issues or Issue Description

Related work: Refs #13619, Refs #11753. Those PRs bound scans or back
off unchanged candidates. This PR uses the transaction work coordinator
to remove the permanent reaper tick. It overlaps their early
terminal-state filtering but does not change their runtime restart
behavior or introduce a second Git scheduler cache.

**What existing behavior does this improve?**

Terminal workspace cleanup and Git inspection for workspace detail
reads.

**Subsystem affected**

Server orchestration, execution workspaces, and idle sleep admission.

**Current behavior**

The heartbeat tick scans workspace candidates every thirty seconds. It
runs Git before rejecting nonterminal trees, cooldowns, and active runs.
Display inspections expire after five seconds.

**Proposed behavior**

Recover workspace rows in bounded startup pages. Wake after relevant
issue, workspace, and pull-request work-product transactions settle.
Schedule the terminal tree cooldown or a five-minute eligibility retry.
Back off unchanged candidates across unrelated wakes, and invalidate
their backoff after relevant committed changes. Park empty and
nonterminal-only lanes. Reuse display inspections for thirty seconds.
Read fresh Git before destructive cleanup.

**Reason and benefit**

Avoid recurring database and Git work when no cleanup is possible.
Retain transaction admission holds, restart recovery, and fresh deletion
checks. Retention eligibility can wait for the next host wake; active
and failed recovery work still blocks sleep.

**Breaking changes**

No schema or API shape changes. Display Git state can lag by thirty
seconds. Cleanup eligibility retries can take five minutes.

## What Changed

- Add a terminal-workspace worker to the existing transaction work
coordinator and scheduler.
- Signal relevant writes before mutation on the owning transaction.
Outer transaction settlement wakes cleanup.
- Derive cooldown deadlines from the full company-scoped terminal task
tree. Continue bounded recovery pages without a permanent sweep.
- Bound unchanged candidate inspections to five minutes, retain at most
256 entries, and invalidate affected entries with scoped transaction
hints. Repeated wakes preserve the earliest inspection deadline.
- Collect deadlines from task trees already read by bounded pages.
Restore the cursor after a failed page and rescan commits received
during a rotation.
- Check terminal state, cooldown, and active runs before Git. Preserve
lifecycle locks, generation fences, and fresh final checks.
- Keep failed or incomplete recovery visible to idle admission. Preserve
the passive retention exception.
- Extend the bounded display cache to thirty seconds and document the
scheduling and wake limits.

## Verification

- Focused workspace, coordinator, cache, and local sleep suites: 111
tests passed with real local PostgreSQL and Git worktrees where
applicable.
- Final worker, display cache, and local sleep suites: 28 tests passed.
- The regressions prove three ineligible workspaces invoke zero Git
scans, cached clean display state cannot remove new untracked files, and
real producer notifications wait for outer transaction settlement.
- `pnpm -r typecheck` and `pnpm build` passed locally. Map and
module-boundary checks passed.
- Work-product and sleep-safety suites: 221 tests passed. Reopen and
issue-tree control suites: 46 tests passed. Final backoff, workspace,
worker, and work-product suites: 87 tests passed.
- Final workspace and worker suites: 81 tests passed. Deadline selection
performs zero database reads. A failed full page retries its unread
terminal workspace before recovery can park.
- Full CI coverage passed on head `2c397970f7`: general tests,
serialized server suites, runner verification, E2E shards, typecheck,
build, and the final verification gate. The duplicate serial local `pnpm
test:run` was stopped after full CI coverage passed; it is not reported
as a completed local pass.
- Greptile: 5/5 with both findings resolved. The security scan also
passed. No failing or pending PR checks remain.

## Risks

- Display state can be stale for thirty seconds. Destructive guards
bypass the display cache.
- Undelivered or otherwise ineligible terminal work retries after five
minutes instead of each heartbeat tick.
- Transaction signals cover the owning `createDb` scope. Direct SQL and
independent server processes require startup recovery.
- Local cooldown timers are maintenance hints. They do not establish an
external wake alarm or whole-instance sleep safety. Existing blockers
remain in force; failed bounded recovery keeps the host awake.

## Model Used

OpenAI GPT-6, model ID `gpt-6`, through Codex. Reasoning, repository
inspection, shell execution, and browser tools assisted the change. The
runtime does not expose the context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-14 08:28:45 -05:00

402 lines
16 KiB
TypeScript

import { and, desc, eq, inArray } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import { executionWorkspaces, heartbeatRunEvents, issueWorkProducts, workspaceRuntimeServices } from "@paperclipai/db";
import type { IssueWorkProduct } from "@paperclipai/shared";
import { unprocessable } from "../errors.js";
import { insertRowsInChunks } from "./batch-insert.js";
import { notifyTerminalWorkspaceWork } from "./terminal-workspace-work.js";
import {
createPullRequestMergeDetailsResolver,
extractGitHubPullRequestReferences,
type PullRequestMergeDetailsResolver,
} from "./github-pull-request-merge.js";
import {
createGitHubCommitDiffDetailsResolver,
extractGitHubCommitReference,
type GitHubCommitDiffDetailsResolver,
} from "./github-commit-details.js";
import type { ImportIssueWorkProductRow } from "./import-write-types.js";
type IssueWorkProductRow = typeof issueWorkProducts.$inferSelect;
type WorkProductTransaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
async function validateExecutionWorkspace(
tx: WorkProductTransaction,
companyId: string,
executionWorkspaceId: string | null | undefined,
): Promise<void> {
if (!executionWorkspaceId) return;
const [workspace] = await tx
.select({ id: executionWorkspaces.id })
.from(executionWorkspaces)
.where(and(
eq(executionWorkspaces.id, executionWorkspaceId),
eq(executionWorkspaces.companyId, companyId),
))
// Keep a concurrent delete from invalidating the reference before the write.
.for("key share");
if (!workspace) {
throw unprocessable(
"executionWorkspaceId must identify an execution workspace in this company. " +
"Project workspace IDs are not execution workspace IDs. Omit this field when no execution workspace is available.",
);
}
}
export interface WorkProductDiffSummary {
additions: number | null;
deletions: number | null;
changedFiles: number;
}
function nonNegativeInteger(value: unknown): number | null {
return typeof value === "number" && Number.isSafeInteger(value) && value >= 0 ? value : null;
}
export function workProductDiffSummaryFromEventPayload(payload: unknown): WorkProductDiffSummary | null {
if (!payload || typeof payload !== "object" || Array.isArray(payload)) return null;
const outer = payload as Record<string, unknown>;
const wrappedEvent = outer.prpEvent && typeof outer.prpEvent === "object" && !Array.isArray(outer.prpEvent)
? outer.prpEvent as Record<string, unknown>
: null;
const eventPayload = wrappedEvent?.payload ?? (outer.schema === "paperclip.prp.event.v1" ? outer.payload : outer);
if (!eventPayload || typeof eventPayload !== "object" || Array.isArray(eventPayload)) return null;
const totals = (eventPayload as Record<string, unknown>).totals;
if (!totals || typeof totals !== "object" || Array.isArray(totals)) return null;
const values = totals as Record<string, unknown>;
const changedFiles = nonNegativeInteger(values.files);
if (changedFiles === null) return null;
return {
additions: values.additions === null ? null : nonNegativeInteger(values.additions),
deletions: values.deletions === null ? null : nonNegativeInteger(values.deletions),
changedFiles,
};
}
export function enrichWorkProductMetadataWithDiff(
metadata: Record<string, unknown> | null | undefined,
summary: WorkProductDiffSummary | null,
): Record<string, unknown> | null {
if (!summary) return metadata ?? null;
return {
...(metadata ?? {}),
...(summary.additions === null ? {} : { additions: summary.additions }),
...(summary.deletions === null ? {} : { deletions: summary.deletions }),
changedFiles: summary.changedFiles,
};
}
export async function refreshPullRequestWorkProductMetadata(
products: IssueWorkProduct[],
resolvePullRequestDetails: PullRequestMergeDetailsResolver,
): Promise<IssueWorkProduct[]> {
return await Promise.all(products.map(async (product) => {
if (product.type !== "pull_request") return product;
const metadata = product.metadata ?? {};
const repo = typeof metadata.repo === "string" ? metadata.repo
: typeof metadata.repository === "string" ? metadata.repository : null;
const number = nonNegativeInteger(metadata.number);
const references = extractGitHubPullRequestReferences([
product.url,
typeof metadata.url === "string" ? metadata.url : null,
repo && number ? `${repo}#${number}` : null,
]);
const reference = references[0];
if (!reference) return product;
try {
const details = await resolvePullRequestDetails(product.companyId, reference);
if (!details.workProductState) return product;
return {
...product,
metadata: {
...metadata,
repo: repo ?? `${reference.owner}/${reference.repo}`,
number: number ?? reference.number,
...(details.baseRef ? { baseRef: details.baseRef } : {}),
...(details.headRef ? { headRef: details.headRef } : {}),
...(nonNegativeInteger(details.additions) === null ? {} : { additions: details.additions }),
...(nonNegativeInteger(details.deletions) === null ? {} : { deletions: details.deletions }),
...(nonNegativeInteger(details.changedFiles) === null ? {} : { changedFiles: details.changedFiles }),
state: details.workProductState,
draft: details.workProductState === "draft" || details.draft === true,
},
};
} catch {
return product;
}
}));
}
function toIssueWorkProduct(row: IssueWorkProductRow): IssueWorkProduct {
return {
id: row.id,
companyId: row.companyId,
projectId: row.projectId ?? null,
issueId: row.issueId,
executionWorkspaceId: row.executionWorkspaceId ?? null,
runtimeServiceId: row.runtimeServiceId ?? null,
type: row.type as IssueWorkProduct["type"],
provider: row.provider,
externalId: row.externalId ?? null,
title: row.title,
url: row.url ?? null,
status: row.status,
reviewState: row.reviewState as IssueWorkProduct["reviewState"],
isPrimary: row.isPrimary,
healthStatus: row.healthStatus as IssueWorkProduct["healthStatus"],
summary: row.summary ?? null,
metadata: (row.metadata as Record<string, unknown> | null) ?? null,
sourceTrust: row.sourceTrust ?? null,
createdByRunId: row.createdByRunId ?? null,
createdAt: row.createdAt,
updatedAt: row.updatedAt,
};
}
/**
* Refresh runtime-service work products from the live runtime rows they point at
* (PAP-17572).
*
* A runtime URL is only valid for as long as the process holds that port. A
* managed restart can relocate it, which used to leave a user-facing preview link
* that answers with somebody else's service or nothing at all. The runtime row is
* the authoritative publication record, so it wins over the stored copy.
*
* Read-path only and deliberately non-destructive: a work product whose runtime
* row is gone keeps its recorded URL and is reported closed rather than silently
* blanked, so the history of what was published survives.
*/
export function reconcileRuntimeServiceWorkProducts(
products: IssueWorkProduct[],
liveRuntimeServices: Array<{
id: string;
url: string | null;
status: string;
healthStatus: string;
}>,
): IssueWorkProduct[] {
if (products.length === 0) return products;
const liveById = new Map(liveRuntimeServices.map((service) => [service.id, service]));
return products.map((product) => {
if (product.type !== "runtime_service" || !product.runtimeServiceId) return product;
const live = liveById.get(product.runtimeServiceId);
if (!live) {
return product.status === "closed" && product.healthStatus === "unhealthy"
? product
: { ...product, status: "closed", healthStatus: "unhealthy" };
}
const isServing = live.status === "running" && live.healthStatus === "healthy";
const url = live.url ?? product.url;
const status = live.status === "running" ? product.status : "closed";
const healthStatus: IssueWorkProduct["healthStatus"] = isServing ? "healthy" : "unhealthy";
if (product.url === url && product.status === status && product.healthStatus === healthStatus) return product;
return { ...product, url, status, healthStatus };
});
}
export function workProductService(
db: Db,
opts: {
resolvePullRequestDetails?: PullRequestMergeDetailsResolver;
resolveCommitDetails?: GitHubCommitDiffDetailsResolver;
} = {},
) {
const resolvePullRequestDetails = opts.resolvePullRequestDetails ?? createPullRequestMergeDetailsResolver(db);
const resolveCommitDetails = opts.resolveCommitDetails ?? createGitHubCommitDiffDetailsResolver(db);
return {
listForIssue: async (issueId: string, options: { refreshPullRequests?: boolean } = {}) => {
const rows = await db
.select()
.from(issueWorkProducts)
.where(eq(issueWorkProducts.issueId, issueId))
.orderBy(desc(issueWorkProducts.isPrimary), desc(issueWorkProducts.updatedAt));
const products = rows.map(toIssueWorkProduct);
const runtimeServiceIds = products
.map((product) => (product.type === "runtime_service" ? product.runtimeServiceId : null))
.filter((value): value is string => Boolean(value));
const reconciled = runtimeServiceIds.length === 0
? products
: reconcileRuntimeServiceWorkProducts(products, await db
.select({
id: workspaceRuntimeServices.id,
url: workspaceRuntimeServices.url,
status: workspaceRuntimeServices.status,
healthStatus: workspaceRuntimeServices.healthStatus,
})
.from(workspaceRuntimeServices)
.where(inArray(workspaceRuntimeServices.id, [...new Set(runtimeServiceIds)])));
return options.refreshPullRequests
? refreshPullRequestWorkProductMetadata(reconciled, resolvePullRequestDetails)
: reconciled;
},
latestRunDiffSummary: async (runId: string): Promise<WorkProductDiffSummary | null> => {
const rows = await db
.select({ payload: heartbeatRunEvents.payload })
.from(heartbeatRunEvents)
.where(and(
eq(heartbeatRunEvents.runId, runId),
inArray(heartbeatRunEvents.eventType, ["workspace.change.updated", "workspace.diff.recorded"]),
))
.orderBy(desc(heartbeatRunEvents.seq))
.limit(20);
for (const row of rows) {
const summary = workProductDiffSummaryFromEventPayload(row.payload);
if (summary) return summary;
}
return null;
},
resolveCommitDiffSummary: async (
companyId: string,
input: { provider: string; url?: string | null; metadata?: Record<string, unknown> | null },
): Promise<WorkProductDiffSummary | null> => {
if (input.provider.toLowerCase() !== "github") return null;
const metadata = input.metadata ?? {};
const repo = typeof metadata.repo === "string" ? metadata.repo : null;
const sha = typeof metadata.sha === "string" ? metadata.sha : null;
const reference = extractGitHubCommitReference([
input.url,
repo && sha ? `${repo}@${sha}` : null,
]);
return reference ? await resolveCommitDetails(companyId, reference) : null;
},
getById: async (id: string) => {
const row = await db
.select()
.from(issueWorkProducts)
.where(eq(issueWorkProducts.id, id))
.then((rows) => rows[0] ?? null);
return row ? toIssueWorkProduct(row) : null;
},
createForIssue: async (issueId: string, companyId: string, data: Omit<typeof issueWorkProducts.$inferInsert, "issueId" | "companyId">) => {
const row = await db.transaction(async (tx) => {
if (data.type === "pull_request") await notifyTerminalWorkspaceWork(tx, `issue:${issueId}`);
await validateExecutionWorkspace(tx, companyId, data.executionWorkspaceId);
if (data.isPrimary) {
await tx
.update(issueWorkProducts)
.set({ isPrimary: false, updatedAt: new Date() })
.where(
and(
eq(issueWorkProducts.companyId, companyId),
eq(issueWorkProducts.issueId, issueId),
eq(issueWorkProducts.type, data.type),
),
);
}
return await tx
.insert(issueWorkProducts)
.values({
...data,
companyId,
issueId,
})
.returning()
.then((rows) => rows[0] ?? null);
});
return row ? toIssueWorkProduct(row) : null;
},
update: async (id: string, patch: Partial<typeof issueWorkProducts.$inferInsert>) => {
const row = await db.transaction(async (tx) => {
const existing = await tx
.select()
.from(issueWorkProducts)
.where(eq(issueWorkProducts.id, id))
.then((rows) => rows[0] ?? null);
if (!existing) return null;
if (existing.type === "pull_request" || patch.type === "pull_request") await notifyTerminalWorkspaceWork(tx, `issue:${existing.issueId}`);
await validateExecutionWorkspace(tx, existing.companyId, patch.executionWorkspaceId);
if (patch.isPrimary === true) {
await tx
.update(issueWorkProducts)
.set({ isPrimary: false, updatedAt: new Date() })
.where(
and(
eq(issueWorkProducts.companyId, existing.companyId),
eq(issueWorkProducts.issueId, existing.issueId),
eq(issueWorkProducts.type, existing.type),
),
);
}
return await tx
.update(issueWorkProducts)
.set({ ...patch, updatedAt: new Date() })
.where(eq(issueWorkProducts.id, id))
.returning()
.then((rows) => rows[0] ?? null);
});
return row ? toIssueWorkProduct(row) : null;
},
/**
* Batched work-product insert for company import.
*
* {@link createForIssue} clears the prior primary of the same type on every
* call; imported issues are brand new, so the only primaries in play are the
* imported rows themselves. We reproduce "last primary wins" within each
* (issue, type) group and insert the whole batch in chunked statements.
*/
createManyForImport: async (rows: ImportIssueWorkProductRow[]): Promise<void> => {
if (rows.length === 0) return;
const lastPrimaryIndexByGroup = new Map<string, number>();
rows.forEach((row, index) => {
if (row.isPrimary) lastPrimaryIndexByGroup.set(`${row.issueId}:${row.type}`, index);
});
const values = rows.map((row, index) => ({
companyId: row.companyId,
issueId: row.issueId,
projectId: row.projectId ?? null,
type: row.type,
provider: row.provider,
externalId: row.externalId ?? null,
title: row.title,
url: row.url ?? null,
status: row.status,
reviewState: row.reviewState,
isPrimary: row.isPrimary
? lastPrimaryIndexByGroup.get(`${row.issueId}:${row.type}`) === index
: false,
healthStatus: row.healthStatus,
summary: row.summary ?? null,
metadata: row.metadata ?? null,
executionWorkspaceId: row.executionWorkspaceId ?? null,
runtimeServiceId: row.runtimeServiceId ?? null,
createdByRunId: row.createdByRunId ?? null,
sourceTrust: row.sourceTrust ?? null,
}));
// Chunked writes are wrapped in a single transaction so a large import
// that spans multiple insert statements is atomic: if a later chunk
// fails, the earlier chunks roll back rather than leaving a partial
// prefix behind (which a retry would then duplicate). Mirrors the
// per-writer transaction the batched issue/document writers use.
await db.transaction(async (tx) => {
for (const value of values) {
if (value.type === "pull_request") await notifyTerminalWorkspaceWork(tx, `issue:${value.issueId}`);
}
await insertRowsInChunks(tx, issueWorkProducts, values);
});
},
remove: async (id: string) => {
const row = await db.transaction(async tx => {
const [existing] = await tx.select({ type: issueWorkProducts.type, issueId: issueWorkProducts.issueId })
.from(issueWorkProducts).where(eq(issueWorkProducts.id, id));
if (existing?.type === "pull_request") await notifyTerminalWorkspaceWork(tx, `issue:${existing.issueId}`);
return tx.delete(issueWorkProducts).where(eq(issueWorkProducts.id, id))
.returning().then(rows => rows[0] ?? null);
});
return row ? toIssueWorkProduct(row) : null;
},
};
}
export { toIssueWorkProduct };