mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The recovery queues check each assigned task to find work that has
no live run and no queued wake
> - On a nearly idle hosted instance, these queues still re-check almost
every task of a company in bursts, at about 22 read-only statements per
task
> - #21831 stopped a run start from re-checking all of its agent's
tasks, but bursts continued after writes to an agent row during a long
run
> - The cause is not the changed-rows backstop. An agent configuration
write publishes the company-wide task recovery hint, so every task queue
reloads and checks every assigned task of the company
> - This pull request makes an agent configuration write signal only the
company-level recovery queues and the run dispatch queues, not the
per-task queues
> - The benefit is that a configuration write during a long run no
longer re-checks hundreds of settled tasks
## Linked Issues or Issue Description
Refs #21831
**What happened?**
A write to an agent's configuration (for example `PATCH /agents/:id`, an
instructions save that rebinds the entry file, a skill or connection
change) calls `notifyRunDispatch({ companyId })`. That function sends
the company hint to the task queues (`stranded-assignments`,
`assigned-work-invariant`, `resolved-dependencies`,
`stale-issue-locks`). A company hint is not a backstop hint, so the
queue loads every eligible task of the company with no changed-rows
filter and checks each one. In a staging-shaped fixture (780 settled
reviews), each configuration write caused 780 checks and about 16,000
statements. The agent row keeps status `running` and its old
`last_heartbeat_at`, which matches the writes seen on staging.
**Expected behavior**
A configuration write does not re-check tasks. Only a change that can
strand a task (pause, resume, hold release, termination) re-checks
tasks, and those changes keep their own lifecycle hint.
**Steps to reproduce**
1. Seed one agent with 700 settled in-review tasks and a second agent
with 80, each with one live native run.
2. Start the recovery workers with the debug flag
`PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments:1`.
3. Update the agent's `runtimeConfig` or `adapterConfig` through
`agentService(db).update`.
4. See 780 `check` events with `wake: "company"`.
**Paperclip version or commit**
`master` at c5218f5ec4.
## What Changed
- `run-dispatch-notifications.ts`: split the dispatch queue hints out of
`notifyRunDispatch` and add `notifyAgentConfigurationDispatch`. It sends
the company-level recovery hints and the dispatch hints, but no
task-queue hint.
- `issue-recovery-notifications.ts`: add `notifyCompanyRecovery`, which
signals only the company-level recovery queues (orphan blockers, active
recovery actions, task watchdogs).
- `agent-configuration.ts`: a configuration write uses
`notifyAgentConfigurationDispatch` instead of `notifyRunDispatch`.
- New census test `issue-recovery-agent-row-write-census.test.ts`. It
uses real workers, a fake clock and a staging-shaped fixture. It drives
five agent-row writers through their real code paths over 35 simulated
minutes, and it adds guard tests.
## Verification
Red/green census (`issue-recovery-agent-row-write-census.test.ts`, 35
simulated minutes, five writes: runtime config on the big agent, adapter
config on the second agent, execution start rewrite, permissions,
adapter config on the big agent):
| | Settled-task checks | Stranded checks by wake | Statements |
| --- | --- | --- | --- |
| Before | 2,340 | company 2,346, hint 204, backstop 6 | 51,497 |
| After | 0 | hint 210, backstop 7 | 4,580 |
The execution start rewrite and the permissions write caused 0 settled
checks before and after. Only the three configuration writes caused the
whole-company reloads.
Guard tests (pass before and after):
- A task of the running agent that becomes stranded after a
configuration write is checked within the backstop bound.
- An idle agent's tasks are still checked within the backstop bound
after its configuration is written (the agent clause of the changed-rows
backstop).
- A hold release through the lifecycle status-change effect checks the
waiting task within one minute.
- Pause and termination through the lifecycle check the agent's tasks
within one minute.
- The hourly full pass still checks every settled task.
Focused suites: `cd server && npx vitest run` on 19 files (recovery
work, census, backstop, debug attribution, signal fanout, run dispatch,
agent configuration, budget configuration, agent timers, lifecycle): 278
passed, 1 skipped. `pnpm verify` returned exit 2 (opaque setup
dependency), so required CI supplies broad coverage. Server `tsc
--noEmit` passed.
## Risks
- A configuration write no longer re-checks the company's tasks at once.
Agent status changes still send the company hint through the lifecycle
effect. For a non-running agent, the changed-rows backstop still picks
up its tasks within the existing 5-minute bound. The hourly full pass is
unchanged.
- The stranded check reads agent status, holds and budget, not
configuration, so a configuration change cannot strand a task by itself.
- Low risk otherwise. No schema change.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
225 lines
13 KiB
TypeScript
225 lines
13 KiB
TypeScript
import { isDeepStrictEqual } from "node:util";
|
|
import { currentPluginInteractionOrigin } from "./plugin-interaction-context.js";
|
|
import { assertAgentInteraction } from "./agent-interaction-policy.js";
|
|
import { preserveAgentInteractionPermissions } from "../lib/agent-interaction-permissions.js";
|
|
import { notifyMuseContactVerification } from "../modules/external-agents/index.js";
|
|
import { assertEphemeralAgentInstructions } from "./agent-instructions.js";
|
|
import { notifyStrandedAgentStatus } from "./live-run-recovery-notifications.js";
|
|
import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js";
|
|
import { notifyAgentConfigurationDispatch } from "./run-dispatch-notifications.js";
|
|
import { notifyAttentionRetentionWork } from "./retention-work-notifications.js";
|
|
import { notifySkillVersionPinChange } from "./skill-version-retention-notifications.js";
|
|
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
|
import { notifyExternalConversationSettlement } from "./external-conversation-settlement-notifications.js";
|
|
import { and, eq } from "drizzle-orm";
|
|
import { agents, assets, agentConfigRevisions, agentTaskSessions, agentRuntimeState, type Db } from "@paperclipai/db";
|
|
import { normalizeAgentUrlKey, extractAgentMentionIds } from "@paperclipai/shared";
|
|
import { normalizePaperclipRunnerAdapterConfig } from "@paperclipai/adapter-utils/server-utils";
|
|
import { conflict, forbidden, notFound, unprocessable } from "../errors.js";
|
|
import { agentRecordQueries, type UpdateAgentOptions, type AgentConfigurationPatch, type AgentConfigurationRecord, isPlainRecord, buildConfigSnapshot,
|
|
hasConfigPatchFields, changedPendingApprovalConfigFields, diffConfigSnapshot, jsonEqual } from "../lib/agent-records.js";
|
|
import { CEO_HIRING_DENIAL_KEY, ceoHiringDenial, normalizeAgentPermissions } from "../lib/agent-permissions.js";
|
|
import type { ActivityPublication } from "../types/activity-publication.js";
|
|
import { secretService, assertClaudeOAuthBindingInvariant } from "./secrets.js";
|
|
import { agentCredentialService } from "./agent-credentials.js";
|
|
import { clearPrimaryAgent } from "./primary-agent.js";
|
|
import { budgetServiceInTransaction } from "./budgets.js";
|
|
|
|
export async function lockAgentConfigurationInteractions(tx: Db, id: string, data: AgentConfigurationPatch, options?: UpdateAgentOptions) {
|
|
const pluginOrigin = currentPluginInteractionOrigin();
|
|
if (pluginOrigin) options = { ...options, recordRevision: { ...options?.recordRevision,
|
|
createdByAgentId: pluginOrigin.agentId, createdByUserId: null, source: options?.recordRevision?.source ?? "plugin_agent" } };
|
|
const interactionActorId = pluginOrigin?.agentId ?? options?.recordRevision?.createdByAgentId;
|
|
if (interactionActorId) {
|
|
const [target] = await tx.select({ companyId: agents.companyId }).from(agents).where(eq(agents.id, id));
|
|
if (!target) return options;
|
|
await assertAgentInteraction(tx, { companyId: target.companyId, actorAgentId: interactionActorId,
|
|
operation: "configure", targetAgentId: id, lockAgentIds: configurationText(data).flatMap(extractAgentMentionIds), lock: "update" });
|
|
}
|
|
return options;
|
|
}
|
|
|
|
export async function prepareAgentConfiguration(db: Db, existing: AgentConfigurationRecord, data: AgentConfigurationPatch, options?: UpdateAgentOptions): Promise<AgentConfigurationPatch> {
|
|
await assertAgentInteraction(db, { companyId: existing.companyId, actorAgentId: options?.recordRevision?.createdByAgentId,
|
|
operation: "mention", texts: changedConfigurationText(data, existing) });
|
|
if (data.permissions !== undefined) data = { ...data, permissions: preserveAgentInteractionPermissions(existing.permissions, data.permissions) };
|
|
if (options?.recordRevision?.createdByAgentId && data.budgetMonthlyCents !== undefined && data.budgetMonthlyCents !== existing.budgetMonthlyCents) {
|
|
throw forbidden("Only board operators can change agent budgets");
|
|
}
|
|
await notifyAttentionRetentionWork(db, { companyId: existing.companyId });
|
|
const id = existing.id;
|
|
const { ensureManager, assertNoCycle, assertCompanyShortnameAvailable, assertBuiltInAgentMetadataMutationAllowed } = agentRecordQueries(db);
|
|
if (existing.status === "pending_approval" && !options?.allowPendingApprovalConfigUpdate) {
|
|
const changedFields = changedPendingApprovalConfigFields(agentRecordQueries(db).normalizeAgentRow(existing), data);
|
|
if (changedFields.length > 0) {
|
|
throw conflict("Pending approval agent configuration cannot be changed before board approval", {
|
|
code: "pending_approval_agent_config_frozen",
|
|
agentId: id,
|
|
fields: changedFields,
|
|
});
|
|
}
|
|
}
|
|
|
|
if (data.reportsTo !== undefined) {
|
|
if (data.reportsTo) {
|
|
await ensureManager(existing.companyId, data.reportsTo);
|
|
}
|
|
await assertNoCycle(id, data.reportsTo);
|
|
}
|
|
|
|
if (data.name !== undefined) {
|
|
const previousShortname = normalizeAgentUrlKey(existing.name);
|
|
const nextShortname = normalizeAgentUrlKey(data.name);
|
|
if (previousShortname !== nextShortname) {
|
|
await assertCompanyShortnameAvailable(existing.companyId, data.name, { excludeAgentId: id });
|
|
}
|
|
}
|
|
|
|
if (Object.prototype.hasOwnProperty.call(data, "metadata")) {
|
|
assertBuiltInAgentMetadataMutationAllowed(existing.metadata, data.metadata, options);
|
|
}
|
|
|
|
if (data.appearance?.customAvatarAssetId) {
|
|
const [asset] = await db.select().from(assets).where(and(
|
|
eq(assets.id, data.appearance.customAvatarAssetId), eq(assets.companyId, existing.companyId),
|
|
eq(assets.createdByAgentId, id),
|
|
));
|
|
if (!asset || asset.contentType !== "image/png" || !asset.objectKey.startsWith(`${existing.companyId}/agent-avatars/${id}/`)) {
|
|
throw unprocessable("Use the avatar upload endpoint to set this agent's image");
|
|
}
|
|
}
|
|
const normalizedPatch = { ...data } as AgentConfigurationPatch;
|
|
if (options?.recordRevision?.createdByAgentId) delete normalizedPatch.budgetMonthlyCents;
|
|
if (data.permissions !== undefined) {
|
|
// Only the permissions control records a CEO hiring restriction. Keep an
|
|
// existing one unless this write turns hiring on, and never take one from input.
|
|
const incoming = data.permissions && typeof data.permissions === "object" && !Array.isArray(data.permissions)
|
|
? Object.fromEntries(Object.entries(data.permissions).filter(([key]) => key !== CEO_HIRING_DENIAL_KEY)) : data.permissions;
|
|
const denial = ceoHiringDenial(existing.permissions);
|
|
normalizedPatch.permissions = normalizeAgentPermissions(
|
|
denial && (incoming as Record<string, unknown> | null)?.canCreateAgents !== true
|
|
? { ...(incoming as Record<string, unknown>), [CEO_HIRING_DENIAL_KEY]: denial } : incoming,
|
|
);
|
|
}
|
|
if (
|
|
Object.prototype.hasOwnProperty.call(normalizedPatch, "adapterConfig") &&
|
|
isPlainRecord(normalizedPatch.adapterConfig)
|
|
) {
|
|
const normalizedAdapterConfig = await secretService(db).normalizeAdapterConfigForPersistence(
|
|
existing.companyId,
|
|
normalizedPatch.adapterConfig,
|
|
{ adapterType: (normalizedPatch.adapterType ?? existing.adapterType) as string },
|
|
);
|
|
normalizedPatch.adapterConfig = normalizePaperclipRunnerAdapterConfig(
|
|
(normalizedPatch.adapterType ?? existing.adapterType) as string,
|
|
normalizedAdapterConfig,
|
|
);
|
|
} else if (
|
|
Object.prototype.hasOwnProperty.call(normalizedPatch, "adapterType")
|
|
&& isPlainRecord(existing.adapterConfig)
|
|
) {
|
|
normalizedPatch.adapterConfig = normalizePaperclipRunnerAdapterConfig(
|
|
normalizedPatch.adapterType as string,
|
|
existing.adapterConfig,
|
|
);
|
|
}
|
|
if(normalizedPatch.adapterType!==undefined&&normalizedPatch.adapterType!==existing.adapterType
|
|
||normalizedPatch.adapterConfig!==undefined&&!jsonEqual(normalizedPatch.adapterConfig,existing.adapterConfig))
|
|
await notifyMuseContactVerification(db,{companyId:existing.companyId,agentId:existing.id});
|
|
await notifyStrandedAgentStatus(db, { companyId: existing.companyId, agentId: existing.id });
|
|
await notifyAgentConfigurationDispatch(db, { companyId: existing.companyId });
|
|
await notifyDeferredExecutionEligibility(db, { companyId: existing.companyId });
|
|
if (normalizedPatch.runtimeConfig !== undefined || normalizedPatch.reportsTo !== undefined) {
|
|
await notifyDeliveryWork(db, DELIVERY_QUEUES.agentTimers);
|
|
}
|
|
if (normalizedPatch.adapterConfig !== undefined) {
|
|
await notifySkillVersionPinChange(db, existing.companyId, existing.adapterConfig ?? {}, normalizedPatch.adapterConfig);
|
|
}
|
|
assertEphemeralAgentInstructions({ ...existing, ...normalizedPatch });
|
|
return normalizedPatch;
|
|
}
|
|
|
|
export async function completeAgentConfiguration(txDb: Db, existing: AgentConfigurationRecord, updated: AgentConfigurationRecord, normalizedPatch: AgentConfigurationPatch, options: UpdateAgentOptions | undefined, publications: ActivityPublication[]) {
|
|
const id = updated.id;
|
|
const shouldRecordRevision = Boolean(options?.recordRevision) && hasConfigPatchFields(normalizedPatch);
|
|
const beforeConfig = shouldRecordRevision
|
|
? buildConfigSnapshot(agentRecordQueries(txDb).normalizeAgentRow(existing))
|
|
: null;
|
|
if (updated.status === "terminated") {
|
|
await clearPrimaryAgent(txDb, updated.companyId, id);
|
|
}
|
|
|
|
const priorAdapterConfig = isPlainRecord(existing.adapterConfig) ? existing.adapterConfig : {};
|
|
const afterConfig = isPlainRecord(updated.adapterConfig) ? updated.adapterConfig : {};
|
|
const changedExecution = updated.adapterType !== existing.adapterType
|
|
|| (updated.adapterType === "paperclip_runner" && ["provider", "acpxAgent", "model"].some(
|
|
(key) => priorAdapterConfig[key] !== afterConfig[key],
|
|
));
|
|
if (changedExecution) {
|
|
await notifyExternalConversationSettlement(txDb, { companyId: existing.companyId });
|
|
await txDb.delete(agentTaskSessions).where(and(eq(agentTaskSessions.companyId, existing.companyId), eq(agentTaskSessions.agentId, id)));
|
|
await txDb.update(agentRuntimeState).set({ adapterType: updated.adapterType, sessionId: null, stateJson: {}, updatedAt: new Date() })
|
|
.where(and(eq(agentRuntimeState.companyId, existing.companyId), eq(agentRuntimeState.agentId, id)));
|
|
}
|
|
|
|
if (Object.prototype.hasOwnProperty.call(normalizedPatch, "adapterConfig")) {
|
|
await agentCredentialService(txDb).enforceClaudeOAuthBindingClaim(txDb, {
|
|
companyId: existing.companyId, consume: false, environmentId: null,
|
|
childAdapterConfig: updated.adapterConfig, claudeLogin: options?.claudeLogin,
|
|
decision: assertClaudeOAuthBindingInvariant({ adapterType: updated.adapterType,
|
|
nextConfig: updated.adapterConfig, priorConfig: existing.adapterConfig }),
|
|
});
|
|
await agentCredentialService(txDb).syncAgentSecretBindings(
|
|
updated, txDb,
|
|
existing.adapterConfig,
|
|
options?.recordRevision,
|
|
);
|
|
}
|
|
|
|
if (normalizedPatch.budgetMonthlyCents !== undefined) {
|
|
await budgetServiceInTransaction(txDb, publications).upsertPolicy(existing.companyId, {
|
|
scopeType: "agent", scopeId: id, amount: normalizedPatch.budgetMonthlyCents,
|
|
isActive: normalizedPatch.budgetMonthlyCents > 0, windowKind: "calendar_month_utc",
|
|
}, options?.recordRevision?.createdByUserId ?? null);
|
|
}
|
|
const normalizedUpdated = await agentRecordQueries(txDb).getById(updated.id);
|
|
if (!normalizedUpdated) {
|
|
throw notFound("Agent not found");
|
|
}
|
|
|
|
if (shouldRecordRevision && beforeConfig) {
|
|
const afterConfig = buildConfigSnapshot(normalizedUpdated);
|
|
const changedKeys = diffConfigSnapshot(beforeConfig, afterConfig);
|
|
if (changedKeys.length > 0) {
|
|
await txDb.insert(agentConfigRevisions).values({
|
|
companyId: normalizedUpdated.companyId,
|
|
agentId: normalizedUpdated.id,
|
|
createdByAgentId: options?.recordRevision?.createdByAgentId ?? null,
|
|
createdByUserId: options?.recordRevision?.createdByUserId ?? null,
|
|
source: options?.recordRevision?.source ?? "patch",
|
|
rolledBackFromRevisionId: options?.recordRevision?.rolledBackFromRevisionId ?? null,
|
|
changedKeys,
|
|
beforeConfig: beforeConfig as unknown as Record<string, unknown>,
|
|
afterConfig: afterConfig as unknown as Record<string, unknown>,
|
|
});
|
|
}
|
|
}
|
|
}
|
|
|
|
function configurationText(value: unknown): string[] {
|
|
if (typeof value === "string") return [value];
|
|
if (Array.isArray(value)) return value.flatMap(configurationText);
|
|
if (value && typeof value === "object") return Object.values(value).flatMap(configurationText);
|
|
return [];
|
|
}
|
|
|
|
function changedConfigurationText(next: unknown, previous: unknown): string[] {
|
|
if (isDeepStrictEqual(next, previous)) return [];
|
|
if (typeof next === "string") return [next];
|
|
if (next && typeof next === "object") {
|
|
const prior = previous && typeof previous === "object" ? previous as Record<string, unknown> : {};
|
|
return Object.entries(next).flatMap(([key, value]) => changedConfigurationText(value, prior[key]));
|
|
}
|
|
return [];
|
|
}
|