Files
PaperClipAI/server/src/services/agent-invokability.ts
T
Tattoofamily f672a9e2e5 fix(runtime): keep agent pause durable at execution-start (#8317)
**Issue (described inline; no existing tracking issue):** Pausing an
agent is not durable. Pausing cancels the in-flight run, but a queued or
recovery-dispatched run can clobber the agent back to `running` because
the execution-start status update is unconditional — so a "paused" agent
silently resumes work while `paused_at` is still set.

## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies; agent
work executes as "runs" tracked in `heartbeat_runs`, with a
recovery/automation layer that re-dispatches work when a run disappears.
> - The agent lifecycle has a pause control (status `paused`,
`paused_at` set) meant to stop an agent from taking or continuing work.
> - The problem: pause is not durable. Pausing cancels the in-flight
run, but the execution-start path then sets `agents.status = 'running'`
with an unconditional `UPDATE ... WHERE id = ?`, so any queued or
recovery-dispatched run can clobber the paused agent back to `running`
and execute.
> - Why it matters: a "paused" agent silently resuming undermines the
core operational control operators rely on to halt runaway,
cost-sensitive, or unsafe work.
> - This pull request guards the execution-start status flip with an
atomic conditional UPDATE, and tags pause-cancellations for
observability without changing resume behaviour.
> - The benefit is that a paused agent can no longer transition back to
`running`; queued/recovery-dispatched runs are cancelled cleanly instead
of clobbering status, while un-pausing still resumes in-flight work.

## What Changed

- Execution-start guard: replaced the unconditional `UPDATE agents SET
status='running' WHERE id = ?` with an atomic conditional `UPDATE ...
WHERE id = ? AND status NOT IN
('paused','terminated','pending_approval')`. On a zero-row match the run
is cancelled (`errorCode: "agent_not_invokable"`), the issue execution
lock is released, and the path returns — instead of clobbering status.
- Exported `DIRECT_NON_INVOKABLE_STATUSES` from `agent-invokability.ts`
and reused it in `heartbeat.ts` as the single source of truth for the
guard.
- Pause observability: `cancelActiveForAgentInternal` now accepts an
`errorCode` (default `"cancelled"`); the pause-route wrapper
`cancelActiveForAgent` passes `"agent_paused"`. This is
classification-neutral — `agent_paused` is NOT added to
`NON_RETRYABLE_CONTINUATION_ERROR_CODES`, so on un-pause the issue's
continuation re-enqueues and work resumes.
- Exported `classifyContinuationFailure` from `recovery/service.ts` for
unit testing (no logic change).
- Added `server/src/services/recovery/service.pause-durability.test.ts`
covering continuation classification.

## Verification

- `pnpm --filter @paperclipai/server typecheck` — clean.
- `pnpm exec vitest run
server/src/services/recovery/service.pause-durability.test.ts` — 5
passed.
- `pnpm exec vitest run server` — full server suite passes locally. The
only failures are pre-existing and environment-specific, unrelated to
this change (a git default-branch test fixture, and a known
checkout-lock race) — both reproduce identically on clean `master` with
this change stashed out.
- Behavioural: a paused agent's execution-start now aborts cleanly with
no status clobber; non-pause cancellations keep `errorCode "cancelled"`
and existing behaviour; un-pausing resumes the in-flight issue.

## Risks

- Low risk. No schema change, no migration, no new dependency; four
files. The change narrows a single UPDATE to be conditional and adds a
rarely-taken abort branch on the run-start path; behaviour for invokable
agents is unchanged. The abort's `agent_not_invokable` code is already
in `NON_RETRYABLE_CONTINUATION_ERROR_CODES`. The only caller of
`cancelActiveForAgent` is the pause route.

## Related upstream work — not duplicates

This area has prior and in-flight PRs; #8317 was checked against them
and is intentionally distinct:

- **#4503** (`fix(heartbeat): make agent pause status guard atomic with
status update`) targets a different TOCTOU race on the **post-run /
finalize** path (`finalizeAgentStatus`). #8317 targets the
**execution-start** race, where a recovery-dispatched run flips a paused
agent back to `running` *before the run begins*. #4503 does not cover
the proven failure path here: `pause → active run cancelled → recovery
dispatches a new run → execution-start overwrites the paused state`.
#4503 also does not add the resume semantics below.
- **#4356** (`honor system/manual/auto pause at all heartbeat-run
enqueue sites`) and **#1067** (`pause guard on queue drain`) protect the
**enqueue / queue-drain** layer. They are complementary to — not
substitutes for — the execution-start guard, which is the last gate
before a run actually starts.
- **#6944** (`guard executeRun against paused agent`), **#7140**, and
**#7141** attempted similar execution-start ideas but were closed for
implementation hygiene / build issues, not because the guard concept was
wrong. #8317 implements that concept cleanly: a single atomic
conditional UPDATE, a clean abort with `errorCode:
"agent_not_invokable"`, a shared `DIRECT_NON_INVOKABLE_STATUSES` source
of truth, and passing tests + typecheck.

Intentional, minor difference (not a criticism of #4503): #8317's
execution-start deny-list is `paused`, `terminated`, and
`pending_approval` — the full non-invokable set for run-start
invokability — whereas #4503 appears focused on `paused`/`terminated`.
The broader set is deliberate for the execution-start guard.

Resume semantics: #8317 keeps `agent_paused` as observability-only and
classification-neutral (retryable), so a paused agent's in-flight work
resumes on un-pause rather than escalating to blocked.

## Model Used

- Provider: Anthropic. Model: Claude Opus 4 (`claude-opus-4-8`), via the
Claude desktop "Cowork" agent. Mode: agentic/extended reasoning with
tool use (shell, file editing, running `tsc`/`vitest`, git). Used to
investigate the root cause in source, design the fix, implement it, and
validate locally.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots (N/A — no UI change)
- [ ] I have updated relevant documentation to reflect my changes (N/A —
no doc-facing change)
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge
- [x] I have searched the open and closed PR list for similar/duplicate
PRs and found none
2026-06-19 08:35:53 -07:00

165 lines
4.8 KiB
TypeScript

import type { Db } from "@paperclipai/db";
import { agents } from "@paperclipai/db";
import { getAgentWorkEligibility, type AgentEligibilityAgent, type AgentOrgChainHealth } from "@paperclipai/shared";
import { eq } from "drizzle-orm";
type AgentStatus = (typeof agents.$inferSelect)["status"];
export type AgentOrgRow = Pick<
typeof agents.$inferSelect,
"id" | "companyId" | "name" | "reportsTo" | "status"
>;
export type AgentInvokabilityBlockReason =
| "missing"
| "paused"
| "terminated"
| "pending_approval"
| "unknown_status"
| "manager_missing"
| "manager_company_mismatch"
| "manager_terminated"
| "reporting_cycle"
| "reporting_chain_too_deep";
export type AgentInvokability =
| { invokable: true }
| {
invokable: false;
reason: AgentInvokabilityBlockReason;
message: string;
details: Record<string, unknown>;
invalidOrgChain: boolean;
};
export const DIRECT_NON_INVOKABLE_STATUSES = new Set<AgentStatus>([
"paused",
"terminated",
"pending_approval",
]);
function blocked(
reason: AgentInvokabilityBlockReason,
message: string,
details: Record<string, unknown>,
invalidOrgChain = false,
): AgentInvokability {
return { invokable: false, reason, message, details, invalidOrgChain };
}
function statusBlockReason(status: AgentStatus): AgentInvokabilityBlockReason | null {
if (status === "paused") return "paused";
if (status === "terminated") return "terminated";
if (status === "pending_approval") return "pending_approval";
return null;
}
function toEligibilityAgent(row: AgentOrgRow): AgentEligibilityAgent {
return {
id: row.id,
companyId: row.companyId,
name: row.name,
status: row.status,
reportsTo: row.reportsTo,
};
}
function invalidChainReason(health: AgentOrgChainHealth): AgentInvokabilityBlockReason {
if (health.reason === "terminated_ancestor") return "manager_terminated";
if (health.reason === "cycle") return "reporting_cycle";
return "manager_missing";
}
export function evaluateAgentInvokability(
agent: AgentOrgRow | null | undefined,
companyAgents: AgentOrgRow[],
): AgentInvokability {
if (!agent) {
return blocked("missing", "Agent no longer exists", {}, false);
}
const eligibility = getAgentWorkEligibility({
agent: toEligibilityAgent(agent),
agents: companyAgents.map(toEligibilityAgent),
});
if (eligibility.invokable) return { invokable: true };
const directStatusReason = eligibility.invokabilityReason === "unknown_status"
? "unknown_status"
: statusBlockReason(agent.status);
if (directStatusReason) {
return blocked(
directStatusReason,
"Agent is not invokable in its current state",
{ agentId: agent.id, agentStatus: agent.status },
false,
);
}
const health = eligibility.orgChainHealth;
const firstInvalidAncestor = health.firstInvalidAncestor;
return blocked(
invalidChainReason(health),
"Agent is not invokable because its reporting chain is invalid",
{
agentId: agent.id,
managerId: firstInvalidAncestor?.id ?? null,
managerStatus: firstInvalidAncestor?.status ?? null,
reportingChainAgentIds: health.fullChain
.filter((entry) => entry.relation === "ancestor")
.map((entry) => entry.id),
orgChainHealth: health,
},
true,
);
}
export async function evaluateAgentInvokabilityFromDb(
db: Db,
agent: AgentOrgRow | null | undefined,
): Promise<AgentInvokability> {
if (!agent) return evaluateAgentInvokability(agent, []);
const companyAgents = await db
.select({
id: agents.id,
companyId: agents.companyId,
name: agents.name,
reportsTo: agents.reportsTo,
status: agents.status,
})
.from(agents)
.where(eq(agents.companyId, agent.companyId));
return evaluateAgentInvokability(agent, companyAgents);
}
export function listInvalidOrgChainDescendantIds(
terminatedAgentId: string,
companyAgents: AgentOrgRow[],
): string[] {
const byManager = new Map<string | null, AgentOrgRow[]>();
for (const row of companyAgents) {
const siblings = byManager.get(row.reportsTo ?? null) ?? [];
siblings.push(row);
byManager.set(row.reportsTo ?? null, siblings);
}
const invalidDescendantIds: string[] = [];
const stack = [...(byManager.get(terminatedAgentId) ?? [])];
const seen = new Set<string>([terminatedAgentId]);
while (stack.length > 0) {
const current = stack.pop();
if (!current || seen.has(current.id)) continue;
seen.add(current.id);
if (current.status !== "terminated") {
invalidDescendantIds.push(current.id);
}
stack.push(...(byManager.get(current.id) ?? []));
}
return invalidDescendantIds;
}
export function shouldCancelRunsForNonInvokableAgent(result: AgentInvokability) {
return !result.invokable && (result.reason === "terminated" || result.invalidOrgChain);
}