mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-10 20:50:08 +02:00
**Issue (described inline; no existing tracking issue):** Pausing an
agent is not durable. Pausing cancels the in-flight run, but a queued or
recovery-dispatched run can clobber the agent back to `running` because
the execution-start status update is unconditional — so a "paused" agent
silently resumes work while `paused_at` is still set.
## Thinking Path
> - Paperclip orchestrates AI agents for zero-human companies; agent
work executes as "runs" tracked in `heartbeat_runs`, with a
recovery/automation layer that re-dispatches work when a run disappears.
> - The agent lifecycle has a pause control (status `paused`,
`paused_at` set) meant to stop an agent from taking or continuing work.
> - The problem: pause is not durable. Pausing cancels the in-flight
run, but the execution-start path then sets `agents.status = 'running'`
with an unconditional `UPDATE ... WHERE id = ?`, so any queued or
recovery-dispatched run can clobber the paused agent back to `running`
and execute.
> - Why it matters: a "paused" agent silently resuming undermines the
core operational control operators rely on to halt runaway,
cost-sensitive, or unsafe work.
> - This pull request guards the execution-start status flip with an
atomic conditional UPDATE, and tags pause-cancellations for
observability without changing resume behaviour.
> - The benefit is that a paused agent can no longer transition back to
`running`; queued/recovery-dispatched runs are cancelled cleanly instead
of clobbering status, while un-pausing still resumes in-flight work.
## What Changed
- Execution-start guard: replaced the unconditional `UPDATE agents SET
status='running' WHERE id = ?` with an atomic conditional `UPDATE ...
WHERE id = ? AND status NOT IN
('paused','terminated','pending_approval')`. On a zero-row match the run
is cancelled (`errorCode: "agent_not_invokable"`), the issue execution
lock is released, and the path returns — instead of clobbering status.
- Exported `DIRECT_NON_INVOKABLE_STATUSES` from `agent-invokability.ts`
and reused it in `heartbeat.ts` as the single source of truth for the
guard.
- Pause observability: `cancelActiveForAgentInternal` now accepts an
`errorCode` (default `"cancelled"`); the pause-route wrapper
`cancelActiveForAgent` passes `"agent_paused"`. This is
classification-neutral — `agent_paused` is NOT added to
`NON_RETRYABLE_CONTINUATION_ERROR_CODES`, so on un-pause the issue's
continuation re-enqueues and work resumes.
- Exported `classifyContinuationFailure` from `recovery/service.ts` for
unit testing (no logic change).
- Added `server/src/services/recovery/service.pause-durability.test.ts`
covering continuation classification.
## Verification
- `pnpm --filter @paperclipai/server typecheck` — clean.
- `pnpm exec vitest run
server/src/services/recovery/service.pause-durability.test.ts` — 5
passed.
- `pnpm exec vitest run server` — full server suite passes locally. The
only failures are pre-existing and environment-specific, unrelated to
this change (a git default-branch test fixture, and a known
checkout-lock race) — both reproduce identically on clean `master` with
this change stashed out.
- Behavioural: a paused agent's execution-start now aborts cleanly with
no status clobber; non-pause cancellations keep `errorCode "cancelled"`
and existing behaviour; un-pausing resumes the in-flight issue.
## Risks
- Low risk. No schema change, no migration, no new dependency; four
files. The change narrows a single UPDATE to be conditional and adds a
rarely-taken abort branch on the run-start path; behaviour for invokable
agents is unchanged. The abort's `agent_not_invokable` code is already
in `NON_RETRYABLE_CONTINUATION_ERROR_CODES`. The only caller of
`cancelActiveForAgent` is the pause route.
## Related upstream work — not duplicates
This area has prior and in-flight PRs; #8317 was checked against them
and is intentionally distinct:
- **#4503** (`fix(heartbeat): make agent pause status guard atomic with
status update`) targets a different TOCTOU race on the **post-run /
finalize** path (`finalizeAgentStatus`). #8317 targets the
**execution-start** race, where a recovery-dispatched run flips a paused
agent back to `running` *before the run begins*. #4503 does not cover
the proven failure path here: `pause → active run cancelled → recovery
dispatches a new run → execution-start overwrites the paused state`.
#4503 also does not add the resume semantics below.
- **#4356** (`honor system/manual/auto pause at all heartbeat-run
enqueue sites`) and **#1067** (`pause guard on queue drain`) protect the
**enqueue / queue-drain** layer. They are complementary to — not
substitutes for — the execution-start guard, which is the last gate
before a run actually starts.
- **#6944** (`guard executeRun against paused agent`), **#7140**, and
**#7141** attempted similar execution-start ideas but were closed for
implementation hygiene / build issues, not because the guard concept was
wrong. #8317 implements that concept cleanly: a single atomic
conditional UPDATE, a clean abort with `errorCode:
"agent_not_invokable"`, a shared `DIRECT_NON_INVOKABLE_STATUSES` source
of truth, and passing tests + typecheck.
Intentional, minor difference (not a criticism of #4503): #8317's
execution-start deny-list is `paused`, `terminated`, and
`pending_approval` — the full non-invokable set for run-start
invokability — whereas #4503 appears focused on `paused`/`terminated`.
The broader set is deliberate for the execution-start guard.
Resume semantics: #8317 keeps `agent_paused` as observability-only and
classification-neutral (retryable), so a paused agent's in-flight work
resumes on un-pause rather than escalating to blocked.
## Model Used
- Provider: Anthropic. Model: Claude Opus 4 (`claude-opus-4-8`), via the
Claude desktop "Cowork" agent. Mode: agentic/extended reasoning with
tool use (shell, file editing, running `tsc`/`vitest`, git). Used to
investigate the root cause in source, design the fix, implement it, and
validate locally.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots (N/A — no UI change)
- [ ] I have updated relevant documentation to reflect my changes (N/A —
no doc-facing change)
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge
- [x] I have searched the open and closed PR list for similar/duplicate
PRs and found none
165 lines
4.8 KiB
TypeScript
165 lines
4.8 KiB
TypeScript
import type { Db } from "@paperclipai/db";
|
|
import { agents } from "@paperclipai/db";
|
|
import { getAgentWorkEligibility, type AgentEligibilityAgent, type AgentOrgChainHealth } from "@paperclipai/shared";
|
|
import { eq } from "drizzle-orm";
|
|
|
|
type AgentStatus = (typeof agents.$inferSelect)["status"];
|
|
|
|
export type AgentOrgRow = Pick<
|
|
typeof agents.$inferSelect,
|
|
"id" | "companyId" | "name" | "reportsTo" | "status"
|
|
>;
|
|
|
|
export type AgentInvokabilityBlockReason =
|
|
| "missing"
|
|
| "paused"
|
|
| "terminated"
|
|
| "pending_approval"
|
|
| "unknown_status"
|
|
| "manager_missing"
|
|
| "manager_company_mismatch"
|
|
| "manager_terminated"
|
|
| "reporting_cycle"
|
|
| "reporting_chain_too_deep";
|
|
|
|
export type AgentInvokability =
|
|
| { invokable: true }
|
|
| {
|
|
invokable: false;
|
|
reason: AgentInvokabilityBlockReason;
|
|
message: string;
|
|
details: Record<string, unknown>;
|
|
invalidOrgChain: boolean;
|
|
};
|
|
|
|
export const DIRECT_NON_INVOKABLE_STATUSES = new Set<AgentStatus>([
|
|
"paused",
|
|
"terminated",
|
|
"pending_approval",
|
|
]);
|
|
|
|
function blocked(
|
|
reason: AgentInvokabilityBlockReason,
|
|
message: string,
|
|
details: Record<string, unknown>,
|
|
invalidOrgChain = false,
|
|
): AgentInvokability {
|
|
return { invokable: false, reason, message, details, invalidOrgChain };
|
|
}
|
|
|
|
function statusBlockReason(status: AgentStatus): AgentInvokabilityBlockReason | null {
|
|
if (status === "paused") return "paused";
|
|
if (status === "terminated") return "terminated";
|
|
if (status === "pending_approval") return "pending_approval";
|
|
return null;
|
|
}
|
|
|
|
function toEligibilityAgent(row: AgentOrgRow): AgentEligibilityAgent {
|
|
return {
|
|
id: row.id,
|
|
companyId: row.companyId,
|
|
name: row.name,
|
|
status: row.status,
|
|
reportsTo: row.reportsTo,
|
|
};
|
|
}
|
|
|
|
function invalidChainReason(health: AgentOrgChainHealth): AgentInvokabilityBlockReason {
|
|
if (health.reason === "terminated_ancestor") return "manager_terminated";
|
|
if (health.reason === "cycle") return "reporting_cycle";
|
|
return "manager_missing";
|
|
}
|
|
|
|
export function evaluateAgentInvokability(
|
|
agent: AgentOrgRow | null | undefined,
|
|
companyAgents: AgentOrgRow[],
|
|
): AgentInvokability {
|
|
if (!agent) {
|
|
return blocked("missing", "Agent no longer exists", {}, false);
|
|
}
|
|
|
|
const eligibility = getAgentWorkEligibility({
|
|
agent: toEligibilityAgent(agent),
|
|
agents: companyAgents.map(toEligibilityAgent),
|
|
});
|
|
|
|
if (eligibility.invokable) return { invokable: true };
|
|
|
|
const directStatusReason = eligibility.invokabilityReason === "unknown_status"
|
|
? "unknown_status"
|
|
: statusBlockReason(agent.status);
|
|
if (directStatusReason) {
|
|
return blocked(
|
|
directStatusReason,
|
|
"Agent is not invokable in its current state",
|
|
{ agentId: agent.id, agentStatus: agent.status },
|
|
false,
|
|
);
|
|
}
|
|
|
|
const health = eligibility.orgChainHealth;
|
|
const firstInvalidAncestor = health.firstInvalidAncestor;
|
|
return blocked(
|
|
invalidChainReason(health),
|
|
"Agent is not invokable because its reporting chain is invalid",
|
|
{
|
|
agentId: agent.id,
|
|
managerId: firstInvalidAncestor?.id ?? null,
|
|
managerStatus: firstInvalidAncestor?.status ?? null,
|
|
reportingChainAgentIds: health.fullChain
|
|
.filter((entry) => entry.relation === "ancestor")
|
|
.map((entry) => entry.id),
|
|
orgChainHealth: health,
|
|
},
|
|
true,
|
|
);
|
|
}
|
|
|
|
export async function evaluateAgentInvokabilityFromDb(
|
|
db: Db,
|
|
agent: AgentOrgRow | null | undefined,
|
|
): Promise<AgentInvokability> {
|
|
if (!agent) return evaluateAgentInvokability(agent, []);
|
|
const companyAgents = await db
|
|
.select({
|
|
id: agents.id,
|
|
companyId: agents.companyId,
|
|
name: agents.name,
|
|
reportsTo: agents.reportsTo,
|
|
status: agents.status,
|
|
})
|
|
.from(agents)
|
|
.where(eq(agents.companyId, agent.companyId));
|
|
return evaluateAgentInvokability(agent, companyAgents);
|
|
}
|
|
|
|
export function listInvalidOrgChainDescendantIds(
|
|
terminatedAgentId: string,
|
|
companyAgents: AgentOrgRow[],
|
|
): string[] {
|
|
const byManager = new Map<string | null, AgentOrgRow[]>();
|
|
for (const row of companyAgents) {
|
|
const siblings = byManager.get(row.reportsTo ?? null) ?? [];
|
|
siblings.push(row);
|
|
byManager.set(row.reportsTo ?? null, siblings);
|
|
}
|
|
|
|
const invalidDescendantIds: string[] = [];
|
|
const stack = [...(byManager.get(terminatedAgentId) ?? [])];
|
|
const seen = new Set<string>([terminatedAgentId]);
|
|
while (stack.length > 0) {
|
|
const current = stack.pop();
|
|
if (!current || seen.has(current.id)) continue;
|
|
seen.add(current.id);
|
|
if (current.status !== "terminated") {
|
|
invalidDescendantIds.push(current.id);
|
|
}
|
|
stack.push(...(byManager.get(current.id) ?? []));
|
|
}
|
|
return invalidDescendantIds;
|
|
}
|
|
|
|
export function shouldCancelRunsForNonInvokableAgent(result: AgentInvokability) {
|
|
return !result.invokable && (result.reason === "terminated" || result.invalidOrgChain);
|
|
}
|