Files
PaperClipAI/server/src/services/conversation-continuation.ts
T
DottaandPaperclip b1efd65edc fix: continue interrupted task conversations with bounded retries (#13237)
## Thinking Path

> - Paperclip manages AI agents and their tasks.
> - A task can outlive a provider process or a server restart.
> - Legacy recovery treated unknown tool outcomes as a permanent
execution hold.
> - That hold could also reject a later user message.
> - A conversation turn can use prior history without replaying prior
tool calls.
> - This pull request lets supported conversation adapters continue
within the existing retry budget.
> - Users can send a new message after automatic attempts stop.

## Linked Issues or Issue Description

**What happened?**

A server restart could interrupt a local ACP run and leave its task
behind a permanent recovery hold. A later user message could be
cancelled before the provider answered. The immediate recovery path
could also create a successor outside the durable failure counter.

**Expected behavior**

Continue with a bounded new conversation turn. Preserve a compatible
provider session or use full task context when it is unavailable. Do not
replay recorded tools. When automatic attempts stop, allow a new user
request through the normal execution gates.

**Steps to reproduce**

1. Start a task with a local conversation adapter.
2. Restart the server while the provider is working.
3. Let the previous run become interrupted.
4. Send a follow-up message and observe the recovery hold on the old
behavior.

Related work: Refs #13075 for durable task recovery. Refs #12946 for
retry-limit and checkout-lock handling. This change routes conversation
recovery through the existing bounded scheduler.

## What Changed

- Mark supported local conversation failures for continuation. Keep
native-runner and non-conversation recovery rules.
- Carry an interruption notice into the next turn. Retain stopped ACP
session history even when a write outcome is unknown.
- Clear unavailable ACP sessions so the next bounded attempt can use
full task context.
- Route immediate failure recovery through the same durable scheduler as
process-loss recovery. Release only the predecessor checkout when its
retry takes ownership.
- Retire obsolete conversation holds using immutable run evidence, in
bounded batches with an activity record. Preserve outcome evidence and
do not wake historical tasks.
- Block actual admission and Resume while a predecessor process or
environment lease is still active. Keep the original interruption notice
after a rejected wake. Preserve the upstream blocked-wake waiting
contract: bounded retry planning can happen during cleanup, while
deferred messages and execution remain gated.
- Add subprocess and database regression tests. Update the execution
contract.
- Add the current thread-status field to the native recovery provider
fixture so its damaged-journal test reaches the intended boundary.
Tolerate an already-exited fixture process during test cleanup while
still asserting both processes terminate.

## Verification

- Workspace typecheck passed: `pnpm -r typecheck`.
- Build passed: `pnpm build`.
- Module boundaries passed: `pnpm check:module-boundaries`.
- Focused tests passed: 293 recovery/session/dispatch tests, 66 retry
and response-gate tests, and 37 native-session tests. Some suites
overlap.
- Tests cover interrupted writes, missing sessions, concurrent retries,
restart persistence, pending questions and approvals, execution gates,
and historical holds.
- Built the Rust test executables with `pnpm --filter
@paperclipai/paperclip-runner build:rust` for native-runner
verification.
- Full Vitest coverage verified locally using the repository’s general
and serialized shards, with focused reruns for failures and files not
reached after a shard stopped. The ownership-gate regression is fixed
and the complete affected server shard passes (1,390 tests). Local
parallel runs also hit temporary-directory, resource, and timing
failures; those suites pass with canonical temporary paths and
sequential reruns. No test timeouts were increased.
- Final merged-branch regression run: 577 tests pass across process
recovery, retry scheduling, liveness, durable chat, wake-queue
application/adapter, dispatch, continuation, native sessions, and task
chat. Earlier focused verification also passed 19 native control tests.
Token gates and whitespace validation pass.
- Browser verification passed all three ACP Stop/continue/pause
scenarios, including a rerun after merging the upstream waiting
behavior: `PAPERCLIP_E2E_PORT=3397 pnpm test:e2e
tests/e2e/acp-stop-continuation.spec.ts`. The interrupted-write case
verifies that follow-up completes without a repeated write.

- Final-head [CI run
34625037394](https://github.com/paperclipai/paperclip/actions/runs/34625037394)
passed on `06ac4bd9d150f8b209a96e5fd609c696958794a0`: all 31 reported
checks are green, including server/workspace suites, all browser shards,
native runner verification, build, typecheck, release dry run, and
aggregate gates. The two conditional Storybook checks were skipped.
Greptile reviewed this exact commit at 5/5; all review threads are
resolved.

## Risks

- A new model turn can choose to repeat an action. Paperclip does not
replay recorded tool calls and does not certify unknown action outcomes.
- Conversation adapters now stop after their retry budget instead of
requiring action reconciliation. Explicit Stop, pause, dependency,
approval, budget, and ownership gates remain in force.
- No schema migration or dependency changes. Historical holds are folded
without changing task status or waking work.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, code
execution, and test execution. The session does not expose a more
specific model build ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-11 12:16:04 -05:00

119 lines
6.1 KiB
TypeScript

import { and, desc, eq, inArray, isNotNull, or, sql } from "drizzle-orm";
import { environmentLeases, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, type Db } from "@paperclipai/db";
import { readProcessStartedAt } from "./hot-restart.js";
// These adapters accept a conversation turn. Retrying a process or webhook can
// replay the action itself, so those adapters retain their recovery contract.
export const CONVERSATION_ADAPTER_TYPES = [
"claude_local", "codex_local", "cursor", "gemini_local", "opencode_local",
"pi_local", "grok_local", "kimi_local", "hermes_local",
] as const;
export function isConversationAdapter(adapterType: string): boolean {
return (CONVERSATION_ADAPTER_TYPES as readonly string[]).includes(adapterType);
}
export const CONVERSATION_CONTINUATION_POLICY = "continue_conversation_v1";
export function hasConversationContinuationPolicy(result: Record<string, unknown> | null | undefined): boolean {
return result?.conversationContinuation === CONVERSATION_CONTINUATION_POLICY;
}
function conversationRunPredicate() {
return or(
sql`${heartbeatRuns.resultJson}->>'conversationContinuation' = ${CONVERSATION_CONTINUATION_POLICY}`,
sql`exists (
select 1 from ${heartbeatRunEvents}
where ${heartbeatRunEvents.companyId} = ${heartbeatRuns.companyId}
and ${heartbeatRunEvents.runId} = ${heartbeatRuns.id}
and ${heartbeatRunEvents.eventType} = 'adapter.invoke'
and ${inArray(sql`${heartbeatRunEvents.payload}->>'adapterType'`, [...CONVERSATION_ADAPTER_TYPES])}
)`,
);
}
/** Recovery must not infer the old adapter from the agent's mutable settings. */
export async function runUsedConversationAdapter(db: Db, run: typeof heartbeatRuns.$inferSelect): Promise<boolean> {
if (hasConversationContinuationPolicy(run.resultJson)) return true;
const [invocation] = await db.select({ payload: heartbeatRunEvents.payload }).from(heartbeatRunEvents)
.where(and(eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id),
eq(heartbeatRunEvents.eventType, "adapter.invoke")))
.orderBy(desc(heartbeatRunEvents.seq)).limit(1);
const adapterType = invocation?.payload?.adapterType;
return typeof adapterType === "string" && isConversationAdapter(adapterType);
}
/** Only immutable run evidence can retire a historical conversation hold.
* An agent's current adapter can differ from the one that executed this run.
* Missing evidence retains the hold; the current agent is never a fallback.
*/
export function conversationRecoveryActionPredicate() {
return and(
eq(issueRecoveryActions.cause, "legacy_execution_requires_reconciliation"),
sql`exists (
select 1 from ${heartbeatRuns}
where ${heartbeatRuns.companyId} = ${issueRecoveryActions.companyId}
and ${heartbeatRuns.id}::text = ${issueRecoveryActions.evidence}->>'runId'
and coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId') = ${issueRecoveryActions.sourceIssueId}::text
and ${heartbeatRuns.runtimeMode} = 'legacy'
and ${inArray(heartbeatRuns.status, ['failed', 'timed_out', 'interrupted', 'cancelled'])}
and ${conversationRunPredicate()}
and ${or(
sql`${heartbeatRuns.resultJson}->>'conversationContinuation' = ${CONVERSATION_CONTINUATION_POLICY}`,
eq(heartbeatRuns.status, "interrupted"),
inArray(heartbeatRuns.errorCode, ["process_lost", "server_shutdown_interrupted", "execution_reconciliation_required"]),
and(eq(heartbeatRuns.status, "cancelled"), sql`${heartbeatRuns.resultJson}->'executionCancellation'->>'state' = 'acknowledged'`),
)}
)`,
);
}
/** OS liveness probes do not signal or stop the process. Unknown ownership holds. */
function processMayBeAlive(pid: number): boolean {
try {
process.kill(pid, 0);
return true;
} catch (error) {
return (error as NodeJS.ErrnoException).code !== "ESRCH";
}
}
/** A terminal conversation row does not prove that its execution authority ended.
* Other adapters keep their existing bootstrap and ownership protocols.
*/
export async function getConversationOwnershipBlocker(db: Db, companyId: string, issueId: string) {
const activeLease = sql`exists (select 1 from ${environmentLeases}
where ${environmentLeases.companyId} = "heartbeat_runs"."company_id"
and ${environmentLeases.heartbeatRunId} = "heartbeat_runs"."id"
and ${environmentLeases.releasedAt} is null)`;
const candidates = await db.select({ run: heartbeatRuns, activeLease }).from(heartbeatRuns)
.where(and(
eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.runtimeMode, "legacy"),
conversationRunPredicate(),
sql`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId') = ${issueId}`,
inArray(heartbeatRuns.status, ["failed", "timed_out", "interrupted", "cancelled"]),
or(isNotNull(heartbeatRuns.processPid), isNotNull(heartbeatRuns.processGroupId), activeLease),
)).orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id));
for (const { run, activeLease: leaseHeld } of candidates) {
let pidAlive = run.processPid !== null && processMayBeAlive(run.processPid);
if (pidAlive && run.processStartedAt) {
// A recycled PID cannot keep an old task blocked. An unreadable identity
// stays conservative; the original process may still own execution.
const observed = await readProcessStartedAt(run.processPid!).catch(() => null);
if (observed && new Date(observed).getTime() !== run.processStartedAt.getTime()) pidAlive = false;
}
const groupAlive = run.processGroupId !== null && processMayBeAlive(-run.processGroupId);
if (pidAlive || groupAlive || leaseHeld) {
return {
runId: run.id,
agentId: run.agentId,
cause: "execution_owner_active",
nextAction: pidAlive || groupAlive
? "The previous provider process is still running. Stop it before continuing this task."
: "The previous execution has not released its environment lease. Wait for cleanup before continuing this task.",
};
}
}
return null;
}