From 3b48e4cf5856be8febf4b0118037434fe70912f1 Mon Sep 17 00:00:00 2001 From: Dotta Date: Sat, 16 May 2026 20:31:04 -0500 Subject: [PATCH] Add graph liveness recovery regressions Co-Authored-By: Paperclip --- doc/SPEC-implementation.md | 2 +- doc/execution-semantics.md | 19 + ...eartbeat-issue-liveness-escalation.test.ts | 345 +++++++++++++++--- .../__tests__/issue-recovery-actions.test.ts | 89 +++++ server/src/services/recovery/service.ts | 11 +- 5 files changed, 413 insertions(+), 53 deletions(-) diff --git a/doc/SPEC-implementation.md b/doc/SPEC-implementation.md index 0f7152f9456..6b05b76cc51 100644 --- a/doc/SPEC-implementation.md +++ b/doc/SPEC-implementation.md @@ -37,7 +37,7 @@ These decisions close open questions from `SPEC.md` for V1. | Visibility | Full visibility to board and all agents in same company | | Communication | Tasks + comments only (no separate chat system) | | Task ownership | Single assignee; atomic checkout required for `in_progress` transition | -| Recovery | Liveness/watchdog recovery preserves explicit ownership: retry lost execution continuity where safe, otherwise open visible source-scoped recovery actions by default, use issue-backed recovery only for independent repair work, or require human escalation (see `doc/execution-semantics.md`) | +| Recovery | Liveness/watchdog recovery preserves explicit ownership: retry lost execution continuity where safe, otherwise open visible source-scoped recovery actions by default, use issue-backed recovery only for independent repair work or safety or ownership boundaries, or require human escalation (see `doc/execution-semantics.md`) | | Agent adapters | Built-in `process`, `http`, local CLI/session adapters, and OpenClaw gateway support; external adapters can also be loaded through the adapter plugin flow | | Plugin framework | Local/self-hosted early plugin runtime is in scope; cloud marketplace and packaged public distribution remain out of scope | | Auth | Mode-dependent human auth (`local_trusted` implicit board in current code; authenticated mode uses sessions), API keys for agents | diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 735c88a6e08..078371ca649 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -390,6 +390,25 @@ The recovery action stays source-scoped by default. The source issue should show Create an issue-backed recovery action only when a separate issue is the right execution object. In that fallback form, the source issue remains visible and is blocked on the recovery issue when blocking is necessary for correctness. The recovery owner must restore a live path, resolve the source issue manually, delegate real follow-up work, or record the reason the signal is a false positive. +### Issue-Graph Liveness Recovery + +`issue_graph_liveness` findings create source-scoped explicit recovery actions by default. The recovery action belongs to the source issue whose dependency path is unhealthy; it should not create a separate "unblock liveness incident" issue merely to say that the source issue is stranded. The action records the unhealthy leaf or edge, selected recovery owner, bounded evidence, idempotency fingerprint, and wake, monitor, timeout, retry, or escalation policy that will restore one valid action-path primitive. + +Default source-scoped handling must let productive work continue. If the source issue or its healthy leaf blockers already have a live action path, Paperclip records the graph finding on the source issue and does not block unrelated work. It moves the source issue to `blocked` or adds blockers only when the unhealthy leaf is a real dependency that prevents the source issue from continuing. + +Repeated scans must be idempotent. For the same source issue, recovery kind, and graph fingerprint, scans update the open source-scoped recovery action or count advisory/out-of-lookback findings; they must not create new sibling issues, new blockers, or new comments solely because the same dependency graph is still non-terminal. A resolved false-positive action stays resolved until new evidence changes the fingerprint. + +Issue-backed graph-liveness recovery is a fallback only when a separate execution object is required. It is allowed for: + +- independent repair work that can complete without mutating the source issue directly +- cross-agent repair where another assignee needs a bounded task, blockers, or child work +- long-running evaluation or investigation that needs its own monitor or retry policy +- security-sensitive review or escalation that should not run in the source issue context +- external-service observation where a separate owner watches GitHub, CI, deployment, provider, or other external state +- unsafe leaf ownership changes, such as invalid or unclear reassignment of a blocker or review participant + +When issue-backed fallback is used, the new issue is the recovery action's execution object. It must link to the source issue, share the idempotency fingerprint, and block the source only when correctness requires the source to wait. + Instance-level issue-graph liveness auto-recovery is disabled by default. When enabled, its lookback window means "dependency paths updated within the last N hours"; older findings remain advisory and are counted as outside the configured lookback instead of creating recovery actions automatically. This is an operator noise control, not the older staleness delay for determining whether a chain is old enough to surface. ### Human Escalation diff --git a/server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts b/server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts index fd335e466c9..545a8215c0d 100644 --- a/server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts +++ b/server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts @@ -346,6 +346,170 @@ describeEmbeddedPostgres("heartbeat issue graph liveness escalation", () => { }); }); + it("opens source-scoped recovery for a stalled in_review leaf without creating a synthetic issue", async () => { + await enableAutoRecovery(); + const { companyId, coderId, blockedIssueId, blockerIssueId } = await seedBlockedChain(); + await db + .update(issues) + .set({ + status: "in_review", + assigneeAgentId: coderId, + executionState: null, + }) + .where(eq(issues.id, blockerIssueId)); + + const result = await heartbeatService(db).reconcileIssueGraphLiveness(); + + expect(result.findings).toBe(1); + expect(result.escalationsCreated).toBe(1); + expect(result.sourceRecoveryActionsOpened).toBe(1); + const escalations = await db + .select() + .from(issues) + .where(and(eq(issues.companyId, companyId), eq(issues.originKind, "harness_liveness_escalation"))); + expect(escalations).toHaveLength(0); + const actions = await db + .select() + .from(issueRecoveryActions) + .where(and(eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, blockerIssueId))); + expect(actions).toHaveLength(1); + expect(actions[0]).toMatchObject({ + companyId, + sourceIssueId: blockerIssueId, + recoveryIssueId: null, + kind: "issue_graph_liveness", + cause: "issue_graph_liveness", + ownerAgentId: coderId, + fingerprint: [ + "harness_liveness_leaf", + companyId, + "in_review_without_action_path", + blockerIssueId, + ].join(":"), + }); + expect(actions[0]?.evidence).toMatchObject({ + incidentKey: [ + "harness_liveness", + companyId, + blockedIssueId, + "in_review_without_action_path", + blockerIssueId, + ].join(":"), + selectedOwnerAgentId: coderId, + }); + }); + + it("opens source-scoped recovery for an invalid review participant", async () => { + await enableAutoRecovery(); + const { companyId, coderId, blockedIssueId, blockerIssueId } = await seedBlockedChain(); + await db + .update(issues) + .set({ + status: "in_review", + assigneeAgentId: coderId, + executionState: { + status: "pending", + currentStageId: "review-1", + currentStageIndex: 0, + currentStageType: "review", + currentParticipant: { type: "agent", agentId: "00000000-0000-0000-0000-000000000404" }, + returnAssignee: { type: "agent", agentId: coderId }, + completedStageIds: [], + lastDecisionId: null, + lastDecisionOutcome: null, + }, + }) + .where(eq(issues.id, blockerIssueId)); + + const result = await heartbeatService(db).reconcileIssueGraphLiveness(); + + expect(result.findings).toBe(1); + expect(result.escalationsCreated).toBe(1); + expect(result.sourceRecoveryActionsOpened).toBe(1); + const escalations = await db + .select() + .from(issues) + .where(and(eq(issues.companyId, companyId), eq(issues.originKind, "harness_liveness_escalation"))); + expect(escalations).toHaveLength(0); + const actions = await db + .select() + .from(issueRecoveryActions) + .where(and(eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, blockerIssueId))); + expect(actions).toHaveLength(1); + expect(actions[0]).toMatchObject({ + companyId, + sourceIssueId: blockerIssueId, + recoveryIssueId: null, + kind: "issue_graph_liveness", + cause: "issue_graph_liveness", + ownerAgentId: coderId, + fingerprint: [ + "harness_liveness_leaf", + companyId, + "invalid_review_participant", + blockerIssueId, + ].join(":"), + }); + expect(actions[0]?.evidence).toMatchObject({ + incidentKey: [ + "harness_liveness", + companyId, + blockedIssueId, + "invalid_review_participant", + "00000000-0000-0000-0000-000000000404", + ].join(":"), + selectedOwnerAgentId: coderId, + }); + }); + + it("opens source-scoped recovery for a cancelled blocker edge", async () => { + await enableAutoRecovery(); + const { companyId, managerId, blockedIssueId, blockerIssueId } = await seedBlockedChain({ + blockerStatus: "cancelled", + blockerAssigneeAgentId: "coder", + }); + + const result = await heartbeatService(db).reconcileIssueGraphLiveness(); + + expect(result.findings).toBe(1); + expect(result.escalationsCreated).toBe(1); + expect(result.sourceRecoveryActionsOpened).toBe(1); + const escalations = await db + .select() + .from(issues) + .where(and(eq(issues.companyId, companyId), eq(issues.originKind, "harness_liveness_escalation"))); + expect(escalations).toHaveLength(0); + const actions = await db + .select() + .from(issueRecoveryActions) + .where(and(eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, blockerIssueId))); + expect(actions).toHaveLength(1); + expect(actions[0]).toMatchObject({ + companyId, + sourceIssueId: blockerIssueId, + recoveryIssueId: null, + kind: "issue_graph_liveness", + cause: "issue_graph_liveness", + ownerAgentId: managerId, + fingerprint: [ + "harness_liveness_leaf", + companyId, + "blocked_by_cancelled_issue", + blockerIssueId, + ].join(":"), + }); + expect(actions[0]?.evidence).toMatchObject({ + incidentKey: [ + "harness_liveness", + companyId, + blockedIssueId, + "blocked_by_cancelled_issue", + blockerIssueId, + ].join(":"), + selectedOwnerAgentId: managerId, + }); + }); + it("opens one manager-owned recovery action, preserves blockers, and records owner selection", async () => { await enableAutoRecovery(); const { companyId, managerId, blockedIssueId, blockerIssueId } = await seedBlockedChain(); @@ -735,6 +899,82 @@ describeEmbeddedPostgres("heartbeat issue graph liveness escalation", () => { }); }); + it("uses issue-backed graph-liveness fallback when the leaf already has a different active recovery action", async () => { + await enableAutoRecovery(); + const { companyId, managerId, blockedIssueId, blockerIssueId } = await seedBlockedChain(); + await db.insert(issueRecoveryActions).values({ + companyId, + sourceIssueId: blockerIssueId, + kind: "manual_recovery", + status: "active", + ownerType: "agent", + ownerAgentId: managerId, + cause: "operator_request", + fingerprint: "manual-recovery:blocks-source-scoped-graph-action", + evidence: { reason: "manual action already owns the source issue surface" }, + nextAction: "Finish the operator-requested recovery first.", + wakePolicy: { type: "wake_owner" }, + }); + + const result = await heartbeatService(db).reconcileIssueGraphLiveness(); + + expect(result.findings).toBe(1); + expect(result.escalationsCreated).toBe(1); + expect(result.sourceRecoveryActionsOpened).toBe(0); + expect(result.escalationIssueIds).toHaveLength(1); + const escalationId = result.escalationIssueIds[0]!; + const escalations = await db + .select() + .from(issues) + .where(and(eq(issues.companyId, companyId), eq(issues.originKind, "harness_liveness_escalation"))); + expect(escalations).toHaveLength(1); + expect(escalations[0]).toMatchObject({ + id: escalationId, + companyId, + parentId: blockerIssueId, + assigneeAgentId: managerId, + originId: [ + "harness_liveness", + companyId, + blockedIssueId, + "blocked_by_unassigned_issue", + blockerIssueId, + ].join(":"), + originFingerprint: [ + "harness_liveness_leaf", + companyId, + "blocked_by_unassigned_issue", + blockerIssueId, + ].join(":"), + }); + const graphActions = await db + .select() + .from(issueRecoveryActions) + .where( + and( + eq(issueRecoveryActions.companyId, companyId), + eq(issueRecoveryActions.sourceIssueId, blockerIssueId), + eq(issueRecoveryActions.kind, "issue_graph_liveness"), + ), + ); + expect(graphActions).toHaveLength(0); + const blockers = await db + .select({ blockerIssueId: issueRelations.issueId }) + .from(issueRelations) + .where(eq(issueRelations.relatedIssueId, blockedIssueId)); + expect(blockers.map((row) => row.blockerIssueId).sort()).toEqual( + [blockerIssueId, escalationId].sort(), + ); + const events = await db.select().from(activityLog).where(eq(activityLog.companyId, companyId)); + expect(events.find((event) => event.action === "issue.harness_liveness_escalation_created")?.details) + .toMatchObject({ + fallbackReason: "source_has_different_active_recovery_action", + sourceIssueId: blockedIssueId, + recoveryIssueId: blockerIssueId, + escalationIssueId: escalationId, + }); + }); + it("reuses one open recovery action for multiple dependents with the same leaf blocker", async () => { await enableAutoRecovery(); const { companyId, blockedIssueId, blockerIssueId } = await seedBlockedChain(); @@ -779,61 +1019,64 @@ describeEmbeddedPostgres("heartbeat issue graph liveness escalation", () => { expect(actions).toHaveLength(1); }); - it("does not create a fresh recovery object when the same-fingerprint action was already resolved", async () => { - await enableAutoRecovery(); - const { companyId, managerId, blockedIssueId, blockerIssueId } = await seedBlockedChain(); - const heartbeat = heartbeatService(db); - const incidentKey = [ - "harness_liveness", - companyId, - blockedIssueId, - "blocked_by_unassigned_issue", - blockerIssueId, - ].join(":"); - const resolvedAt = new Date(); - - await db.insert(issueRecoveryActions).values({ - companyId, - sourceIssueId: blockerIssueId, - kind: "issue_graph_liveness", - status: "resolved", - ownerType: "agent", - ownerAgentId: managerId, - cause: "issue_graph_liveness", - fingerprint: [ - "harness_liveness_leaf", + it.each(["resolved", "cancelled"] as const)( + "does not create a fresh recovery object when the same-fingerprint action was already %s", + async (status) => { + await enableAutoRecovery(); + const { companyId, managerId, blockedIssueId, blockerIssueId } = await seedBlockedChain(); + const heartbeat = heartbeatService(db); + const incidentKey = [ + "harness_liveness", companyId, + blockedIssueId, "blocked_by_unassigned_issue", blockerIssueId, - ].join(":"), - evidence: { incidentKey }, - nextAction: "Already handled", - outcome: "restored", - resolvedAt, - }); + ].join(":"); + const resolvedAt = new Date(); - const result = await heartbeat.reconcileIssueGraphLiveness(); + await db.insert(issueRecoveryActions).values({ + companyId, + sourceIssueId: blockerIssueId, + kind: "issue_graph_liveness", + status, + ownerType: "agent", + ownerAgentId: managerId, + cause: "issue_graph_liveness", + fingerprint: [ + "harness_liveness_leaf", + companyId, + "blocked_by_unassigned_issue", + blockerIssueId, + ].join(":"), + evidence: { incidentKey }, + nextAction: "Already handled", + outcome: "restored", + resolvedAt, + }); - expect(result.escalationsCreated).toBe(0); - expect(result.existingEscalations).toBe(0); - expect(result.resolvedUnchangedSourceRecoveryActions).toBe(1); + const result = await heartbeat.reconcileIssueGraphLiveness(); - const openEscalations = await db - .select() - .from(issues) - .where( - and( - eq(issues.companyId, companyId), - eq(issues.originKind, "harness_liveness_escalation"), - eq(issues.originId, incidentKey), - ), - ); - expect(openEscalations).toHaveLength(0); - const actions = await db - .select() - .from(issueRecoveryActions) - .where(and(eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, blockerIssueId))); - expect(actions).toHaveLength(1); - expect(actions[0]?.status).toBe("resolved"); - }); + expect(result.escalationsCreated).toBe(0); + expect(result.existingEscalations).toBe(0); + expect(result.resolvedUnchangedSourceRecoveryActions).toBe(1); + + const openEscalations = await db + .select() + .from(issues) + .where( + and( + eq(issues.companyId, companyId), + eq(issues.originKind, "harness_liveness_escalation"), + eq(issues.originId, incidentKey), + ), + ); + expect(openEscalations).toHaveLength(0); + const actions = await db + .select() + .from(issueRecoveryActions) + .where(and(eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, blockerIssueId))); + expect(actions).toHaveLength(1); + expect(actions[0]?.status).toBe(status); + }, + ); }); diff --git a/server/src/__tests__/issue-recovery-actions.test.ts b/server/src/__tests__/issue-recovery-actions.test.ts index cc533998ba0..1bfa9d48beb 100644 --- a/server/src/__tests__/issue-recovery-actions.test.ts +++ b/server/src/__tests__/issue-recovery-actions.test.ts @@ -726,6 +726,95 @@ describeEmbeddedPostgres("issue recovery actions", () => { }); }); + it("rejects same-company peer agents from resolving unassigned recovery actions they do not own", async () => { + const { companyId, managerId, coderId, sourceIssueId } = await seedCompany(); + await db.update(issues).set({ assigneeAgentId: null, status: "todo" }).where(eq(issues.id, sourceIssueId)); + const recoveryActionSvc = issueRecoveryActionService(db); + const action = await recoveryActionSvc.upsertSourceScoped({ + companyId, + sourceIssueId, + kind: "issue_graph_liveness", + ownerType: "agent", + ownerAgentId: managerId, + cause: "issue_graph_liveness", + fingerprint: "graph-liveness:owner-required", + evidence: { latestIssueStatus: "in_progress" }, + nextAction: "Only the assigned recovery owner should resolve this action.", + wakePolicy: { type: "wake_owner" }, + }); + const app = createApp({ + type: "agent", + agentId: coderId, + companyId, + runId: randomUUID(), + source: "agent_jwt", + }); + + await request(app) + .post(`/api/issues/${sourceIssueId}/recovery-actions/resolve`) + .send({ + actionId: action.id, + outcome: "restored", + sourceIssueStatus: "done", + }) + .expect(403); + + const [sourceIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId)); + expect(sourceIssue?.status).toBe("todo"); + expect(await recoveryActionSvc.getActiveForIssue(companyId, sourceIssueId)).toMatchObject({ + id: action.id, + status: "active", + ownerAgentId: managerId, + }); + }); + + it("allows the assigned recovery owner to resolve an unassigned recovery action", async () => { + const { companyId, managerId, sourceIssueId } = await seedCompany(); + await db.update(issues).set({ assigneeAgentId: null, status: "todo" }).where(eq(issues.id, sourceIssueId)); + const recoveryActionSvc = issueRecoveryActionService(db); + const action = await recoveryActionSvc.upsertSourceScoped({ + companyId, + sourceIssueId, + kind: "issue_graph_liveness", + ownerType: "agent", + ownerAgentId: managerId, + cause: "issue_graph_liveness", + fingerprint: "graph-liveness:owner-can-resolve", + evidence: { latestIssueStatus: "in_progress" }, + nextAction: "Recovery owner confirms the source issue is complete.", + wakePolicy: { type: "wake_owner" }, + }); + const app = createApp({ + type: "agent", + agentId: managerId, + companyId, + source: "agent_jwt", + }); + + const resolved = await request(app) + .post(`/api/issues/${sourceIssueId}/recovery-actions/resolve`) + .send({ + actionId: action.id, + outcome: "restored", + sourceIssueStatus: "done", + resolutionNote: "Recovery owner completed the source issue cleanup.", + }); + expect(resolved.status).toBe(200); + + expect(resolved.body.issue).toMatchObject({ + id: sourceIssueId, + status: "done", + activeRecoveryAction: null, + }); + expect(resolved.body.recoveryAction).toMatchObject({ + id: action.id, + status: "resolved", + outcome: "restored", + resolutionNote: "Recovery owner completed the source issue cleanup.", + }); + expect(await recoveryActionSvc.getActiveForIssue(companyId, sourceIssueId)).toBeNull(); + }); + it("enforces company scope when resolving recovery actions", async () => { const { companyId, managerId, sourceIssueId } = await seedCompany(); const recoveryActionSvc = issueRecoveryActionService(db); diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index 8162b38457b..0962de4ed88 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -2366,6 +2366,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) .select({ companyId: issueRecoveryActions.companyId, issueId: issueRecoveryActions.sourceIssueId, + kind: issueRecoveryActions.kind, status: issueRecoveryActions.status, }) .from(issueRecoveryActions) @@ -2415,7 +2416,15 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) })), pendingInteractions: interactionRows, pendingApprovals: approvalRows, - openRecoveryIssues: openRecoveryIssues.concat(recoveryActionRows), + openRecoveryIssues: openRecoveryIssues.concat( + recoveryActionRows + .filter((row) => row.kind === "issue_graph_liveness") + .map((row) => ({ + companyId: row.companyId, + issueId: row.issueId, + status: row.status, + })), + ), now: new Date(), }); }