fix(runner): settle accepted results after shutdown failures (#15217)

## Thinking Path > - Paperclip manages AI agents and the tasks they perform. > - The native runner saves tool results and completion reports before it releases a session. > - Large project discovery responses can exceed the durable command limit. > - A shutdown failure can leave a saved answer waiting for workspace repair. > - Recovery reused the old assessment for a different status decision, which violated a database constraint. > - This pull request bounds discovery responses and lets recovery commit the saved result after workspace repair. > - The benefit is a task that reaches its correct final status without another provider turn. ## Linked Issues or Issue Description **What happened?** A native run can save its final answer, fail during shutdown, and leave the task In Progress after workspace repair succeeds. Reconciliation tries to reuse the failed-workspace assessment for a new decision. The one-decision-per-assessment constraint rejects the write. Replaying the old decision can also retain a fresh coordinator lease. Separately, retained session cleanup only recognizes the old `adapter_failed` label. The project-list tool returns full project records, including large descriptions and workspace configuration. A large response exceeds the runner's durable command limit. The settlement diagnostic previously recorded only a failure flag. **Expected behavior** Project discovery stays within the command limit. Recovery finishes the saved result after workspace repair, releases its lease, and preserves the original error for inspection. It does not repeat provider work or relax session ownership checks. **Steps to reproduce** 1. Return large project records from `list_projects` and observe an oversized semantic result. 2. Persist an accepted native completion result, then record a shutdown failure. 3. Finalize with a failed workspace, repeat that attempt, then record successful workspace repair. 4. Reconcile the run. Before this fix, the issue stays In Progress. **Paperclip version or commit** Reproduced on `a386a599983519eb1d399f8b770bfccdb2a74762`. **Deployment mode** Self-hosted server with Paperclip Runner. Related transport work: #12208 drains queued events; #12241 resumes interrupted semantic calls. This change addresses bounded project discovery and accepted-result finalization. ## What Changed - Read bounded project summary projections from the database and return at most 50 authorized summaries with a continuation cursor and explicit description truncation. Agent and run trust boundaries narrow the database candidates; project-specific policies still receive full authorization. Only visible projects determine continuations. The default project-list API remains unchanged. - Record bounded, content-free settlement failure causes for command limits, storage errors, and rejected dispatch. - Include workspace state in assessment identity. Preserve the initial assessment for interrupted finalization, and commit replacement assessment and decision references together. - Release the coordinator lease when an existing decision is replayed, without repeating its effects. - Clear stale errors when recovery succeeds and retain them in `recoveredExecutionFailure`. - Accept both current and legacy close-failure labels in the existing exact-state cleanup path. - Add regression tests and update the tool contracts and recovery documentation. ## Verification - Red: the new project paging, settlement diagnostic, current cleanup label, and repaired-workspace regressions failed on the original implementation. - Green: protocol/catalog/tool checks (117 tests), the full project-tool and finalizer suites (47 tests), cleanup ownership cases (70 tests), and cleanup sweep cases (4 tests) pass. Database-backed pagination covers large descriptions/configuration, complete enumeration, uppercase cursors, agent/run restrictions, project-policy scope contributions, and identical results/cursors when hidden projects are added. - The initial CI failures in interrupted Board waits, contended endpoint proof, and semantic schema validation were reproduced and fixed; all affected cases pass locally. - `pnpm -r typecheck` — passed. - `pnpm build` — passed. - `pnpm test:run` — started before the review corrections; it spanned several source revisions and was stopped after reporting old-behavior and timing failures. It is not claimed green. Fresh project and recovery suites pass; the recovery suite also passes all 25 cases with the broad runner’s isolated home/config. The Slack timing case passed in isolation. Latest-head CI is the authoritative complete test matrix. - Existing authorization suite — 66 tests passed. - Latest-head CI on `8e5763d915aa6f375bdab6601996899ea01496fc` — 55 checks passed, 4 intentionally skipped, no pending or failing checks. - Greptile — 5/5, zero unresolved threads on the same commit. - `git diff --check` — passed. ## Risks - `list_projects` now returns summaries. Callers must follow `nextCursor` and use the authorized project API for full records. Candidate narrowing is only an optimization: project policy and responsible-user authorization remain authoritative. - Assessment identity changes for workspace finalization. Existing evidence remains intact; no migration is needed. - Cleanup still requires matching identities, settled tool evidence, and verified process ownership. Unknown tool outcomes remain blocked from session reuse. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, code execution, and GitHub tools. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 20:21:50 +01:00 · 2026-10-05 10:31:11 -05:00
1 parent a386a59998
commit a65ca09508
@@ -240,7 +240,7 @@ export function reconcileRetainedNativeSessionCleanups(
            ),
            // The accepted-result projector preserves a recovered close failure
            // privately after clearing the visible successful run's stale error.
-            sql`coalesce(${heartbeatRuns.errorCode}, ${heartbeatRuns.resultJson}->'recoveredExecutionFailure'->>'errorCode') = 'adapter_failed'`,
+            sql`coalesce(${heartbeatRuns.errorCode}, ${heartbeatRuns.resultJson}->'recoveredExecutionFailure'->>'errorCode') in ('adapter_failed', 'provider_transport_failed')`,
            sql`coalesce(${heartbeatRuns.error}, ${heartbeatRuns.resultJson}->'recoveredExecutionFailure'->>'error') = 'provider_transport_failed: runner did not durably suspend before checkpoint'`,
            sql`not (${nativeRunFinalizations.recoveryHistory} @> '[{"kind":"native_cleanup_runner_epoch"}]'::jsonb)`,
            sql`not (${nativeRunFinalizations.recoveryHistory} @> '[{"kind":"native_cleanup_source_archive","phase":"operator_required"}]'::jsonb)`,
@@ -12,6 +12,7 @@ import {
  nativeRunFinalizations,
  nativeRunResults,
  workAssessments,
+  statusDecisions,
  issueRecoveryActions,
  agentWakeupRequests,
  workspaceOperations,
@@ -184,6 +185,42 @@ describeEmbeddedPostgres("native run finalizer / status decision committer — a
    });
  }

+  it("finishes an accepted result after shutdown failed and workspace repair succeeded", async () => {
+    const fixture = await seedNativeRun();
+    await db.update(completionContracts).set({ risk: "low", completionAuthority: "agent_claim_policy",
+      contractJson: { revision: CONTROL_PLANE_CONFORMANCE_RESULT.completionClaim.contractRevision,
+        objective: "Finish the work", criteria: [{ id: "objective", requirement: "Complete the task" }] } })
+      .where(eq(completionContracts.id, fixture.contractId));
+    await driveToCompleteResult(fixture);
+    await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(),
+      errorCode: "provider_transport_failed", error: "provider_transport_failed: runner did not durably suspend before checkpoint" })
+      .where(eq(heartbeatRuns.id, fixture.runId));
+    await db.update(nativeRunFinalizations).set({ phase: "retryable_failure", leaseOwner: null,
+      leaseExpiresAt: null, nextAttemptAt: null }).where(eq(nativeRunFinalizations.runId, fixture.runId));
+    const failed = await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "failed" });
+    // An unchanged failed workspace must replay its existing decision.
+    await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "failed" });
+    expect(await db.select().from(statusDecisions).where(eq(statusDecisions.runId, fixture.runId))).toHaveLength(1);
+    await db.insert(workspaceOperations).values({ companyId, issueId: fixture.issueId,
+      heartbeatRunId: fixture.runId, phase: "workspace_finalize", status: "succeeded" });
+    await db.update(nativeRunFinalizations).set({ nextAttemptAt: null }).where(eq(nativeRunFinalizations.runId, fixture.runId));
+    await reconcileNativeFinalizations(db, [fixture.runId]);
+    expect((await db.select().from(issues).where(eq(issues.id, fixture.issueId)))[0]).toMatchObject({
+      status: "done", executionRunId: null, checkoutRunId: null,
+    });
+    expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)))[0]).toMatchObject({
+      status: "succeeded", errorCode: null, error: null,
+      resultJson: { recoveredExecutionFailure: { errorCode: "provider_transport_failed" } },
+    });
+    const assessments = await db.select().from(workAssessments).where(eq(workAssessments.runId, fixture.runId));
+    expect(assessments).toHaveLength(2);
+    expect(assessments.find(row => row.id !== failed.assessmentId)?.supersedesAssessmentId).toBe(failed.assessmentId);
+    await reconcileNativeFinalizations(db, [fixture.runId]);
+    expect(await db.select().from(statusDecisions).where(eq(statusDecisions.runId, fixture.runId))).toHaveLength(2);
+    expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, fixture.runId))).toHaveLength(1);
+    expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId))).toHaveLength(0);
+  });
+
  it("emits exactly one agent.task_run event when the native finalizer commits a terminal status", async () => {
    const fixture = await seedNativeRun();
    await driveToCompleteResult(fixture);
@@ -615,6 +615,16 @@ async function resolveCommittedFinalizationRecovery(db: Db, run: typeof heartbea
  });
 }

+function recoveredExecutionFailureMetadata() {
+  // Read the row being updated so a concurrent cleanup diagnostic is retained.
+  return sql`case when ${heartbeatRuns.error} is not null or ${heartbeatRuns.errorCode} is not null
+    then jsonb_build_object('recoveredExecutionFailure', jsonb_build_object(
+      'schema', 'paperclip.recovered_execution_failure.v1',
+      'errorCode', ${heartbeatRuns.errorCode}, 'error', ${heartbeatRuns.error},
+      'observedAt', ${heartbeatRuns.updatedAt}
+    )) else '{}'::jsonb end`;
+}
+
 async function projectCommittedRun(input: {
  db: Db;
  run: typeof heartbeatRuns.$inferSelect;
@@ -658,18 +668,9 @@ async function projectCommittedRun(input: {
            // Capture the row being updated, not the earlier admission read:
            // cleanup may have recorded a new diagnostic in the meantime.
            // Other result metadata and all physical-owner evidence stay put.
-            resultJson: sql`case
-              when ${heartbeatRuns.error} is not null or ${heartbeatRuns.errorCode} is not null
-              then coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || jsonb_build_object(
-                'recoveredExecutionFailure', jsonb_build_object(
-                  'schema', 'paperclip.recovered_execution_failure.v1',
-                  'errorCode', ${heartbeatRuns.errorCode},
-                  'error', ${heartbeatRuns.error},
-                  'observedAt', ${heartbeatRuns.updatedAt}
-                )
-              )
-              else coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb)
-            end || jsonb_build_object('finalizationPhase', 'committed', 'failureCode', null, 'originalFailureCode', null, 'nextAttemptAt', null)`,
+            resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb)
+              || ${recoveredExecutionFailureMetadata()}
+              || jsonb_build_object('finalizationPhase', 'committed', 'failureCode', null, 'originalFailureCode', null, 'nextAttemptAt', null)`,
          }
        : { resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || jsonb_build_object('finalizationPhase', 'committed', 'failureCode', null, 'originalFailureCode', null, 'nextAttemptAt', null)` }),
      updatedAt: now,
@@ -1197,7 +1198,7 @@ export async function finalizeNativeRun(input: {
  // One follow-up may repair an incomplete report. Repeated incomplete results
  // require a visible recovery action instead of an unbounded wake loop.
  const allowIncompleteContinuation = record(sourceWake?.payload).continuationIdempotencyKey !== "native-completion-incomplete";
-  let supersedesAssessmentId: string | null = null;
+  let supersedesAssessmentId: string | null = coordinator.assessmentId;
  for (let attempt = 0; attempt < 3; attempt += 1) {
    const authoritativeIssue = await input.db
      .select()
@@ -1322,12 +1323,18 @@ export async function finalizeNativeRun(input: {
      policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
      assessment,
      supersedesAssessmentId,
+      workspaceFinalizeStatus: input.workspaceFinalizeStatus,
    });
    await input.db
      .update(nativeRunFinalizations)
      .set({
        phase: "arbitrating",
-        assessmentId: assessmentRow.id,
+        // The old decision refers to its original assessment through a
+        // composite foreign key. Replace this pair together in the status
+        // committer, after the new decision and its effects are durable.
+        // Before any decision exists, retain the assessment for crash recovery.
+        assessmentId: sql`case when ${nativeRunFinalizations.decisionId} is null
+          then ${assessmentRow.id}::uuid else ${nativeRunFinalizations.assessmentId} end`,
        updatedAt: new Date(),
      })
      .where(
@@ -1391,6 +1398,40 @@ export async function finalizeNativeRun(input: {
      const alreadyEmittedByCommittedDecision = decision.effects.some(
        (effect) => effect.kind === "cancel_continuations",
      );
+      const clearExecutionFailure = input.projectRunStatus && finalizationPhase === "committed" && terminalState === "succeeded";
+      const finalizationMetadata = {
+        finalizationPhase,
+        ...(finalizationPhase === "committed" ? { failureCode: null, originalFailureCode: null, nextAttemptAt: null } : {}),
+        assessmentId: assessmentRow.id,
+        decisionId: committed.decision.id,
+        authoritativeDecision: decision.toStatus,
+        finalizationPolicyVersion: decision.policyVersion,
+        finalizationReasonCode: decision.reasonCode,
+        // Only the locked status transaction can mint this presentation
+        // proof. Never carry a runner-provided marker forward.
+        externalChatReviewPresentation: record(
+          committed.decision.decisionJson,
+        ).externalChatReviewPresentation
+          ? {
+              ...record(
+                record(committed.decision.decisionJson)
+                  .externalChatReviewPresentation,
+              ),
+              decisionId: committed.decision.id,
+            }
+          : null,
+        ...(record(committed.decision.decisionJson)
+          .externalChatReviewPresentation
+          ? { nativeResult: result }
+          : {}),
+        verificationCaveats: assessment.verificationCaveats,
+        ignoredAttentionRequests: assessment.ignoredAttentionRequests,
+        issueStatusBefore: authoritativeIssue.status,
+        issueStatusAfter: committed.issue.status,
+        statusVersionBefore: Number(authoritativeIssue.statusVersion),
+        statusVersionAfter: Number(committed.issue.statusVersion),
+        workspaceFinalizeStatus: input.workspaceFinalizeStatus,
+      };
      const [updatedRun] = await input.db
        .update(heartbeatRuns)
        .set({
@@ -1408,40 +1449,10 @@ export async function finalizeNativeRun(input: {
            : {}),
          nativePhase: finalizationPhase,
          nativePhaseUpdatedAt: now,
-          resultJson: {
-            ...record(run.resultJson),
-            finalizationPhase,
-            ...(finalizationPhase === "committed" ? { failureCode: null, originalFailureCode: null, nextAttemptAt: null } : {}),
-            assessmentId: assessmentRow.id,
-            decisionId: committed.decision.id,
-            authoritativeDecision: decision.toStatus,
-            finalizationPolicyVersion: decision.policyVersion,
-            finalizationReasonCode: decision.reasonCode,
-            // Only the locked status transaction can mint this presentation
-            // proof. Never carry a runner-provided marker forward.
-            externalChatReviewPresentation: record(
-              committed.decision.decisionJson,
-            ).externalChatReviewPresentation
-              ? {
-                  ...record(
-                    record(committed.decision.decisionJson)
-                      .externalChatReviewPresentation,
-                  ),
-                  decisionId: committed.decision.id,
-                }
-              : null,
-            ...(record(committed.decision.decisionJson)
-              .externalChatReviewPresentation
-              ? { nativeResult: result }
-              : {}),
-            verificationCaveats: assessment.verificationCaveats,
-            ignoredAttentionRequests: assessment.ignoredAttentionRequests,
-            issueStatusBefore: authoritativeIssue.status,
-            issueStatusAfter: committed.issue.status,
-            statusVersionBefore: Number(authoritativeIssue.statusVersion),
-            statusVersionAfter: Number(committed.issue.statusVersion),
-            workspaceFinalizeStatus: input.workspaceFinalizeStatus,
-          },
+          ...(clearExecutionFailure ? { error: null, errorCode: null } : {}),
+          resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb)
+            || ${JSON.stringify(finalizationMetadata)}::jsonb
+            || ${clearExecutionFailure ? recoveredExecutionFailureMetadata() : sql`'{}'::jsonb`}`,
          updatedAt: now,
        })
        .where(eq(heartbeatRuns.id, run.id))
@@ -2641,6 +2641,7 @@ const execution = {
 describe("retained native cleanup activation", () => {
  it.each([
    "settled",
+    "provider_transport_failed",
    "canonical_source",
    "canonical_live_owner",
    "canonical_foreign_owner",
@@ -2955,7 +2956,7 @@ describe("retained native cleanup activation", () => {
      processGroupId: mode.endsWith("live_owner") ? process.pid : 99_999_999,
      completionContractId: "contract",
      completionContractSha256: "sha",
-      errorCode: "adapter_failed",
+      errorCode: mode === "provider_transport_failed" ? "provider_transport_failed" : "adapter_failed",
      error:
        "provider_transport_failed: runner did not durably suspend before checkpoint",
      runnerProfileJson: {
@@ -3822,6 +3823,7 @@ describe("retained native cleanup activation", () => {
        "provider_home",
        "legacy_copy",
        "settled",
+        "provider_transport_failed",
        "activation_commit_stalled",
        "empty_root",
        "distinct_provider_account",
@@ -2829,7 +2829,7 @@ export async function reconcileRetainedNativeSessionCleanup(
      const errorCode = run.errorCode ?? failure.errorCode;
      const error = run.error ?? failure.error;
      if (
-        errorCode !== "adapter_failed" ||
+        !["adapter_failed", "provider_transport_failed"].includes(String(errorCode)) ||
        error !==
          "provider_transport_failed: runner did not durably suspend before checkpoint" ||
        execution.binding.companyId !== run.companyId ||
@@ -1746,6 +1746,34 @@ export async function commitNativeStatusDecision(input: {
        throw error;
      }
    }
+    const recordCoordinatorDecision = async (decisionId: string) => {
+      const finalizationError = input.decision.effects.find(
+        effect => effect.kind === "record_finalization_error",
+      );
+      await tx
+        .update(nativeRunFinalizations)
+        .set({
+          phase: finalizationError ? "retryable_failure" : "committed",
+          assessmentId: input.assessmentId,
+          decisionId,
+          leaseOwner: null,
+          leaseExpiresAt: null,
+          failureCode: finalizationError?.cause ?? null,
+          failureDetail: finalizationError
+            ? {
+                originalFailureCode: finalizationError.cause,
+                recoveryOwner: {
+                  kind: "agent",
+                  agentId: finalizationError.agentId,
+                },
+                nextAction: finalizationError.nextAction,
+              }
+            : null,
+          nextAttemptAt: finalizationError ? new Date(Date.now() + 30_000) : null,
+          updatedAt: new Date(),
+        })
+        .where(eq(nativeRunFinalizations.runId, input.runId));
+    };
    const decisionJson = {
      statusAction: input.decision.statusAction,
      toStatus: input.decision.toStatus,
@@ -1789,6 +1817,9 @@ export async function commitNativeStatusDecision(input: {
      decisionRow?.applicationState === "applied" ||
      decisionRow?.applicationState === "proposed"
    ) {
+      // The effects already committed, but this retry owns a fresh coordinator
+      // lease. Release it and restore the retry deadline without replaying effects.
+      await recordCoordinatorDecision(decisionRow.id);
      return { decision: decisionRow, issue, replayed: true };
    }
    if (!decisionRow) {
@@ -2042,9 +2073,6 @@ export async function commitNativeStatusDecision(input: {
    const shadowOnly = input.decision.effects.some(
      (effect) => effect.kind === "record_shadow_decision",
    );
-    const finalizationError = input.decision.effects.find(
-      (effect) => effect.kind === "record_finalization_error",
-    );
    const applicationState = shadowOnly ? "proposed" : "applied";
    await tx
      .update(statusDecisions)
@@ -2053,29 +2081,7 @@ export async function commitNativeStatusDecision(input: {
        appliedAt: shadowOnly ? null : new Date(),
      })
      .where(eq(statusDecisions.id, decisionRow.id));
-    await tx
-      .update(nativeRunFinalizations)
-      .set({
-        phase: finalizationError ? "retryable_failure" : "committed",
-        assessmentId: input.assessmentId,
-        decisionId: decisionRow.id,
-        leaseOwner: null,
-        leaseExpiresAt: null,
-        failureCode: finalizationError?.cause ?? null,
-        failureDetail: finalizationError
-          ? {
-              originalFailureCode: finalizationError.cause,
-              recoveryOwner: {
-                kind: "agent",
-                agentId: finalizationError.agentId,
-              },
-              nextAction: finalizationError.nextAction,
-            }
-          : null,
-        nextAttemptAt: finalizationError ? new Date(Date.now() + 30_000) : null,
-        updatedAt: new Date(),
-      })
-      .where(eq(nativeRunFinalizations.runId, input.runId));
+    await recordCoordinatorDecision(decisionRow.id);
    if (externalChatReviewPresentation && input.reviewResponsePresentation) {
      await restoreNativeChatReviewPresentationInTransaction(
        tx as unknown as Db,
@@ -19,6 +19,8 @@ export async function recordNativeWorkAssessment(input: {
  priorDecisionId: string | null;
  policyVersion: string;
  assessment: NativeEvidenceAssessment;
+  /** Workspace repair changes the status decision even when work evidence is unchanged. */
+  workspaceFinalizeStatus?: "succeeded" | "failed";
  supersedesAssessmentId?: string | null;
 }) {
  const assessmentJson = input.assessment as unknown as Record<string, unknown>;
@@ -36,6 +38,7 @@ export async function recordNativeWorkAssessment(input: {
    triggerCapability: "server_native_finalizer",
    policyVersion: input.policyVersion,
    assessment: assessmentJson,
+    ...(input.workspaceFinalizeStatus ? { workspaceFinalizeStatus: input.workspaceFinalizeStatus } : {}),
  });
  const existing = await input.db.select().from(workAssessments).where(and(
    eq(workAssessments.issueId, input.issueId),