diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index ad2b42ec320..25d01ce03f5 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1501,6 +1501,27 @@ Historical legacy interruption holds for conversational adapters no longer block The server projection remains available for diagnostics. Normal working, finishing, and interaction waits add no badges or cards to task lists or feeds. Active transcript headers keep saying Working during automatic retry and execution confirmation; attempts, causes, and recovery decisions belong in the run log. Recovery uses the existing transcript and run log rather than adding a reconciliation form. A cancelled run that never started says “Couldn't start” instead of implying that the agent answered. +Native workspace export failures offer **Inspect recovery** on the run page. +The explicit read lists recovery actions for that exact task and source run; +selecting an action observes its current recorded-source identity. It includes +resolved actions whose retained work may still need attention. This observation +does not prove provider termination, local descriptor integrity, or successful +file preservation, and it does not explain an earlier owner state retroactively. +Recorded system recovery failures retain their original attempt and intent. +New source-binding failures record a closed reason and, for a rejected lease +policy, an allowlisted policy value. Neither detail includes paths, provider +allocation identifiers, credentials, or raw provider errors. + +The company/task-authorized `GET /api/issues/{id}/recovery-actions` accepts +`inspectSourceRunId` to list at most 20 matching actions and an optional +`inspectSourceActionId` to inspect one exact action. This opt-in path bypasses +read revalidation and performs no recovery writes, provider calls, filesystem +inspection, or retries. It reads the existing source verifier without changing +its guards. Inspection of more than eight recorded leases returns +`not_inspected/source_limit` before per-lease lookup; physical recovery retains +its existing acceptance rules. The observation spans multiple database reads +and grants no execution or preservation authority. + ### Codex startup and provider state Paperclip trusts the server-selected startup execution root in the isolated diff --git a/maps/agent-access-map/activity.md b/maps/agent-access-map/activity.md index f4275793426..5e0bea6f4a2 100644 --- a/maps/agent-access-map/activity.md +++ b/maps/agent-access-map/activity.md @@ -124,6 +124,18 @@ activity links and the original task. Existing credential redaction remains. Bridge admission grants no cancellation, retry, recovery resolution or provider-trace administration authority. +The recovery-actions GET also accepts `inspectSourceRunId` and optional +`inspectSourceActionId`. The explicit path requires the same readable task and +exact company/task/run/action binding, including resolved retained actions. +Run-only selection lists at most 20 actions; action selection observes current +recorded-source identity with an eight-lease ceiling. This path does not run +read revalidation, mutate recovery/Retry state, call providers or inspect files. +Only closed reason/policy values supplement the existing recorded failure; +the observation is neither historical cause nor permission to execute. Local +[HTTP/database regressions](../../server/src/services/workspace-recovery-source.test.ts) +cover exact selection, foreign action denial, reused allocations, redaction, +bounded lookup and unchanged persisted state. Deployed agent use is unverified. + Evidence: `source-only` — scoped review of `f10210990a3c9cad1a4359318388f237de3169dd`: [run routes](../../server/src/routes/agents.ts), [task links](../../server/src/routes/activity.ts), diff --git a/maps/feature-map/activity.md b/maps/feature-map/activity.md index 7c2c03524e8..0cb2ada22bd 100644 --- a/maps/feature-map/activity.md +++ b/maps/feature-map/activity.md @@ -47,6 +47,17 @@ Automated: [activity service](../../server/src/__tests__/activity-service.test.t Manual: Find a known mutation by actor/resource and a known run by status. Change filters, load more, and follow the source links. Reload the chosen view and verify no company leakage or duplicate events. Inspect a routine run from its audit entry. +On a native workspace export failure's run page, choose **Inspect recovery**, +then **Inspect current source** on the matching action. Compare the recorded +failure with the current recorded-source identity. Inspection is read-only and +does not confirm provider termination or file preservation. Resolved retained +actions remain inspectable. The control makes no inspection request until used. +[Source/HTTP regressions](../../server/src/services/workspace-recovery-source.test.ts) +cover current identity, rejected policy, foreign/reused allocation, bounded +lookup, redaction and unchanged recovery state; [component regressions](../../ui/src/components/RunWorkspaceRecoverySurface.test.tsx) +cover explicit selection, loading, failure and identity-only success. These +checks do not establish recovery of a deployed task. + Case detail's **View in Audit** opens `/activity?entityType=case&entityId=`. The Activity filters include **Cases** actions and **Case** entities; the Case search accepts a UUID or human identifier such as `CAS-C1` (Enter or blur applies diff --git a/maps/feature-map/recovery.md b/maps/feature-map/recovery.md index ef5c2dc136d..f56db5f99ae 100644 --- a/maps/feature-map/recovery.md +++ b/maps/feature-map/recovery.md @@ -78,10 +78,13 @@ fresh workspaces, bounded retries, historical adoption and service incidents. ### `run-detail` Open the agent's run (`/agents/:agentId/runs/:runId`) from activity or the task's -run link. The run page hosts workspace recovery controls only for a failed run +run link. The run page hosts workspace repair controls only for a failed run with error code `workspace_validation_failed` and a linked source task that still has a live `workspace_validation` recovery action. Use the source task for other -stopped-run recovery checks. +stopped-run recovery checks. Native workspace export failures also offer the +read-only **Inspect recovery** control described in the +[run diagnostic recipe](activity.md#activity-audit-1). It observes one selected +source's recorded identity and does not retry, repair, or authorize execution. ### `operator-retry` diff --git a/packages/shared/src/index.ts b/packages/shared/src/index.ts index 5d760d874c8..c698a2904aa 100644 --- a/packages/shared/src/index.ts +++ b/packages/shared/src/index.ts @@ -1611,7 +1611,8 @@ export type { QuotaWindow, ProviderQuotaResult, } from "./types/index.js"; -export { WORKSPACE_READINESS_STATES } from "./types/index.js"; +export { WORKSPACE_READINESS_STATES, WORKSPACE_SOURCE_VERIFICATION_REASONS, + type WorkspaceSourceVerificationFailure, type WorkspaceSourceInspection } from "./types/index.js"; export { COMPANY_SEARCH_EXTRACT_KINDS, COMPANY_SEARCH_EXTRACT_SCOPES, diff --git a/packages/shared/src/types/index.ts b/packages/shared/src/types/index.ts index 9da3eff2ee2..fe0ed6ec356 100644 --- a/packages/shared/src/types/index.ts +++ b/packages/shared/src/types/index.ts @@ -1,4 +1,6 @@ export { decisionEffectTargetIssueIds } from "./decision.js"; +export { WORKSPACE_SOURCE_VERIFICATION_REASONS, type WorkspaceSourceVerificationFailure, + type WorkspaceSourceInspection } from "./workspace-source-verification.js"; export { NATIVE_FINALIZATION_SCHEMA, type NativeFinalizationResult, diff --git a/packages/shared/src/types/workspace-source-verification.ts b/packages/shared/src/types/workspace-source-verification.ts new file mode 100644 index 00000000000..647f5c88175 --- /dev/null +++ b/packages/shared/src/types/workspace-source-verification.ts @@ -0,0 +1,29 @@ +import type { EnvironmentLeasePolicy } from "../constants.js"; + +export const WORKSPACE_SOURCE_VERIFICATION_REASONS = [ + "source_run_missing", "recovery_action_missing", "recovery_action_run_mismatch", "source_run_not_terminal", + "source_issue_mismatch", "native_reference_invalid", "restore_reference_invalid", "historical_reference_invalid", + "lease_reference_invalid", "source_lease_not_bound", "lease_issue_mismatch", "lease_environment_missing", + "lease_allocation_missing", "lease_provider_invalid", "lease_policy_not_retained", "lease_cwd_invalid", + "allocation_reference_mismatch", "cwd_reference_mismatch", "workspace_reference_mismatch", + "recovery_lease_not_selected", "recovery_environment_mismatch", "recovery_lease_not_claimed", + "recovery_owner_mismatch", "recovery_stop_intent_missing", "local_descriptor_invalid", "execution_target_invalid", + "recovery_stop_intent_invalid", "reopened_allocation_mismatch", "reopened_cwd_mismatch", +] as const; + +export interface WorkspaceSourceVerificationFailure { + reason: (typeof WORKSPACE_SOURCE_VERIFICATION_REASONS)[number]; + observedLeasePolicy?: EnvironmentLeasePolicy | "unrecognized"; +} + +export type WorkspaceSourceInspection = { + schema: "paperclip.workspace-source-inspection.v1"; + actionId: string; + runId: string; + observedAt: string; + scope: "recorded_source_identity"; +} & ( + | { status: "verified" } + | { status: "blocked"; failureCode: string; sourceVerification?: WorkspaceSourceVerificationFailure } + | { status: "not_inspected"; reason: "source_limit" } +); diff --git a/server/src/routes/issues.ts b/server/src/routes/issues.ts index dccefa9a355..9c4beef7251 100644 --- a/server/src/routes/issues.ts +++ b/server/src/routes/issues.ts @@ -29,6 +29,7 @@ import { reconstructWorkspaceSchema } from "@paperclipai/shared"; import { queuedInteractionId, readQueuedInteractionResponse, hasQueuedInteractionResponse } from "../services/queued-interaction-response.js"; import { deliverConversationComments, isConversation } from "../services/agent-conversations.js"; import { issueRecoveryActionReadModel } from "../services/issue-recovery-actions.js"; +import { inspectWorkspaceSource } from "../services/workspace-source-inspection.js"; import { getExecutionBlocker } from "../services/execution-blocker.js"; import { EXECUTION_RECONCILIATION_CAUSES, requiresExecutionReconciliation } from "@paperclipai/shared"; import { @@ -9590,6 +9591,25 @@ export function issueRoutes( ); if (!issue) return; if (!(await assertIssueReadAllowed(req, res, issue))) return; + if (req.query.inspectSourceRunId !== undefined || req.query.inspectSourceActionId !== undefined) { + const selected = z.object({ inspectSourceRunId: z.string().uuid(), inspectSourceActionId: z.string().uuid().optional() }) + .safeParse(req.query); + if (!selected.success) throw badRequest("A valid source run and optional recovery action are required."); + const { inspectSourceRunId: runId, inspectSourceActionId: actionId } = selected.data; + const rows = await db.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, issue.companyId), eq(issueRecoveryActions.sourceIssueId, issue.id), + sql`${issueRecoveryActions.evidence}->>'runId' = ${runId}`, + actionId ? eq(issueRecoveryActions.id, actionId) : undefined, + )).orderBy(desc(issueRecoveryActions.createdAt)).limit(actionId ? 1 : 20); + if (actionId && !rows[0]) throw notFound("Recovery source not found."); + const actions = rows.map(issueRecoveryActionReadModel); + const sourceInspection = actionId ? await inspectWorkspaceSource(db, { + companyId: issue.companyId, issueId: issue.id, runId, actionId, + }) : undefined; + res.json({ active: actions.find(action => ["active", "escalated"].includes(action.status)) ?? null, + actions, ...(sourceInspection ? { sourceInspection } : {}) }); + return; + } const active = await revalidateActiveSourceRecoveryForRead({ issue, trigger: "read_projection", diff --git a/server/src/services/system-recovery-state.test.ts b/server/src/services/system-recovery-state.test.ts index cab57be2639..96357950a6f 100644 --- a/server/src/services/system-recovery-state.test.ts +++ b/server/src/services/system-recovery-state.test.ts @@ -45,6 +45,19 @@ describe("durable system recovery budget", () => { expect(state).toMatchObject({ state: "exhausted", nextAttemptAt: null, failure: { code: "source_binding_invalid", retryClass: "service_required" } }); }); + it("persists only allowlisted source-verification evidence without changing the failure budget", () => { + const state = createSystemRecovery(randomUUID()); + const failure = new SystemRecoveryFailure("source_binding_invalid", "service_required", undefined, + { reason: "lease_policy_not_retained", observedLeasePolicy: "reuse_by_environment", token: "secret" } as never); + const result = failSystemRecovery(state, failure, { now: 1_000 }); + expect(readSystemRecovery({ systemRecovery: result })?.failure?.sourceVerification).toEqual({ + reason: "lease_policy_not_retained", observedLeasePolicy: "reuse_by_environment" }); + expect(JSON.stringify(result)).not.toContain("secret"); + expect(result).toMatchObject({ state: "exhausted", attempts: 0, nextAttemptAt: null, intentId: state.intentId }); + const invalid = failSystemRecovery(state, new SystemRecoveryFailure("source_binding_invalid", "service_required", undefined, + { reason: "arbitrary secret", observedLeasePolicy: "private policy" } as never)); + expect(invalid.failure?.sourceVerification).toBeUndefined(); + }); it("reads prior incident metadata without losing the local recovery budget or failure", () => { const state = failSystemRecovery(createSystemRecovery(randomUUID()), new SystemRecoveryFailure("source_unavailable", "service_required"), { now: 1_000 }); diff --git a/server/src/services/system-recovery-state.ts b/server/src/services/system-recovery-state.ts index 5dd8fdd5aed..d29e20f4150 100644 --- a/server/src/services/system-recovery-state.ts +++ b/server/src/services/system-recovery-state.ts @@ -1,5 +1,7 @@ import { createHash, randomUUID } from "node:crypto"; import { z } from "zod"; +import { ENVIRONMENT_LEASE_POLICIES, WORKSPACE_SOURCE_VERIFICATION_REASONS, + type WorkspaceSourceVerificationFailure } from "@paperclipai/shared"; export const SYSTEM_RECOVERY_POLICY_VERSION = 1; export const SYSTEM_RECOVERY_MAX_RETRIES = 5; @@ -21,11 +23,17 @@ export const recoveryRetryClasses = [ export type RecoveryRetryClass = typeof recoveryRetryClasses[number]; export class SystemRecoveryFailure extends Error { - constructor(readonly code: RecoveryFailureCode, readonly retryClass: RecoveryRetryClass, message = code) { + constructor(readonly code: RecoveryFailureCode, readonly retryClass: RecoveryRetryClass, message = code, + readonly sourceVerification?: WorkspaceSourceVerificationFailure) { super(message); } } +export const workspaceSourceVerificationFailureSchema = z.object({ + reason: z.enum(WORKSPACE_SOURCE_VERIFICATION_REASONS), + observedLeasePolicy: z.enum([...ENVIRONMENT_LEASE_POLICIES, "unrecognized"]).optional(), +}); + export const systemRecoverySchema = z.object({ schema: z.literal("paperclip.system-recovery.v1"), policyVersion: z.literal(SYSTEM_RECOVERY_POLICY_VERSION), @@ -46,6 +54,7 @@ export const systemRecoverySchema = z.object({ code: z.enum(recoveryFailureCodes), retryClass: z.enum(recoveryRetryClasses), evidenceFingerprint: z.string(), + sourceVerification: workspaceSourceVerificationFailureSchema.optional(), }).nullable(), preservationOperationIds: z.array(z.string().uuid()), freshAttemptRunId: z.string().uuid().nullable(), @@ -94,11 +103,13 @@ export function failSystemRecovery( const nextAttemptAt = Math.max(now + delay, options.retryAfterAt ?? 0); const retryable = ["retry_transient", "recheck_evidence"].includes(failure.retryClass); const retry = retryable && retryNumber <= SYSTEM_RECOVERY_MAX_RETRIES && nextAttemptAt < deadlineAt; + const sourceVerification = workspaceSourceVerificationFailureSchema.safeParse(failure.sourceVerification); return { ...state, revision: state.revision + 1, firstFailureAt, deadlineAt, providerCooldownAt: options.retryAfterAt ?? null, state: retry ? failure.retryClass === "recheck_evidence" ? "awaiting_evidence" : "scheduled" : "exhausted", nextAttemptAt: retry ? nextAttemptAt : null, failure: { code: failure.code, retryClass: failure.retryClass, + ...(sourceVerification.success ? { sourceVerification: sourceVerification.data } : {}), evidenceFingerprint: options.evidenceFingerprint ?? createHash("sha256").update(`${state.sourceRunId}:${failure.code}`).digest("hex") }, }; } diff --git a/server/src/services/workspace-recovery-source.test.ts b/server/src/services/workspace-recovery-source.test.ts new file mode 100644 index 00000000000..92e62fa74d9 --- /dev/null +++ b/server/src/services/workspace-recovery-source.test.ts @@ -0,0 +1,183 @@ +import { randomUUID } from "node:crypto"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import express from "express"; +import request from "supertest"; +import { eq, sql } from "drizzle-orm"; +import { activityLog, agents, agentWakeupRequests, applyPendingMigrations, companies, createDb, environmentLeases, environments, + heartbeatRuns, issueRecoveryActions, issues, workspaceOperations } from "@paperclipai/db"; +import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js"; +import { inspectWorkspaceRecoverySource } from "./workspace-recovery-source.js"; +import { createSystemRecovery, failSystemRecovery, SystemRecoveryFailure } from "./system-recovery-state.js"; +import { inspectWorkspaceSource } from "./workspace-source-inspection.js"; +import { issueRoutes } from "../routes/issues.js"; +import { errorHandler } from "../middleware/error-handler.js"; + +describe("workspace source verification diagnostics", () => { + let db: ReturnType; + let temporary: Awaited>; + beforeAll(async () => { + const external = resolveExternalTestDatabaseUrl("PAPERCLIP_WORKSPACE_RECOVERY_TEST_DATABASE_URL"); + if (external) { + await applyPendingMigrations(external); + temporary = { connectionString: external, cleanup: async () => {} }; + } else temporary = await startEmbeddedPostgresTestDatabase("source-verification-"); + db = createDb(temporary.connectionString); + if (external) await db.execute(sql`TRUNCATE TABLE companies CASCADE`); + }, 30_000); + afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); }); + + async function seed() { + const companyId = randomUUID(), issueId = randomUUID(), runId = randomUUID(), actionId = randomUUID(), agentId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Source verification", issuePrefix: `S${companyId.slice(0, 6)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Worker", adapterType: "paperclip_runner" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Preserve the source", status: "blocked", assigneeAgentId: agentId }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, nativeIssueId: issueId, + runtimeMode: "native", status: "failed", finishedAt: new Date() }); + const [environment] = await db.insert(environments).values({ name: runId, driver: "sandbox" }).returning(); + const [lease] = await db.insert(environmentLeases).values({ companyId, issueId, heartbeatRunId: runId, + environmentId: environment.id, provider: "daytona", providerLeaseId: randomUUID(), leasePolicy: "retain_on_failure", + status: "released", cleanupStatus: "success", metadata: { remoteCwd: "/workspace", token: "secret-must-not-appear" } }).returning(); + const reference = { schema: "paperclip.native-workspace-sync/v3", state: "prepared", leaseId: lease.id, + providerLeaseId: lease.providerLeaseId, workspaceId: issueId, remoteCwd: "/workspace", resourceDisposition: "keep_running", + descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null }; + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceSync: reference } }).where(eq(heartbeatRuns.id, runId)); + const systemRecovery = failSystemRecovery(createSystemRecovery(runId, randomUUID()), + new SystemRecoveryFailure("source_binding_invalid", "service_required")); + await db.insert(issueRecoveryActions).values({ id: actionId, companyId, sourceIssueId: issueId, + kind: "active_run_watchdog", ownerType: "board", cause: "native_workspace_sync_out_retry_exhausted", fingerprint: runId, + nextAction: "Inspect the retained source", evidence: { runId, systemRecovery, + workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId, status: "blocked", + stage: "verifying_stop", attempt: 1, attemptLimit: 6, requestIds: [] } } }); + return { db, companyId, issueId, runId, actionId, agentId, environment, lease, reference }; + } + + function app(actor: express.Request["actor"] = { type: "board", source: "local_implicit" }) { + const result = express(); + result.use((req, _res, next) => { req.actor = actor; next(); }); + result.use("/api", issueRoutes(db, {} as Parameters[1])); + result.use(errorHandler); + return result; + } + + function path(f: Awaited>, actionId: string = f.actionId, runId: string = f.runId) { + return `/api/issues/${f.issueId}/recovery-actions?inspectSourceRunId=${runId}&inspectSourceActionId=${actionId}`; + } + + async function snapshot(f: Awaited>) { + return { + actions: await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, f.companyId)), + runs: await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId)), + leases: await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.companyId)), + issues: await db.select().from(issues).where(eq(issues.companyId, f.companyId)), + wakeups: await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId)), + operations: await db.select().from(workspaceOperations).where(eq(workspaceOperations.companyId, f.companyId)), + activity: await db.select().from(activityLog).where(eq(activityLog.companyId, f.companyId)), + }; + } + + it.each(["lease_policy_not_retained", "source_lease_not_bound"])("identifies the rejected contract: %s", async reason => { + const f = await seed(); + if (reason === "lease_policy_not_retained") { + await db.update(environmentLeases).set({ leasePolicy: "reuse_by_environment" }).where(eq(environmentLeases.id, f.lease.id)); + } else { + const otherRunId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: otherRunId, companyId: f.companyId, agentId: f.agentId, + issueId: f.issueId, status: "running" }); + await db.update(environmentLeases).set({ heartbeatRunId: otherRunId }).where(eq(environmentLeases.id, f.lease.id)); + } + await expect(inspectWorkspaceRecoverySource(db, f)).rejects.toMatchObject({ + code: "source_binding_invalid", retryClass: "service_required", sourceVerification: { reason }, + }); + }); + + it("inspects a resolved retained source through HTTP without revalidating or changing recorded recovery", async () => { + const f = await seed(); + await db.update(environmentLeases).set({ leasePolicy: "reuse_by_environment" }).where(eq(environmentLeases.id, f.lease.id)); + await db.update(issueRecoveryActions).set({ status: "resolved", outcome: "blocked", evidence: { + ...(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId)))[0].evidence, + automaticRecovery: { policy: "preserve_without_replay_v1", replay: "blocked", runId: f.runId }, + } }).where(eq(issueRecoveryActions.id, f.actionId)); + const before = await snapshot(f); + const response = await request(app()).get(path(f)).expect(200); + expect(response.body.active).toBeNull(); + expect(response.body.actions[0]).toMatchObject({ id: f.actionId, status: "resolved", evidence: { + systemRecovery: { state: "exhausted", failure: { code: "source_binding_invalid" } }, + } }); + expect(response.body.sourceInspection).toMatchObject({ actionId: f.actionId, runId: f.runId, + scope: "recorded_source_identity", status: "blocked", failureCode: "source_binding_invalid", + sourceVerification: { reason: "lease_policy_not_retained", observedLeasePolicy: "reuse_by_environment" } }); + expect(response.body.actions[0].evidence.systemRecovery.failure.sourceVerification).toBeUndefined(); + expect(JSON.stringify(response.body)).not.toContain("secret-must-not-appear"); + expect(JSON.stringify(response.body)).not.toContain("/workspace"); + expect(JSON.stringify(response.body)).not.toContain(f.lease.providerLeaseId); + expect(await snapshot(f)).toEqual(before); + }); + + it("only verifies current recorded identity for a valid retained source", async () => { + const f = await seed(); + const before = await snapshot(f); + const response = await request(app()).get(path(f)).expect(200); + expect(response.body.sourceInspection).toEqual({ schema: "paperclip.workspace-source-inspection.v1", + actionId: f.actionId, runId: f.runId, scope: "recorded_source_identity", status: "verified", + observedAt: expect.any(String) }); + expect(await snapshot(f)).toEqual(before); + }); + + it("keeps a reused physical allocation blocked without touching either lease", async () => { + const f = await seed(); + await db.insert(environmentLeases).values({ companyId: f.companyId, environmentId: f.environment.id, + provider: "daytona", providerLeaseId: f.lease.providerLeaseId, leasePolicy: "reuse_by_environment", status: "active" }); + const before = await snapshot(f); + const response = await request(app()).get(path(f)).expect(200); + expect(response.body.sourceInspection).toMatchObject({ status: "blocked", failureCode: "allocation_ownership_ambiguous" }); + expect(response.body.sourceInspection.sourceVerification).toBeUndefined(); + expect(await snapshot(f)).toEqual(before); + }); + + it("rejects a foreign action, mismatched run, and malformed selection without inspection", async () => { + const f = await seed(), foreign = await seed(); + const before = await snapshot(f); + await request(app()).get(path(f, foreign.actionId)).expect(404); + await request(app()).get(path(f, f.actionId, foreign.runId)).expect(404); + await request(app()).get(path(f, "not-a-uuid")).expect(400); + expect(await snapshot(f)).toEqual(before); + }); + + it("does not let another company's agent inspect the source", async () => { + const f = await seed(), foreign = await seed(); + const response = await request(app({ type: "agent", agentId: foreign.agentId, companyId: foreign.companyId, + source: "agent_key" })).get(path(f)); + expect([403, 404]).toContain(response.status); + expect(response.body.sourceInspection).toBeUndefined(); + }); + + it("lists only the explicitly requested run without inspecting it", async () => { + const f = await seed(); + await db.update(environmentLeases).set({ leasePolicy: "reuse_by_environment" }).where(eq(environmentLeases.id, f.lease.id)); + const response = await request(app()).get(`/api/issues/${f.issueId}/recovery-actions?inspectSourceRunId=${f.runId}`).expect(200); + expect(response.body.actions.map((action: { id: string }) => action.id)).toEqual([f.actionId]); + expect(response.body.sourceInspection).toBeUndefined(); + }); + + it("bounds explicit inspection before lease lookup without changing worker verification", async () => { + const f = await seed(); + const leaseIds = [f.lease.id]; + for (let index = 0; index < 8; index++) { + const [lease] = await db.insert(environmentLeases).values({ companyId: f.companyId, issueId: f.issueId, + heartbeatRunId: f.runId, environmentId: f.environment.id, provider: "daytona", providerLeaseId: randomUUID(), + leasePolicy: "retain_on_failure", status: "released", metadata: { remoteCwd: "/workspace" } }).returning(); + leaseIds.push(lease.id); + } + await db.update(heartbeatRuns).set({ resultJson: { workspaceRestoreRecovery: { + schema: "paperclip.workspace-restore-recovery.v1", leaseIds } } }).where(eq(heartbeatRuns.id, f.runId)); + const before = await snapshot(f); + const select = vi.spyOn(db, "select"); + try { + expect(await inspectWorkspaceSource(db, f)).toMatchObject({ status: "not_inspected", reason: "source_limit" }); + expect(select).toHaveBeenCalledTimes(2); + } finally { select.mockRestore(); } + expect((await inspectWorkspaceRecoverySource(db, f)).sources).toHaveLength(9); + expect(await snapshot(f)).toEqual(before); + }); +}); diff --git a/server/src/services/workspace-recovery-source.ts b/server/src/services/workspace-recovery-source.ts index 61b9320b7d7..7b98811ab12 100644 --- a/server/src/services/workspace-recovery-source.ts +++ b/server/src/services/workspace-recovery-source.ts @@ -5,11 +5,23 @@ import { readNativeWorkspaceSyncReference } from "./native-runtime/native-worksp import { SystemRecoveryFailure } from "./system-recovery-state.js"; import type { NativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js"; import { readStopOnlyCleanup } from "./sandbox-stop-and-retain.js"; +import { ENVIRONMENT_LEASE_POLICIES, type EnvironmentLeasePolicy, + type WorkspaceSourceVerificationFailure } from "@paperclipai/shared"; export const workspaceExportCauses = ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"]; export const terminalWorkspaceRunStates = ["failed", "interrupted", "timed_out", "cancelled", "succeeded"]; export type WorkspaceRecoverySourceIdentity = { companyId: string; issueId: string; runId: string; actionId: string }; +export function workspaceSourceBindingFailure(reason: WorkspaceSourceVerificationFailure["reason"], policy?: unknown) { + const observedLeasePolicy = typeof policy === "string" && (ENVIRONMENT_LEASE_POLICIES as readonly string[]).includes(policy) + ? policy as EnvironmentLeasePolicy : "unrecognized"; + return new SystemRecoveryFailure("source_binding_invalid", "service_required", undefined, { + reason, ...(reason === "lease_policy_not_retained" ? { observedLeasePolicy } : {}), + }); +} + +export class WorkspaceSourceInspectionLimit extends Error {} + export function workspaceRecoverySourceEligible(includeCompleted = false) { const actionEligible = sql`(${issueRecoveryActions.status} in ('active', 'escalated') or (${issueRecoveryActions.status} = 'resolved' and ( @@ -41,39 +53,46 @@ export async function ownedWorkspaceRecoveryLease(db: Db, input: WorkspaceRecove await input.ownership.assertHeld(); const inspected = await inspectWorkspaceRecoverySource(db, input); const source = inspected.sources.find(item => item.lease.id === input.leaseId); - if (!source || source.environment.id !== input.environmentId || source.lease.status !== "pending_cleanup" - || source.lease.metadata?.pendingCleanupInFlight !== true - || source.lease.metadata?.pendingCleanupAttemptId !== input.ownership.token - || !readStopOnlyCleanup(source.lease)) { - throw new SystemRecoveryFailure("source_binding_invalid", "service_required"); - } + if (!source) throw workspaceSourceBindingFailure("recovery_lease_not_selected"); + if (source.environment.id !== input.environmentId) throw workspaceSourceBindingFailure("recovery_environment_mismatch"); + if (source.lease.status !== "pending_cleanup" || source.lease.metadata?.pendingCleanupInFlight !== true) + throw workspaceSourceBindingFailure("recovery_lease_not_claimed"); + if (source.lease.metadata?.pendingCleanupAttemptId !== input.ownership.token) + throw workspaceSourceBindingFailure("recovery_owner_mismatch"); + if (!readStopOnlyCleanup(source.lease)) throw workspaceSourceBindingFailure("recovery_stop_intent_missing"); await input.ownership.assertHeld(); return source; } /** Resolve only the immutable source recorded by the failed run. Never discover * a candidate by recency, mutable adapter configuration, or a provider search. */ -export async function inspectWorkspaceRecoverySource(db: Db, input: WorkspaceRecoverySourceIdentity) { - const invalid = () => new SystemRecoveryFailure("source_binding_invalid", "service_required"); +export async function inspectWorkspaceRecoverySource(db: Db, input: WorkspaceRecoverySourceIdentity, + options: { maximumSources?: number } = {}) { + const invalid = workspaceSourceBindingFailure; const [run] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId))); const [action] = await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId), eq(issueRecoveryActions.id, input.actionId))); - if (!run || !action || action.evidence.runId !== run.id || !terminalWorkspaceRunStates.includes(run.status) || - (run.nativeIssueId ?? run.issueId ?? run.contextSnapshot?.issueId) !== input.issueId) throw invalid(); + if (!run) throw invalid("source_run_missing"); + if (!action) throw invalid("recovery_action_missing"); + if (action.evidence.runId !== run.id) throw invalid("recovery_action_run_mismatch"); + if (!terminalWorkspaceRunStates.includes(run.status)) throw invalid("source_run_not_terminal"); + if ((run.nativeIssueId ?? run.issueId ?? run.contextSnapshot?.issueId) !== input.issueId) throw invalid("source_issue_mismatch"); const rawReference = run.runnerProfileJson?.nativeWorkspaceSync; const reference = readNativeWorkspaceSyncReference(rawReference); - if (rawReference != null && !reference) throw invalid(); + if (rawReference != null && !reference) throw invalid("native_reference_invalid"); const recorded = run.resultJson?.workspaceRestoreRecovery as { schema?: unknown; leaseIds?: unknown } | undefined; - if (recorded != null && recorded.schema !== "paperclip.workspace-restore-recovery.v1") throw invalid(); + if (recorded != null && recorded.schema !== "paperclip.workspace-restore-recovery.v1") throw invalid("restore_reference_invalid"); const historical = action.evidence.historicalWorkspaceSource as { schema?: unknown; runId?: unknown; leaseIds?: unknown } | undefined; - if (historical != null && (historical.schema !== "paperclip.historical-workspace-source.v1" || historical.runId !== run.id)) throw invalid(); + if (historical != null && (historical.schema !== "paperclip.historical-workspace-source.v1" || historical.runId !== run.id)) throw invalid("historical_reference_invalid"); const leaseIds = recorded ? recorded.leaseIds : reference && workspaceExportCauses.includes(action.cause) ? [reference.leaseId] : historical?.leaseIds; if (!Array.isArray(leaseIds) || leaseIds.length === 0) throw new SystemRecoveryFailure("historical_source_missing", "use_alternative_strategy"); + if (options.maximumSources !== undefined && leaseIds.length > options.maximumSources) + throw new WorkspaceSourceInspectionLimit(); if (leaseIds.some(id => typeof id !== "string" || !/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(id)) || - new Set(leaseIds).size !== leaseIds.length || (reference && !leaseIds.includes(reference.leaseId))) throw invalid(); + new Set(leaseIds).size !== leaseIds.length || (reference && !leaseIds.includes(reference.leaseId))) throw invalid("lease_reference_invalid"); const [coordinator] = await db.select().from(nativeRunFinalizations).where(and( eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, run.id))); if (coordinator?.leaseOwner || adapterExecutionControls.has(run.id) || @@ -85,11 +104,17 @@ export async function inspectWorkspaceRecoverySource(db: Db, input: WorkspaceRec eq(environmentLeases.id, leaseId), eq(environmentLeases.heartbeatRunId, run.id))); const descriptor = reference?.leaseId === leaseId ? reference : null; const remoteCwd = lease?.metadata?.remoteCwd; - if (!lease || lease.issueId !== input.issueId || !lease.environmentId || !lease.providerLeaseId || - !lease.provider || lease.provider === "local" || !["retain_on_failure", "ephemeral"].includes(lease.leasePolicy) || - typeof remoteCwd !== "string" || !remoteCwd.startsWith("/") || - (descriptor && (lease.providerLeaseId !== descriptor.providerLeaseId || remoteCwd !== descriptor.remoteCwd || - (lease.executionWorkspaceId && lease.executionWorkspaceId !== descriptor.workspaceId)))) throw invalid(); + if (!lease) throw invalid("source_lease_not_bound"); + if (lease.issueId !== input.issueId) throw invalid("lease_issue_mismatch"); + if (!lease.environmentId) throw invalid("lease_environment_missing"); + if (!lease.providerLeaseId) throw invalid("lease_allocation_missing"); + if (!lease.provider || lease.provider === "local") throw invalid("lease_provider_invalid"); + if (!["retain_on_failure", "ephemeral"].includes(lease.leasePolicy)) throw invalid("lease_policy_not_retained", lease.leasePolicy); + if (typeof remoteCwd !== "string" || !remoteCwd.startsWith("/")) throw invalid("lease_cwd_invalid"); + if (descriptor && lease.providerLeaseId !== descriptor.providerLeaseId) throw invalid("allocation_reference_mismatch"); + if (descriptor && remoteCwd !== descriptor.remoteCwd) throw invalid("cwd_reference_mismatch"); + if (descriptor && lease.executionWorkspaceId && lease.executionWorkspaceId !== descriptor.workspaceId) + throw invalid("workspace_reference_mismatch"); const [other] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and( ne(environmentLeases.id, lease.id), eq(environmentLeases.provider, lease.provider), eq(environmentLeases.providerLeaseId, lease.providerLeaseId))).limit(1); diff --git a/server/src/services/workspace-recovery.ts b/server/src/services/workspace-recovery.ts index 28b0f301d28..179ddc478ad 100644 --- a/server/src/services/workspace-recovery.ts +++ b/server/src/services/workspace-recovery.ts @@ -21,7 +21,7 @@ import { classifyNativeWorkspaceFailure } from "./native-runtime/native-workspac import { adoptSystemRecovery, loadSystemRecovery, updateSystemRecovery } from "./system-recovery.js"; import { createSystemRecovery, failSystemRecovery, readSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure, type SystemRecoveryState } from "./system-recovery-state.js"; -import { inspectWorkspaceRecoverySource, workspaceRecoverySourceEligible } from "./workspace-recovery-source.js"; +import { inspectWorkspaceRecoverySource, workspaceRecoverySourceEligible, workspaceSourceBindingFailure } from "./workspace-recovery-source.js"; import { preserveWorkspaceSnapshot } from "./workspace-recovery-snapshot.js"; import { adoptHistoricalWorkspaceRecoveries } from "./workspace-recovery-backfill.js"; @@ -132,7 +132,7 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ // Missing local merge metadata permits a separate snapshot of the // exact bound allocation. Corrupt metadata never weakens binding. if ((error as NodeJS.ErrnoException)?.code === "ENOENT") reference = null; - else throw new SystemRecoveryFailure("source_binding_invalid", "service_required"); + else throw workspaceSourceBindingFailure("local_descriptor_invalid"); } } let stoppedLease = lease; @@ -168,7 +168,7 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ environment, leaseId: lease.id, leaseMetadata: stoppedLease.metadata, lease: stoppedLease as EnvironmentLease, environmentRuntime: input.environmentRuntime }); if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner || target.remoteCwd !== remoteCwd) - throw new SystemRecoveryFailure("source_binding_invalid", "service_required"); + throw workspaceSourceBindingFailure("execution_target_invalid"); await ownership.assertHeld(); const cleanupLease = await db.transaction(async tx => { const [fresh] = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId), @@ -176,7 +176,7 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ if (!fresh || !hasRemoteTerminationReceipt(fresh) || fresh.providerLeaseId !== lease.providerLeaseId) throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence"); const pending = await prepareStoppedSandboxForRecovery(tx as unknown as Db, fresh); - if (!pending) throw new SystemRecoveryFailure("source_binding_invalid", "service_required"); + if (!pending) throw workspaceSourceBindingFailure("recovery_stop_intent_invalid"); const [claimed] = await tx.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({ pendingCleanupAttemptId: ownership.token, pendingCleanupInFlight: true, pendingCleanupLeaseExpiresAtMs: Date.now() + 15 * 60_000, })}::jsonb` }).where(eq(environmentLeases.id, fresh.id)).returning(); @@ -192,8 +192,8 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ try { const opened = await input.environmentRuntime.resumeWorkspaceRecoveryLease({ source: input, leaseId: cleanupLease.id, environmentId: environment.id, ownership }); - if (opened?.providerLeaseId !== lease.providerLeaseId || opened.metadata?.remoteCwd !== remoteCwd) - throw new SystemRecoveryFailure("source_binding_invalid", "service_required"); + if (opened?.providerLeaseId !== lease.providerLeaseId) throw workspaceSourceBindingFailure("reopened_allocation_mismatch"); + if (opened.metadata?.remoteCwd !== remoteCwd) throw workspaceSourceBindingFailure("reopened_cwd_mismatch"); await inspectWorkspaceRecoverySource(db, input); await progress({ phase: "preserve_files" }); let operationId: string; diff --git a/server/src/services/workspace-source-inspection.ts b/server/src/services/workspace-source-inspection.ts new file mode 100644 index 00000000000..f835d749493 --- /dev/null +++ b/server/src/services/workspace-source-inspection.ts @@ -0,0 +1,22 @@ +import type { Db } from "@paperclipai/db"; +import type { WorkspaceSourceInspection } from "@paperclipai/shared"; +import { inspectWorkspaceRecoverySource, WorkspaceSourceInspectionLimit, + type WorkspaceRecoverySourceIdentity } from "./workspace-recovery-source.js"; +import { SystemRecoveryFailure, workspaceSourceVerificationFailureSchema } from "./system-recovery-state.js"; + +export const MAX_INSPECTED_WORKSPACE_SOURCES = 8; + +export async function inspectWorkspaceSource(db: Db, input: WorkspaceRecoverySourceIdentity): Promise { + const base = { schema: "paperclip.workspace-source-inspection.v1" as const, actionId: input.actionId, + runId: input.runId, observedAt: new Date().toISOString(), scope: "recorded_source_identity" as const }; + try { + await inspectWorkspaceRecoverySource(db, input, { maximumSources: MAX_INSPECTED_WORKSPACE_SOURCES }); + return { ...base, status: "verified" }; + } catch (error) { + if (error instanceof WorkspaceSourceInspectionLimit) return { ...base, status: "not_inspected", reason: "source_limit" }; + if (!(error instanceof SystemRecoveryFailure)) throw error; + const detail = workspaceSourceVerificationFailureSchema.safeParse(error.sourceVerification); + return { ...base, status: "blocked", failureCode: error.code, + ...(detail.success ? { sourceVerification: detail.data } : {}) }; + } +} diff --git a/ui/src/api/issues.ts b/ui/src/api/issues.ts index 1f19291f103..159c215ce7a 100644 --- a/ui/src/api/issues.ts +++ b/ui/src/api/issues.ts @@ -26,6 +26,7 @@ import type { IssueDocument, IssueLabel, IssueRecoveryAction, + WorkspaceSourceInspection, IssueRetryNowResponse, StalledReviewDecision, StalledReviewDecisionResponse, @@ -317,8 +318,12 @@ export const issuesApi = { reconstructWorkspace: (id: string, data: { actionId: string; runId: string; expectedStatusVersion: number; attachmentIds: string[]; missingWork: string; actionOutcome: "completed" | "not_performed" | "mixed"; outcomeEvidence: string; acknowledgeLostWork: true }) => api.post<{ runId: string; actionId: string; status: "queued"; replayed: boolean }>(`/issues/${id}/recovery-actions/reconstruct-workspace`, data), - recoveryActions: (id: string) => - api.get<{ active: IssueRecoveryAction | null; actions: IssueRecoveryAction[] }>(`/issues/${id}/recovery-actions`), + recoveryActions: (id: string, source?: { runId: string; actionId?: string }) => { + const query = source ? new URLSearchParams({ inspectSourceRunId: source.runId, + ...(source.actionId ? { inspectSourceActionId: source.actionId } : {}) }) : null; + return api.get<{ active: IssueRecoveryAction | null; actions: IssueRecoveryAction[]; + sourceInspection?: WorkspaceSourceInspection }>(`/issues/${id}/recovery-actions${query ? `?${query}` : ""}`); + }, resolveRecoveryAction: ( id: string, data: { diff --git a/ui/src/components/RunWorkspaceRecoverySurface.test.tsx b/ui/src/components/RunWorkspaceRecoverySurface.test.tsx index 10aa936f0f7..6c6514696b9 100644 --- a/ui/src/components/RunWorkspaceRecoverySurface.test.tsx +++ b/ui/src/components/RunWorkspaceRecoverySurface.test.tsx @@ -15,6 +15,7 @@ const issueCreateMock = vi.hoisted(() => vi.fn()); const resolveRecoveryMock = vi.hoisted(() => vi.fn()); const reconcileMock = vi.hoisted(() => vi.fn()); const boardAccessMock = vi.hoisted(() => vi.fn()); +const recoveryActionsMock = vi.hoisted(() => vi.fn()); vi.mock("@/lib/router", () => ({ useNavigate: () => navigateMock, @@ -28,6 +29,7 @@ vi.mock("../api/issues", () => ({ get: issueGetMock, create: issueCreateMock, resolveRecoveryAction: resolveRecoveryMock, + recoveryActions: recoveryActionsMock, }, })); @@ -68,7 +70,7 @@ afterEach(() => { root = null; container?.remove(); container = null; - vi.clearAllMocks(); + vi.resetAllMocks(); }); function buildRun(overrides: Partial = {}): HeartbeatRun { @@ -178,6 +180,104 @@ function click(element: Element | null) { } describe("RunWorkspaceRecoverySurface", () => { + it("inspects a selected resolved source only on request and separates current identity from the recorded failure", async () => { + const action = { ...buildRecoveryAction(), status: "resolved", cause: "native_workspace_sync_out_retry_exhausted", + evidence: { systemRecovery: { failure: { code: "source_binding_invalid" } } } }; + recoveryActionsMock.mockResolvedValueOnce({ active: null, actions: [action] }).mockResolvedValueOnce({ active: null, + actions: [action], sourceInspection: { actionId: action.id, runId: "run-aaaa1111", scope: "recorded_source_identity", + observedAt: "2026-10-26T17:00:00.000Z", status: "blocked", failureCode: "source_binding_invalid", + sourceVerification: { reason: "lease_policy_not_retained", observedLeasePolicy: "reuse_by_environment" } } }); + const node = await renderSurface(buildRun({ errorCode: "native_workspace_sync_out_retry_exhausted" })); + expect(recoveryActionsMock).not.toHaveBeenCalled(); + expect(issueGetMock).not.toHaveBeenCalled(); + click(node.querySelector("button")); + await flush(); + expect(recoveryActionsMock).toHaveBeenLastCalledWith("issue-1", { runId: "run-aaaa1111" }); + expect(node.textContent).toContain("Recorded failure: source_binding_invalid"); + const inspect = [...node.querySelectorAll("button")].find(button => button.textContent === "Inspect current source"); + click(inspect ?? null); + await flush(); + expect(recoveryActionsMock).toHaveBeenLastCalledWith("issue-1", { runId: "run-aaaa1111", actionId: "action-1" }); + expect(node.textContent).toContain("Current recorded-source identity"); + expect(node.textContent).toContain("lease_policy_not_retained"); + expect(node.textContent).toContain("reuse_by_environment"); + expect(node.textContent).toContain("does not confirm provider termination, local files, or successful preservation"); + expect(issueCreateMock).not.toHaveBeenCalled(); + expect(resolveRecoveryMock).not.toHaveBeenCalled(); + expect(reconcileMock).not.toHaveBeenCalled(); + }); + + it("shows verified identity without claiming preservation and reports an inspection error inline", async () => { + const action = buildRecoveryAction(); + recoveryActionsMock.mockResolvedValueOnce({ active: null, actions: [action] }) + .mockResolvedValueOnce({ active: null, actions: [action], sourceInspection: { + actionId: action.id, status: "verified", observedAt: "2026-10-26T17:00:00.000Z" } }) + .mockRejectedValueOnce(new Error("Source inspection unavailable")); + const node = await renderSurface(buildRun({ errorCode: "native_workspace_sync_out_unrecoverable" })); + click(node.querySelector("button")); + await flush(); + click([...node.querySelectorAll("button")].find(button => button.textContent === "Inspect current source") ?? null); + await flush(); + expect(recoveryActionsMock).toHaveBeenLastCalledWith("issue-1", { runId: "run-aaaa1111", actionId: action.id }); + expect(node.textContent).toContain("Recorded-source identity verified."); + expect(node.textContent).toContain("does not confirm provider termination"); + click([...node.querySelectorAll("button")].find(button => button.textContent === "Inspect current source") ?? null); + await flush(); + expect(node.querySelector("[role='alert']")?.textContent).toContain("Source inspection unavailable"); + }); + + it("keeps inspection pending and explains a bounded source limit", async () => { + const action = buildRecoveryAction(); + let complete!: (value: unknown) => void; + recoveryActionsMock.mockResolvedValueOnce({ active: null, actions: [action] }) + .mockImplementationOnce(() => new Promise(resolve => { complete = resolve; })); + const node = await renderSurface(buildRun({ errorCode: "native_workspace_sync_out_retry_exhausted" })); + click(node.querySelector("button")); + await flush(); + click([...node.querySelectorAll("button")].find(button => button.textContent === "Inspect current source") ?? null); + await flush(); + expect(node.querySelector("button")?.disabled).toBe(true); + expect(recoveryActionsMock).toHaveBeenLastCalledWith("issue-1", { runId: "run-aaaa1111", actionId: action.id }); + complete({ active: null, actions: [action], sourceInspection: { actionId: action.id, + observedAt: "2026-10-26T17:00:00.000Z", status: "not_inspected", reason: "source_limit" } }); + await flush(); + expect(node.textContent).toContain("exceeds the inspection limit"); + expect(node.querySelector("button")?.disabled).toBe(false); + }); + + it("keeps saved reasons separate from changed observations and preserves other source selections", async () => { + const first = { ...buildRecoveryAction(), evidence: { systemRecovery: { failure: { + code: "source_binding_invalid", sourceVerification: { reason: "lease_policy_not_retained", observedLeasePolicy: "reuse_by_environment" }, + } } } }; + const second = { ...first, id: "action-2", evidence: { systemRecovery: { failure: { code: "source_binding_invalid", + sourceVerification: { reason: "secret-reason", observedLeasePolicy: "secret-policy" } } } } }; + recoveryActionsMock.mockResolvedValueOnce({ active: null, actions: [first, second] }) + .mockResolvedValueOnce({ active: null, actions: [first], sourceInspection: { actionId: first.id, + status: "blocked", observedAt: "2026-10-26T17:00:00.000Z", failureCode: "source_binding_invalid", + sourceVerification: { reason: "source_lease_not_bound" } } }) + .mockResolvedValueOnce({ active: null, actions: [second], sourceInspection: { actionId: second.id, + status: "verified", observedAt: "2026-10-26T17:01:00.000Z" } }); + const node = await renderSurface(buildRun({ errorCode: "native_workspace_sync_out_retry_exhausted" })); + click(node.querySelector("button")); + await flush(); + expect(node.textContent).toContain("Recorded failure: source_binding_invalid · lease_policy_not_retained · lease policy reuse_by_environment"); + expect(node.textContent).not.toContain("secret-reason"); + expect(node.textContent).not.toContain("secret-policy"); + const choices = () => [...node.querySelectorAll("button")].filter(button => button.textContent === "Inspect current source"); + click(choices()[0]); + await flush(); + expect(node.textContent).toContain("source_lease_not_bound"); + expect(node.textContent).toContain("Action action-1"); + expect(node.textContent).toContain("lease_policy_not_retained"); + expect(choices()).toHaveLength(2); + click(choices()[1]); + await flush(); + expect(recoveryActionsMock).toHaveBeenLastCalledWith("issue-1", { runId: "run-aaaa1111", actionId: "action-2" }); + expect(node.textContent).toContain("Action action-2"); + expect(node.textContent).toContain("Recorded-source identity verified."); + expect(choices()).toHaveLength(2); + }); + it("renders nothing for a run that is not a workspace-validation failure", async () => { issueGetMock.mockResolvedValue(buildIssue(buildRecoveryAction())); boardAccessMock.mockResolvedValue(undefined); diff --git a/ui/src/components/RunWorkspaceRecoverySurface.tsx b/ui/src/components/RunWorkspaceRecoverySurface.tsx index bfae52e2761..d57b8a4cb12 100644 --- a/ui/src/components/RunWorkspaceRecoverySurface.tsx +++ b/ui/src/components/RunWorkspaceRecoverySurface.tsx @@ -1,6 +1,8 @@ import { useCallback } from "react"; import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; import type { HeartbeatRun } from "@paperclipai/shared"; +import { isNativeWorkspaceExportRepairCause } from "@paperclipai/shared"; +import { RunWorkspaceSourceInspection } from "./RunWorkspaceSourceInspection"; import { useNavigate } from "@/lib/router"; import { issuesApi } from "../api/issues"; import { executionWorkspacesApi } from "../api/execution-workspaces"; @@ -52,6 +54,14 @@ function readRunIssueId(run: HeartbeatRun): string | null { * a live `workspace_validation` recovery action. */ export function RunWorkspaceRecoverySurface({ run }: { run: HeartbeatRun }) { + const issueId = readRunIssueId(run); + const cause = run.execution?.cause ?? run.errorCode; + if (issueId && (isNativeWorkspaceExportRepairCause(cause) || cause === "native_workspace_sync_out_unrecoverable")) + return ; + return ; +} + +function WorkspaceValidationRecoverySurface({ run }: { run: HeartbeatRun }) { const navigate = useNavigate(); const queryClient = useQueryClient(); const { pushToast } = useToastActions(); diff --git a/ui/src/components/RunWorkspaceSourceInspection.tsx b/ui/src/components/RunWorkspaceSourceInspection.tsx new file mode 100644 index 00000000000..9cea2578912 --- /dev/null +++ b/ui/src/components/RunWorkspaceSourceInspection.tsx @@ -0,0 +1,58 @@ +import { useMutation } from "@tanstack/react-query"; +import { ENVIRONMENT_LEASE_POLICIES, WORKSPACE_SOURCE_VERIFICATION_REASONS } from "@paperclipai/shared"; +import { issuesApi } from "../api/issues"; +import { Button } from "./ui/button"; +import { formatDateTime } from "../lib/utils"; + +function recordedFailure(evidence: Record): string { + const recovery = evidence.systemRecovery; + if (!recovery || typeof recovery !== "object" || !("failure" in recovery)) return "No recorded system recovery failure."; + const failure = recovery.failure; + if (!failure || typeof failure !== "object" || !("code" in failure) || typeof failure.code !== "string") + return "No recorded system recovery failure."; + const detail = "sourceVerification" in failure ? failure.sourceVerification : null; + if (!detail || typeof detail !== "object" || !("reason" in detail) || typeof detail.reason !== "string" || + !(WORKSPACE_SOURCE_VERIFICATION_REASONS as readonly string[]).includes(detail.reason)) return failure.code; + const policy = "observedLeasePolicy" in detail ? detail.observedLeasePolicy : null; + const allowedPolicy = typeof policy === "string" && + (policy === "unrecognized" || (ENVIRONMENT_LEASE_POLICIES as readonly string[]).includes(policy)); + return [failure.code, detail.reason, ...(allowedPolicy ? [`lease policy ${policy}`] : [])].join(" · "); +} + +export function RunWorkspaceSourceInspection({ issueId, runId }: { issueId: string; runId: string }) { + const listing = useMutation({ mutationFn: () => issuesApi.recoveryActions(issueId, { runId }) }); + const inspection = useMutation({ mutationFn: (actionId: string) => issuesApi.recoveryActions(issueId, { runId, actionId }) }); + const pending = listing.isPending || inspection.isPending; + const error = listing.error ?? inspection.error; + const current = inspection.data?.sourceInspection; + return ( +
+ + {error ?

+ {error instanceof Error ? error.message : "Recovery inspection failed. Try inspecting again."} +

: null} + {listing.data?.actions.length === 0 ?

No recorded recovery action for this run.

: null} + {listing.data?.actions.map(action => ( +
+

Recorded failure: {recordedFailure(action.evidence)}

+

{action.id} · {action.status}

+ +
+ ))} + {current ?
+

Current recorded-source identity

+

Action {current.actionId}

+

This read-only observation checks recorded identity. It does not confirm provider termination, local files, or successful preservation.

+ {current.status === "verified" ?

Recorded-source identity verified.

+ : current.status === "not_inspected" ?

Source not inspected: this record exceeds the inspection limit. Service attention is required.

+ :

Blocked: {current.sourceVerification?.reason ?? current.failureCode} + {current.sourceVerification?.observedLeasePolicy ? <> · lease policy {current.sourceVerification.observedLeasePolicy} : null}

} +

Observed {formatDateTime(current.observedAt)}

+
: null} +
+ ); +}