mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-28 12:04:41 +01:00
fix(recovery): explain retained source verification failures (#17773)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Workspace recovery must prove the identity of retained work before it uses an allocation. > - The verifier rejects several different contracts with the same `source_binding_invalid` code. > - A saved Retry can therefore stop before file preservation without enough evidence for a safe repair. > - Operators need a bounded observation of the selected source without another attempt or weaker guards. > - This pull request records closed failure reasons and adds an explicit read-only inspection on the run page. > - The benefit is a precise next diagnostic step while the saved work, intent, and ownership fences remain intact. ## Linked Issues or Issue Description Refs: #17741, #17053. This is a diagnostic follow-up. It does not change recovery admission or the retained-source acceptance policy. **What happened?** A retained-source recovery exhausted in `verify_execution` with only `source_binding_invalid`. Run and task diagnostics could not distinguish an invalid reference, lease policy, source binding, or reopened allocation. The normal recovery-actions read can also revalidate recovery state, so it was not a safe substitute for a current read-only inspection. **Expected behavior** Record an allowlisted reason at the rejected guard. Allow a reader to inspect one exact task/run/action without changing its attempts, saved input, receipts, or ownership. Clearly separate the recorded failure from current identity observations. **Steps to reproduce** 1. Prepare a terminal native run with a retained-source action and a lease policy of `reuse_by_environment`. 2. Attempt source verification. The guard must still reject that policy. 3. Open the failed run and choose Inspect recovery, then Inspect current source. The diagnostic should name `lease_policy_not_retained` and the allowlisted policy. ## What Changed - Split existing source-binding guards into closed reason codes. Keep each acceptance predicate and physical recovery action unchanged. - Persist validated, path-free reason/policy detail with future system recovery failures. Keep historical generic records compatible. - Add an explicit query on the existing task recovery-actions GET. Exact company/task/run/action selection bypasses read revalidation. Run selection lists at most 20 actions; inspection stops before per-lease lookup when a record has more than eight sources. - Add a manual inspection control on native export-failure run pages. Keep resolved retained actions inspectable. Show recorded failure and current recorded-source identity separately. - Update execution semantics and feature/access recipes. The shared type is the normalized server/UI contract; it contains no provider payload or secret. ## Verification - RED: the prior verifier failed both reason assertions for rejected warm policy and foreign-run lease binding. GREEN: the final focused source/HTTP, system-state, and UI suites pass 23 tests. The UI follow-up passes 8 tests after the review fixes. - HTTP tests prove exact selection, cross-company denial, resolved-action access, reused-allocation rejection, redaction, and unchanged actions, runs, leases, tasks, wakeup receipts, workspace operations, and activity. A nine-source case proves bounded inspection while the existing worker verifier still accepts all nine valid sources. - Review follow-up: show allowlisted saved detail beside a changed current observation, preserve all listed actions, and follow exact-action selection in verified/limited/error tests. The prior component fails the saved/current drift regression in an in-memory RED control; the current component passes. UI typecheck/build pass again. - `pnpm -r typecheck` and `pnpm build` pass. Token gates, maps, module boundaries, and `git diff --check` pass. - Rebased onto master `46de15ff4b` without conflicts. Range-diff proves both diagnostic commits are unchanged. On new head `a7e2bc87fcea75e76b6b736304d1287257f71174`, all 23 focused tests, full typecheck/build, token gates, maps, and module checks pass again. Both fresh reviews are 5/5 with zero unresolved threads. Final exact-head GitHub status is CLEAN: 55 success, 2 skipped, 1 neutral, no pending checks or failures. The [ci / verify aggregate](https://github.com/paperclipai/paperclip/actions/runs/40387002325/job/121802415824) passed in eight seconds. All test/build/typecheck/E2E lanes passed. - Existing full workspace recovery suite: 56/56 pass (103.5 seconds) before the byte-equivalent rebase. `pnpm test:run` hit repeated embedded PostgreSQL initialization failure before database-suite assertions and was stopped. No full local suite pass is claimed. The owned external `_test` database provided the focused HTTP/source verification. The prior browser attempt also stalled at Network.enable before its click; that attempt did not establish browser acceptance. - Current-head local browser acceptance passes through the actual run page: Inspect recovery lists the exact action, Inspect current source shows recorded generic failure separately from the current rejected policy, and reload resets inspection to the explicit control. Explicit selection works again after reload. The app used isolated data, an empty inherited environment, a paused synthetic agent, and no provider credentials. Final state remains failed/exhausted with zero attempts, zero wakeup receipts, zero workspace operations, and unchanged original generic failure. The local app and temporary tab were stopped. This proves the diagnostic journey, not the deployed source's actual lease policy or its recovery. - Final merge verification at `a7e2bc87fcea75e76b6b736304d1287257f71174`: GitHub reports CLEAN and mergeable. Both repository-required checks, `ci / verify` and `ci / e2e`, pass, as does the exact-head Greptile check (5/5, zero unresolved threads). The separate [Entire trail](https://entire.com/gh/paperclipai/paperclip/trails/7) has no recorded reviewer approval; no such approval is claimed. It is not a configured GitHub merge gate. - Initial local setup failures were distinct from product assertions: localhost sandbox EPERM required scoped local access; a browser harness import path was corrected. A test UUID type inference error and historical native-export UI selection gap were corrected before the green run. The live task remains blocked; this diagnostic does not claim it recovered. ## Risks - This change improves evidence. It does not prove provider termination, local descriptor integrity, or successful file preservation, and it does not repair the original source. - Current inspection spans multiple database reads. It is an observation, not a historical cause or execution authority. It performs no provider/filesystem calls and starts no retry or executor. - The optional persisted detail is backward compatible. Unknown values and extra fields are excluded. Oversized source lists remain uninspected through this explicit diagnostic path; physical recovery rules do not change. ## Model Used OpenAI Codex based on GPT-6, with code execution and browser tools. The exact serving model ID and context window were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
0c9d10d695
commit
bd382eee53
18 files changed
+558
-32
No files matched your search
@@ -1501,6 +1501,27 @@ Historical legacy interruption holds for conversational adapters no longer block
|
||||
|
||||
The server projection remains available for diagnostics. Normal working, finishing, and interaction waits add no badges or cards to task lists or feeds. Active transcript headers keep saying Working during automatic retry and execution confirmation; attempts, causes, and recovery decisions belong in the run log. Recovery uses the existing transcript and run log rather than adding a reconciliation form. A cancelled run that never started says “Couldn't start” instead of implying that the agent answered.
|
||||
|
||||
Native workspace export failures offer **Inspect recovery** on the run page.
|
||||
The explicit read lists recovery actions for that exact task and source run;
|
||||
selecting an action observes its current recorded-source identity. It includes
|
||||
resolved actions whose retained work may still need attention. This observation
|
||||
does not prove provider termination, local descriptor integrity, or successful
|
||||
file preservation, and it does not explain an earlier owner state retroactively.
|
||||
Recorded system recovery failures retain their original attempt and intent.
|
||||
New source-binding failures record a closed reason and, for a rejected lease
|
||||
policy, an allowlisted policy value. Neither detail includes paths, provider
|
||||
allocation identifiers, credentials, or raw provider errors.
|
||||
|
||||
The company/task-authorized `GET /api/issues/{id}/recovery-actions` accepts
|
||||
`inspectSourceRunId` to list at most 20 matching actions and an optional
|
||||
`inspectSourceActionId` to inspect one exact action. This opt-in path bypasses
|
||||
read revalidation and performs no recovery writes, provider calls, filesystem
|
||||
inspection, or retries. It reads the existing source verifier without changing
|
||||
its guards. Inspection of more than eight recorded leases returns
|
||||
`not_inspected/source_limit` before per-lease lookup; physical recovery retains
|
||||
its existing acceptance rules. The observation spans multiple database reads
|
||||
and grants no execution or preservation authority.
|
||||
|
||||
### Codex startup and provider state
|
||||
|
||||
Paperclip trusts the server-selected startup execution root in the isolated
|
||||
|
||||
@@ -124,6 +124,18 @@ activity links and the original task. Existing credential redaction remains.
|
||||
Bridge admission grants no cancellation, retry, recovery resolution or
|
||||
provider-trace administration authority.
|
||||
|
||||
The recovery-actions GET also accepts `inspectSourceRunId` and optional
|
||||
`inspectSourceActionId`. The explicit path requires the same readable task and
|
||||
exact company/task/run/action binding, including resolved retained actions.
|
||||
Run-only selection lists at most 20 actions; action selection observes current
|
||||
recorded-source identity with an eight-lease ceiling. This path does not run
|
||||
read revalidation, mutate recovery/Retry state, call providers or inspect files.
|
||||
Only closed reason/policy values supplement the existing recorded failure;
|
||||
the observation is neither historical cause nor permission to execute. Local
|
||||
[HTTP/database regressions](../../server/src/services/workspace-recovery-source.test.ts)
|
||||
cover exact selection, foreign action denial, reused allocations, redaction,
|
||||
bounded lookup and unchanged persisted state. Deployed agent use is unverified.
|
||||
|
||||
Evidence: `source-only` — scoped review of `f10210990a3c9cad1a4359318388f237de3169dd`:
|
||||
[run routes](../../server/src/routes/agents.ts),
|
||||
[task links](../../server/src/routes/activity.ts),
|
||||
|
||||
@@ -47,6 +47,17 @@ Automated: [activity service](../../server/src/__tests__/activity-service.test.t
|
||||
|
||||
Manual: Find a known mutation by actor/resource and a known run by status. Change filters, load more, and follow the source links. Reload the chosen view and verify no company leakage or duplicate events. Inspect a routine run from its audit entry.
|
||||
|
||||
On a native workspace export failure's run page, choose **Inspect recovery**,
|
||||
then **Inspect current source** on the matching action. Compare the recorded
|
||||
failure with the current recorded-source identity. Inspection is read-only and
|
||||
does not confirm provider termination or file preservation. Resolved retained
|
||||
actions remain inspectable. The control makes no inspection request until used.
|
||||
[Source/HTTP regressions](../../server/src/services/workspace-recovery-source.test.ts)
|
||||
cover current identity, rejected policy, foreign/reused allocation, bounded
|
||||
lookup, redaction and unchanged recovery state; [component regressions](../../ui/src/components/RunWorkspaceRecoverySurface.test.tsx)
|
||||
cover explicit selection, loading, failure and identity-only success. These
|
||||
checks do not establish recovery of a deployed task.
|
||||
|
||||
Case detail's **View in Audit** opens `/activity?entityType=case&entityId=<case UUID>`.
|
||||
The Activity filters include **Cases** actions and **Case** entities; the Case
|
||||
search accepts a UUID or human identifier such as `CAS-C1` (Enter or blur applies
|
||||
|
||||
@@ -78,10 +78,13 @@ fresh workspaces, bounded retries, historical adoption and service incidents.
|
||||
### `run-detail`
|
||||
|
||||
Open the agent's run (`/agents/:agentId/runs/:runId`) from activity or the task's
|
||||
run link. The run page hosts workspace recovery controls only for a failed run
|
||||
run link. The run page hosts workspace repair controls only for a failed run
|
||||
with error code `workspace_validation_failed` and a linked source task that still
|
||||
has a live `workspace_validation` recovery action. Use the source task for other
|
||||
stopped-run recovery checks.
|
||||
stopped-run recovery checks. Native workspace export failures also offer the
|
||||
read-only **Inspect recovery** control described in the
|
||||
[run diagnostic recipe](activity.md#activity-audit-1). It observes one selected
|
||||
source's recorded identity and does not retry, repair, or authorize execution.
|
||||
|
||||
### `operator-retry`
|
||||
|
||||
|
||||
@@ -1611,7 +1611,8 @@ export type {
|
||||
QuotaWindow,
|
||||
ProviderQuotaResult,
|
||||
} from "./types/index.js";
|
||||
export { WORKSPACE_READINESS_STATES } from "./types/index.js";
|
||||
export { WORKSPACE_READINESS_STATES, WORKSPACE_SOURCE_VERIFICATION_REASONS,
|
||||
type WorkspaceSourceVerificationFailure, type WorkspaceSourceInspection } from "./types/index.js";
|
||||
export {
|
||||
COMPANY_SEARCH_EXTRACT_KINDS,
|
||||
COMPANY_SEARCH_EXTRACT_SCOPES,
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
export { decisionEffectTargetIssueIds } from "./decision.js";
|
||||
export { WORKSPACE_SOURCE_VERIFICATION_REASONS, type WorkspaceSourceVerificationFailure,
|
||||
type WorkspaceSourceInspection } from "./workspace-source-verification.js";
|
||||
export {
|
||||
NATIVE_FINALIZATION_SCHEMA,
|
||||
type NativeFinalizationResult,
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
import type { EnvironmentLeasePolicy } from "../constants.js";
|
||||
|
||||
export const WORKSPACE_SOURCE_VERIFICATION_REASONS = [
|
||||
"source_run_missing", "recovery_action_missing", "recovery_action_run_mismatch", "source_run_not_terminal",
|
||||
"source_issue_mismatch", "native_reference_invalid", "restore_reference_invalid", "historical_reference_invalid",
|
||||
"lease_reference_invalid", "source_lease_not_bound", "lease_issue_mismatch", "lease_environment_missing",
|
||||
"lease_allocation_missing", "lease_provider_invalid", "lease_policy_not_retained", "lease_cwd_invalid",
|
||||
"allocation_reference_mismatch", "cwd_reference_mismatch", "workspace_reference_mismatch",
|
||||
"recovery_lease_not_selected", "recovery_environment_mismatch", "recovery_lease_not_claimed",
|
||||
"recovery_owner_mismatch", "recovery_stop_intent_missing", "local_descriptor_invalid", "execution_target_invalid",
|
||||
"recovery_stop_intent_invalid", "reopened_allocation_mismatch", "reopened_cwd_mismatch",
|
||||
] as const;
|
||||
|
||||
export interface WorkspaceSourceVerificationFailure {
|
||||
reason: (typeof WORKSPACE_SOURCE_VERIFICATION_REASONS)[number];
|
||||
observedLeasePolicy?: EnvironmentLeasePolicy | "unrecognized";
|
||||
}
|
||||
|
||||
export type WorkspaceSourceInspection = {
|
||||
schema: "paperclip.workspace-source-inspection.v1";
|
||||
actionId: string;
|
||||
runId: string;
|
||||
observedAt: string;
|
||||
scope: "recorded_source_identity";
|
||||
} & (
|
||||
| { status: "verified" }
|
||||
| { status: "blocked"; failureCode: string; sourceVerification?: WorkspaceSourceVerificationFailure }
|
||||
| { status: "not_inspected"; reason: "source_limit" }
|
||||
);
|
||||
@@ -29,6 +29,7 @@ import { reconstructWorkspaceSchema } from "@paperclipai/shared";
|
||||
import { queuedInteractionId, readQueuedInteractionResponse, hasQueuedInteractionResponse } from "../services/queued-interaction-response.js";
|
||||
import { deliverConversationComments, isConversation } from "../services/agent-conversations.js";
|
||||
import { issueRecoveryActionReadModel } from "../services/issue-recovery-actions.js";
|
||||
import { inspectWorkspaceSource } from "../services/workspace-source-inspection.js";
|
||||
import { getExecutionBlocker } from "../services/execution-blocker.js";
|
||||
import { EXECUTION_RECONCILIATION_CAUSES, requiresExecutionReconciliation } from "@paperclipai/shared";
|
||||
import {
|
||||
@@ -9590,6 +9591,25 @@ export function issueRoutes(
|
||||
);
|
||||
if (!issue) return;
|
||||
if (!(await assertIssueReadAllowed(req, res, issue))) return;
|
||||
if (req.query.inspectSourceRunId !== undefined || req.query.inspectSourceActionId !== undefined) {
|
||||
const selected = z.object({ inspectSourceRunId: z.string().uuid(), inspectSourceActionId: z.string().uuid().optional() })
|
||||
.safeParse(req.query);
|
||||
if (!selected.success) throw badRequest("A valid source run and optional recovery action are required.");
|
||||
const { inspectSourceRunId: runId, inspectSourceActionId: actionId } = selected.data;
|
||||
const rows = await db.select().from(issueRecoveryActions).where(and(
|
||||
eq(issueRecoveryActions.companyId, issue.companyId), eq(issueRecoveryActions.sourceIssueId, issue.id),
|
||||
sql`${issueRecoveryActions.evidence}->>'runId' = ${runId}`,
|
||||
actionId ? eq(issueRecoveryActions.id, actionId) : undefined,
|
||||
)).orderBy(desc(issueRecoveryActions.createdAt)).limit(actionId ? 1 : 20);
|
||||
if (actionId && !rows[0]) throw notFound("Recovery source not found.");
|
||||
const actions = rows.map(issueRecoveryActionReadModel);
|
||||
const sourceInspection = actionId ? await inspectWorkspaceSource(db, {
|
||||
companyId: issue.companyId, issueId: issue.id, runId, actionId,
|
||||
}) : undefined;
|
||||
res.json({ active: actions.find(action => ["active", "escalated"].includes(action.status)) ?? null,
|
||||
actions, ...(sourceInspection ? { sourceInspection } : {}) });
|
||||
return;
|
||||
}
|
||||
const active = await revalidateActiveSourceRecoveryForRead({
|
||||
issue,
|
||||
trigger: "read_projection",
|
||||
|
||||
@@ -45,6 +45,19 @@ describe("durable system recovery budget", () => {
|
||||
expect(state).toMatchObject({ state: "exhausted", nextAttemptAt: null,
|
||||
failure: { code: "source_binding_invalid", retryClass: "service_required" } });
|
||||
});
|
||||
it("persists only allowlisted source-verification evidence without changing the failure budget", () => {
|
||||
const state = createSystemRecovery(randomUUID());
|
||||
const failure = new SystemRecoveryFailure("source_binding_invalid", "service_required", undefined,
|
||||
{ reason: "lease_policy_not_retained", observedLeasePolicy: "reuse_by_environment", token: "secret" } as never);
|
||||
const result = failSystemRecovery(state, failure, { now: 1_000 });
|
||||
expect(readSystemRecovery({ systemRecovery: result })?.failure?.sourceVerification).toEqual({
|
||||
reason: "lease_policy_not_retained", observedLeasePolicy: "reuse_by_environment" });
|
||||
expect(JSON.stringify(result)).not.toContain("secret");
|
||||
expect(result).toMatchObject({ state: "exhausted", attempts: 0, nextAttemptAt: null, intentId: state.intentId });
|
||||
const invalid = failSystemRecovery(state, new SystemRecoveryFailure("source_binding_invalid", "service_required", undefined,
|
||||
{ reason: "arbitrary secret", observedLeasePolicy: "private policy" } as never));
|
||||
expect(invalid.failure?.sourceVerification).toBeUndefined();
|
||||
});
|
||||
it("reads prior incident metadata without losing the local recovery budget or failure", () => {
|
||||
const state = failSystemRecovery(createSystemRecovery(randomUUID()),
|
||||
new SystemRecoveryFailure("source_unavailable", "service_required"), { now: 1_000 });
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { z } from "zod";
|
||||
import { ENVIRONMENT_LEASE_POLICIES, WORKSPACE_SOURCE_VERIFICATION_REASONS,
|
||||
type WorkspaceSourceVerificationFailure } from "@paperclipai/shared";
|
||||
|
||||
export const SYSTEM_RECOVERY_POLICY_VERSION = 1;
|
||||
export const SYSTEM_RECOVERY_MAX_RETRIES = 5;
|
||||
@@ -21,11 +23,17 @@ export const recoveryRetryClasses = [
|
||||
export type RecoveryRetryClass = typeof recoveryRetryClasses[number];
|
||||
|
||||
export class SystemRecoveryFailure extends Error {
|
||||
constructor(readonly code: RecoveryFailureCode, readonly retryClass: RecoveryRetryClass, message = code) {
|
||||
constructor(readonly code: RecoveryFailureCode, readonly retryClass: RecoveryRetryClass, message = code,
|
||||
readonly sourceVerification?: WorkspaceSourceVerificationFailure) {
|
||||
super(message);
|
||||
}
|
||||
}
|
||||
|
||||
export const workspaceSourceVerificationFailureSchema = z.object({
|
||||
reason: z.enum(WORKSPACE_SOURCE_VERIFICATION_REASONS),
|
||||
observedLeasePolicy: z.enum([...ENVIRONMENT_LEASE_POLICIES, "unrecognized"]).optional(),
|
||||
});
|
||||
|
||||
export const systemRecoverySchema = z.object({
|
||||
schema: z.literal("paperclip.system-recovery.v1"),
|
||||
policyVersion: z.literal(SYSTEM_RECOVERY_POLICY_VERSION),
|
||||
@@ -46,6 +54,7 @@ export const systemRecoverySchema = z.object({
|
||||
code: z.enum(recoveryFailureCodes),
|
||||
retryClass: z.enum(recoveryRetryClasses),
|
||||
evidenceFingerprint: z.string(),
|
||||
sourceVerification: workspaceSourceVerificationFailureSchema.optional(),
|
||||
}).nullable(),
|
||||
preservationOperationIds: z.array(z.string().uuid()),
|
||||
freshAttemptRunId: z.string().uuid().nullable(),
|
||||
@@ -94,11 +103,13 @@ export function failSystemRecovery(
|
||||
const nextAttemptAt = Math.max(now + delay, options.retryAfterAt ?? 0);
|
||||
const retryable = ["retry_transient", "recheck_evidence"].includes(failure.retryClass);
|
||||
const retry = retryable && retryNumber <= SYSTEM_RECOVERY_MAX_RETRIES && nextAttemptAt < deadlineAt;
|
||||
const sourceVerification = workspaceSourceVerificationFailureSchema.safeParse(failure.sourceVerification);
|
||||
return { ...state, revision: state.revision + 1, firstFailureAt, deadlineAt,
|
||||
providerCooldownAt: options.retryAfterAt ?? null,
|
||||
state: retry ? failure.retryClass === "recheck_evidence" ? "awaiting_evidence" : "scheduled" : "exhausted",
|
||||
nextAttemptAt: retry ? nextAttemptAt : null,
|
||||
failure: { code: failure.code, retryClass: failure.retryClass,
|
||||
...(sourceVerification.success ? { sourceVerification: sourceVerification.data } : {}),
|
||||
evidenceFingerprint: options.evidenceFingerprint ?? createHash("sha256").update(`${state.sourceRunId}:${failure.code}`).digest("hex") },
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,183 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import express from "express";
|
||||
import request from "supertest";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { activityLog, agents, agentWakeupRequests, applyPendingMigrations, companies, createDb, environmentLeases, environments,
|
||||
heartbeatRuns, issueRecoveryActions, issues, workspaceOperations } from "@paperclipai/db";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
|
||||
import { inspectWorkspaceRecoverySource } from "./workspace-recovery-source.js";
|
||||
import { createSystemRecovery, failSystemRecovery, SystemRecoveryFailure } from "./system-recovery-state.js";
|
||||
import { inspectWorkspaceSource } from "./workspace-source-inspection.js";
|
||||
import { issueRoutes } from "../routes/issues.js";
|
||||
import { errorHandler } from "../middleware/error-handler.js";
|
||||
|
||||
describe("workspace source verification diagnostics", () => {
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
beforeAll(async () => {
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_WORKSPACE_RECOVERY_TEST_DATABASE_URL");
|
||||
if (external) {
|
||||
await applyPendingMigrations(external);
|
||||
temporary = { connectionString: external, cleanup: async () => {} };
|
||||
} else temporary = await startEmbeddedPostgresTestDatabase("source-verification-");
|
||||
db = createDb(temporary.connectionString);
|
||||
if (external) await db.execute(sql`TRUNCATE TABLE companies CASCADE`);
|
||||
}, 30_000);
|
||||
afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); });
|
||||
|
||||
async function seed() {
|
||||
const companyId = randomUUID(), issueId = randomUUID(), runId = randomUUID(), actionId = randomUUID(), agentId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Source verification", issuePrefix: `S${companyId.slice(0, 6)}` });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Worker", adapterType: "paperclip_runner" });
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Preserve the source", status: "blocked", assigneeAgentId: agentId });
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, nativeIssueId: issueId,
|
||||
runtimeMode: "native", status: "failed", finishedAt: new Date() });
|
||||
const [environment] = await db.insert(environments).values({ name: runId, driver: "sandbox" }).returning();
|
||||
const [lease] = await db.insert(environmentLeases).values({ companyId, issueId, heartbeatRunId: runId,
|
||||
environmentId: environment.id, provider: "daytona", providerLeaseId: randomUUID(), leasePolicy: "retain_on_failure",
|
||||
status: "released", cleanupStatus: "success", metadata: { remoteCwd: "/workspace", token: "secret-must-not-appear" } }).returning();
|
||||
const reference = { schema: "paperclip.native-workspace-sync/v3", state: "prepared", leaseId: lease.id,
|
||||
providerLeaseId: lease.providerLeaseId, workspaceId: issueId, remoteCwd: "/workspace", resourceDisposition: "keep_running",
|
||||
descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null };
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceSync: reference } }).where(eq(heartbeatRuns.id, runId));
|
||||
const systemRecovery = failSystemRecovery(createSystemRecovery(runId, randomUUID()),
|
||||
new SystemRecoveryFailure("source_binding_invalid", "service_required"));
|
||||
await db.insert(issueRecoveryActions).values({ id: actionId, companyId, sourceIssueId: issueId,
|
||||
kind: "active_run_watchdog", ownerType: "board", cause: "native_workspace_sync_out_retry_exhausted", fingerprint: runId,
|
||||
nextAction: "Inspect the retained source", evidence: { runId, systemRecovery,
|
||||
workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId, status: "blocked",
|
||||
stage: "verifying_stop", attempt: 1, attemptLimit: 6, requestIds: [] } } });
|
||||
return { db, companyId, issueId, runId, actionId, agentId, environment, lease, reference };
|
||||
}
|
||||
|
||||
function app(actor: express.Request["actor"] = { type: "board", source: "local_implicit" }) {
|
||||
const result = express();
|
||||
result.use((req, _res, next) => { req.actor = actor; next(); });
|
||||
result.use("/api", issueRoutes(db, {} as Parameters<typeof issueRoutes>[1]));
|
||||
result.use(errorHandler);
|
||||
return result;
|
||||
}
|
||||
|
||||
function path(f: Awaited<ReturnType<typeof seed>>, actionId: string = f.actionId, runId: string = f.runId) {
|
||||
return `/api/issues/${f.issueId}/recovery-actions?inspectSourceRunId=${runId}&inspectSourceActionId=${actionId}`;
|
||||
}
|
||||
|
||||
async function snapshot(f: Awaited<ReturnType<typeof seed>>) {
|
||||
return {
|
||||
actions: await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, f.companyId)),
|
||||
runs: await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId)),
|
||||
leases: await db.select().from(environmentLeases).where(eq(environmentLeases.companyId, f.companyId)),
|
||||
issues: await db.select().from(issues).where(eq(issues.companyId, f.companyId)),
|
||||
wakeups: await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId)),
|
||||
operations: await db.select().from(workspaceOperations).where(eq(workspaceOperations.companyId, f.companyId)),
|
||||
activity: await db.select().from(activityLog).where(eq(activityLog.companyId, f.companyId)),
|
||||
};
|
||||
}
|
||||
|
||||
it.each(["lease_policy_not_retained", "source_lease_not_bound"])("identifies the rejected contract: %s", async reason => {
|
||||
const f = await seed();
|
||||
if (reason === "lease_policy_not_retained") {
|
||||
await db.update(environmentLeases).set({ leasePolicy: "reuse_by_environment" }).where(eq(environmentLeases.id, f.lease.id));
|
||||
} else {
|
||||
const otherRunId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: otherRunId, companyId: f.companyId, agentId: f.agentId,
|
||||
issueId: f.issueId, status: "running" });
|
||||
await db.update(environmentLeases).set({ heartbeatRunId: otherRunId }).where(eq(environmentLeases.id, f.lease.id));
|
||||
}
|
||||
await expect(inspectWorkspaceRecoverySource(db, f)).rejects.toMatchObject({
|
||||
code: "source_binding_invalid", retryClass: "service_required", sourceVerification: { reason },
|
||||
});
|
||||
});
|
||||
|
||||
it("inspects a resolved retained source through HTTP without revalidating or changing recorded recovery", async () => {
|
||||
const f = await seed();
|
||||
await db.update(environmentLeases).set({ leasePolicy: "reuse_by_environment" }).where(eq(environmentLeases.id, f.lease.id));
|
||||
await db.update(issueRecoveryActions).set({ status: "resolved", outcome: "blocked", evidence: {
|
||||
...(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId)))[0].evidence,
|
||||
automaticRecovery: { policy: "preserve_without_replay_v1", replay: "blocked", runId: f.runId },
|
||||
} }).where(eq(issueRecoveryActions.id, f.actionId));
|
||||
const before = await snapshot(f);
|
||||
const response = await request(app()).get(path(f)).expect(200);
|
||||
expect(response.body.active).toBeNull();
|
||||
expect(response.body.actions[0]).toMatchObject({ id: f.actionId, status: "resolved", evidence: {
|
||||
systemRecovery: { state: "exhausted", failure: { code: "source_binding_invalid" } },
|
||||
} });
|
||||
expect(response.body.sourceInspection).toMatchObject({ actionId: f.actionId, runId: f.runId,
|
||||
scope: "recorded_source_identity", status: "blocked", failureCode: "source_binding_invalid",
|
||||
sourceVerification: { reason: "lease_policy_not_retained", observedLeasePolicy: "reuse_by_environment" } });
|
||||
expect(response.body.actions[0].evidence.systemRecovery.failure.sourceVerification).toBeUndefined();
|
||||
expect(JSON.stringify(response.body)).not.toContain("secret-must-not-appear");
|
||||
expect(JSON.stringify(response.body)).not.toContain("/workspace");
|
||||
expect(JSON.stringify(response.body)).not.toContain(f.lease.providerLeaseId);
|
||||
expect(await snapshot(f)).toEqual(before);
|
||||
});
|
||||
|
||||
it("only verifies current recorded identity for a valid retained source", async () => {
|
||||
const f = await seed();
|
||||
const before = await snapshot(f);
|
||||
const response = await request(app()).get(path(f)).expect(200);
|
||||
expect(response.body.sourceInspection).toEqual({ schema: "paperclip.workspace-source-inspection.v1",
|
||||
actionId: f.actionId, runId: f.runId, scope: "recorded_source_identity", status: "verified",
|
||||
observedAt: expect.any(String) });
|
||||
expect(await snapshot(f)).toEqual(before);
|
||||
});
|
||||
|
||||
it("keeps a reused physical allocation blocked without touching either lease", async () => {
|
||||
const f = await seed();
|
||||
await db.insert(environmentLeases).values({ companyId: f.companyId, environmentId: f.environment.id,
|
||||
provider: "daytona", providerLeaseId: f.lease.providerLeaseId, leasePolicy: "reuse_by_environment", status: "active" });
|
||||
const before = await snapshot(f);
|
||||
const response = await request(app()).get(path(f)).expect(200);
|
||||
expect(response.body.sourceInspection).toMatchObject({ status: "blocked", failureCode: "allocation_ownership_ambiguous" });
|
||||
expect(response.body.sourceInspection.sourceVerification).toBeUndefined();
|
||||
expect(await snapshot(f)).toEqual(before);
|
||||
});
|
||||
|
||||
it("rejects a foreign action, mismatched run, and malformed selection without inspection", async () => {
|
||||
const f = await seed(), foreign = await seed();
|
||||
const before = await snapshot(f);
|
||||
await request(app()).get(path(f, foreign.actionId)).expect(404);
|
||||
await request(app()).get(path(f, f.actionId, foreign.runId)).expect(404);
|
||||
await request(app()).get(path(f, "not-a-uuid")).expect(400);
|
||||
expect(await snapshot(f)).toEqual(before);
|
||||
});
|
||||
|
||||
it("does not let another company's agent inspect the source", async () => {
|
||||
const f = await seed(), foreign = await seed();
|
||||
const response = await request(app({ type: "agent", agentId: foreign.agentId, companyId: foreign.companyId,
|
||||
source: "agent_key" })).get(path(f));
|
||||
expect([403, 404]).toContain(response.status);
|
||||
expect(response.body.sourceInspection).toBeUndefined();
|
||||
});
|
||||
|
||||
it("lists only the explicitly requested run without inspecting it", async () => {
|
||||
const f = await seed();
|
||||
await db.update(environmentLeases).set({ leasePolicy: "reuse_by_environment" }).where(eq(environmentLeases.id, f.lease.id));
|
||||
const response = await request(app()).get(`/api/issues/${f.issueId}/recovery-actions?inspectSourceRunId=${f.runId}`).expect(200);
|
||||
expect(response.body.actions.map((action: { id: string }) => action.id)).toEqual([f.actionId]);
|
||||
expect(response.body.sourceInspection).toBeUndefined();
|
||||
});
|
||||
|
||||
it("bounds explicit inspection before lease lookup without changing worker verification", async () => {
|
||||
const f = await seed();
|
||||
const leaseIds = [f.lease.id];
|
||||
for (let index = 0; index < 8; index++) {
|
||||
const [lease] = await db.insert(environmentLeases).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
heartbeatRunId: f.runId, environmentId: f.environment.id, provider: "daytona", providerLeaseId: randomUUID(),
|
||||
leasePolicy: "retain_on_failure", status: "released", metadata: { remoteCwd: "/workspace" } }).returning();
|
||||
leaseIds.push(lease.id);
|
||||
}
|
||||
await db.update(heartbeatRuns).set({ resultJson: { workspaceRestoreRecovery: {
|
||||
schema: "paperclip.workspace-restore-recovery.v1", leaseIds } } }).where(eq(heartbeatRuns.id, f.runId));
|
||||
const before = await snapshot(f);
|
||||
const select = vi.spyOn(db, "select");
|
||||
try {
|
||||
expect(await inspectWorkspaceSource(db, f)).toMatchObject({ status: "not_inspected", reason: "source_limit" });
|
||||
expect(select).toHaveBeenCalledTimes(2);
|
||||
} finally { select.mockRestore(); }
|
||||
expect((await inspectWorkspaceRecoverySource(db, f)).sources).toHaveLength(9);
|
||||
expect(await snapshot(f)).toEqual(before);
|
||||
});
|
||||
});
|
||||
@@ -5,11 +5,23 @@ import { readNativeWorkspaceSyncReference } from "./native-runtime/native-worksp
|
||||
import { SystemRecoveryFailure } from "./system-recovery-state.js";
|
||||
import type { NativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
|
||||
import { readStopOnlyCleanup } from "./sandbox-stop-and-retain.js";
|
||||
import { ENVIRONMENT_LEASE_POLICIES, type EnvironmentLeasePolicy,
|
||||
type WorkspaceSourceVerificationFailure } from "@paperclipai/shared";
|
||||
|
||||
export const workspaceExportCauses = ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"];
|
||||
export const terminalWorkspaceRunStates = ["failed", "interrupted", "timed_out", "cancelled", "succeeded"];
|
||||
export type WorkspaceRecoverySourceIdentity = { companyId: string; issueId: string; runId: string; actionId: string };
|
||||
|
||||
export function workspaceSourceBindingFailure(reason: WorkspaceSourceVerificationFailure["reason"], policy?: unknown) {
|
||||
const observedLeasePolicy = typeof policy === "string" && (ENVIRONMENT_LEASE_POLICIES as readonly string[]).includes(policy)
|
||||
? policy as EnvironmentLeasePolicy : "unrecognized";
|
||||
return new SystemRecoveryFailure("source_binding_invalid", "service_required", undefined, {
|
||||
reason, ...(reason === "lease_policy_not_retained" ? { observedLeasePolicy } : {}),
|
||||
});
|
||||
}
|
||||
|
||||
export class WorkspaceSourceInspectionLimit extends Error {}
|
||||
|
||||
export function workspaceRecoverySourceEligible(includeCompleted = false) {
|
||||
const actionEligible = sql`(${issueRecoveryActions.status} in ('active', 'escalated') or
|
||||
(${issueRecoveryActions.status} = 'resolved' and (
|
||||
@@ -41,39 +53,46 @@ export async function ownedWorkspaceRecoveryLease(db: Db, input: WorkspaceRecove
|
||||
await input.ownership.assertHeld();
|
||||
const inspected = await inspectWorkspaceRecoverySource(db, input);
|
||||
const source = inspected.sources.find(item => item.lease.id === input.leaseId);
|
||||
if (!source || source.environment.id !== input.environmentId || source.lease.status !== "pending_cleanup"
|
||||
|| source.lease.metadata?.pendingCleanupInFlight !== true
|
||||
|| source.lease.metadata?.pendingCleanupAttemptId !== input.ownership.token
|
||||
|| !readStopOnlyCleanup(source.lease)) {
|
||||
throw new SystemRecoveryFailure("source_binding_invalid", "service_required");
|
||||
}
|
||||
if (!source) throw workspaceSourceBindingFailure("recovery_lease_not_selected");
|
||||
if (source.environment.id !== input.environmentId) throw workspaceSourceBindingFailure("recovery_environment_mismatch");
|
||||
if (source.lease.status !== "pending_cleanup" || source.lease.metadata?.pendingCleanupInFlight !== true)
|
||||
throw workspaceSourceBindingFailure("recovery_lease_not_claimed");
|
||||
if (source.lease.metadata?.pendingCleanupAttemptId !== input.ownership.token)
|
||||
throw workspaceSourceBindingFailure("recovery_owner_mismatch");
|
||||
if (!readStopOnlyCleanup(source.lease)) throw workspaceSourceBindingFailure("recovery_stop_intent_missing");
|
||||
await input.ownership.assertHeld();
|
||||
return source;
|
||||
}
|
||||
|
||||
/** Resolve only the immutable source recorded by the failed run. Never discover
|
||||
* a candidate by recency, mutable adapter configuration, or a provider search. */
|
||||
export async function inspectWorkspaceRecoverySource(db: Db, input: WorkspaceRecoverySourceIdentity) {
|
||||
const invalid = () => new SystemRecoveryFailure("source_binding_invalid", "service_required");
|
||||
export async function inspectWorkspaceRecoverySource(db: Db, input: WorkspaceRecoverySourceIdentity,
|
||||
options: { maximumSources?: number } = {}) {
|
||||
const invalid = workspaceSourceBindingFailure;
|
||||
const [run] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId)));
|
||||
const [action] = await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, input.companyId),
|
||||
eq(issueRecoveryActions.sourceIssueId, input.issueId), eq(issueRecoveryActions.id, input.actionId)));
|
||||
if (!run || !action || action.evidence.runId !== run.id || !terminalWorkspaceRunStates.includes(run.status) ||
|
||||
(run.nativeIssueId ?? run.issueId ?? run.contextSnapshot?.issueId) !== input.issueId) throw invalid();
|
||||
if (!run) throw invalid("source_run_missing");
|
||||
if (!action) throw invalid("recovery_action_missing");
|
||||
if (action.evidence.runId !== run.id) throw invalid("recovery_action_run_mismatch");
|
||||
if (!terminalWorkspaceRunStates.includes(run.status)) throw invalid("source_run_not_terminal");
|
||||
if ((run.nativeIssueId ?? run.issueId ?? run.contextSnapshot?.issueId) !== input.issueId) throw invalid("source_issue_mismatch");
|
||||
const rawReference = run.runnerProfileJson?.nativeWorkspaceSync;
|
||||
const reference = readNativeWorkspaceSyncReference(rawReference);
|
||||
if (rawReference != null && !reference) throw invalid();
|
||||
if (rawReference != null && !reference) throw invalid("native_reference_invalid");
|
||||
const recorded = run.resultJson?.workspaceRestoreRecovery as { schema?: unknown; leaseIds?: unknown } | undefined;
|
||||
if (recorded != null && recorded.schema !== "paperclip.workspace-restore-recovery.v1") throw invalid();
|
||||
if (recorded != null && recorded.schema !== "paperclip.workspace-restore-recovery.v1") throw invalid("restore_reference_invalid");
|
||||
const historical = action.evidence.historicalWorkspaceSource as { schema?: unknown; runId?: unknown; leaseIds?: unknown } | undefined;
|
||||
if (historical != null && (historical.schema !== "paperclip.historical-workspace-source.v1" || historical.runId !== run.id)) throw invalid();
|
||||
if (historical != null && (historical.schema !== "paperclip.historical-workspace-source.v1" || historical.runId !== run.id)) throw invalid("historical_reference_invalid");
|
||||
const leaseIds = recorded ? recorded.leaseIds
|
||||
: reference && workspaceExportCauses.includes(action.cause) ? [reference.leaseId]
|
||||
: historical?.leaseIds;
|
||||
if (!Array.isArray(leaseIds) || leaseIds.length === 0)
|
||||
throw new SystemRecoveryFailure("historical_source_missing", "use_alternative_strategy");
|
||||
if (options.maximumSources !== undefined && leaseIds.length > options.maximumSources)
|
||||
throw new WorkspaceSourceInspectionLimit();
|
||||
if (leaseIds.some(id => typeof id !== "string" || !/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(id)) ||
|
||||
new Set(leaseIds).size !== leaseIds.length || (reference && !leaseIds.includes(reference.leaseId))) throw invalid();
|
||||
new Set(leaseIds).size !== leaseIds.length || (reference && !leaseIds.includes(reference.leaseId))) throw invalid("lease_reference_invalid");
|
||||
const [coordinator] = await db.select().from(nativeRunFinalizations).where(and(
|
||||
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, run.id)));
|
||||
if (coordinator?.leaseOwner || adapterExecutionControls.has(run.id) ||
|
||||
@@ -85,11 +104,17 @@ export async function inspectWorkspaceRecoverySource(db: Db, input: WorkspaceRec
|
||||
eq(environmentLeases.id, leaseId), eq(environmentLeases.heartbeatRunId, run.id)));
|
||||
const descriptor = reference?.leaseId === leaseId ? reference : null;
|
||||
const remoteCwd = lease?.metadata?.remoteCwd;
|
||||
if (!lease || lease.issueId !== input.issueId || !lease.environmentId || !lease.providerLeaseId ||
|
||||
!lease.provider || lease.provider === "local" || !["retain_on_failure", "ephemeral"].includes(lease.leasePolicy) ||
|
||||
typeof remoteCwd !== "string" || !remoteCwd.startsWith("/") ||
|
||||
(descriptor && (lease.providerLeaseId !== descriptor.providerLeaseId || remoteCwd !== descriptor.remoteCwd ||
|
||||
(lease.executionWorkspaceId && lease.executionWorkspaceId !== descriptor.workspaceId)))) throw invalid();
|
||||
if (!lease) throw invalid("source_lease_not_bound");
|
||||
if (lease.issueId !== input.issueId) throw invalid("lease_issue_mismatch");
|
||||
if (!lease.environmentId) throw invalid("lease_environment_missing");
|
||||
if (!lease.providerLeaseId) throw invalid("lease_allocation_missing");
|
||||
if (!lease.provider || lease.provider === "local") throw invalid("lease_provider_invalid");
|
||||
if (!["retain_on_failure", "ephemeral"].includes(lease.leasePolicy)) throw invalid("lease_policy_not_retained", lease.leasePolicy);
|
||||
if (typeof remoteCwd !== "string" || !remoteCwd.startsWith("/")) throw invalid("lease_cwd_invalid");
|
||||
if (descriptor && lease.providerLeaseId !== descriptor.providerLeaseId) throw invalid("allocation_reference_mismatch");
|
||||
if (descriptor && remoteCwd !== descriptor.remoteCwd) throw invalid("cwd_reference_mismatch");
|
||||
if (descriptor && lease.executionWorkspaceId && lease.executionWorkspaceId !== descriptor.workspaceId)
|
||||
throw invalid("workspace_reference_mismatch");
|
||||
const [other] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
|
||||
ne(environmentLeases.id, lease.id), eq(environmentLeases.provider, lease.provider),
|
||||
eq(environmentLeases.providerLeaseId, lease.providerLeaseId))).limit(1);
|
||||
|
||||
@@ -21,7 +21,7 @@ import { classifyNativeWorkspaceFailure } from "./native-runtime/native-workspac
|
||||
|
||||
import { adoptSystemRecovery, loadSystemRecovery, updateSystemRecovery } from "./system-recovery.js";
|
||||
import { createSystemRecovery, failSystemRecovery, readSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure, type SystemRecoveryState } from "./system-recovery-state.js";
|
||||
import { inspectWorkspaceRecoverySource, workspaceRecoverySourceEligible } from "./workspace-recovery-source.js";
|
||||
import { inspectWorkspaceRecoverySource, workspaceRecoverySourceEligible, workspaceSourceBindingFailure } from "./workspace-recovery-source.js";
|
||||
import { preserveWorkspaceSnapshot } from "./workspace-recovery-snapshot.js";
|
||||
import { adoptHistoricalWorkspaceRecoveries } from "./workspace-recovery-backfill.js";
|
||||
|
||||
@@ -132,7 +132,7 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ
|
||||
// Missing local merge metadata permits a separate snapshot of the
|
||||
// exact bound allocation. Corrupt metadata never weakens binding.
|
||||
if ((error as NodeJS.ErrnoException)?.code === "ENOENT") reference = null;
|
||||
else throw new SystemRecoveryFailure("source_binding_invalid", "service_required");
|
||||
else throw workspaceSourceBindingFailure("local_descriptor_invalid");
|
||||
}
|
||||
}
|
||||
let stoppedLease = lease;
|
||||
@@ -168,7 +168,7 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ
|
||||
environment, leaseId: lease.id, leaseMetadata: stoppedLease.metadata,
|
||||
lease: stoppedLease as EnvironmentLease, environmentRuntime: input.environmentRuntime });
|
||||
if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner || target.remoteCwd !== remoteCwd)
|
||||
throw new SystemRecoveryFailure("source_binding_invalid", "service_required");
|
||||
throw workspaceSourceBindingFailure("execution_target_invalid");
|
||||
await ownership.assertHeld();
|
||||
const cleanupLease = await db.transaction(async tx => {
|
||||
const [fresh] = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
|
||||
@@ -176,7 +176,7 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ
|
||||
if (!fresh || !hasRemoteTerminationReceipt(fresh) || fresh.providerLeaseId !== lease.providerLeaseId)
|
||||
throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
|
||||
const pending = await prepareStoppedSandboxForRecovery(tx as unknown as Db, fresh);
|
||||
if (!pending) throw new SystemRecoveryFailure("source_binding_invalid", "service_required");
|
||||
if (!pending) throw workspaceSourceBindingFailure("recovery_stop_intent_invalid");
|
||||
const [claimed] = await tx.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({
|
||||
pendingCleanupAttemptId: ownership.token, pendingCleanupInFlight: true, pendingCleanupLeaseExpiresAtMs: Date.now() + 15 * 60_000,
|
||||
})}::jsonb` }).where(eq(environmentLeases.id, fresh.id)).returning();
|
||||
@@ -192,8 +192,8 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ
|
||||
try {
|
||||
const opened = await input.environmentRuntime.resumeWorkspaceRecoveryLease({ source: input,
|
||||
leaseId: cleanupLease.id, environmentId: environment.id, ownership });
|
||||
if (opened?.providerLeaseId !== lease.providerLeaseId || opened.metadata?.remoteCwd !== remoteCwd)
|
||||
throw new SystemRecoveryFailure("source_binding_invalid", "service_required");
|
||||
if (opened?.providerLeaseId !== lease.providerLeaseId) throw workspaceSourceBindingFailure("reopened_allocation_mismatch");
|
||||
if (opened.metadata?.remoteCwd !== remoteCwd) throw workspaceSourceBindingFailure("reopened_cwd_mismatch");
|
||||
await inspectWorkspaceRecoverySource(db, input);
|
||||
await progress({ phase: "preserve_files" });
|
||||
let operationId: string;
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import type { WorkspaceSourceInspection } from "@paperclipai/shared";
|
||||
import { inspectWorkspaceRecoverySource, WorkspaceSourceInspectionLimit,
|
||||
type WorkspaceRecoverySourceIdentity } from "./workspace-recovery-source.js";
|
||||
import { SystemRecoveryFailure, workspaceSourceVerificationFailureSchema } from "./system-recovery-state.js";
|
||||
|
||||
export const MAX_INSPECTED_WORKSPACE_SOURCES = 8;
|
||||
|
||||
export async function inspectWorkspaceSource(db: Db, input: WorkspaceRecoverySourceIdentity): Promise<WorkspaceSourceInspection> {
|
||||
const base = { schema: "paperclip.workspace-source-inspection.v1" as const, actionId: input.actionId,
|
||||
runId: input.runId, observedAt: new Date().toISOString(), scope: "recorded_source_identity" as const };
|
||||
try {
|
||||
await inspectWorkspaceRecoverySource(db, input, { maximumSources: MAX_INSPECTED_WORKSPACE_SOURCES });
|
||||
return { ...base, status: "verified" };
|
||||
} catch (error) {
|
||||
if (error instanceof WorkspaceSourceInspectionLimit) return { ...base, status: "not_inspected", reason: "source_limit" };
|
||||
if (!(error instanceof SystemRecoveryFailure)) throw error;
|
||||
const detail = workspaceSourceVerificationFailureSchema.safeParse(error.sourceVerification);
|
||||
return { ...base, status: "blocked", failureCode: error.code,
|
||||
...(detail.success ? { sourceVerification: detail.data } : {}) };
|
||||
}
|
||||
}
|
||||
@@ -26,6 +26,7 @@ import type {
|
||||
IssueDocument,
|
||||
IssueLabel,
|
||||
IssueRecoveryAction,
|
||||
WorkspaceSourceInspection,
|
||||
IssueRetryNowResponse,
|
||||
StalledReviewDecision,
|
||||
StalledReviewDecisionResponse,
|
||||
@@ -317,8 +318,12 @@ export const issuesApi = {
|
||||
reconstructWorkspace: (id: string, data: { actionId: string; runId: string; expectedStatusVersion: number; attachmentIds: string[];
|
||||
missingWork: string; actionOutcome: "completed" | "not_performed" | "mixed"; outcomeEvidence: string; acknowledgeLostWork: true }) =>
|
||||
api.post<{ runId: string; actionId: string; status: "queued"; replayed: boolean }>(`/issues/${id}/recovery-actions/reconstruct-workspace`, data),
|
||||
recoveryActions: (id: string) =>
|
||||
api.get<{ active: IssueRecoveryAction | null; actions: IssueRecoveryAction[] }>(`/issues/${id}/recovery-actions`),
|
||||
recoveryActions: (id: string, source?: { runId: string; actionId?: string }) => {
|
||||
const query = source ? new URLSearchParams({ inspectSourceRunId: source.runId,
|
||||
...(source.actionId ? { inspectSourceActionId: source.actionId } : {}) }) : null;
|
||||
return api.get<{ active: IssueRecoveryAction | null; actions: IssueRecoveryAction[];
|
||||
sourceInspection?: WorkspaceSourceInspection }>(`/issues/${id}/recovery-actions${query ? `?${query}` : ""}`);
|
||||
},
|
||||
resolveRecoveryAction: (
|
||||
id: string,
|
||||
data: {
|
||||
|
||||
@@ -15,6 +15,7 @@ const issueCreateMock = vi.hoisted(() => vi.fn());
|
||||
const resolveRecoveryMock = vi.hoisted(() => vi.fn());
|
||||
const reconcileMock = vi.hoisted(() => vi.fn());
|
||||
const boardAccessMock = vi.hoisted(() => vi.fn());
|
||||
const recoveryActionsMock = vi.hoisted(() => vi.fn());
|
||||
|
||||
vi.mock("@/lib/router", () => ({
|
||||
useNavigate: () => navigateMock,
|
||||
@@ -28,6 +29,7 @@ vi.mock("../api/issues", () => ({
|
||||
get: issueGetMock,
|
||||
create: issueCreateMock,
|
||||
resolveRecoveryAction: resolveRecoveryMock,
|
||||
recoveryActions: recoveryActionsMock,
|
||||
},
|
||||
}));
|
||||
|
||||
@@ -68,7 +70,7 @@ afterEach(() => {
|
||||
root = null;
|
||||
container?.remove();
|
||||
container = null;
|
||||
vi.clearAllMocks();
|
||||
vi.resetAllMocks();
|
||||
});
|
||||
|
||||
function buildRun(overrides: Partial<HeartbeatRun> = {}): HeartbeatRun {
|
||||
@@ -178,6 +180,104 @@ function click(element: Element | null) {
|
||||
}
|
||||
|
||||
describe("RunWorkspaceRecoverySurface", () => {
|
||||
it("inspects a selected resolved source only on request and separates current identity from the recorded failure", async () => {
|
||||
const action = { ...buildRecoveryAction(), status: "resolved", cause: "native_workspace_sync_out_retry_exhausted",
|
||||
evidence: { systemRecovery: { failure: { code: "source_binding_invalid" } } } };
|
||||
recoveryActionsMock.mockResolvedValueOnce({ active: null, actions: [action] }).mockResolvedValueOnce({ active: null,
|
||||
actions: [action], sourceInspection: { actionId: action.id, runId: "run-aaaa1111", scope: "recorded_source_identity",
|
||||
observedAt: "2026-10-26T17:00:00.000Z", status: "blocked", failureCode: "source_binding_invalid",
|
||||
sourceVerification: { reason: "lease_policy_not_retained", observedLeasePolicy: "reuse_by_environment" } } });
|
||||
const node = await renderSurface(buildRun({ errorCode: "native_workspace_sync_out_retry_exhausted" }));
|
||||
expect(recoveryActionsMock).not.toHaveBeenCalled();
|
||||
expect(issueGetMock).not.toHaveBeenCalled();
|
||||
click(node.querySelector("button"));
|
||||
await flush();
|
||||
expect(recoveryActionsMock).toHaveBeenLastCalledWith("issue-1", { runId: "run-aaaa1111" });
|
||||
expect(node.textContent).toContain("Recorded failure: source_binding_invalid");
|
||||
const inspect = [...node.querySelectorAll("button")].find(button => button.textContent === "Inspect current source");
|
||||
click(inspect ?? null);
|
||||
await flush();
|
||||
expect(recoveryActionsMock).toHaveBeenLastCalledWith("issue-1", { runId: "run-aaaa1111", actionId: "action-1" });
|
||||
expect(node.textContent).toContain("Current recorded-source identity");
|
||||
expect(node.textContent).toContain("lease_policy_not_retained");
|
||||
expect(node.textContent).toContain("reuse_by_environment");
|
||||
expect(node.textContent).toContain("does not confirm provider termination, local files, or successful preservation");
|
||||
expect(issueCreateMock).not.toHaveBeenCalled();
|
||||
expect(resolveRecoveryMock).not.toHaveBeenCalled();
|
||||
expect(reconcileMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("shows verified identity without claiming preservation and reports an inspection error inline", async () => {
|
||||
const action = buildRecoveryAction();
|
||||
recoveryActionsMock.mockResolvedValueOnce({ active: null, actions: [action] })
|
||||
.mockResolvedValueOnce({ active: null, actions: [action], sourceInspection: {
|
||||
actionId: action.id, status: "verified", observedAt: "2026-10-26T17:00:00.000Z" } })
|
||||
.mockRejectedValueOnce(new Error("Source inspection unavailable"));
|
||||
const node = await renderSurface(buildRun({ errorCode: "native_workspace_sync_out_unrecoverable" }));
|
||||
click(node.querySelector("button"));
|
||||
await flush();
|
||||
click([...node.querySelectorAll("button")].find(button => button.textContent === "Inspect current source") ?? null);
|
||||
await flush();
|
||||
expect(recoveryActionsMock).toHaveBeenLastCalledWith("issue-1", { runId: "run-aaaa1111", actionId: action.id });
|
||||
expect(node.textContent).toContain("Recorded-source identity verified.");
|
||||
expect(node.textContent).toContain("does not confirm provider termination");
|
||||
click([...node.querySelectorAll("button")].find(button => button.textContent === "Inspect current source") ?? null);
|
||||
await flush();
|
||||
expect(node.querySelector("[role='alert']")?.textContent).toContain("Source inspection unavailable");
|
||||
});
|
||||
|
||||
it("keeps inspection pending and explains a bounded source limit", async () => {
|
||||
const action = buildRecoveryAction();
|
||||
let complete!: (value: unknown) => void;
|
||||
recoveryActionsMock.mockResolvedValueOnce({ active: null, actions: [action] })
|
||||
.mockImplementationOnce(() => new Promise(resolve => { complete = resolve; }));
|
||||
const node = await renderSurface(buildRun({ errorCode: "native_workspace_sync_out_retry_exhausted" }));
|
||||
click(node.querySelector("button"));
|
||||
await flush();
|
||||
click([...node.querySelectorAll("button")].find(button => button.textContent === "Inspect current source") ?? null);
|
||||
await flush();
|
||||
expect(node.querySelector("button")?.disabled).toBe(true);
|
||||
expect(recoveryActionsMock).toHaveBeenLastCalledWith("issue-1", { runId: "run-aaaa1111", actionId: action.id });
|
||||
complete({ active: null, actions: [action], sourceInspection: { actionId: action.id,
|
||||
observedAt: "2026-10-26T17:00:00.000Z", status: "not_inspected", reason: "source_limit" } });
|
||||
await flush();
|
||||
expect(node.textContent).toContain("exceeds the inspection limit");
|
||||
expect(node.querySelector("button")?.disabled).toBe(false);
|
||||
});
|
||||
|
||||
it("keeps saved reasons separate from changed observations and preserves other source selections", async () => {
|
||||
const first = { ...buildRecoveryAction(), evidence: { systemRecovery: { failure: {
|
||||
code: "source_binding_invalid", sourceVerification: { reason: "lease_policy_not_retained", observedLeasePolicy: "reuse_by_environment" },
|
||||
} } } };
|
||||
const second = { ...first, id: "action-2", evidence: { systemRecovery: { failure: { code: "source_binding_invalid",
|
||||
sourceVerification: { reason: "secret-reason", observedLeasePolicy: "secret-policy" } } } } };
|
||||
recoveryActionsMock.mockResolvedValueOnce({ active: null, actions: [first, second] })
|
||||
.mockResolvedValueOnce({ active: null, actions: [first], sourceInspection: { actionId: first.id,
|
||||
status: "blocked", observedAt: "2026-10-26T17:00:00.000Z", failureCode: "source_binding_invalid",
|
||||
sourceVerification: { reason: "source_lease_not_bound" } } })
|
||||
.mockResolvedValueOnce({ active: null, actions: [second], sourceInspection: { actionId: second.id,
|
||||
status: "verified", observedAt: "2026-10-26T17:01:00.000Z" } });
|
||||
const node = await renderSurface(buildRun({ errorCode: "native_workspace_sync_out_retry_exhausted" }));
|
||||
click(node.querySelector("button"));
|
||||
await flush();
|
||||
expect(node.textContent).toContain("Recorded failure: source_binding_invalid · lease_policy_not_retained · lease policy reuse_by_environment");
|
||||
expect(node.textContent).not.toContain("secret-reason");
|
||||
expect(node.textContent).not.toContain("secret-policy");
|
||||
const choices = () => [...node.querySelectorAll("button")].filter(button => button.textContent === "Inspect current source");
|
||||
click(choices()[0]);
|
||||
await flush();
|
||||
expect(node.textContent).toContain("source_lease_not_bound");
|
||||
expect(node.textContent).toContain("Action action-1");
|
||||
expect(node.textContent).toContain("lease_policy_not_retained");
|
||||
expect(choices()).toHaveLength(2);
|
||||
click(choices()[1]);
|
||||
await flush();
|
||||
expect(recoveryActionsMock).toHaveBeenLastCalledWith("issue-1", { runId: "run-aaaa1111", actionId: "action-2" });
|
||||
expect(node.textContent).toContain("Action action-2");
|
||||
expect(node.textContent).toContain("Recorded-source identity verified.");
|
||||
expect(choices()).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("renders nothing for a run that is not a workspace-validation failure", async () => {
|
||||
issueGetMock.mockResolvedValue(buildIssue(buildRecoveryAction()));
|
||||
boardAccessMock.mockResolvedValue(undefined);
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
import { useCallback } from "react";
|
||||
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import type { HeartbeatRun } from "@paperclipai/shared";
|
||||
import { isNativeWorkspaceExportRepairCause } from "@paperclipai/shared";
|
||||
import { RunWorkspaceSourceInspection } from "./RunWorkspaceSourceInspection";
|
||||
import { useNavigate } from "@/lib/router";
|
||||
import { issuesApi } from "../api/issues";
|
||||
import { executionWorkspacesApi } from "../api/execution-workspaces";
|
||||
@@ -52,6 +54,14 @@ function readRunIssueId(run: HeartbeatRun): string | null {
|
||||
* a live `workspace_validation` recovery action.
|
||||
*/
|
||||
export function RunWorkspaceRecoverySurface({ run }: { run: HeartbeatRun }) {
|
||||
const issueId = readRunIssueId(run);
|
||||
const cause = run.execution?.cause ?? run.errorCode;
|
||||
if (issueId && (isNativeWorkspaceExportRepairCause(cause) || cause === "native_workspace_sync_out_unrecoverable"))
|
||||
return <RunWorkspaceSourceInspection key={run.id} issueId={issueId} runId={run.id} />;
|
||||
return <WorkspaceValidationRecoverySurface run={run} />;
|
||||
}
|
||||
|
||||
function WorkspaceValidationRecoverySurface({ run }: { run: HeartbeatRun }) {
|
||||
const navigate = useNavigate();
|
||||
const queryClient = useQueryClient();
|
||||
const { pushToast } = useToastActions();
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
import { useMutation } from "@tanstack/react-query";
|
||||
import { ENVIRONMENT_LEASE_POLICIES, WORKSPACE_SOURCE_VERIFICATION_REASONS } from "@paperclipai/shared";
|
||||
import { issuesApi } from "../api/issues";
|
||||
import { Button } from "./ui/button";
|
||||
import { formatDateTime } from "../lib/utils";
|
||||
|
||||
function recordedFailure(evidence: Record<string, unknown>): string {
|
||||
const recovery = evidence.systemRecovery;
|
||||
if (!recovery || typeof recovery !== "object" || !("failure" in recovery)) return "No recorded system recovery failure.";
|
||||
const failure = recovery.failure;
|
||||
if (!failure || typeof failure !== "object" || !("code" in failure) || typeof failure.code !== "string")
|
||||
return "No recorded system recovery failure.";
|
||||
const detail = "sourceVerification" in failure ? failure.sourceVerification : null;
|
||||
if (!detail || typeof detail !== "object" || !("reason" in detail) || typeof detail.reason !== "string" ||
|
||||
!(WORKSPACE_SOURCE_VERIFICATION_REASONS as readonly string[]).includes(detail.reason)) return failure.code;
|
||||
const policy = "observedLeasePolicy" in detail ? detail.observedLeasePolicy : null;
|
||||
const allowedPolicy = typeof policy === "string" &&
|
||||
(policy === "unrecognized" || (ENVIRONMENT_LEASE_POLICIES as readonly string[]).includes(policy));
|
||||
return [failure.code, detail.reason, ...(allowedPolicy ? [`lease policy ${policy}`] : [])].join(" · ");
|
||||
}
|
||||
|
||||
export function RunWorkspaceSourceInspection({ issueId, runId }: { issueId: string; runId: string }) {
|
||||
const listing = useMutation({ mutationFn: () => issuesApi.recoveryActions(issueId, { runId }) });
|
||||
const inspection = useMutation({ mutationFn: (actionId: string) => issuesApi.recoveryActions(issueId, { runId, actionId }) });
|
||||
const pending = listing.isPending || inspection.isPending;
|
||||
const error = listing.error ?? inspection.error;
|
||||
const current = inspection.data?.sourceInspection;
|
||||
return (
|
||||
<div className="space-y-2" data-testid="run-workspace-source-inspection">
|
||||
<Button variant="outline" size="sm" disabled={pending} onClick={() => { inspection.reset(); listing.mutate(); }}>
|
||||
{pending ? "Inspecting recovery…" : "Inspect recovery"}
|
||||
</Button>
|
||||
{error ? <p role="alert" className="text-xs text-destructive">
|
||||
{error instanceof Error ? error.message : "Recovery inspection failed. Try inspecting again."}
|
||||
</p> : null}
|
||||
{listing.data?.actions.length === 0 ? <p className="text-xs text-muted-foreground">No recorded recovery action for this run.</p> : null}
|
||||
{listing.data?.actions.map(action => (
|
||||
<div key={action.id} className="space-y-2 text-xs">
|
||||
<p>Recorded failure: <code className="font-mono">{recordedFailure(action.evidence)}</code></p>
|
||||
<p className="font-mono text-muted-foreground">{action.id} · {action.status}</p>
|
||||
<Button variant="ghost" size="sm" disabled={pending} onClick={() => inspection.mutate(action.id)}>
|
||||
Inspect current source
|
||||
</Button>
|
||||
</div>
|
||||
))}
|
||||
{current ? <div className="space-y-2 text-xs" role="status">
|
||||
<p className="font-medium">Current recorded-source identity</p>
|
||||
<p className="font-mono text-muted-foreground">Action {current.actionId}</p>
|
||||
<p className="text-muted-foreground">This read-only observation checks recorded identity. It does not confirm provider termination, local files, or successful preservation.</p>
|
||||
{current.status === "verified" ? <p>Recorded-source identity verified.</p>
|
||||
: current.status === "not_inspected" ? <p>Source not inspected: this record exceeds the inspection limit. Service attention is required.</p>
|
||||
: <p>Blocked: <code className="font-mono">{current.sourceVerification?.reason ?? current.failureCode}</code>
|
||||
{current.sourceVerification?.observedLeasePolicy ? <> · lease policy <code className="font-mono">{current.sourceVerification.observedLeasePolicy}</code></> : null}</p>}
|
||||
<p className="font-mono text-muted-foreground">Observed {formatDateTime(current.observedAt)}</p>
|
||||
</div> : null}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
Reference in new issue
Block a user