fix(workspaces): name incompatible project storage when a volume-free server refuses a run (#22398)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A volume-free server admits a run only when its task files have
durable storage.
> - A project can keep its task files in plugin storage, such as an
agent-computers plugin. Only environments of that plugin can use that
storage.
> - A task keeps its selected source when it moves to another project.
So the source project's storage still applies after the move.
> - When an agent in a Daytona sandbox got such a task, the volume-free
check refused the run with a generic message. The message said "native
Paperclip Runner with remote workspace storage". The agent already used
native Paperclip Runner.
> - The run ended `setup_failed`. The task offered Retry, but Retry
always failed the same way. Sentry received the refusal as an unexpected
run failure.
> - This pull request finds the real cause with the existing
project-storage compatibility rule. It records the refusal as a
configuration problem that an owner must repair.
> - The benefit is a message that names the storage and the environment,
a task notice with the correct owner action, and no Sentry noise for
this known configuration state.

## Linked Issues or Issue Description

Related: #17552 (a task project and its selected workspace are separate
settings, so a moved task keeps its source).

**What happened?**

On a volume-free server, a task's source project used plugin storage.
The assigned agent used a sandbox environment that cannot use that
storage. Workspace preparation refused the run with "Volume-free servers
require native Paperclip Runner execution with remote workspace
storage." The run failed with `setup_failed`. The task notice told the
user to press Retry. Each retry failed the same way. Sentry received
each refusal as an unexpected run failure.

**Expected behavior**

The refusal names the real cause: the project's workspace storage does
not support the agent's environment. The run fails
`configuration_incomplete`. The task notice tells the owner to use an
agent in a compatible environment, or a source from a project for this
environment. Sentry does not receive this known configuration state.

**Steps to reproduce**

1. Start the server with `PAPERCLIP_EPHEMERAL_SERVER=true`.
2. Create a project whose storage is a plugin workspace provider.
3. Assign a task in that project to a Paperclip Runner agent in a
Daytona sandbox environment.
4. Wake the agent. The run fails `setup_failed` with the generic
volume-free message.
5. Move the task to a project without plugin storage and comment "try
again". The task keeps its source, and the run fails the same way.

**Paperclip version or commit**

Seen on a hosted volume-free staging server. The new regression test
fails on `master` at 201abacd6e and passes with this change.

## What Changed

- Workspace preparation keeps the volume-free admission rule. When that
rule refuses a run, it checks the task source project's storage provider
against the selected environment with the existing
`assertProjectStorageEnvironment` rule.
- On a mismatch, the run fails with `ConfigurationIncompleteFailure`,
reason `project_storage_environment_incompatible`. The message is the
existing message that names the storage and the environment. The payload
records the provider, the environment, and a stable fingerprint.
- Refusals with any other cause keep the original volume-free error.
- The configuration-incomplete task notice and the recovery next action
name this owner action. Before this change, an unknown reason fell back
to the "missing secret" text.
- Run-failure reporting skips this exact pre-dispatch outcome, like
`ai_connection_required` and `workspace_source_restricted`. A marker
after provider work started, during execution, or without
`configuration_incomplete` is still reported.
- `doc/project-environment-compatibility.md` and
`doc/workspace-storage.md` describe the new outcome.

## Verification

- Red before the fix, green after:
- `server/src/services/heartbeat/workspace-preparation.test.ts` › "names
an incompatible project storage environment when a volume-free server
rejects the run". Before the fix, the run failed with HTTP 422
`ephemeral_execution_configuration_required`. After the fix, it fails
with `configuration_incomplete` and the storage and environment message.
The same test confirms that an unavailable provider keeps the original
error. It also confirms that no workspace is resolved or acquired.
- `server/src/services/__tests__/run-failure-report.test.ts` › "keeps a
run refused for incompatible project workspace storage actionable on the
task". Before the fix, Sentry received the run.
- `server/src/services/recovery/stranded-notice.test.ts` › "names
incompatible project workspace storage instead of a missing secret".
Before the fix, the notice said that a secret was missing.
- Safety cases: three marker variants are still reported (executing
stage, `setup_failed`, provider work started).
- Commands (local, Node 25, embedded Postgres):
- `pnpm --filter @paperclipai/server exec vitest run
src/services/heartbeat/workspace-preparation.test.ts
src/services/__tests__/run-failure-report.test.ts
src/services/recovery/stranded-notice.test.ts` (482 passed)
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/ephemeral-server-contract.test.ts
src/__tests__/project-environments.test.ts
src/__tests__/task-workspace-binding.test.ts
src/__tests__/agent-project-access.test.ts` (passed)
  - `pnpm --filter @paperclipai/server typecheck`

## Risks

- Low risk. The new path runs only after the volume-free rule has
already refused the run, so no run that started before can fail now.
- The only behavior change for those refused runs is the failure code,
the message, the notice, and the Sentry decision.
- One extra plugin-provider read happens only on that refusal path.
- A task that moved projects still keeps its selected source. This
change does not alter that contract. The owner must choose a compatible
agent or a new source.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in a sandboxed workspace.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-11-20 05:17:37 -08:00
1 parent c3260b1c03
commit 7ef731ec7e
10 files changed
+143 -6

No files matched your search

@@ -949,6 +949,34 @@ describeEmbeddedPostgres("reportRunFailure", () => {
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
});
function projectStorageEnvironmentRun(overrides: Partial<typeof heartbeatRuns.$inferSelect> = {}) {
return buildRun({
errorCode: "configuration_incomplete",
executionStage: "preparing",
resultJson: {
configurationIncomplete: { reason: "project_storage_environment_incompatible", actionUrl: "/agents/fixture/runtime" },
executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
},
...overrides,
});
}
it("keeps a run refused for incompatible project workspace storage actionable on the task", async () => {
await seedCompanyAndAgent();
await reportRunFailure(db, projectStorageEnvironmentRun(), { phase: "setup" });
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
});
it.each([
{ executionStage: "executing" },
{ errorCode: "setup_failed" },
{ resultJson: { configurationIncomplete: { reason: "project_storage_environment_incompatible" }, executionRecovery: { kind: "bootstrap", providerWorkStarted: true } } },
])("reports project storage markers without an exact pre-dispatch outcome: %j", async (overrides) => {
await seedCompanyAndAgent();
await reportRunFailure(db, projectStorageEnvironmentRun(overrides), { phase: "setup" });
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
});
function localPathConfigurationRun(overrides: Partial<typeof heartbeatRuns.$inferSelect> = {}) {
return buildRun({
errorCode: "workspace_validation_failed",
@@ -38,6 +38,7 @@ import { buildProjectMentionHref, readRemoteWorkspaceAuthority } from "@papercli
import { migrateRemoteWorkspace } from "../remote-workspace-migration.js";
import { trace } from "@opentelemetry/api";
import { effectiveProjectWorkspacePolicy } from "../project-workspace-provider.js";
import { createPluginWorkspaceFixture } from "../../__tests__/helpers/plugin-workspaces.js";
import { resolveExecutionWorkspaceMode, buildExecutionWorkspaceAdapterConfig } from "../execution-workspace-policy.js";
import { createLocalDiskStorageProvider } from "../../storage/local-disk-provider.js";
import { createStorageService } from "../../storage/service.js";
@@ -633,6 +634,33 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
expect(acquire).not.toHaveBeenCalled();
});
it("names an incompatible project storage environment when a volume-free server rejects the run", async () => {
const { input, envOrchestrator } = await fixture();
const storage = await createPluginWorkspaceFixture(db, input.agent.companyId);
input.agent.adapterType = "paperclip_runner";
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
name: "Daytona", driver: "sandbox", config: { provider: "daytona" } };
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = true;
configureRemotePersistence();
process.env.PAPERCLIP_EPHEMERAL_SERVER = "true";
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun");
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
input.policy.projectExecutionWorkspacePolicy = { enabled: true, storageProviderPluginId: storage.pluginId };
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({ name: "ConfigurationIncompleteFailure",
code: "configuration_incomplete",
message: `This project's workspace storage (Storage) does not support the environment "Daytona". Use an agent in a compatible environment, or create a project for this environment.`,
resultJson: { configurationIncomplete: { reason: "project_storage_environment_incompatible",
storageProviderPluginId: storage.pluginId, environmentId: input.environment.selectedEnvironmentForConfig.id,
fingerprint: `project_storage_environment:${storage.pluginId}:${input.environment.selectedEnvironmentForConfig.id}` } } });
// An unavailable provider cannot explain the rejection; keep the original reason.
input.policy.projectExecutionWorkspacePolicy = { enabled: true, storageProviderPluginId: randomUUID() };
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow(
"Volume-free servers require native Paperclip Runner execution with remote workspace storage.");
expect(resolve).not.toHaveBeenCalled();
expect(acquire).not.toHaveBeenCalled();
expect(await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.companyId, input.agent.companyId))).toHaveLength(0);
});
it.each(["configured-folder", "explicit-task-directory", "enrolled-remote"] as const)(
"preserves effective configured cwd authority for %s", async scenario => {
const { input, source, issue, envOrchestrator } = await fixture();
@@ -2,6 +2,9 @@ import { NATIVE_REMOTE_RECOVERY_LEASE_MISMATCH, remoteRecoveryLeaseMismatch } fr
import { readInheritedRunnerRetainedSandbox } from "../native-runtime/native-inherited-runner-retry.js";
import { resolveProviderApiPlacement } from "../native-runtime/provider-api-placement.js";
import { assertEphemeralWorkspaceExecution, readRemoteWorkspaceAuthority, RemoteWorkspaceWriterActiveError, reserveRemoteWorkspace, shouldUseRemoteWorkspaces } from "../remote-workspaces.js";
import { assertProjectStorageEnvironment, listProjectWorkspaceProviders } from "../project-workspace-provider.js";
import { PROJECT_STORAGE_ENVIRONMENT_INCOMPATIBLE } from "../project-storage-environment-reason.js";
import { ConfigurationIncompleteFailure } from "./run-preparation.js";
import { WorkspaceBusyDeferral } from "./retries.js";
import { remoteWorkspaceHoldRefusesStart } from "../remote-workspace-writer-hold.js";
import { WORKSPACE_BUSY_RETRY_REASON } from "../../modules/run-dispatch/index.js";
@@ -283,10 +286,35 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
&& !projectExecutionWorkspacePolicy?.storageProviderPluginId,
existingWorkspace: reusableExistingExecutionWorkspace ?? existingExecutionWorkspace,
});
if (!providerApiPlacement) assertEphemeralWorkspaceExecution({ externalConversation: isDotRun,
adapterType: agent.adapterType,
environmentDriver: selectedEnvironmentForConfig?.driver,
pluginTaskWorkspace, remoteAuthority });
// A volume-free server rejects execution without durable task storage. When
// the source project's storage provider cannot serve the selected
// environment, that mismatch is the cause and only an owner can repair it.
const assertEphemeralExecution = async () => {
if (providerApiPlacement) return;
try {
assertEphemeralWorkspaceExecution({ externalConversation: isDotRun, adapterType: agent.adapterType,
environmentDriver: selectedEnvironmentForConfig?.driver, pluginTaskWorkspace, remoteAuthority });
} catch (error) {
const storagePluginId = projectExecutionWorkspacePolicy?.storageProviderPluginId;
const provider = storagePluginId && selectedEnvironmentForConfig
? (await listProjectWorkspaceProviders(db, agent.companyId)).find(value => value.id === storagePluginId) : undefined;
if (provider && selectedEnvironmentForConfig) {
try {
assertProjectStorageEnvironment(provider, selectedEnvironmentForConfig);
} catch (mismatch) {
throw new ConfigurationIncompleteFailure(mismatch instanceof Error ? mismatch.message : String(mismatch), {
configurationIncomplete: { reason: PROJECT_STORAGE_ENVIRONMENT_INCOMPATIBLE, companyId: agent.companyId,
agentId: agent.id, issueId: issueId ?? null, projectId: issueRef?.projectId ?? null,
storageProviderPluginId: provider.id, environmentId: selectedEnvironmentForConfig.id,
fingerprint: `project_storage_environment:${provider.id}:${selectedEnvironmentForConfig.id}`,
actionUrl: `/agents/${agent.id}/runtime`, missingBindings: [] },
});
}
}
throw error;
}
};
await assertEphemeralExecution();
const assertRemotePreparationOwned = run.runtimeMode === "native" ? () => assertNativeSessionPreparationAuthority({
db, companyId: agent.companyId, runId: run.id,
leaseOwner: runOptions.nativeLeaseOwner ?? runOptions.nativeRestartRecovery?.leaseOwner,
@@ -370,8 +398,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
remoteAuthority = false;
// Source resolution can retain a configured host folder. Check the
// resolved authority before materializing it or acquiring a sandbox.
if (!providerApiPlacement) assertEphemeralWorkspaceExecution({ externalConversation: isDotRun,
adapterType: agent.adapterType, environmentDriver: selectedEnvironmentForConfig?.driver, pluginTaskWorkspace, remoteAuthority });
await assertEphemeralExecution();
}
else {
remoteSourceState.value = sourceState;
@@ -0,0 +1,5 @@
// Dependency-free so run-failure reporting and recovery notices can classify
// this outcome without loading the plugin and workspace graph.
/** The task source's project storage provider cannot serve the run's environment. */
export const PROJECT_STORAGE_ENVIRONMENT_INCOMPATIBLE = "project_storage_environment_incompatible";
+4
View File
@@ -24,6 +24,7 @@ import { isAgentAwaitingSetup } from "../../modules/agent-lifecycle/index.js";
import { onboardingFirstTaskAwaitingUserSql } from "./onboarding-first-task-wait.js";
import { isAiAuthenticationBlocked } from "../ai-auth-failure.js";
import { RESTRICTED_WORKSPACE_SOURCE_REASON } from "../workspace-source-restriction.js";
import { PROJECT_STORAGE_ENVIRONMENT_INCOMPATIBLE } from "../project-storage-environment-reason.js";
import { hasCommittedNativePlanWait } from "../native-runtime/native-plan-wait.js";
import { hasCommittedNativeConversationTurnWait, hasCommittedNativeResponseWakeWait } from "../native-runtime/native-response-wake-wait.js";
import { listExternalReviewWaits } from "../native-runtime/external-review-wait.js";
@@ -173,6 +174,7 @@ import {
} from "./successful-run-handoff.js";
import {
SANDBOX_PROVIDER_PLUGIN_NOT_READY_REASON,
PROJECT_STORAGE_ENVIRONMENT_NEXT_ACTION,
sandboxProviderPluginRemedy,
buildExecutionReviewParticipantRecoveryNoticeSeed,
buildExecutionReviewParticipantUnavailableNoticeSeed,
@@ -2802,6 +2804,8 @@ export function recoveryService(
? "Reconnect the selected AI account or choose an available connection, then continue the task."
: readConfigurationIncompletePayload(input.latestRun)?.reason === RESTRICTED_WORKSPACE_SOURCE_REASON
? "Retry the task."
: readConfigurationIncompletePayload(input.latestRun)?.reason === PROJECT_STORAGE_ENVIRONMENT_INCOMPATIBLE
? PROJECT_STORAGE_ENVIRONMENT_NEXT_ACTION
: readConfigurationIncompletePayload(input.latestRun)
?.reason === SANDBOX_PROVIDER_PLUGIN_NOT_READY_REASON
? `The plugin that starts this agent's sandbox is not running. To fix it, ${sandboxProviderPluginRemedy(
@@ -395,6 +395,14 @@ it("names the managed AI connection a plugin Runner-task environment requires",
expect(notice.body).not.toContain("secret/env");
});
it("names incompatible project workspace storage instead of a missing secret", () => {
const notice = buildConfigurationIncompleteRecoveryNoticeSeed({ reason: "project_storage_environment_incompatible" });
expect(notice.title).toBe("Workspace storage unavailable");
expect(notice.body).toContain("workspace storage that the agent's environment cannot use");
expect(notice.nextAction).toContain("Assign the task to an agent whose environment can use");
expect(notice.body).not.toContain("secret");
});
it("tells the operator to switch a preconfigured Runner home to an isolated one", () => {
const notice = buildConfigurationIncompleteRecoveryNoticeSeed({ reason: "ai_connection_required", homeMode: "preconfigured" });
expect(notice.title).toBe("Isolated Runner home required");
@@ -1,5 +1,6 @@
import type { IssueCommentMetadata, IssueCommentPresentation } from "@paperclipai/shared";
import { ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE } from "../environment-provider-unavailable-error.js";
import { PROJECT_STORAGE_ENVIRONMENT_INCOMPATIBLE } from "../project-storage-environment-reason.js";
import {
agentLinkRow,
keyValueRow,
@@ -128,6 +129,7 @@ export function buildWorkspaceValidationRecoveryNoticeSeed(): StrandedRecoveryNo
}
export const SANDBOX_PROVIDER_PLUGIN_NOT_READY_REASON = "sandbox_provider_plugin_not_ready";
export const PROJECT_STORAGE_ENVIRONMENT_NEXT_ACTION = "Assign the task to an agent whose environment can use this project's workspace storage, or choose a workspace from a project for this agent's environment, then press Retry.";
function readNonEmptyStringField(payload: Record<string, unknown> | null | undefined, key: string): string | null {
const value = payload?.[key];
@@ -226,6 +228,14 @@ export function buildConfigurationIncompleteRecoveryNoticeSeed(
tone: "danger",
};
}
if (readNonEmptyStringField(configurationIncomplete, "reason") === PROJECT_STORAGE_ENVIRONMENT_INCOMPATIBLE) {
return {
title: "Workspace storage unavailable",
body: "The agent did not start because the project that stores this task's files uses workspace storage that the agent's environment cannot use.",
nextAction: PROJECT_STORAGE_ENVIRONMENT_NEXT_ACTION,
tone: "danger",
};
}
if (readNonEmptyStringField(configurationIncomplete, "reason") === SANDBOX_PROVIDER_PLUGIN_NOT_READY_REASON) {
const pluginKey = readNonEmptyStringField(configurationIncomplete, "pluginKey") ?? "the sandbox provider plugin";
const pluginStatus = readNonEmptyStringField(configurationIncomplete, "pluginStatus") ?? "not ready";
+15
View File
@@ -18,6 +18,7 @@ import { isOwnedAiConnectionSelectionRejection } from "./ai-connection-configura
import { readRunFailureReportingMode, type RunFailureReportingMode } from "../sentry-run-failure-policy.js";
import { RESTRICTED_WORKSPACE_SOURCE_REASON } from "./workspace-source-restriction.js";
import { PLUGIN_RUNNER_AI_CONNECTION_REQUIRED } from "./plugin-runner-ai-connection.js";
import { PROJECT_STORAGE_ENVIRONMENT_INCOMPATIBLE } from "./project-storage-environment-reason.js";
import { NATIVE_PROVIDER_OVERLOADED_CODE } from "./native-runtime/native-provider-failure.js";
import { PROJECT_ACCESS_DENIED } from "./project-access-denial.js";
import { REVIEWED_CHAT_BINDING_NOT_READY } from "./reviewed-chat-binding-diagnostics.js";
@@ -176,6 +177,19 @@ function isPluginRunnerAiConnectionRequiredBlocker(run: HeartbeatRun, options: R
recovery?.kind === "bootstrap" && recovery.providerWorkStarted === false;
}
/** The source project's storage cannot serve the agent's environment, refused before dispatch; the task notice names the owner action. */
function isProjectStorageEnvironmentBlocker(run: HeartbeatRun, options: RunFailureReportOptions): boolean {
if (
run.status !== "failed" || run.errorCode !== "configuration_incomplete" ||
run.executionStage !== "preparing" || options.phase !== "setup" ||
run.exitCode != null || run.signal != null
) return false;
const configuration = asRecord(run.resultJson?.configurationIncomplete);
const recovery = asRecord(run.resultJson?.executionRecovery);
return configuration?.reason === PROJECT_STORAGE_ENVIRONMENT_INCOMPATIBLE &&
recovery?.kind === "bootstrap" && recovery.providerWorkStarted === false;
}
function isProjectAccessDenialBlocker(run: HeartbeatRun, options: RunFailureReportOptions): boolean {
if (
run.status !== "failed" || run.errorCode !== PROJECT_ACCESS_DENIED ||
@@ -382,6 +396,7 @@ export function reportRunFailure(db: Db, run: HeartbeatRun, options: RunFailureR
isLocalPathWorkspaceConfigurationBlocker(run, options) ||
isRestrictedWorkspaceSourceBlocker(run, options) ||
isPluginRunnerAiConnectionRequiredBlocker(run, options) ||
isProjectStorageEnvironmentBlocker(run, options) ||
isProjectAccessDenialBlocker(run, options) ||
isGitConnectionFailure(run, options) ||
isProviderQuotaFailure(run, options) ||