fix(workspaces): name incompatible project storage when a volume-free server refuses a run (#22398)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A volume-free server admits a run only when its task files have
durable storage.
> - A project can keep its task files in plugin storage, such as an
agent-computers plugin. Only environments of that plugin can use that
storage.
> - A task keeps its selected source when it moves to another project.
So the source project's storage still applies after the move.
> - When an agent in a Daytona sandbox got such a task, the volume-free
check refused the run with a generic message. The message said "native
Paperclip Runner with remote workspace storage". The agent already used
native Paperclip Runner.
> - The run ended `setup_failed`. The task offered Retry, but Retry
always failed the same way. Sentry received the refusal as an unexpected
run failure.
> - This pull request finds the real cause with the existing
project-storage compatibility rule. It records the refusal as a
configuration problem that an owner must repair.
> - The benefit is a message that names the storage and the environment,
a task notice with the correct owner action, and no Sentry noise for
this known configuration state.

## Linked Issues or Issue Description

Related: #17552 (a task project and its selected workspace are separate
settings, so a moved task keeps its source).

**What happened?**

On a volume-free server, a task's source project used plugin storage.
The assigned agent used a sandbox environment that cannot use that
storage. Workspace preparation refused the run with "Volume-free servers
require native Paperclip Runner execution with remote workspace
storage." The run failed with `setup_failed`. The task notice told the
user to press Retry. Each retry failed the same way. Sentry received
each refusal as an unexpected run failure.

**Expected behavior**

The refusal names the real cause: the project's workspace storage does
not support the agent's environment. The run fails
`configuration_incomplete`. The task notice tells the owner to use an
agent in a compatible environment, or a source from a project for this
environment. Sentry does not receive this known configuration state.

**Steps to reproduce**

1. Start the server with `PAPERCLIP_EPHEMERAL_SERVER=true`.
2. Create a project whose storage is a plugin workspace provider.
3. Assign a task in that project to a Paperclip Runner agent in a
Daytona sandbox environment.
4. Wake the agent. The run fails `setup_failed` with the generic
volume-free message.
5. Move the task to a project without plugin storage and comment "try
again". The task keeps its source, and the run fails the same way.

**Paperclip version or commit**

Seen on a hosted volume-free staging server. The new regression test
fails on `master` at 201abacd6e and passes with this change.

## What Changed

- Workspace preparation keeps the volume-free admission rule. When that
rule refuses a run, it checks the task source project's storage provider
against the selected environment with the existing
`assertProjectStorageEnvironment` rule.
- On a mismatch, the run fails with `ConfigurationIncompleteFailure`,
reason `project_storage_environment_incompatible`. The message is the
existing message that names the storage and the environment. The payload
records the provider, the environment, and a stable fingerprint.
- Refusals with any other cause keep the original volume-free error.
- The configuration-incomplete task notice and the recovery next action
name this owner action. Before this change, an unknown reason fell back
to the "missing secret" text.
- Run-failure reporting skips this exact pre-dispatch outcome, like
`ai_connection_required` and `workspace_source_restricted`. A marker
after provider work started, during execution, or without
`configuration_incomplete` is still reported.
- `doc/project-environment-compatibility.md` and
`doc/workspace-storage.md` describe the new outcome.

## Verification

- Red before the fix, green after:
- `server/src/services/heartbeat/workspace-preparation.test.ts` › "names
an incompatible project storage environment when a volume-free server
rejects the run". Before the fix, the run failed with HTTP 422
`ephemeral_execution_configuration_required`. After the fix, it fails
with `configuration_incomplete` and the storage and environment message.
The same test confirms that an unavailable provider keeps the original
error. It also confirms that no workspace is resolved or acquired.
- `server/src/services/__tests__/run-failure-report.test.ts` › "keeps a
run refused for incompatible project workspace storage actionable on the
task". Before the fix, Sentry received the run.
- `server/src/services/recovery/stranded-notice.test.ts` › "names
incompatible project workspace storage instead of a missing secret".
Before the fix, the notice said that a secret was missing.
- Safety cases: three marker variants are still reported (executing
stage, `setup_failed`, provider work started).
- Commands (local, Node 25, embedded Postgres):
- `pnpm --filter @paperclipai/server exec vitest run
src/services/heartbeat/workspace-preparation.test.ts
src/services/__tests__/run-failure-report.test.ts
src/services/recovery/stranded-notice.test.ts` (482 passed)
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/ephemeral-server-contract.test.ts
src/__tests__/project-environments.test.ts
src/__tests__/task-workspace-binding.test.ts
src/__tests__/agent-project-access.test.ts` (passed)
  - `pnpm --filter @paperclipai/server typecheck`

## Risks

- Low risk. The new path runs only after the volume-free rule has
already refused the run, so no run that started before can fail now.
- The only behavior change for those refused runs is the failure code,
the message, the notice, and the Sentry decision.
- One extra plugin-provider read happens only on that refusal path.
- A task that moved projects still keeps its selected source. This
change does not alter that contract. The owner must choose a compatible
agent or a new source.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in a sandboxed workspace.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-11-20 05:17:37 -08:00
1 parent c3260b1c03
commit 7ef731ec7e
10 files changed
+143 -6

No files matched your search

+7
View File
@@ -73,6 +73,13 @@ projects keep their stored owner. This change does not migrate managed storage
or detach projects that contain existing work. A managed project run in an
incompatible environment fails with `project_environment_incompatible` before a
workspace provider operation, with a message naming the storage and environment.
On a volume-free server, execution admission refuses that run earlier. When the
task source's project storage cannot serve the selected environment, the run
fails `configuration_incomplete` with reason
`project_storage_environment_incompatible` and the same message. The task notice
names the owner action, and run-failure reporting does not send it to Sentry.
A task that moves to another project keeps its selected source, so the source
project's storage still decides compatibility.
## Project Configuration
+5
View File
@@ -363,6 +363,11 @@ selector checks the resolved native warm lifecycle before admission and rejects
legacy or per-turn execution. External conversation providers keep their existing
exemption because they own no task filesystem. Source resolution checks this
contract again before a configured host folder can fall back to local storage.
When the task source's project uses plugin storage that the selected
environment cannot serve, the refusal names that storage and environment and
ends the run `configuration_incomplete` (reason
`project_storage_environment_incompatible`). See
[project-environment-compatibility.md](project-environment-compatibility.md).
A replacement controller reconstructs an active remote agent-directory cache
from the registered sandbox, including unsaved files. The remote directory,
@@ -949,6 +949,34 @@ describeEmbeddedPostgres("reportRunFailure", () => {
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
});
function projectStorageEnvironmentRun(overrides: Partial<typeof heartbeatRuns.$inferSelect> = {}) {
return buildRun({
errorCode: "configuration_incomplete",
executionStage: "preparing",
resultJson: {
configurationIncomplete: { reason: "project_storage_environment_incompatible", actionUrl: "/agents/fixture/runtime" },
executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
},
...overrides,
});
}
it("keeps a run refused for incompatible project workspace storage actionable on the task", async () => {
await seedCompanyAndAgent();
await reportRunFailure(db, projectStorageEnvironmentRun(), { phase: "setup" });
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
});
it.each([
{ executionStage: "executing" },
{ errorCode: "setup_failed" },
{ resultJson: { configurationIncomplete: { reason: "project_storage_environment_incompatible" }, executionRecovery: { kind: "bootstrap", providerWorkStarted: true } } },
])("reports project storage markers without an exact pre-dispatch outcome: %j", async (overrides) => {
await seedCompanyAndAgent();
await reportRunFailure(db, projectStorageEnvironmentRun(overrides), { phase: "setup" });
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
});
function localPathConfigurationRun(overrides: Partial<typeof heartbeatRuns.$inferSelect> = {}) {
return buildRun({
errorCode: "workspace_validation_failed",
@@ -38,6 +38,7 @@ import { buildProjectMentionHref, readRemoteWorkspaceAuthority } from "@papercli
import { migrateRemoteWorkspace } from "../remote-workspace-migration.js";
import { trace } from "@opentelemetry/api";
import { effectiveProjectWorkspacePolicy } from "../project-workspace-provider.js";
import { createPluginWorkspaceFixture } from "../../__tests__/helpers/plugin-workspaces.js";
import { resolveExecutionWorkspaceMode, buildExecutionWorkspaceAdapterConfig } from "../execution-workspace-policy.js";
import { createLocalDiskStorageProvider } from "../../storage/local-disk-provider.js";
import { createStorageService } from "../../storage/service.js";
@@ -633,6 +634,33 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace
expect(acquire).not.toHaveBeenCalled();
});
it("names an incompatible project storage environment when a volume-free server rejects the run", async () => {
const { input, envOrchestrator } = await fixture();
const storage = await createPluginWorkspaceFixture(db, input.agent.companyId);
input.agent.adapterType = "paperclip_runner";
input.environment.selectedEnvironmentForConfig = { ...input.environment.localEnvironment,
name: "Daytona", driver: "sandbox", config: { provider: "daytona" } };
input.environment.resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled = true;
configureRemotePersistence();
process.env.PAPERCLIP_EPHEMERAL_SERVER = "true";
const resolve = vi.spyOn(input.services, "resolveWorkspaceForRun");
const acquire = vi.spyOn(envOrchestrator, "acquireForRun");
input.policy.projectExecutionWorkspacePolicy = { enabled: true, storageProviderPluginId: storage.pluginId };
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toMatchObject({ name: "ConfigurationIncompleteFailure",
code: "configuration_incomplete",
message: `This project's workspace storage (Storage) does not support the environment "Daytona". Use an agent in a compatible environment, or create a project for this environment.`,
resultJson: { configurationIncomplete: { reason: "project_storage_environment_incompatible",
storageProviderPluginId: storage.pluginId, environmentId: input.environment.selectedEnvironmentForConfig.id,
fingerprint: `project_storage_environment:${storage.pluginId}:${input.environment.selectedEnvironmentForConfig.id}` } } });
// An unavailable provider cannot explain the rejection; keep the original reason.
input.policy.projectExecutionWorkspacePolicy = { enabled: true, storageProviderPluginId: randomUUID() };
await expect(prepareHeartbeatWorkspace(db, input)).rejects.toThrow(
"Volume-free servers require native Paperclip Runner execution with remote workspace storage.");
expect(resolve).not.toHaveBeenCalled();
expect(acquire).not.toHaveBeenCalled();
expect(await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.companyId, input.agent.companyId))).toHaveLength(0);
});
it.each(["configured-folder", "explicit-task-directory", "enrolled-remote"] as const)(
"preserves effective configured cwd authority for %s", async scenario => {
const { input, source, issue, envOrchestrator } = await fixture();
@@ -2,6 +2,9 @@ import { NATIVE_REMOTE_RECOVERY_LEASE_MISMATCH, remoteRecoveryLeaseMismatch } fr
import { readInheritedRunnerRetainedSandbox } from "../native-runtime/native-inherited-runner-retry.js";
import { resolveProviderApiPlacement } from "../native-runtime/provider-api-placement.js";
import { assertEphemeralWorkspaceExecution, readRemoteWorkspaceAuthority, RemoteWorkspaceWriterActiveError, reserveRemoteWorkspace, shouldUseRemoteWorkspaces } from "../remote-workspaces.js";
import { assertProjectStorageEnvironment, listProjectWorkspaceProviders } from "../project-workspace-provider.js";
import { PROJECT_STORAGE_ENVIRONMENT_INCOMPATIBLE } from "../project-storage-environment-reason.js";
import { ConfigurationIncompleteFailure } from "./run-preparation.js";
import { WorkspaceBusyDeferral } from "./retries.js";
import { remoteWorkspaceHoldRefusesStart } from "../remote-workspace-writer-hold.js";
import { WORKSPACE_BUSY_RETRY_REASON } from "../../modules/run-dispatch/index.js";
@@ -283,10 +286,35 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
&& !projectExecutionWorkspacePolicy?.storageProviderPluginId,
existingWorkspace: reusableExistingExecutionWorkspace ?? existingExecutionWorkspace,
});
if (!providerApiPlacement) assertEphemeralWorkspaceExecution({ externalConversation: isDotRun,
adapterType: agent.adapterType,
environmentDriver: selectedEnvironmentForConfig?.driver,
pluginTaskWorkspace, remoteAuthority });
// A volume-free server rejects execution without durable task storage. When
// the source project's storage provider cannot serve the selected
// environment, that mismatch is the cause and only an owner can repair it.
const assertEphemeralExecution = async () => {
if (providerApiPlacement) return;
try {
assertEphemeralWorkspaceExecution({ externalConversation: isDotRun, adapterType: agent.adapterType,
environmentDriver: selectedEnvironmentForConfig?.driver, pluginTaskWorkspace, remoteAuthority });
} catch (error) {
const storagePluginId = projectExecutionWorkspacePolicy?.storageProviderPluginId;
const provider = storagePluginId && selectedEnvironmentForConfig
? (await listProjectWorkspaceProviders(db, agent.companyId)).find(value => value.id === storagePluginId) : undefined;
if (provider && selectedEnvironmentForConfig) {
try {
assertProjectStorageEnvironment(provider, selectedEnvironmentForConfig);
} catch (mismatch) {
throw new ConfigurationIncompleteFailure(mismatch instanceof Error ? mismatch.message : String(mismatch), {
configurationIncomplete: { reason: PROJECT_STORAGE_ENVIRONMENT_INCOMPATIBLE, companyId: agent.companyId,
agentId: agent.id, issueId: issueId ?? null, projectId: issueRef?.projectId ?? null,
storageProviderPluginId: provider.id, environmentId: selectedEnvironmentForConfig.id,
fingerprint: `project_storage_environment:${provider.id}:${selectedEnvironmentForConfig.id}`,
actionUrl: `/agents/${agent.id}/runtime`, missingBindings: [] },
});
}
}
throw error;
}
};
await assertEphemeralExecution();
const assertRemotePreparationOwned = run.runtimeMode === "native" ? () => assertNativeSessionPreparationAuthority({
db, companyId: agent.companyId, runId: run.id,
leaseOwner: runOptions.nativeLeaseOwner ?? runOptions.nativeRestartRecovery?.leaseOwner,
@@ -370,8 +398,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace
remoteAuthority = false;
// Source resolution can retain a configured host folder. Check the
// resolved authority before materializing it or acquiring a sandbox.
if (!providerApiPlacement) assertEphemeralWorkspaceExecution({ externalConversation: isDotRun,
adapterType: agent.adapterType, environmentDriver: selectedEnvironmentForConfig?.driver, pluginTaskWorkspace, remoteAuthority });
await assertEphemeralExecution();
}
else {
remoteSourceState.value = sourceState;
@@ -0,0 +1,5 @@
// Dependency-free so run-failure reporting and recovery notices can classify
// this outcome without loading the plugin and workspace graph.
/** The task source's project storage provider cannot serve the run's environment. */
export const PROJECT_STORAGE_ENVIRONMENT_INCOMPATIBLE = "project_storage_environment_incompatible";
+4
View File
@@ -24,6 +24,7 @@ import { isAgentAwaitingSetup } from "../../modules/agent-lifecycle/index.js";
import { onboardingFirstTaskAwaitingUserSql } from "./onboarding-first-task-wait.js";
import { isAiAuthenticationBlocked } from "../ai-auth-failure.js";
import { RESTRICTED_WORKSPACE_SOURCE_REASON } from "../workspace-source-restriction.js";
import { PROJECT_STORAGE_ENVIRONMENT_INCOMPATIBLE } from "../project-storage-environment-reason.js";
import { hasCommittedNativePlanWait } from "../native-runtime/native-plan-wait.js";
import { hasCommittedNativeConversationTurnWait, hasCommittedNativeResponseWakeWait } from "../native-runtime/native-response-wake-wait.js";
import { listExternalReviewWaits } from "../native-runtime/external-review-wait.js";
@@ -173,6 +174,7 @@ import {
} from "./successful-run-handoff.js";
import {
SANDBOX_PROVIDER_PLUGIN_NOT_READY_REASON,
PROJECT_STORAGE_ENVIRONMENT_NEXT_ACTION,
sandboxProviderPluginRemedy,
buildExecutionReviewParticipantRecoveryNoticeSeed,
buildExecutionReviewParticipantUnavailableNoticeSeed,
@@ -2802,6 +2804,8 @@ export function recoveryService(
? "Reconnect the selected AI account or choose an available connection, then continue the task."
: readConfigurationIncompletePayload(input.latestRun)?.reason === RESTRICTED_WORKSPACE_SOURCE_REASON
? "Retry the task."
: readConfigurationIncompletePayload(input.latestRun)?.reason === PROJECT_STORAGE_ENVIRONMENT_INCOMPATIBLE
? PROJECT_STORAGE_ENVIRONMENT_NEXT_ACTION
: readConfigurationIncompletePayload(input.latestRun)
?.reason === SANDBOX_PROVIDER_PLUGIN_NOT_READY_REASON
? `The plugin that starts this agent's sandbox is not running. To fix it, ${sandboxProviderPluginRemedy(
@@ -395,6 +395,14 @@ it("names the managed AI connection a plugin Runner-task environment requires",
expect(notice.body).not.toContain("secret/env");
});
it("names incompatible project workspace storage instead of a missing secret", () => {
const notice = buildConfigurationIncompleteRecoveryNoticeSeed({ reason: "project_storage_environment_incompatible" });
expect(notice.title).toBe("Workspace storage unavailable");
expect(notice.body).toContain("workspace storage that the agent's environment cannot use");
expect(notice.nextAction).toContain("Assign the task to an agent whose environment can use");
expect(notice.body).not.toContain("secret");
});
it("tells the operator to switch a preconfigured Runner home to an isolated one", () => {
const notice = buildConfigurationIncompleteRecoveryNoticeSeed({ reason: "ai_connection_required", homeMode: "preconfigured" });
expect(notice.title).toBe("Isolated Runner home required");
@@ -1,5 +1,6 @@
import type { IssueCommentMetadata, IssueCommentPresentation } from "@paperclipai/shared";
import { ENVIRONMENT_PROVIDER_UNAVAILABLE_CODE } from "../environment-provider-unavailable-error.js";
import { PROJECT_STORAGE_ENVIRONMENT_INCOMPATIBLE } from "../project-storage-environment-reason.js";
import {
agentLinkRow,
keyValueRow,
@@ -128,6 +129,7 @@ export function buildWorkspaceValidationRecoveryNoticeSeed(): StrandedRecoveryNo
}
export const SANDBOX_PROVIDER_PLUGIN_NOT_READY_REASON = "sandbox_provider_plugin_not_ready";
export const PROJECT_STORAGE_ENVIRONMENT_NEXT_ACTION = "Assign the task to an agent whose environment can use this project's workspace storage, or choose a workspace from a project for this agent's environment, then press Retry.";
function readNonEmptyStringField(payload: Record<string, unknown> | null | undefined, key: string): string | null {
const value = payload?.[key];
@@ -226,6 +228,14 @@ export function buildConfigurationIncompleteRecoveryNoticeSeed(
tone: "danger",
};
}
if (readNonEmptyStringField(configurationIncomplete, "reason") === PROJECT_STORAGE_ENVIRONMENT_INCOMPATIBLE) {
return {
title: "Workspace storage unavailable",
body: "The agent did not start because the project that stores this task's files uses workspace storage that the agent's environment cannot use.",
nextAction: PROJECT_STORAGE_ENVIRONMENT_NEXT_ACTION,
tone: "danger",
};
}
if (readNonEmptyStringField(configurationIncomplete, "reason") === SANDBOX_PROVIDER_PLUGIN_NOT_READY_REASON) {
const pluginKey = readNonEmptyStringField(configurationIncomplete, "pluginKey") ?? "the sandbox provider plugin";
const pluginStatus = readNonEmptyStringField(configurationIncomplete, "pluginStatus") ?? "not ready";
+15
View File
@@ -18,6 +18,7 @@ import { isOwnedAiConnectionSelectionRejection } from "./ai-connection-configura
import { readRunFailureReportingMode, type RunFailureReportingMode } from "../sentry-run-failure-policy.js";
import { RESTRICTED_WORKSPACE_SOURCE_REASON } from "./workspace-source-restriction.js";
import { PLUGIN_RUNNER_AI_CONNECTION_REQUIRED } from "./plugin-runner-ai-connection.js";
import { PROJECT_STORAGE_ENVIRONMENT_INCOMPATIBLE } from "./project-storage-environment-reason.js";
import { NATIVE_PROVIDER_OVERLOADED_CODE } from "./native-runtime/native-provider-failure.js";
import { PROJECT_ACCESS_DENIED } from "./project-access-denial.js";
import { REVIEWED_CHAT_BINDING_NOT_READY } from "./reviewed-chat-binding-diagnostics.js";
@@ -176,6 +177,19 @@ function isPluginRunnerAiConnectionRequiredBlocker(run: HeartbeatRun, options: R
recovery?.kind === "bootstrap" && recovery.providerWorkStarted === false;
}
/** The source project's storage cannot serve the agent's environment, refused before dispatch; the task notice names the owner action. */
function isProjectStorageEnvironmentBlocker(run: HeartbeatRun, options: RunFailureReportOptions): boolean {
if (
run.status !== "failed" || run.errorCode !== "configuration_incomplete" ||
run.executionStage !== "preparing" || options.phase !== "setup" ||
run.exitCode != null || run.signal != null
) return false;
const configuration = asRecord(run.resultJson?.configurationIncomplete);
const recovery = asRecord(run.resultJson?.executionRecovery);
return configuration?.reason === PROJECT_STORAGE_ENVIRONMENT_INCOMPATIBLE &&
recovery?.kind === "bootstrap" && recovery.providerWorkStarted === false;
}
function isProjectAccessDenialBlocker(run: HeartbeatRun, options: RunFailureReportOptions): boolean {
if (
run.status !== "failed" || run.errorCode !== PROJECT_ACCESS_DENIED ||
@@ -382,6 +396,7 @@ export function reportRunFailure(db: Db, run: HeartbeatRun, options: RunFailureR
isLocalPathWorkspaceConfigurationBlocker(run, options) ||
isRestrictedWorkspaceSourceBlocker(run, options) ||
isPluginRunnerAiConnectionRequiredBlocker(run, options) ||
isProjectStorageEnvironmentBlocker(run, options) ||
isProjectAccessDenialBlocker(run, options) ||
isGitConnectionFailure(run, options) ||
isProviderQuotaFailure(run, options) ||