Files
PaperClipAI/server/src/services/heartbeat/environment-selection.ts
T
Devin FoleyandPaperclip edb9d29942 fix(recovery): preserve Retry authority and reopen cross-task cases (#18997)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Tasks keep execution identity and saved workspace files separate
from control-plane updates.
> - A scanner can have permission to reopen a case while its run cannot
restore that case's files.
> - Retry can also follow a preparation failure that has no
recovery-action record.
> - Both paths can accept valid work but leave the target unable to run.
> - This change restores files under the target executor and saves
admitted Retry authority on the canonical receipt.
> - Existing workspace, user, source and successor checks still apply.

## Linked Issues or Issue Description

**What happened?**

An authorized agent updates another closed task. The route tries to
restore the target workspace under the caller's run identity and returns
`403 Task workspace run identity is unavailable`.

A separate failure occurs when a normal board Retry follows a
pre-provider preparation failure. Admission has no recovery-action row
on which to save continuation authority. Dispatch then fails with
`continuation_user_authorization_missing`.

**Expected behavior**

Accept the authorized task update. Restore its terminal workspace when
the target's admitted executor starts. An admitted Retry must retain its
exact dispatch authority even when the predecessor has no
recovery-action record.

**Steps to reproduce**

1. Complete a task with an isolated workspace. From a different running
task, use an authorized comment or PATCH with `resume: true` on that
closed task.
2. Separately, fail native preparation before provider work starts, with
no recovery-action row. Submit board Retry, admit its successor, and
build the execution continuation.
3. Before this change, the first path returns 403 and the second rejects
its admitted authority.

**Paperclip version or commit**

Reproduced against `e968131efa7e63d321f3bcca8d47a62513c2efb4`. Rebased
onto `afbc5a885b5`.

**Deployment mode**

Server-managed native task execution. Tests use real PostgreSQL and
temporary workspace files.

Related PRs: #16927 covers terminal workspace races. #17916 retains
exact native export sources. Neither covers these route and Retry
receipt failures. A GitHub search found no duplicate. ROADMAP.md was
checked; this is a recovery bug fix.

## What Changed

- Defer workspace restoration only for authorized cross-task comments
and updates from a live run. Checkout keeps its existing workspace
identity checks.
- Restore the target's selected terminal workspace under its own
admitted run. Preserve its saved files and existing source checks. Leave
manually archived workspaces unchanged.
- Store server-created continuation authority on the canonical admitted
Retry receipt when no source recovery-action record exists. Existing
action records remain authoritative. Read the fallback only after
matching the company, task, agent, user, source, generation and
successor.
- Add route and PostgreSQL regression tests. Document both paths.

## Verification

- Red: both cross-task route tests returned the observed 403 before the
route fix.
- Red: the target executor could not reuse its archived terminal
workspace before the restoration fix.
- Red: an admitted preparation Retry with no recovery-action row failed
with `continuation_user_authorization_missing` before the receipt fix.
- Green: 419 recovery and continuation tests pass, plus 183 focused
route, environment and Retry tests. The assembled heartbeat test starts
the agent in the same terminal worktree and verifies both its unpushed
commit and uncommitted file bytes. It fails before the restoration fix.
- Negative cases reject wrong Retry runs, users, source runs,
generations, unadmitted receipts and missing authority. The original
failed run stays unchanged. Workspace tests verify saved file bytes.
- Full `pnpm build`, `pnpm -r typecheck`, `pnpm check:module-boundaries`
and `git diff --check` pass.
- The full suite found an existing assembled chat authority regression:
clearing a recovery-action record could be bypassed by the new fallback.
The fix now writes the fallback only when that source has no action
record. That assembled regression passes with the original test
unchanged.
- Full `pnpm test:run` is running again. Its first attempt also hit an
unrelated ancestor skill-root lookup for a missing Slack skill. The
rerun uses a temporary untracked link to the repository's own bundled
skills. This link is excluded from the PR. Full-suite completion and
hosted CI are pending.
- Hosted CI has multiple runner interruptions. E2e shard 1/8, server
shards 3/12, 8/12 and 12/12, and workspaces-a shard 1/2 show runner
shutdown signals. E2e shard 5/8 and server shards 1/12 and 2/12 have
GitHub annotations reporting lost runner communication; their logs
return 404. The `ci / verify` assertion failed because the general-test
matrix was cancelled. GitHub rejected job and failed-workflow reruns
while this workflow is still running. Rerun the interrupted checks after
it settles.
- Greptile reviewed exact head
`a1e3929aedb1ee3bfcd001f4ebb1510106bf4070` at 2026-11-04 17:37 UTC: 5/5,
with no new blocking findings and no unresolved review threads.

## Risks

- A deferred workspace restore can still fail at target admission. The
run keeps the existing workspace failure path; this change does not
bypass file recovery or provider stop checks.
- Retry receipts now store server-created authority. Existing action
receipts remain valid. No schema migration is required.
- Local regression tests establish the software behavior. The new
revision has not been deployed to a hosted instance.

## Model Used

OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, with xhigh
reasoning, repository inspection, tool use and code execution. The
context-window size is not exposed by the runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 11:55:16 -06:00

552 lines
29 KiB
TypeScript

import { assertProviderApiPlacementPolicy, resolveProviderApiPlacement } from "../native-runtime/provider-api-placement.js";
import { heartbeatExternalProvider, nativeExecutionUsesTaskWorkspace } from "./external-provider.js";
import { heartbeatRuns, issues, type Db, type agents } from "@paperclipai/db";
import { and, eq } from "drizzle-orm";
import { readGitCacheExpiration } from "../git-workspace-restoration.js";
import type { ExecutionWorkspace, IssueExecutionWorkspaceSettings, ProjectExecutionWorkspacePolicy } from "@paperclipai/shared";
import { parseObject } from "../../adapters/utils.js";
import { logger } from "../../middleware/logger.js";
import { parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
import { isExecutionForcedToKubernetes } from "../execution-allowlist.js";
import { parseExecutionPolicyBootstrapEnv } from "../execution-policy-bootstrap.js";
import {
parseIssueExecutionWorkspaceSettings,
resolveEffectiveWorkspaceStrategyType,
resolveExecutionWorkspaceEnvironmentId,
resolveExecutionWorkspaceMode,
resolveSharedWorkspaceConcurrency,
type ParsedExecutionWorkspaceMode,
} from "../execution-workspace-policy.js";
import { assertTaskWorkspaceExecutionAccess, assertTaskWorkspaceSourceProjectAccess } from "../task-workspace-access.js";
import { isBoundIsolatedTaskDirectory, shouldUseIsolatedTaskDirectory } from "../isolated-task-directory.js";
import { readTaskDirectoryRetention } from "../isolated-task-directory-retention.js";
import { isBoundFreshPluginTaskDirectory } from "../provider-fresh-workspace.js";
import {
findNativeChatWorkspaceScope,
nativeChatWorkspaceCwd,
nativeChatWorkspaceMatches,
} from "../native-runtime/native-chat-workspace.js";
import { NativeRunnerOwnershipUnverifiedError } from "../native-runtime/native-runner-ownership.js";
import type { environmentService } from "../environments.js";
import type { executionWorkspaceService } from "../execution-workspaces.js";
import type { instanceSettingsService } from "../instance-settings.js";
import type { TrustPresetResolution } from "../trust-preset-resolver.js";
import { WORKSPACE_BUSY_RETRY_REASON } from "../../modules/run-dispatch/index.js";
import { WorkspaceBusyDeferral, type createHeartbeatRetries } from "./retries.js";
import { ConfigurationIncompleteFailure } from "./run-preparation.js";
import { resolveNativeRecoveryExecutionWorkspaceBinding, resolveExecutionWorkspaceReuseRequestForIssue, WorkspaceValidationFailure } from "./workspaces.js";
type Issue = typeof issues.$inferSelect;
/** Selects policy and reuse inputs; lease acquisition and cleanup belong to the executor. */
export interface HeartbeatEnvironmentSelectionInput {
run: Pick<typeof heartbeatRuns.$inferSelect,
"id" | "runtimeMode" | "runnerProfileJson" | "scheduledRetryReason" | "scheduledRetryAttempt">;
agent: Pick<typeof agents.$inferSelect,
"id" | "companyId" | "adapterType" | "adapterConfig" | "defaultEnvironmentId">;
task: {
freshWorkspace?: import("../fresh-workspace-continuation.js").ContinuationReadiness | null;
issueRef: Pick<Issue, "id" | "projectWorkspaceId" | "executionWorkspaceId" | "executionWorkspacePreference"> | null;
issueContext: Pick<Issue, "assigneeAgentId" | "executionWorkspaceSettings"> | null;
issueId: string | null;
context: Record<string, unknown>;
executionProjectId: string | null;
workspaceProjectContext: { hasWorkspace: boolean } | null;
boundSourceWorkspace: ExecutionWorkspace | null;
hasSelectedWorkspaceSource: boolean;
taskWorkspaceIntent: Issue["workspaceSelection"];
responsibleUserId: string | null;
};
policy: {
trustPreset: TrustPresetResolution;
requestedExecutionWorkspaceMode: ParsedExecutionWorkspaceMode;
isolatedWorkspacesEnabled: boolean;
projectExecutionWorkspacePolicy: ProjectExecutionWorkspacePolicy | null;
issueExecutionWorkspaceSettings: IssueExecutionWorkspaceSettings | null;
};
config: {
config: Record<string, unknown>;
sourceWorkspaceConfig: Record<string, unknown>;
issueAssigneeOverrides: { adapterConfig: Record<string, unknown> | null } | null;
};
services: {
executionWorkspacesSvc: Pick<ReturnType<typeof executionWorkspaceService>, "getStoredById" | "validateStoredSelectionForRun"
| "reopenClosedIsolatedExecutionWorkspaceForIssue" | "clearReopenPendingConsumptionForUnconsumedReopen">;
environmentsSvc: Pick<ReturnType<typeof environmentService>,
"ensureLocalEnvironment" | "findManagedSandboxEnvironment" | "findKubernetesEnvironment" | "ensureKubernetesEnvironment" | "getById">;
instanceSettings: Pick<ReturnType<typeof instanceSettingsService>, "get" | "getExperimental">;
findSharedWorkspaceHolder: ReturnType<typeof createHeartbeatRetries>["findSharedWorkspaceHolder"];
findExecutionWorkspaceHolder: ReturnType<typeof createHeartbeatRetries>["findExecutionWorkspaceHolder"];
};
}
function readNonEmptyString(value: unknown): string | null {
return typeof value === "string" && value.trim().length > 0 ? value : null;
}
export async function selectHeartbeatEnvironment(db: Db, input: HeartbeatEnvironmentSelectionInput) {
const { run, agent } = input;
const { issueRef, issueContext, issueId, context, executionProjectId, workspaceProjectContext, boundSourceWorkspace, hasSelectedWorkspaceSource, taskWorkspaceIntent, responsibleUserId } = input.task;
const {
trustPreset, requestedExecutionWorkspaceMode, isolatedWorkspacesEnabled: runtimeWorkspaceSelectionEnabled,
projectExecutionWorkspacePolicy, issueExecutionWorkspaceSettings,
} = input.policy;
const { config, sourceWorkspaceConfig, issueAssigneeOverrides } = input.config;
const { executionWorkspacesSvc, environmentsSvc, instanceSettings, findSharedWorkspaceHolder, findExecutionWorkspaceHolder } = input.services;
// A native run's execution input is immutable once persisted. Recovery must therefore
// restore the workspace bound to that input rather than consulting the issue's current
// workspace pointer: a newer run may already have moved or cleared the issue binding while
// this older provider session is still recoverable.
const persistedRunnerProfile = parseObject(run.runnerProfileJson);
const persistedNativeExecutionInput =
run.runtimeMode === "native" &&
persistedRunnerProfile.nativeExecutionInput !== undefined
? parseNativeExecutionInput(
persistedRunnerProfile.nativeExecutionInput,
)
: null;
// This retained name covers both external bridges; neither adopts task files.
const isDotRun = heartbeatExternalProvider({ run, agent }) !== null;
const providerApiPlacement = resolveProviderApiPlacement({ run, agent });
const controllerOnly = isDotRun || providerApiPlacement !== null;
const persistedNativeExecutionWorkspaceId =
providerApiPlacement ? null : persistedNativeExecutionInput?.binding.executionWorkspaceId ?? null;
const fresh = input.task.freshWorkspace?.mode === "fresh_isolated_workspace";
const requestedExecutionWorkspaceId = controllerOnly ? null :
persistedNativeExecutionWorkspaceId ??
readNonEmptyString(issueRef?.executionWorkspaceId);
let existingExecutionWorkspace = requestedExecutionWorkspaceId
? await executionWorkspacesSvc.getStoredById(agent.companyId, requestedExecutionWorkspaceId)
: null;
if (existingExecutionWorkspace && issueId && !controllerOnly && !fresh
&& existingExecutionWorkspace.id === readNonEmptyString(issueRef?.executionWorkspaceId)
&& (existingExecutionWorkspace.status === "archived" || existingExecutionWorkspace.status === "cleanup_failed")
&& (readGitCacheExpiration(existingExecutionWorkspace as Parameters<typeof readGitCacheExpiration>[0])?.workspaceRemoved
|| readTaskDirectoryRetention(existingExecutionWorkspace) !== null
|| (existingExecutionWorkspace.sourceIssueId === issueId
&& existingExecutionWorkspace.cleanupReason?.startsWith("issue_terminal") === true))) {
const [boundIssue] = await db.select({ id: issues.id, companyId: issues.companyId, projectId: issues.projectId })
.from(issues).where(and(eq(issues.id, issueId), eq(issues.companyId, agent.companyId)));
if (!boundIssue) throw new Error("Execution workspace issue is unavailable");
const actor = { agentId: agent.id, actorType: "agent" };
const reopened = await executionWorkspacesSvc.reopenClosedIsolatedExecutionWorkspaceForIssue({
workspaceId: existingExecutionWorkspace.id,
issue: boundIssue,
actor,
authorizationActor: { type: "agent", agentId: agent.id, companyId: agent.companyId, source: "agent_jwt", runId: run.id,
onBehalfOfUserId: responsibleUserId === "local-board" ? null : responsibleUserId },
});
if (!reopened.ok) throw new Error(reopened.message);
if (reopened.reopened) {
await executionWorkspacesSvc.clearReopenPendingConsumptionForUnconsumedReopen({
workspaceId: existingExecutionWorkspace.id, issue: boundIssue, actor, expectedGeneration: reopened.generation,
});
}
existingExecutionWorkspace = await executionWorkspacesSvc.getStoredById(agent.companyId, existingExecutionWorkspace.id);
}
const retainedDirectory = input.task.freshWorkspace?.mode === "reuse_current_workspace" ? input.task.freshWorkspace : null;
if (retainedDirectory && (issueRef?.executionWorkspaceId !== retainedDirectory.workspaceId ||
existingExecutionWorkspace?.id !== retainedDirectory.workspaceId || existingExecutionWorkspace.cwd !== retainedDirectory.cwd)) {
throw new WorkspaceValidationFailure("The saved Retry workspace binding changed.", {
workspaceValidation: { reason: "isolated_task_directory_binding_mismatch", issueId },
});
}
const nativeRecoveryExecutionWorkspaceId =
resolveNativeRecoveryExecutionWorkspaceBinding({
bindingId: persistedNativeExecutionWorkspaceId,
persistedWorkspaceFound: existingExecutionWorkspace !== null,
});
const workspaceReuseRequest =
resolveExecutionWorkspaceReuseRequestForIssue({
issueExecutionWorkspaceId: requestedExecutionWorkspaceId,
issueExecutionWorkspacePreference: controllerOnly ? null : nativeRecoveryExecutionWorkspaceId
? "reuse_existing"
: (issueRef?.executionWorkspacePreference ?? null),
existingExecutionWorkspaceStatus:
existingExecutionWorkspace?.status ?? null,
});
const requestedShouldReuseExisting =
workspaceReuseRequest.requestedShouldReuseExisting;
const reusableExistingExecutionWorkspace =
workspaceReuseRequest.existingExecutionWorkspaceAvailable
? existingExecutionWorkspace
: null;
const workspaceAuthorizationActor = { type: "agent" as const, agentId: agent.id, companyId: agent.companyId,
source: "agent_jwt" as const, runId: run.id, onBehalfOfUserId: responsibleUserId === "local-board" ? null : responsibleUserId };
if (!controllerOnly && !persistedNativeExecutionInput && !nativeRecoveryExecutionWorkspaceId) {
const workspaceRequiringAccess = reusableExistingExecutionWorkspace ?? boundSourceWorkspace;
if (workspaceRequiringAccess && issueId) {
await assertTaskWorkspaceExecutionAccess(db, workspaceAuthorizationActor, agent.companyId, { issueId, workspaceId: workspaceRequiringAccess.id });
} else if (!controllerOnly && executionProjectId && (hasSelectedWorkspaceSource || workspaceProjectContext?.hasWorkspace)) {
await assertTaskWorkspaceSourceProjectAccess(db, workspaceAuthorizationActor, agent.companyId, executionProjectId);
}
}
const requestedReusableExecutionWorkspaceConfig =
reusableExistingExecutionWorkspace?.config ?? null;
const localEnvironment = await environmentsSvc.ensureLocalEnvironment(
agent.companyId,
);
const resolvedInstanceSettings = await instanceSettings.get();
// Managed-sandbox-only policy: a run that would land on the local
// environment is redirected onto the platform-managed sandbox row, and
// with no active managed row the resolution fails closed
// (ManagedSandboxUnavailableError) — never local. Mirrors the forced
// kubernetes execution mode below, which takes precedence when both
// regimes are active.
const managedSandboxOnly =
(await instanceSettings.getExperimental()).enableManagedSandboxOnly ===
true;
const managedSandboxEnvironment = managedSandboxOnly && !providerApiPlacement
? await environmentsSvc.findManagedSandboxEnvironment(agent.companyId)
: null;
const environmentResolution = providerApiPlacement
? { environmentId: null, source: "default" as const }
: resolveExecutionWorkspaceEnvironmentId({
lowTrustIssueEnvironmentId: trustPreset.kind === "low_trust_review"
? parseIssueExecutionWorkspaceSettings(issueContext?.executionWorkspaceSettings, {includeEnvironmentId: true})?.environmentId : null,
agentDefaultEnvironmentId: agent.defaultEnvironmentId,
instanceDefaultEnvironmentId:
resolvedInstanceSettings.defaultEnvironmentId ?? null,
localDefaultEnvironmentId: localEnvironment.id,
managedSandboxOnly,
managedSandboxEnvironmentId: managedSandboxEnvironment?.id ?? null,
});
const effectiveExecutionWorkspaceMode: ReturnType<
typeof resolveExecutionWorkspaceMode
> = requestedExecutionWorkspaceMode;
const executionPolicy = {
executionMode: resolvedInstanceSettings.general.executionMode,
// Backstop behind the resolver's local→managed redirect: the run-time
// allowlist below fails any run that still resolved to a `local`
// environment under managed-sandbox-only, so no selection path or
// tenant-set env var can land untrusted execution on the tenant
// container.
managedSandboxOnly,
};
const executionForcedToKubernetes =
isExecutionForcedToKubernetes(executionPolicy);
if (providerApiPlacement) assertProviderApiPlacementPolicy(executionPolicy);
let selectedEnvironmentId = environmentResolution.environmentId;
if (executionForcedToKubernetes && !providerApiPlacement) {
let kubernetesEnvironment =
await environmentsSvc.findKubernetesEnvironment(agent.companyId);
if (!kubernetesEnvironment) {
// Lazy recovery for companies created after the startup bootstrap ran
// (the boot hook only provisions environments for companies that exist
// at boot). Re-derive the managed-env config from the bootstrap env.
// If the process env no longer forces Kubernetes (rollback / config
// drift relative to the persisted executionMode setting), skip the
// provisioning gracefully: the guard below still refuses local
// fallback with the explicit error, instead of crashing here on
// undefined config.
let bootstrap: ReturnType<typeof parseExecutionPolicyBootstrapEnv> =
null;
let bootstrapSkipReason: string | null = null;
try {
bootstrap = parseExecutionPolicyBootstrapEnv(process.env);
if (!bootstrap) {
bootstrapSkipReason =
'PAPERCLIP_EXECUTION_MODE bootstrap env is not kubernetes-forced (absent or "any")';
}
} catch (err) {
bootstrapSkipReason = `PAPERCLIP_EXECUTION_MODE bootstrap env failed to parse: ${
err instanceof Error ? err.message : String(err)
}`;
}
if (bootstrap) {
await environmentsSvc.ensureKubernetesEnvironment(
agent.companyId,
bootstrap.kubernetesConfig,
);
kubernetesEnvironment =
await environmentsSvc.findKubernetesEnvironment(agent.companyId);
} else {
logger.warn(
{
runId: run.id,
agentId: agent.id,
companyId: agent.companyId,
reason: bootstrapSkipReason,
},
"executionMode=kubernetes is persisted but the bootstrap env cannot provision a managed Kubernetes environment; skipping lazy provisioning for this company (the run will fail with the explicit no-managed-environment error)",
);
}
}
if (!kubernetesEnvironment) {
throw new Error(
"Instance execution policy requires the Kubernetes sandbox provider " +
"(executionMode=kubernetes) but no managed Kubernetes environment is " +
"configured for this company. Configure one (PAPERCLIP_K8S_* env on the " +
"cloud instance) before running agents; refusing to fall back to local execution.",
);
}
if (kubernetesEnvironment.id !== selectedEnvironmentId) {
logger.info(
{
runId: run.id,
issueId,
agentId: agent.id,
resolvedEnvironmentId: selectedEnvironmentId,
forcedKubernetesEnvironmentId: kubernetesEnvironment.id,
},
"Forcing run onto the managed Kubernetes environment (executionMode=kubernetes)",
);
}
selectedEnvironmentId = kubernetesEnvironment.id;
}
const selectedEnvironmentForConfig =
selectedEnvironmentId === localEnvironment.id
? localEnvironment
: selectedEnvironmentId
? await environmentsSvc.getById(selectedEnvironmentId)
: null;
const nativeChatWorkspaceScope = controllerOnly ? null : await findNativeChatWorkspaceScope(db, {
adapterType: agent.adapterType,
environmentDriver: selectedEnvironmentForConfig?.driver ?? null,
companyId: agent.companyId,
agentId: agent.id,
issueId,
...(fresh ? { freshAttemptId: run.id } : {}),
admittedCwd: nativeExecutionUsesTaskWorkspace(persistedNativeExecutionInput)
? persistedNativeExecutionInput.workspace.cwd : null,
});
let nativeChatExpectedCwd = nativeChatWorkspaceScope
? nativeChatWorkspaceCwd(
nativeChatWorkspaceScope,
reusableExistingExecutionWorkspace,
requestedShouldReuseExisting,
)
: null;
if (
nativeChatWorkspaceScope &&
nativeExecutionUsesTaskWorkspace(persistedNativeExecutionInput) &&
!nativeChatWorkspaceMatches({
scope: nativeChatWorkspaceScope,
expectedCwd: nativeChatExpectedCwd,
execution: persistedNativeExecutionInput,
})
) {
// Never rewrite an admitted provider input or release ownership of an
// older process whose permissions still include the shared agent home.
throw new NativeRunnerOwnershipUnverifiedError(
"native_chat_workspace_scope_mismatch",
);
}
const materializeNativeChatWorktree = Boolean(
nativeChatWorkspaceScope?.projectId && !persistedNativeExecutionInput && !existingExecutionWorkspace &&
taskWorkspaceIntent?.selection.kind === "configured_source" &&
taskWorkspaceIntent.selection.mode === "managed_isolated" &&
requestedExecutionWorkspaceMode === "isolated_workspace" &&
resolveEffectiveWorkspaceStrategyType(requestedExecutionWorkspaceMode, sourceWorkspaceConfig) === "git_worktree",
);
if (
nativeChatWorkspaceScope &&
((!nativeChatExpectedCwd && !materializeNativeChatWorktree) ||
executionProjectId !== nativeChatWorkspaceScope.projectId)
) {
throw new ConfigurationIncompleteFailure(
"External chat requires a task-owned isolated workspace. Select a managed isolated source or an existing task-owned isolated worktree; shared project workspaces cannot be used for external chat.",
{
configurationIncomplete: {
reason: "native_chat_workspace_isolation_required",
issueId,
},
},
);
}
const sharedWorkspaceConcurrency = resolveSharedWorkspaceConcurrency({
projectPolicy: projectExecutionWorkspacePolicy,
issueSettings: issueExecutionWorkspaceSettings,
});
// A reused shared execution workspace (for example one a child task inherited from a
// parent whose run is still live) follows the same policy as a shared project workspace:
// only an explicit "allow" is unsupported outside local/SSH; otherwise wait for the holder.
// A holder silent past the staleness cutoff is still running, so its folder is not free:
// fail rather than overlap it or wait on it forever.
if (!nativeRecoveryExecutionWorkspaceId && reusableExistingExecutionWorkspace &&
effectiveExecutionWorkspaceMode === "shared_workspace" &&
selectedEnvironmentForConfig?.driver !== "local" && selectedEnvironmentForConfig?.driver !== "ssh") {
const executionWorkspaceHolder = await findExecutionWorkspaceHolder({
companyId: agent.companyId,
executionWorkspaceId: reusableExistingExecutionWorkspace.id,
excludeRunId: run.id,
});
if (executionWorkspaceHolder) {
if (executionWorkspaceHolder.stale) {
throw new ConfigurationIncompleteFailure(`This environment cannot place concurrent agents in one physical shared folder, and run ${executionWorkspaceHolder.runId} still holds this shared execution workspace after more than an hour without output. Use local/SSH sharing or isolated task workspaces.`, {
configurationIncomplete: { reason: "shared_realization_unsupported", issueId, holderRunId: executionWorkspaceHolder.runId, holderStale: true },
});
}
if (sharedWorkspaceConcurrency === "allow") {
throw new ConfigurationIncompleteFailure("This environment cannot place concurrent agents in one physical shared folder. Use local/SSH sharing or isolated task workspaces.", {
configurationIncomplete: { reason: "shared_realization_unsupported", issueId },
});
}
throw new WorkspaceBusyDeferral({
holder: executionWorkspaceHolder,
executionWorkspaceId: reusableExistingExecutionWorkspace.id,
deferralAttempt:
run.scheduledRetryReason === WORKSPACE_BUSY_RETRY_REASON
? (run.scheduledRetryAttempt ?? 0)
: 0,
wasIssueAssignee: issueContext?.assigneeAgentId === agent.id,
});
}
}
// A live holder is always consulted for shared workspaces. Depending on policy and the final
// execution target it either remains the existing deferral gate or becomes dispatch context.
// Local/SSH folders never take an exclusive workspace lock, including when older
// project or issue settings request serialization. Sandbox protection still uses
// the existing holder staleness and workspace_busy retry ladder.
if (
!controllerOnly && issueRef?.projectWorkspaceId &&
effectiveExecutionWorkspaceMode === "shared_workspace"
) {
const workspaceHolder = await findSharedWorkspaceHolder({
companyId: agent.companyId,
projectWorkspaceId: issueRef.projectWorkspaceId,
excludeIssueId: issueRef.id,
excludeRunId: run.id,
honorIsolatedWorkspaceModes: runtimeWorkspaceSelectionEnabled,
});
if (workspaceHolder) {
const environmentDriver =
selectedEnvironmentForConfig?.driver ?? null;
if (!nativeRecoveryExecutionWorkspaceId && sharedWorkspaceConcurrency === "allow" &&
(executionForcedToKubernetes || (environmentDriver !== "local" && environmentDriver !== "ssh"))) {
throw new ConfigurationIncompleteFailure("This environment cannot place concurrent agents in one physical shared folder. Use local/SSH sharing or isolated task workspaces.", {
configurationIncomplete: { reason: "shared_realization_unsupported", issueId },
});
}
const shouldSerialize =
sharedWorkspaceConcurrency !== "allow" &&
(executionForcedToKubernetes ||
(environmentDriver !== "local" &&
environmentDriver !== "ssh"));
if (shouldSerialize) {
throw new WorkspaceBusyDeferral({
holder: workspaceHolder,
projectWorkspaceId: issueRef.projectWorkspaceId,
deferralAttempt:
run.scheduledRetryReason === WORKSPACE_BUSY_RETRY_REASON
? (run.scheduledRetryAttempt ?? 0)
: 0,
wasIssueAssignee: issueContext?.assigneeAgentId === agent.id,
});
}
const holderIssueLabel =
workspaceHolder.issueIdentifier ?? workspaceHolder.issueId;
const concurrentWorkspaceNote =
`shared workspace is concurrently held by run ${workspaceHolder.runId} (issue ${holderIssueLabel}); ` +
"expect concurrent mutations, coordinate via commits";
const appendConcurrentWorkspaceNote = (value: unknown) => {
const existing = typeof value === "string" ? value.trimEnd() : "";
return existing
? `${existing}\n${concurrentWorkspaceNote}`
: concurrentWorkspaceNote;
};
context.paperclipTaskMarkdown = appendConcurrentWorkspaceNote(
context.paperclipTaskMarkdown,
);
context.paperclipTaskMarkdownAssignment = appendConcurrentWorkspaceNote(
context.paperclipTaskMarkdownAssignment,
);
if (typeof context.paperclipTaskMarkdownCompact === "string") {
context.paperclipTaskMarkdownCompact =
appendConcurrentWorkspaceNote(
context.paperclipTaskMarkdownCompact,
);
}
if (typeof context.paperclipTaskMarkdownAssignmentCompact === "string") {
context.paperclipTaskMarkdownAssignmentCompact =
appendConcurrentWorkspaceNote(
context.paperclipTaskMarkdownAssignmentCompact,
);
}
logger.info(
{
event: "shared_workspace_concurrent_dispatch",
runId: run.id,
issueId: issueRef.id,
projectWorkspaceId: issueRef.projectWorkspaceId,
holderRunId: workspaceHolder.runId,
holderIssueId: workspaceHolder.issueId,
sharedWorkspaceConcurrency,
environmentDriver,
executionForcedToKubernetes,
},
"Dispatching alongside a live shared-workspace holder",
);
}
}
if (!controllerOnly && taskWorkspaceIntent && !boundSourceWorkspace && !nativeRecoveryExecutionWorkspaceId) {
await executionWorkspacesSvc.validateStoredSelectionForRun({ companyId: agent.companyId,
actor: { type: "agent", agentId: agent.id, companyId: agent.companyId, runId: run.id, source: "agent_jwt", onBehalfOfUserId: responsibleUserId === "local-board" ? null : responsibleUserId },
selection: taskWorkspaceIntent.selection, issueId,
preserveImplicitDefaultSettings: ["project", "task_default"].includes(taskWorkspaceIntent.source) });
}
const explicitTaskDirectory = !controllerOnly && taskWorkspaceIntent?.selection.kind === "task_directory";
const explicitDirectoryStrategies = [
...(reusableExistingExecutionWorkspace?.providerType === "adapter_managed" ? [config.workspaceStrategy] : []),
issueAssigneeOverrides?.adapterConfig?.workspaceStrategy,
projectExecutionWorkspacePolicy?.workspaceStrategy, issueExecutionWorkspaceSettings?.workspaceStrategy];
const boundTaskDirectory = !controllerOnly && !fresh && !nativeRecoveryExecutionWorkspaceId && issueId !== null && issueRef !== null &&
requestedShouldReuseExisting && requestedExecutionWorkspaceMode === "isolated_workspace" &&
!workspaceProjectContext?.hasWorkspace && !issueRef.projectWorkspaceId &&
taskWorkspaceIntent?.selection.kind !== "configured_source" &&
(taskWorkspaceIntent?.selection.kind !== "existing" || taskWorkspaceIntent.selection.workspaceId === issueRef.executionWorkspaceId) &&
(await isBoundIsolatedTaskDirectory({ companyId: agent.companyId, issueId,
workspaceId: issueRef.executionWorkspaceId, workspace: reusableExistingExecutionWorkspace }) ||
(selectedEnvironmentForConfig?.driver === "plugin" && agent.adapterType === "paperclip_runner" &&
isBoundFreshPluginTaskDirectory({ companyId: agent.companyId, issueId,
workspaceId: issueRef.executionWorkspaceId, workspace: reusableExistingExecutionWorkspace,
owner: { agentId: agent.id, environmentId: selectedEnvironmentForConfig.id,
pluginKey: parseObject(selectedEnvironmentForConfig.config).pluginKey ?? null,
driverKey: parseObject(selectedEnvironmentForConfig.config).driverKey ?? null } })));
const reuseTaskDirectory = boundTaskDirectory && explicitDirectoryStrategies.every(value => {
const strategy = parseObject(value);
return Object.keys(strategy).length === 0 || (Object.keys(strategy).length === 1 && strategy.type === "project_primary");
});
if (retainedDirectory && !boundTaskDirectory) throw new WorkspaceValidationFailure("The saved Retry task directory is unavailable.", {
workspaceValidation: { reason: "isolated_task_directory_binding_mismatch", issueId },
});
const useIsolatedTaskDirectory = !controllerOnly && issueRef !== null && (reuseTaskDirectory || (fresh && !workspaceProjectContext?.hasWorkspace && !issueRef.projectWorkspaceId) || shouldUseIsolatedTaskDirectory({
trustPreset: trustPreset.kind,
environmentDriver: selectedEnvironmentForConfig?.driver ?? null,
mode: requestedExecutionWorkspaceMode,
hasProjectWorkspace: workspaceProjectContext?.hasWorkspace ?? false,
projectWorkspaceId: issueRef.projectWorkspaceId,
workspaceStrategies: [
config.workspaceStrategy,
issueAssigneeOverrides?.adapterConfig?.workspaceStrategy,
projectExecutionWorkspacePolicy?.workspaceStrategy,
issueExecutionWorkspaceSettings?.workspaceStrategy,
],
}));
return {
persistedRunnerProfile,
persistedNativeExecutionInput,
isDotRun,
providerApiPlacement,
existingExecutionWorkspace,
nativeRecoveryExecutionWorkspaceId,
workspaceReuseRequest,
requestedShouldReuseExisting,
reusableExistingExecutionWorkspace,
requestedReusableExecutionWorkspaceConfig,
localEnvironment,
resolvedInstanceSettings,
environmentResolution,
effectiveExecutionWorkspaceMode,
executionPolicy,
selectedEnvironmentId,
selectedEnvironmentForConfig,
nativeChatWorkspaceScope,
nativeChatExpectedCwd,
materializeNativeChatWorktree,
explicitTaskDirectory,
useIsolatedTaskDirectory,
};
}