mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 07:10:34 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Tasks keep execution identity and saved workspace files separate from control-plane updates. > - A scanner can have permission to reopen a case while its run cannot restore that case's files. > - Retry can also follow a preparation failure that has no recovery-action record. > - Both paths can accept valid work but leave the target unable to run. > - This change restores files under the target executor and saves admitted Retry authority on the canonical receipt. > - Existing workspace, user, source and successor checks still apply. ## Linked Issues or Issue Description **What happened?** An authorized agent updates another closed task. The route tries to restore the target workspace under the caller's run identity and returns `403 Task workspace run identity is unavailable`. A separate failure occurs when a normal board Retry follows a pre-provider preparation failure. Admission has no recovery-action row on which to save continuation authority. Dispatch then fails with `continuation_user_authorization_missing`. **Expected behavior** Accept the authorized task update. Restore its terminal workspace when the target's admitted executor starts. An admitted Retry must retain its exact dispatch authority even when the predecessor has no recovery-action record. **Steps to reproduce** 1. Complete a task with an isolated workspace. From a different running task, use an authorized comment or PATCH with `resume: true` on that closed task. 2. Separately, fail native preparation before provider work starts, with no recovery-action row. Submit board Retry, admit its successor, and build the execution continuation. 3. Before this change, the first path returns 403 and the second rejects its admitted authority. **Paperclip version or commit** Reproduced against `e968131efa7e63d321f3bcca8d47a62513c2efb4`. Rebased onto `afbc5a885b5`. **Deployment mode** Server-managed native task execution. Tests use real PostgreSQL and temporary workspace files. Related PRs: #16927 covers terminal workspace races. #17916 retains exact native export sources. Neither covers these route and Retry receipt failures. A GitHub search found no duplicate. ROADMAP.md was checked; this is a recovery bug fix. ## What Changed - Defer workspace restoration only for authorized cross-task comments and updates from a live run. Checkout keeps its existing workspace identity checks. - Restore the target's selected terminal workspace under its own admitted run. Preserve its saved files and existing source checks. Leave manually archived workspaces unchanged. - Store server-created continuation authority on the canonical admitted Retry receipt when no source recovery-action record exists. Existing action records remain authoritative. Read the fallback only after matching the company, task, agent, user, source, generation and successor. - Add route and PostgreSQL regression tests. Document both paths. ## Verification - Red: both cross-task route tests returned the observed 403 before the route fix. - Red: the target executor could not reuse its archived terminal workspace before the restoration fix. - Red: an admitted preparation Retry with no recovery-action row failed with `continuation_user_authorization_missing` before the receipt fix. - Green: 419 recovery and continuation tests pass, plus 183 focused route, environment and Retry tests. The assembled heartbeat test starts the agent in the same terminal worktree and verifies both its unpushed commit and uncommitted file bytes. It fails before the restoration fix. - Negative cases reject wrong Retry runs, users, source runs, generations, unadmitted receipts and missing authority. The original failed run stays unchanged. Workspace tests verify saved file bytes. - Full `pnpm build`, `pnpm -r typecheck`, `pnpm check:module-boundaries` and `git diff --check` pass. - The full suite found an existing assembled chat authority regression: clearing a recovery-action record could be bypassed by the new fallback. The fix now writes the fallback only when that source has no action record. That assembled regression passes with the original test unchanged. - Full `pnpm test:run` is running again. Its first attempt also hit an unrelated ancestor skill-root lookup for a missing Slack skill. The rerun uses a temporary untracked link to the repository's own bundled skills. This link is excluded from the PR. Full-suite completion and hosted CI are pending. - Hosted CI has multiple runner interruptions. E2e shard 1/8, server shards 3/12, 8/12 and 12/12, and workspaces-a shard 1/2 show runner shutdown signals. E2e shard 5/8 and server shards 1/12 and 2/12 have GitHub annotations reporting lost runner communication; their logs return 404. The `ci / verify` assertion failed because the general-test matrix was cancelled. GitHub rejected job and failed-workflow reruns while this workflow is still running. Rerun the interrupted checks after it settles. - Greptile reviewed exact head `a1e3929aedb1ee3bfcd001f4ebb1510106bf4070` at 2026-11-04 17:37 UTC: 5/5, with no new blocking findings and no unresolved review threads. ## Risks - A deferred workspace restore can still fail at target admission. The run keeps the existing workspace failure path; this change does not bypass file recovery or provider stop checks. - Retry receipts now store server-created authority. Existing action receipts remain valid. No schema migration is required. - Local regression tests establish the software behavior. The new revision has not been deployed to a hosted instance. ## Model Used OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, with xhigh reasoning, repository inspection, tool use and code execution. The context-window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
552 lines
29 KiB
TypeScript
552 lines
29 KiB
TypeScript
import { assertProviderApiPlacementPolicy, resolveProviderApiPlacement } from "../native-runtime/provider-api-placement.js";
|
|
import { heartbeatExternalProvider, nativeExecutionUsesTaskWorkspace } from "./external-provider.js";
|
|
import { heartbeatRuns, issues, type Db, type agents } from "@paperclipai/db";
|
|
import { and, eq } from "drizzle-orm";
|
|
import { readGitCacheExpiration } from "../git-workspace-restoration.js";
|
|
import type { ExecutionWorkspace, IssueExecutionWorkspaceSettings, ProjectExecutionWorkspacePolicy } from "@paperclipai/shared";
|
|
import { parseObject } from "../../adapters/utils.js";
|
|
import { logger } from "../../middleware/logger.js";
|
|
import { parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
|
|
import { isExecutionForcedToKubernetes } from "../execution-allowlist.js";
|
|
import { parseExecutionPolicyBootstrapEnv } from "../execution-policy-bootstrap.js";
|
|
import {
|
|
parseIssueExecutionWorkspaceSettings,
|
|
resolveEffectiveWorkspaceStrategyType,
|
|
resolveExecutionWorkspaceEnvironmentId,
|
|
resolveExecutionWorkspaceMode,
|
|
resolveSharedWorkspaceConcurrency,
|
|
type ParsedExecutionWorkspaceMode,
|
|
} from "../execution-workspace-policy.js";
|
|
import { assertTaskWorkspaceExecutionAccess, assertTaskWorkspaceSourceProjectAccess } from "../task-workspace-access.js";
|
|
import { isBoundIsolatedTaskDirectory, shouldUseIsolatedTaskDirectory } from "../isolated-task-directory.js";
|
|
import { readTaskDirectoryRetention } from "../isolated-task-directory-retention.js";
|
|
import { isBoundFreshPluginTaskDirectory } from "../provider-fresh-workspace.js";
|
|
import {
|
|
findNativeChatWorkspaceScope,
|
|
nativeChatWorkspaceCwd,
|
|
nativeChatWorkspaceMatches,
|
|
} from "../native-runtime/native-chat-workspace.js";
|
|
import { NativeRunnerOwnershipUnverifiedError } from "../native-runtime/native-runner-ownership.js";
|
|
import type { environmentService } from "../environments.js";
|
|
import type { executionWorkspaceService } from "../execution-workspaces.js";
|
|
import type { instanceSettingsService } from "../instance-settings.js";
|
|
import type { TrustPresetResolution } from "../trust-preset-resolver.js";
|
|
import { WORKSPACE_BUSY_RETRY_REASON } from "../../modules/run-dispatch/index.js";
|
|
import { WorkspaceBusyDeferral, type createHeartbeatRetries } from "./retries.js";
|
|
import { ConfigurationIncompleteFailure } from "./run-preparation.js";
|
|
import { resolveNativeRecoveryExecutionWorkspaceBinding, resolveExecutionWorkspaceReuseRequestForIssue, WorkspaceValidationFailure } from "./workspaces.js";
|
|
|
|
type Issue = typeof issues.$inferSelect;
|
|
|
|
/** Selects policy and reuse inputs; lease acquisition and cleanup belong to the executor. */
|
|
export interface HeartbeatEnvironmentSelectionInput {
|
|
run: Pick<typeof heartbeatRuns.$inferSelect,
|
|
"id" | "runtimeMode" | "runnerProfileJson" | "scheduledRetryReason" | "scheduledRetryAttempt">;
|
|
agent: Pick<typeof agents.$inferSelect,
|
|
"id" | "companyId" | "adapterType" | "adapterConfig" | "defaultEnvironmentId">;
|
|
task: {
|
|
freshWorkspace?: import("../fresh-workspace-continuation.js").ContinuationReadiness | null;
|
|
issueRef: Pick<Issue, "id" | "projectWorkspaceId" | "executionWorkspaceId" | "executionWorkspacePreference"> | null;
|
|
issueContext: Pick<Issue, "assigneeAgentId" | "executionWorkspaceSettings"> | null;
|
|
issueId: string | null;
|
|
context: Record<string, unknown>;
|
|
executionProjectId: string | null;
|
|
workspaceProjectContext: { hasWorkspace: boolean } | null;
|
|
boundSourceWorkspace: ExecutionWorkspace | null;
|
|
hasSelectedWorkspaceSource: boolean;
|
|
taskWorkspaceIntent: Issue["workspaceSelection"];
|
|
responsibleUserId: string | null;
|
|
};
|
|
policy: {
|
|
trustPreset: TrustPresetResolution;
|
|
requestedExecutionWorkspaceMode: ParsedExecutionWorkspaceMode;
|
|
isolatedWorkspacesEnabled: boolean;
|
|
projectExecutionWorkspacePolicy: ProjectExecutionWorkspacePolicy | null;
|
|
issueExecutionWorkspaceSettings: IssueExecutionWorkspaceSettings | null;
|
|
};
|
|
config: {
|
|
config: Record<string, unknown>;
|
|
sourceWorkspaceConfig: Record<string, unknown>;
|
|
issueAssigneeOverrides: { adapterConfig: Record<string, unknown> | null } | null;
|
|
};
|
|
services: {
|
|
executionWorkspacesSvc: Pick<ReturnType<typeof executionWorkspaceService>, "getStoredById" | "validateStoredSelectionForRun"
|
|
| "reopenClosedIsolatedExecutionWorkspaceForIssue" | "clearReopenPendingConsumptionForUnconsumedReopen">;
|
|
environmentsSvc: Pick<ReturnType<typeof environmentService>,
|
|
"ensureLocalEnvironment" | "findManagedSandboxEnvironment" | "findKubernetesEnvironment" | "ensureKubernetesEnvironment" | "getById">;
|
|
instanceSettings: Pick<ReturnType<typeof instanceSettingsService>, "get" | "getExperimental">;
|
|
findSharedWorkspaceHolder: ReturnType<typeof createHeartbeatRetries>["findSharedWorkspaceHolder"];
|
|
findExecutionWorkspaceHolder: ReturnType<typeof createHeartbeatRetries>["findExecutionWorkspaceHolder"];
|
|
};
|
|
}
|
|
|
|
function readNonEmptyString(value: unknown): string | null {
|
|
return typeof value === "string" && value.trim().length > 0 ? value : null;
|
|
}
|
|
|
|
export async function selectHeartbeatEnvironment(db: Db, input: HeartbeatEnvironmentSelectionInput) {
|
|
const { run, agent } = input;
|
|
const { issueRef, issueContext, issueId, context, executionProjectId, workspaceProjectContext, boundSourceWorkspace, hasSelectedWorkspaceSource, taskWorkspaceIntent, responsibleUserId } = input.task;
|
|
const {
|
|
trustPreset, requestedExecutionWorkspaceMode, isolatedWorkspacesEnabled: runtimeWorkspaceSelectionEnabled,
|
|
projectExecutionWorkspacePolicy, issueExecutionWorkspaceSettings,
|
|
} = input.policy;
|
|
const { config, sourceWorkspaceConfig, issueAssigneeOverrides } = input.config;
|
|
const { executionWorkspacesSvc, environmentsSvc, instanceSettings, findSharedWorkspaceHolder, findExecutionWorkspaceHolder } = input.services;
|
|
// A native run's execution input is immutable once persisted. Recovery must therefore
|
|
// restore the workspace bound to that input rather than consulting the issue's current
|
|
// workspace pointer: a newer run may already have moved or cleared the issue binding while
|
|
// this older provider session is still recoverable.
|
|
const persistedRunnerProfile = parseObject(run.runnerProfileJson);
|
|
const persistedNativeExecutionInput =
|
|
run.runtimeMode === "native" &&
|
|
persistedRunnerProfile.nativeExecutionInput !== undefined
|
|
? parseNativeExecutionInput(
|
|
persistedRunnerProfile.nativeExecutionInput,
|
|
)
|
|
: null;
|
|
// This retained name covers both external bridges; neither adopts task files.
|
|
const isDotRun = heartbeatExternalProvider({ run, agent }) !== null;
|
|
const providerApiPlacement = resolveProviderApiPlacement({ run, agent });
|
|
const controllerOnly = isDotRun || providerApiPlacement !== null;
|
|
const persistedNativeExecutionWorkspaceId =
|
|
providerApiPlacement ? null : persistedNativeExecutionInput?.binding.executionWorkspaceId ?? null;
|
|
const fresh = input.task.freshWorkspace?.mode === "fresh_isolated_workspace";
|
|
const requestedExecutionWorkspaceId = controllerOnly ? null :
|
|
persistedNativeExecutionWorkspaceId ??
|
|
readNonEmptyString(issueRef?.executionWorkspaceId);
|
|
let existingExecutionWorkspace = requestedExecutionWorkspaceId
|
|
? await executionWorkspacesSvc.getStoredById(agent.companyId, requestedExecutionWorkspaceId)
|
|
: null;
|
|
if (existingExecutionWorkspace && issueId && !controllerOnly && !fresh
|
|
&& existingExecutionWorkspace.id === readNonEmptyString(issueRef?.executionWorkspaceId)
|
|
&& (existingExecutionWorkspace.status === "archived" || existingExecutionWorkspace.status === "cleanup_failed")
|
|
&& (readGitCacheExpiration(existingExecutionWorkspace as Parameters<typeof readGitCacheExpiration>[0])?.workspaceRemoved
|
|
|| readTaskDirectoryRetention(existingExecutionWorkspace) !== null
|
|
|| (existingExecutionWorkspace.sourceIssueId === issueId
|
|
&& existingExecutionWorkspace.cleanupReason?.startsWith("issue_terminal") === true))) {
|
|
const [boundIssue] = await db.select({ id: issues.id, companyId: issues.companyId, projectId: issues.projectId })
|
|
.from(issues).where(and(eq(issues.id, issueId), eq(issues.companyId, agent.companyId)));
|
|
if (!boundIssue) throw new Error("Execution workspace issue is unavailable");
|
|
const actor = { agentId: agent.id, actorType: "agent" };
|
|
const reopened = await executionWorkspacesSvc.reopenClosedIsolatedExecutionWorkspaceForIssue({
|
|
workspaceId: existingExecutionWorkspace.id,
|
|
issue: boundIssue,
|
|
actor,
|
|
authorizationActor: { type: "agent", agentId: agent.id, companyId: agent.companyId, source: "agent_jwt", runId: run.id,
|
|
onBehalfOfUserId: responsibleUserId === "local-board" ? null : responsibleUserId },
|
|
});
|
|
if (!reopened.ok) throw new Error(reopened.message);
|
|
if (reopened.reopened) {
|
|
await executionWorkspacesSvc.clearReopenPendingConsumptionForUnconsumedReopen({
|
|
workspaceId: existingExecutionWorkspace.id, issue: boundIssue, actor, expectedGeneration: reopened.generation,
|
|
});
|
|
}
|
|
existingExecutionWorkspace = await executionWorkspacesSvc.getStoredById(agent.companyId, existingExecutionWorkspace.id);
|
|
}
|
|
const retainedDirectory = input.task.freshWorkspace?.mode === "reuse_current_workspace" ? input.task.freshWorkspace : null;
|
|
if (retainedDirectory && (issueRef?.executionWorkspaceId !== retainedDirectory.workspaceId ||
|
|
existingExecutionWorkspace?.id !== retainedDirectory.workspaceId || existingExecutionWorkspace.cwd !== retainedDirectory.cwd)) {
|
|
throw new WorkspaceValidationFailure("The saved Retry workspace binding changed.", {
|
|
workspaceValidation: { reason: "isolated_task_directory_binding_mismatch", issueId },
|
|
});
|
|
}
|
|
const nativeRecoveryExecutionWorkspaceId =
|
|
resolveNativeRecoveryExecutionWorkspaceBinding({
|
|
bindingId: persistedNativeExecutionWorkspaceId,
|
|
persistedWorkspaceFound: existingExecutionWorkspace !== null,
|
|
});
|
|
const workspaceReuseRequest =
|
|
resolveExecutionWorkspaceReuseRequestForIssue({
|
|
issueExecutionWorkspaceId: requestedExecutionWorkspaceId,
|
|
issueExecutionWorkspacePreference: controllerOnly ? null : nativeRecoveryExecutionWorkspaceId
|
|
? "reuse_existing"
|
|
: (issueRef?.executionWorkspacePreference ?? null),
|
|
existingExecutionWorkspaceStatus:
|
|
existingExecutionWorkspace?.status ?? null,
|
|
});
|
|
const requestedShouldReuseExisting =
|
|
workspaceReuseRequest.requestedShouldReuseExisting;
|
|
const reusableExistingExecutionWorkspace =
|
|
workspaceReuseRequest.existingExecutionWorkspaceAvailable
|
|
? existingExecutionWorkspace
|
|
: null;
|
|
const workspaceAuthorizationActor = { type: "agent" as const, agentId: agent.id, companyId: agent.companyId,
|
|
source: "agent_jwt" as const, runId: run.id, onBehalfOfUserId: responsibleUserId === "local-board" ? null : responsibleUserId };
|
|
if (!controllerOnly && !persistedNativeExecutionInput && !nativeRecoveryExecutionWorkspaceId) {
|
|
const workspaceRequiringAccess = reusableExistingExecutionWorkspace ?? boundSourceWorkspace;
|
|
if (workspaceRequiringAccess && issueId) {
|
|
await assertTaskWorkspaceExecutionAccess(db, workspaceAuthorizationActor, agent.companyId, { issueId, workspaceId: workspaceRequiringAccess.id });
|
|
} else if (!controllerOnly && executionProjectId && (hasSelectedWorkspaceSource || workspaceProjectContext?.hasWorkspace)) {
|
|
await assertTaskWorkspaceSourceProjectAccess(db, workspaceAuthorizationActor, agent.companyId, executionProjectId);
|
|
}
|
|
}
|
|
const requestedReusableExecutionWorkspaceConfig =
|
|
reusableExistingExecutionWorkspace?.config ?? null;
|
|
const localEnvironment = await environmentsSvc.ensureLocalEnvironment(
|
|
agent.companyId,
|
|
);
|
|
const resolvedInstanceSettings = await instanceSettings.get();
|
|
// Managed-sandbox-only policy: a run that would land on the local
|
|
// environment is redirected onto the platform-managed sandbox row, and
|
|
// with no active managed row the resolution fails closed
|
|
// (ManagedSandboxUnavailableError) — never local. Mirrors the forced
|
|
// kubernetes execution mode below, which takes precedence when both
|
|
// regimes are active.
|
|
const managedSandboxOnly =
|
|
(await instanceSettings.getExperimental()).enableManagedSandboxOnly ===
|
|
true;
|
|
const managedSandboxEnvironment = managedSandboxOnly && !providerApiPlacement
|
|
? await environmentsSvc.findManagedSandboxEnvironment(agent.companyId)
|
|
: null;
|
|
const environmentResolution = providerApiPlacement
|
|
? { environmentId: null, source: "default" as const }
|
|
: resolveExecutionWorkspaceEnvironmentId({
|
|
lowTrustIssueEnvironmentId: trustPreset.kind === "low_trust_review"
|
|
? parseIssueExecutionWorkspaceSettings(issueContext?.executionWorkspaceSettings, {includeEnvironmentId: true})?.environmentId : null,
|
|
agentDefaultEnvironmentId: agent.defaultEnvironmentId,
|
|
instanceDefaultEnvironmentId:
|
|
resolvedInstanceSettings.defaultEnvironmentId ?? null,
|
|
localDefaultEnvironmentId: localEnvironment.id,
|
|
managedSandboxOnly,
|
|
managedSandboxEnvironmentId: managedSandboxEnvironment?.id ?? null,
|
|
});
|
|
const effectiveExecutionWorkspaceMode: ReturnType<
|
|
typeof resolveExecutionWorkspaceMode
|
|
> = requestedExecutionWorkspaceMode;
|
|
const executionPolicy = {
|
|
executionMode: resolvedInstanceSettings.general.executionMode,
|
|
// Backstop behind the resolver's local→managed redirect: the run-time
|
|
// allowlist below fails any run that still resolved to a `local`
|
|
// environment under managed-sandbox-only, so no selection path or
|
|
// tenant-set env var can land untrusted execution on the tenant
|
|
// container.
|
|
managedSandboxOnly,
|
|
};
|
|
const executionForcedToKubernetes =
|
|
isExecutionForcedToKubernetes(executionPolicy);
|
|
if (providerApiPlacement) assertProviderApiPlacementPolicy(executionPolicy);
|
|
let selectedEnvironmentId = environmentResolution.environmentId;
|
|
if (executionForcedToKubernetes && !providerApiPlacement) {
|
|
let kubernetesEnvironment =
|
|
await environmentsSvc.findKubernetesEnvironment(agent.companyId);
|
|
if (!kubernetesEnvironment) {
|
|
// Lazy recovery for companies created after the startup bootstrap ran
|
|
// (the boot hook only provisions environments for companies that exist
|
|
// at boot). Re-derive the managed-env config from the bootstrap env.
|
|
// If the process env no longer forces Kubernetes (rollback / config
|
|
// drift relative to the persisted executionMode setting), skip the
|
|
// provisioning gracefully: the guard below still refuses local
|
|
// fallback with the explicit error, instead of crashing here on
|
|
// undefined config.
|
|
let bootstrap: ReturnType<typeof parseExecutionPolicyBootstrapEnv> =
|
|
null;
|
|
let bootstrapSkipReason: string | null = null;
|
|
try {
|
|
bootstrap = parseExecutionPolicyBootstrapEnv(process.env);
|
|
if (!bootstrap) {
|
|
bootstrapSkipReason =
|
|
'PAPERCLIP_EXECUTION_MODE bootstrap env is not kubernetes-forced (absent or "any")';
|
|
}
|
|
} catch (err) {
|
|
bootstrapSkipReason = `PAPERCLIP_EXECUTION_MODE bootstrap env failed to parse: ${
|
|
err instanceof Error ? err.message : String(err)
|
|
}`;
|
|
}
|
|
if (bootstrap) {
|
|
await environmentsSvc.ensureKubernetesEnvironment(
|
|
agent.companyId,
|
|
bootstrap.kubernetesConfig,
|
|
);
|
|
kubernetesEnvironment =
|
|
await environmentsSvc.findKubernetesEnvironment(agent.companyId);
|
|
} else {
|
|
logger.warn(
|
|
{
|
|
runId: run.id,
|
|
agentId: agent.id,
|
|
companyId: agent.companyId,
|
|
reason: bootstrapSkipReason,
|
|
},
|
|
"executionMode=kubernetes is persisted but the bootstrap env cannot provision a managed Kubernetes environment; skipping lazy provisioning for this company (the run will fail with the explicit no-managed-environment error)",
|
|
);
|
|
}
|
|
}
|
|
if (!kubernetesEnvironment) {
|
|
throw new Error(
|
|
"Instance execution policy requires the Kubernetes sandbox provider " +
|
|
"(executionMode=kubernetes) but no managed Kubernetes environment is " +
|
|
"configured for this company. Configure one (PAPERCLIP_K8S_* env on the " +
|
|
"cloud instance) before running agents; refusing to fall back to local execution.",
|
|
);
|
|
}
|
|
if (kubernetesEnvironment.id !== selectedEnvironmentId) {
|
|
logger.info(
|
|
{
|
|
runId: run.id,
|
|
issueId,
|
|
agentId: agent.id,
|
|
resolvedEnvironmentId: selectedEnvironmentId,
|
|
forcedKubernetesEnvironmentId: kubernetesEnvironment.id,
|
|
},
|
|
"Forcing run onto the managed Kubernetes environment (executionMode=kubernetes)",
|
|
);
|
|
}
|
|
selectedEnvironmentId = kubernetesEnvironment.id;
|
|
}
|
|
const selectedEnvironmentForConfig =
|
|
selectedEnvironmentId === localEnvironment.id
|
|
? localEnvironment
|
|
: selectedEnvironmentId
|
|
? await environmentsSvc.getById(selectedEnvironmentId)
|
|
: null;
|
|
const nativeChatWorkspaceScope = controllerOnly ? null : await findNativeChatWorkspaceScope(db, {
|
|
adapterType: agent.adapterType,
|
|
environmentDriver: selectedEnvironmentForConfig?.driver ?? null,
|
|
companyId: agent.companyId,
|
|
agentId: agent.id,
|
|
issueId,
|
|
...(fresh ? { freshAttemptId: run.id } : {}),
|
|
admittedCwd: nativeExecutionUsesTaskWorkspace(persistedNativeExecutionInput)
|
|
? persistedNativeExecutionInput.workspace.cwd : null,
|
|
});
|
|
let nativeChatExpectedCwd = nativeChatWorkspaceScope
|
|
? nativeChatWorkspaceCwd(
|
|
nativeChatWorkspaceScope,
|
|
reusableExistingExecutionWorkspace,
|
|
requestedShouldReuseExisting,
|
|
)
|
|
: null;
|
|
if (
|
|
nativeChatWorkspaceScope &&
|
|
nativeExecutionUsesTaskWorkspace(persistedNativeExecutionInput) &&
|
|
!nativeChatWorkspaceMatches({
|
|
scope: nativeChatWorkspaceScope,
|
|
expectedCwd: nativeChatExpectedCwd,
|
|
execution: persistedNativeExecutionInput,
|
|
})
|
|
) {
|
|
// Never rewrite an admitted provider input or release ownership of an
|
|
// older process whose permissions still include the shared agent home.
|
|
throw new NativeRunnerOwnershipUnverifiedError(
|
|
"native_chat_workspace_scope_mismatch",
|
|
);
|
|
}
|
|
const materializeNativeChatWorktree = Boolean(
|
|
nativeChatWorkspaceScope?.projectId && !persistedNativeExecutionInput && !existingExecutionWorkspace &&
|
|
taskWorkspaceIntent?.selection.kind === "configured_source" &&
|
|
taskWorkspaceIntent.selection.mode === "managed_isolated" &&
|
|
requestedExecutionWorkspaceMode === "isolated_workspace" &&
|
|
resolveEffectiveWorkspaceStrategyType(requestedExecutionWorkspaceMode, sourceWorkspaceConfig) === "git_worktree",
|
|
);
|
|
if (
|
|
nativeChatWorkspaceScope &&
|
|
((!nativeChatExpectedCwd && !materializeNativeChatWorktree) ||
|
|
executionProjectId !== nativeChatWorkspaceScope.projectId)
|
|
) {
|
|
throw new ConfigurationIncompleteFailure(
|
|
"External chat requires a task-owned isolated workspace. Select a managed isolated source or an existing task-owned isolated worktree; shared project workspaces cannot be used for external chat.",
|
|
{
|
|
configurationIncomplete: {
|
|
reason: "native_chat_workspace_isolation_required",
|
|
issueId,
|
|
},
|
|
},
|
|
);
|
|
}
|
|
const sharedWorkspaceConcurrency = resolveSharedWorkspaceConcurrency({
|
|
projectPolicy: projectExecutionWorkspacePolicy,
|
|
issueSettings: issueExecutionWorkspaceSettings,
|
|
});
|
|
// A reused shared execution workspace (for example one a child task inherited from a
|
|
// parent whose run is still live) follows the same policy as a shared project workspace:
|
|
// only an explicit "allow" is unsupported outside local/SSH; otherwise wait for the holder.
|
|
// A holder silent past the staleness cutoff is still running, so its folder is not free:
|
|
// fail rather than overlap it or wait on it forever.
|
|
if (!nativeRecoveryExecutionWorkspaceId && reusableExistingExecutionWorkspace &&
|
|
effectiveExecutionWorkspaceMode === "shared_workspace" &&
|
|
selectedEnvironmentForConfig?.driver !== "local" && selectedEnvironmentForConfig?.driver !== "ssh") {
|
|
const executionWorkspaceHolder = await findExecutionWorkspaceHolder({
|
|
companyId: agent.companyId,
|
|
executionWorkspaceId: reusableExistingExecutionWorkspace.id,
|
|
excludeRunId: run.id,
|
|
});
|
|
if (executionWorkspaceHolder) {
|
|
if (executionWorkspaceHolder.stale) {
|
|
throw new ConfigurationIncompleteFailure(`This environment cannot place concurrent agents in one physical shared folder, and run ${executionWorkspaceHolder.runId} still holds this shared execution workspace after more than an hour without output. Use local/SSH sharing or isolated task workspaces.`, {
|
|
configurationIncomplete: { reason: "shared_realization_unsupported", issueId, holderRunId: executionWorkspaceHolder.runId, holderStale: true },
|
|
});
|
|
}
|
|
if (sharedWorkspaceConcurrency === "allow") {
|
|
throw new ConfigurationIncompleteFailure("This environment cannot place concurrent agents in one physical shared folder. Use local/SSH sharing or isolated task workspaces.", {
|
|
configurationIncomplete: { reason: "shared_realization_unsupported", issueId },
|
|
});
|
|
}
|
|
throw new WorkspaceBusyDeferral({
|
|
holder: executionWorkspaceHolder,
|
|
executionWorkspaceId: reusableExistingExecutionWorkspace.id,
|
|
deferralAttempt:
|
|
run.scheduledRetryReason === WORKSPACE_BUSY_RETRY_REASON
|
|
? (run.scheduledRetryAttempt ?? 0)
|
|
: 0,
|
|
wasIssueAssignee: issueContext?.assigneeAgentId === agent.id,
|
|
});
|
|
}
|
|
}
|
|
// A live holder is always consulted for shared workspaces. Depending on policy and the final
|
|
// execution target it either remains the existing deferral gate or becomes dispatch context.
|
|
// Local/SSH folders never take an exclusive workspace lock, including when older
|
|
// project or issue settings request serialization. Sandbox protection still uses
|
|
// the existing holder staleness and workspace_busy retry ladder.
|
|
if (
|
|
!controllerOnly && issueRef?.projectWorkspaceId &&
|
|
effectiveExecutionWorkspaceMode === "shared_workspace"
|
|
) {
|
|
const workspaceHolder = await findSharedWorkspaceHolder({
|
|
companyId: agent.companyId,
|
|
projectWorkspaceId: issueRef.projectWorkspaceId,
|
|
excludeIssueId: issueRef.id,
|
|
excludeRunId: run.id,
|
|
honorIsolatedWorkspaceModes: runtimeWorkspaceSelectionEnabled,
|
|
});
|
|
if (workspaceHolder) {
|
|
const environmentDriver =
|
|
selectedEnvironmentForConfig?.driver ?? null;
|
|
if (!nativeRecoveryExecutionWorkspaceId && sharedWorkspaceConcurrency === "allow" &&
|
|
(executionForcedToKubernetes || (environmentDriver !== "local" && environmentDriver !== "ssh"))) {
|
|
throw new ConfigurationIncompleteFailure("This environment cannot place concurrent agents in one physical shared folder. Use local/SSH sharing or isolated task workspaces.", {
|
|
configurationIncomplete: { reason: "shared_realization_unsupported", issueId },
|
|
});
|
|
}
|
|
const shouldSerialize =
|
|
sharedWorkspaceConcurrency !== "allow" &&
|
|
(executionForcedToKubernetes ||
|
|
(environmentDriver !== "local" &&
|
|
environmentDriver !== "ssh"));
|
|
if (shouldSerialize) {
|
|
throw new WorkspaceBusyDeferral({
|
|
holder: workspaceHolder,
|
|
projectWorkspaceId: issueRef.projectWorkspaceId,
|
|
deferralAttempt:
|
|
run.scheduledRetryReason === WORKSPACE_BUSY_RETRY_REASON
|
|
? (run.scheduledRetryAttempt ?? 0)
|
|
: 0,
|
|
wasIssueAssignee: issueContext?.assigneeAgentId === agent.id,
|
|
});
|
|
}
|
|
|
|
const holderIssueLabel =
|
|
workspaceHolder.issueIdentifier ?? workspaceHolder.issueId;
|
|
const concurrentWorkspaceNote =
|
|
`shared workspace is concurrently held by run ${workspaceHolder.runId} (issue ${holderIssueLabel}); ` +
|
|
"expect concurrent mutations, coordinate via commits";
|
|
const appendConcurrentWorkspaceNote = (value: unknown) => {
|
|
const existing = typeof value === "string" ? value.trimEnd() : "";
|
|
return existing
|
|
? `${existing}\n${concurrentWorkspaceNote}`
|
|
: concurrentWorkspaceNote;
|
|
};
|
|
context.paperclipTaskMarkdown = appendConcurrentWorkspaceNote(
|
|
context.paperclipTaskMarkdown,
|
|
);
|
|
context.paperclipTaskMarkdownAssignment = appendConcurrentWorkspaceNote(
|
|
context.paperclipTaskMarkdownAssignment,
|
|
);
|
|
if (typeof context.paperclipTaskMarkdownCompact === "string") {
|
|
context.paperclipTaskMarkdownCompact =
|
|
appendConcurrentWorkspaceNote(
|
|
context.paperclipTaskMarkdownCompact,
|
|
);
|
|
}
|
|
if (typeof context.paperclipTaskMarkdownAssignmentCompact === "string") {
|
|
context.paperclipTaskMarkdownAssignmentCompact =
|
|
appendConcurrentWorkspaceNote(
|
|
context.paperclipTaskMarkdownAssignmentCompact,
|
|
);
|
|
}
|
|
logger.info(
|
|
{
|
|
event: "shared_workspace_concurrent_dispatch",
|
|
runId: run.id,
|
|
issueId: issueRef.id,
|
|
projectWorkspaceId: issueRef.projectWorkspaceId,
|
|
holderRunId: workspaceHolder.runId,
|
|
holderIssueId: workspaceHolder.issueId,
|
|
sharedWorkspaceConcurrency,
|
|
environmentDriver,
|
|
executionForcedToKubernetes,
|
|
},
|
|
"Dispatching alongside a live shared-workspace holder",
|
|
);
|
|
}
|
|
}
|
|
if (!controllerOnly && taskWorkspaceIntent && !boundSourceWorkspace && !nativeRecoveryExecutionWorkspaceId) {
|
|
await executionWorkspacesSvc.validateStoredSelectionForRun({ companyId: agent.companyId,
|
|
actor: { type: "agent", agentId: agent.id, companyId: agent.companyId, runId: run.id, source: "agent_jwt", onBehalfOfUserId: responsibleUserId === "local-board" ? null : responsibleUserId },
|
|
selection: taskWorkspaceIntent.selection, issueId,
|
|
preserveImplicitDefaultSettings: ["project", "task_default"].includes(taskWorkspaceIntent.source) });
|
|
}
|
|
const explicitTaskDirectory = !controllerOnly && taskWorkspaceIntent?.selection.kind === "task_directory";
|
|
const explicitDirectoryStrategies = [
|
|
...(reusableExistingExecutionWorkspace?.providerType === "adapter_managed" ? [config.workspaceStrategy] : []),
|
|
issueAssigneeOverrides?.adapterConfig?.workspaceStrategy,
|
|
projectExecutionWorkspacePolicy?.workspaceStrategy, issueExecutionWorkspaceSettings?.workspaceStrategy];
|
|
const boundTaskDirectory = !controllerOnly && !fresh && !nativeRecoveryExecutionWorkspaceId && issueId !== null && issueRef !== null &&
|
|
requestedShouldReuseExisting && requestedExecutionWorkspaceMode === "isolated_workspace" &&
|
|
!workspaceProjectContext?.hasWorkspace && !issueRef.projectWorkspaceId &&
|
|
taskWorkspaceIntent?.selection.kind !== "configured_source" &&
|
|
(taskWorkspaceIntent?.selection.kind !== "existing" || taskWorkspaceIntent.selection.workspaceId === issueRef.executionWorkspaceId) &&
|
|
(await isBoundIsolatedTaskDirectory({ companyId: agent.companyId, issueId,
|
|
workspaceId: issueRef.executionWorkspaceId, workspace: reusableExistingExecutionWorkspace }) ||
|
|
(selectedEnvironmentForConfig?.driver === "plugin" && agent.adapterType === "paperclip_runner" &&
|
|
isBoundFreshPluginTaskDirectory({ companyId: agent.companyId, issueId,
|
|
workspaceId: issueRef.executionWorkspaceId, workspace: reusableExistingExecutionWorkspace,
|
|
owner: { agentId: agent.id, environmentId: selectedEnvironmentForConfig.id,
|
|
pluginKey: parseObject(selectedEnvironmentForConfig.config).pluginKey ?? null,
|
|
driverKey: parseObject(selectedEnvironmentForConfig.config).driverKey ?? null } })));
|
|
const reuseTaskDirectory = boundTaskDirectory && explicitDirectoryStrategies.every(value => {
|
|
const strategy = parseObject(value);
|
|
return Object.keys(strategy).length === 0 || (Object.keys(strategy).length === 1 && strategy.type === "project_primary");
|
|
});
|
|
if (retainedDirectory && !boundTaskDirectory) throw new WorkspaceValidationFailure("The saved Retry task directory is unavailable.", {
|
|
workspaceValidation: { reason: "isolated_task_directory_binding_mismatch", issueId },
|
|
});
|
|
const useIsolatedTaskDirectory = !controllerOnly && issueRef !== null && (reuseTaskDirectory || (fresh && !workspaceProjectContext?.hasWorkspace && !issueRef.projectWorkspaceId) || shouldUseIsolatedTaskDirectory({
|
|
trustPreset: trustPreset.kind,
|
|
environmentDriver: selectedEnvironmentForConfig?.driver ?? null,
|
|
mode: requestedExecutionWorkspaceMode,
|
|
hasProjectWorkspace: workspaceProjectContext?.hasWorkspace ?? false,
|
|
projectWorkspaceId: issueRef.projectWorkspaceId,
|
|
workspaceStrategies: [
|
|
config.workspaceStrategy,
|
|
issueAssigneeOverrides?.adapterConfig?.workspaceStrategy,
|
|
projectExecutionWorkspacePolicy?.workspaceStrategy,
|
|
issueExecutionWorkspaceSettings?.workspaceStrategy,
|
|
],
|
|
}));
|
|
return {
|
|
persistedRunnerProfile,
|
|
persistedNativeExecutionInput,
|
|
isDotRun,
|
|
providerApiPlacement,
|
|
existingExecutionWorkspace,
|
|
nativeRecoveryExecutionWorkspaceId,
|
|
workspaceReuseRequest,
|
|
requestedShouldReuseExisting,
|
|
reusableExistingExecutionWorkspace,
|
|
requestedReusableExecutionWorkspaceConfig,
|
|
localEnvironment,
|
|
resolvedInstanceSettings,
|
|
environmentResolution,
|
|
effectiveExecutionWorkspaceMode,
|
|
executionPolicy,
|
|
selectedEnvironmentId,
|
|
selectedEnvironmentForConfig,
|
|
nativeChatWorkspaceScope,
|
|
nativeChatExpectedCwd,
|
|
materializeNativeChatWorktree,
|
|
explicitTaskDirectory,
|
|
useIsolatedTaskDirectory,
|
|
};
|
|
}
|