mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-24 10:13:11 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Tasks can use isolated repository workspaces. > - Core currently prepares those repositories on its own filesystem. > - A storage provider may only expose repositories inside remote agent computers. > - Project storage and task workspace preparation must have one provider owner. > - This pull request adds that contract and keeps execution as a separate plugin interface. > - Core keeps authorization, durable task bindings, and lifecycle fences. ## Linked Issues or Issue Description **Subsystem affected** Plugin SDK, project workspace policy, heartbeat preparation, execution leases, and workspace lifecycle. **Problem or motivation** Core cannot prepare a task workspace when its repository storage is unavailable on the Core host. An execution lease must not be required just to prepare persistent storage. **Proposed solution** Let one plugin provide project resource lifecycle and workspace prepare, inspect, restore, and remove behavior. Core reserves a task workspace before it calls the plugin. The plugin returns an opaque durable reference. The matching execution plugin receives that reference on its task RPC. **Alternatives considered** Preparing repositories on Core would require a provider mount on Core. Preparing them only during task launch would tie persistent workspace lifecycle to execution leases and agent availability. **Roadmap alignment** This extends the remote provider model in ROADMAP.md. Related execution work is in #16978. This PR does not replace that task bridge. Native execution of managed storage requires the bridge; this change fails closed when it is absent. ## What Changed - Add one storage provider declaration with supported modes/strategies, a capability, a validated RPC contract, and a test harness. - Persist provider ownership, repository revision, and execution-workspace identity before external effects. Reconcile retries with a stable operation ID. - Prepare provider storage before environment acquisition. Keep Core filesystem preparation out of this path. - Pin the durable reference to the matching execution plugin and validate the lease binding before task dispatch. - Route inspection, reopen, and removal through the existing lifecycle process. Keep local data when remote recovery is uncertain. - Keep the application database pool free during lifecycle locks. Preserve pending workspace selections, validate source edits before retirement, and retry unfinished closed storage cleanup. - Protect provider metadata and live repository sources from caller edits. Storage edits retire only closed records with confirmed removal; retired records cannot reopen. ## Verification - Current head `8214b3ca5`: all 25 provider lifecycle tests pass with `maxConnections=1`. They cover preparation, inspection, reopening, removal, a paused provider with responsive ordinary SQL and a competing source edit, pending selection preservation, rejected source edits, cleanup retry scheduling, and providers that cache operation results, and committed cleanup notifications for storage intent and recovered state. - Final frozen-tree lifecycle run on `8214b3ca5`: all 318 tests passed in 10 files, covering workspace services, reopen, compaction, task binding, project sources, heartbeat preparation, display caching, and source locks. - Direct server `tsc --noEmit` passes. The SDK build and all 128 SDK tests passed during initial verification; later SDK changes are documentation only. - Full CI on the previous head `ec5c7209a` passed all 55 applicable checks; 2 conditional checks skipped. Standard Greptile reviewed current head `342e6d9c3` at 5/5 with no unresolved threads. All 55 applicable CI checks passed on `342e6d9c3`; 2 conditional checks skipped. The Slack browser and CLI startup jobs passed on their single unchanged-head rerun. Its Apex review returned 4/5 with one pending-restoration issue. Commit `1b58943b1` reconciles that durable intent before cleanup, advances the generation without reopening the Core row, and blocks direct removal or retirement while intent remains unresolved. Standard Greptile reviewed `1b58943b1` at 5/5 with no new findings. All 54 applicable CI checks passed on `1b58943b1`; 3 conditional checks skipped. The chat job passed on its single unchanged-head rerun. Apex then returned 4/5 for a missing cleanup wake after recovery. Commit `8214b3ca5` atomically registers cleanup intent before storage intent and recovery-result writes. Its provider test observes real committed notification hints and the visible advanced generation. Standard Greptile reviewed `8214b3ca5` at 5/5 with no new findings. All 55 applicable current-head CI checks passed on the first run; 2 conditional checks skipped. Final Apex verification completed at 5/5 on this exact head: 38 files reviewed, no new comments, and no unresolved threads. The PR is ready for review and mergeable. - The first Apex review found five issues after a standard 5/5: application-pool starvation, stale pending selections, retirement during rejected source edits, lost cleanup retries, and reused inspection identities. Commits `6d72e7fd4` and `342e6d9c3` fix each with regression coverage. Archive lock waits use dedicated connections, and source edits fail promptly if the lifecycle is busy. Retirement compares normalized values. The normalized-source regression initially failed with the raw-value comparison; after moving the check after normalization, a clean provider-suite rerun passed all 24 tests at that stage. Those threads are resolved; the earlier Apex 0/5 is not a passing acceptance result. - Earlier local integration attempts could not bind localhost inside the sandbox. The same suites passed with permission to start local PostgreSQL. - The earlier broad local `pnpm test:run` overlapped edits and reported stale-revision and checkout-path/Git-fixture failures. It was not a final-tree result. Full CI subsequently passed; the original local failure remains part of verification history. - Local repository-wide typecheck/build cannot complete because `cargo` is unavailable. Direct server TypeScript checks and affected tests pass; full Rust and build validation comes from CI. - An earlier browser run on `ff918c5a8` failed in the unchanged case-filter journey: expected “Done,” received “All statuses,” with an announcement handler recorded during the assertion. The cause is not confirmed. [The failed job](https://github.com/paperclipai/paperclip/actions/runs/39500611857/job/118896954508) remains verification history. The unchanged shard passed in the next full CI run on `ec5c7209a`; no UI or fixture change was made. - Current-head browser shard 2 failed waiting five seconds for the mocked Slack setup Success heading. This PR changes neither that UI nor its fixture. [The failed job](https://github.com/paperclipai/paperclip/actions/runs/39506070513/job/118916021232) remains verification evidence. The cause is not confirmed; the single unchanged-shard rerun passed; no UI or fixture change was made, and the original cause remains unconfirmed. - The workspace test shard passed its 3,978 general tests, then its CLI company-import/export fixture timed out after 30 seconds waiting for startup health before creating any company or workspace. [The failed job](https://github.com/paperclipai/paperclip/actions/runs/39506070513/job/118916021855) remains verification history. The child reached external-PostgreSQL startup without exiting or reporting an exception. Its cause is not confirmed; the single unchanged-job rerun passed, including the CLI fixture. No startup or fixture change was made, and the original cause remains unconfirmed. - The new cached-removal regression initially failed because its fake worker advertised availability for unrelated fixture plugins, so another workspace changed its fake checkout state. Worker availability now matches the fixture plugin; the complete 25-test provider suite passes. - On `1b58943b1`, the chat shard timed out polling for one conversation after an authorized GitHub description edit. [The failed job](https://github.com/paperclipai/paperclip/actions/runs/39510521657/job/118930193418) remains evidence. The exact case passes locally against PostgreSQL (1 passed, 1,380 unrelated cases skipped), and this PR changes neither that test nor its handler. The original cause is unconfirmed; the single unchanged-job CI rerun passed without a code or fixture change. The original cause remains unconfirmed. - The diff was checked for credentials, private references, and generated artifacts. ## Risks This is an additive provider contract with no database migration. Existing projects keep their local behavior unless their policy selects a storage plugin. A selected provider must be ready and enabled. Unknown or unsafe cleanup fails closed. Managed native execution depends on #16978. Core refuses a host Runner fallback. Providers must fence lifecycle generations, preserve ambiguous retries, and report remote recovery honestly. Provider-specific live storage and guest mount qualification remain separate from these Core contract tests. ## Model Used OpenAI Codex (GPT-6), with repository inspection, code execution, and tests. The exact model variant was not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id - [x] I have run tests locally and they pass (affected suites; broad-run limits are documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
332 lines
13 KiB
TypeScript
332 lines
13 KiB
TypeScript
import { workspaceProviderBindingSchema } from "@paperclipai/shared";
|
|
import type {
|
|
Environment,
|
|
EnvironmentLease,
|
|
ExecutionWorkspaceConfig,
|
|
WorkspaceRealizationRecord,
|
|
WorkspaceRealizationRequest,
|
|
} from "@paperclipai/shared";
|
|
import type { RealizedExecutionWorkspace } from "./workspace-runtime.js";
|
|
import { ENVIRONMENT_DRIVER_TRAITS, getEnvironmentDriverTraits } from "./environment-driver-traits.js";
|
|
|
|
function parseObject(value: unknown): Record<string, unknown> {
|
|
return value && typeof value === "object" && !Array.isArray(value)
|
|
? (value as Record<string, unknown>)
|
|
: {};
|
|
}
|
|
|
|
function readString(value: unknown): string | null {
|
|
return typeof value === "string" && value.trim().length > 0 ? value.trim() : null;
|
|
}
|
|
|
|
function readNumber(value: unknown): number | null {
|
|
return typeof value === "number" && Number.isFinite(value) ? value : null;
|
|
}
|
|
|
|
function readStringArray(value: unknown): string[] {
|
|
return Array.isArray(value)
|
|
? value.map(readString).filter((entry): entry is string => entry !== null)
|
|
: [];
|
|
}
|
|
|
|
function readPathAliases(value: unknown): Array<{ path: string; target: string }> {
|
|
if (!Array.isArray(value)) return [];
|
|
return value.flatMap((entry) => {
|
|
const parsed = parseObject(entry);
|
|
const aliasPath = readString(parsed.path);
|
|
const target = readString(parsed.target);
|
|
return aliasPath && target ? [{ path: aliasPath, target }] : [];
|
|
});
|
|
}
|
|
|
|
// Read the additional referenced (mentioned) project sources. Legacy payloads omit the field, so
|
|
// this defaults to an empty array. Each source needs a localPath; entries without one are dropped.
|
|
function readAdditionalSources(
|
|
value: unknown,
|
|
): NonNullable<WorkspaceRealizationRequest["additionalSources"]> {
|
|
if (!Array.isArray(value)) return [];
|
|
return value.flatMap((entry) => {
|
|
const parsed = parseObject(entry);
|
|
const localPath = readString(parsed.localPath);
|
|
if (!localPath) return [];
|
|
return [
|
|
{
|
|
localPath,
|
|
projectId: readString(parsed.projectId),
|
|
projectWorkspaceId: readString(parsed.projectWorkspaceId),
|
|
repoUrl: readString(parsed.repoUrl),
|
|
repoRef: readString(parsed.repoRef),
|
|
},
|
|
];
|
|
});
|
|
}
|
|
|
|
export function readWorkspaceRealizationRequest(value: unknown): WorkspaceRealizationRequest | null {
|
|
const parsed = parseObject(value);
|
|
if (parsed.version !== 1) return null;
|
|
const source = parseObject(parsed.source);
|
|
const runtimeOverlay = parseObject(parsed.runtimeOverlay);
|
|
const localPath = readString(source.localPath);
|
|
const companyId = readString(parsed.companyId);
|
|
const environmentId = readString(parsed.environmentId);
|
|
const heartbeatRunId = readString(parsed.heartbeatRunId);
|
|
const adapterType = readString(parsed.adapterType);
|
|
if (!localPath || !companyId || !environmentId || !heartbeatRunId || !adapterType) return null;
|
|
|
|
return {
|
|
version: 1,
|
|
adapterType,
|
|
companyId,
|
|
environmentId,
|
|
executionWorkspaceId: readString(parsed.executionWorkspaceId),
|
|
issueId: readString(parsed.issueId),
|
|
heartbeatRunId,
|
|
requestedMode: readString(parsed.requestedMode),
|
|
source: {
|
|
kind:
|
|
source.kind === "task_session" || source.kind === "agent_home"
|
|
? source.kind
|
|
: "project_primary",
|
|
localPath,
|
|
...(source.providerBinding ? { providerBinding: workspaceProviderBindingSchema.parse(source.providerBinding) } : {}),
|
|
projectId: readString(source.projectId),
|
|
projectWorkspaceId: readString(source.projectWorkspaceId),
|
|
repoUrl: readString(source.repoUrl),
|
|
repoRef: readString(source.repoRef),
|
|
strategy: source.strategy === "git_worktree" ? "git_worktree" : "project_primary",
|
|
branchName: readString(source.branchName),
|
|
worktreePath: readString(source.worktreePath),
|
|
},
|
|
additionalSources: readAdditionalSources(parsed.additionalSources),
|
|
runtimeOverlay: {
|
|
provisionCommand: readString(runtimeOverlay.provisionCommand),
|
|
runtimeProvisionCommand: readString(runtimeOverlay.runtimeProvisionCommand),
|
|
teardownCommand: readString(runtimeOverlay.teardownCommand),
|
|
cleanupCommand: readString(runtimeOverlay.cleanupCommand),
|
|
workspaceRuntime: Object.keys(parseObject(runtimeOverlay.workspaceRuntime)).length > 0
|
|
? parseObject(runtimeOverlay.workspaceRuntime)
|
|
: null,
|
|
},
|
|
};
|
|
}
|
|
|
|
export function buildWorkspaceRealizationRequest(input: {
|
|
adapterType: string;
|
|
companyId: string;
|
|
environmentId: string;
|
|
executionWorkspaceId: string | null;
|
|
issueId: string | null;
|
|
heartbeatRunId: string;
|
|
requestedMode: string | null;
|
|
workspace: RealizedExecutionWorkspace;
|
|
workspaceConfig: ExecutionWorkspaceConfig | null;
|
|
}): WorkspaceRealizationRequest {
|
|
return {
|
|
version: 1,
|
|
adapterType: input.adapterType,
|
|
companyId: input.companyId,
|
|
environmentId: input.environmentId,
|
|
executionWorkspaceId: input.executionWorkspaceId,
|
|
issueId: input.issueId,
|
|
heartbeatRunId: input.heartbeatRunId,
|
|
requestedMode: input.requestedMode,
|
|
source: {
|
|
...(input.workspace.providerBinding ? { providerBinding: input.workspace.providerBinding } : {}),
|
|
kind: input.workspace.source,
|
|
localPath: input.workspace.cwd,
|
|
projectId: input.workspace.projectId,
|
|
projectWorkspaceId: input.workspace.workspaceId,
|
|
repoUrl: input.workspace.repoUrl,
|
|
repoRef: input.workspace.repoRef,
|
|
strategy: input.workspace.strategy,
|
|
branchName: input.workspace.branchName,
|
|
worktreePath: input.workspace.worktreePath,
|
|
},
|
|
// The additional (referenced) sources carry the read-only referenced-project workspaces. Run
|
|
// preparation resolves them for a local execution target only and exposes each local path to
|
|
// the agent through the workspace-hints channel (`PAPERCLIP_WORKSPACES_JSON`). A remote target
|
|
// never receives a referenced source: run preparation skips referenced-project resolution on a
|
|
// remote target, so this array is empty there. The `sync` block below therefore realizes only
|
|
// the anchor source; a remote-transport sync of the referenced trees is not implemented yet.
|
|
additionalSources: (input.workspace.additionalWorkspaces ?? []).map((additional) => ({
|
|
localPath: additional.cwd,
|
|
projectId: additional.projectId,
|
|
projectWorkspaceId: additional.workspaceId,
|
|
repoUrl: additional.repoUrl,
|
|
repoRef: additional.repoRef,
|
|
})),
|
|
runtimeOverlay: {
|
|
provisionCommand: input.workspaceConfig?.provisionCommand ?? null,
|
|
runtimeProvisionCommand: input.workspaceConfig?.runtimeProvisionCommand ?? null,
|
|
teardownCommand: input.workspaceConfig?.teardownCommand ?? null,
|
|
cleanupCommand: input.workspaceConfig?.cleanupCommand ?? null,
|
|
workspaceRuntime: input.workspaceConfig?.workspaceRuntime ?? null,
|
|
},
|
|
};
|
|
}
|
|
|
|
export function buildWorkspaceRealizationRecord(input: {
|
|
environment: Environment;
|
|
lease: EnvironmentLease;
|
|
request: WorkspaceRealizationRequest;
|
|
realizedCwd?: string | null;
|
|
providerMetadata?: Record<string, unknown> | null;
|
|
}): WorkspaceRealizationRecord {
|
|
const leaseMetadata = input.lease.metadata ?? {};
|
|
const providerMetadata = input.providerMetadata ?? {};
|
|
// An unknown or absent driver reads the "local" row, same as today's fallback.
|
|
const traits = getEnvironmentDriverTraits(input.environment.driver) ?? ENVIRONMENT_DRIVER_TRAITS.local;
|
|
const transport = traits.driver;
|
|
const remotePath =
|
|
readString(providerMetadata.remoteCwd) ??
|
|
readString(leaseMetadata.remoteCwd) ??
|
|
readString(providerMetadata.remotePath) ??
|
|
null;
|
|
const host = readString(leaseMetadata.host);
|
|
const port = readNumber(leaseMetadata.port);
|
|
const username = readString(leaseMetadata.username);
|
|
const sandboxId = readString(leaseMetadata.sandboxId) ?? readString(providerMetadata.sandboxId);
|
|
const realizationMetadata = {
|
|
...parseObject(leaseMetadata.workspaceRealization),
|
|
...parseObject(providerMetadata.workspaceRealization),
|
|
...providerMetadata,
|
|
};
|
|
const mode = realizationMetadata.mode === "in_place" || realizationMetadata.realizationMode === "in_place"
|
|
? "in_place" as const
|
|
: "copy" as const;
|
|
const authoritativeRoot =
|
|
readString(realizationMetadata.authoritativeRoot) ??
|
|
(mode === "in_place" ? remotePath : null) ??
|
|
input.request.source.localPath;
|
|
const pathAliases = readPathAliases(realizationMetadata.pathAliases ?? realizationMetadata.workspaceAliases);
|
|
const outboundRestorePaths = readStringArray(realizationMetadata.outboundRestorePaths);
|
|
|
|
const provider =
|
|
input.lease.provider ??
|
|
(transport === "ssh" ? "ssh" : transport === "local" ? "local" : null);
|
|
const localPath = input.request.source.localPath;
|
|
const summary =
|
|
transport === "local"
|
|
? `Local workspace realized at ${localPath}.`
|
|
: transport === "ssh"
|
|
? `SSH workspace realized at ${username ?? "user"}@${host ?? "host"}:${port ?? 22}:${remotePath ?? input.request.source.localPath}.`
|
|
: transport === "sandbox"
|
|
? `Sandbox workspace realized at ${remotePath ?? "/"}${sandboxId ? ` in ${sandboxId}` : ""}.`
|
|
: `Plugin workspace realized at ${input.realizedCwd ?? remotePath ?? localPath}.`;
|
|
|
|
return {
|
|
version: 1,
|
|
mode,
|
|
authoritativeRoot,
|
|
pathAliases,
|
|
outboundRestorePaths,
|
|
provider,
|
|
environmentId: input.environment.id,
|
|
leaseId: input.lease.id,
|
|
providerLeaseId: input.lease.providerLeaseId,
|
|
local: {
|
|
path: localPath,
|
|
source: input.request.source.kind,
|
|
strategy: input.request.source.strategy,
|
|
projectId: input.request.source.projectId,
|
|
projectWorkspaceId: input.request.source.projectWorkspaceId,
|
|
repoUrl: input.request.source.repoUrl,
|
|
repoRef: input.request.source.repoRef,
|
|
branchName: input.request.source.branchName,
|
|
worktreePath: input.request.source.worktreePath,
|
|
},
|
|
additional: (input.request.additionalSources ?? []).map((additional) => ({
|
|
path: additional.localPath,
|
|
projectId: additional.projectId,
|
|
projectWorkspaceId: additional.projectWorkspaceId,
|
|
repoUrl: additional.repoUrl,
|
|
repoRef: additional.repoRef,
|
|
})),
|
|
remote: {
|
|
path: remotePath,
|
|
...(host ? { host } : {}),
|
|
...(port ? { port } : {}),
|
|
...(username ? { username } : {}),
|
|
...(sandboxId ? { sandboxId } : {}),
|
|
},
|
|
bootstrap: {
|
|
command: input.request.runtimeOverlay.provisionCommand,
|
|
},
|
|
rebuild: {
|
|
executionWorkspaceId: input.request.executionWorkspaceId,
|
|
mode: input.request.requestedMode,
|
|
repoUrl: input.request.source.repoUrl,
|
|
repoRef: input.request.source.repoRef,
|
|
localPath,
|
|
remotePath,
|
|
providerLeaseId: input.lease.providerLeaseId,
|
|
metadata: {
|
|
source: input.request.source,
|
|
runtimeOverlay: input.request.runtimeOverlay,
|
|
environmentDriver: input.environment.driver,
|
|
provider,
|
|
providerMetadata,
|
|
},
|
|
},
|
|
summary,
|
|
};
|
|
}
|
|
|
|
/**
|
|
* Build the workspace-realization record from the run request. The server owns the record;
|
|
* a driver realize handler (built-in or plugin) returns only a realized cwd and provider
|
|
* metadata. Every `realizeWorkspace` exit must route through this helper, so the record carries
|
|
* the referenced (mentioned) project sources in `additional`. The adapter reads `additional` to
|
|
* stage each referenced tree into the target; a realize exit that returns a raw provider result
|
|
* without this helper drops the mentioned projects.
|
|
*/
|
|
export function buildWorkspaceRealizationRecordFromDriverInput(input: {
|
|
environment: Environment;
|
|
lease: EnvironmentLease;
|
|
workspace: {
|
|
localPath?: string;
|
|
remotePath?: string;
|
|
mode?: string;
|
|
metadata?: Record<string, unknown>;
|
|
};
|
|
cwd?: string | null;
|
|
providerMetadata?: Record<string, unknown> | null;
|
|
}): WorkspaceRealizationRecord {
|
|
const request =
|
|
readWorkspaceRealizationRequest(input.workspace.metadata?.workspaceRealizationRequest) ??
|
|
readWorkspaceRealizationRequest(input.workspace.metadata?.request) ??
|
|
buildWorkspaceRealizationRequest({
|
|
adapterType: "unknown",
|
|
companyId: input.lease.companyId,
|
|
environmentId: input.environment.id,
|
|
executionWorkspaceId: input.lease.executionWorkspaceId,
|
|
issueId: input.lease.issueId,
|
|
heartbeatRunId: input.lease.heartbeatRunId ?? "unknown",
|
|
requestedMode: input.workspace.mode ?? null,
|
|
workspace: {
|
|
baseCwd: input.workspace.localPath ?? input.cwd ?? input.workspace.remotePath ?? "/",
|
|
source: "task_session",
|
|
projectId: null,
|
|
workspaceId: null,
|
|
repoUrl: null,
|
|
repoRef: null,
|
|
strategy: "project_primary",
|
|
cwd: input.workspace.localPath ?? input.cwd ?? input.workspace.remotePath ?? "/",
|
|
branchName: null,
|
|
worktreePath: null,
|
|
warnings: [],
|
|
created: false,
|
|
branchCreatedByRuntime: false,
|
|
},
|
|
workspaceConfig: null,
|
|
});
|
|
|
|
return buildWorkspaceRealizationRecord({
|
|
environment: input.environment,
|
|
lease: input.lease,
|
|
request,
|
|
realizedCwd: input.cwd ?? null,
|
|
providerMetadata: input.providerMetadata,
|
|
});
|
|
}
|