mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 01:57:31 +01:00
feat(workspaces): sign the workspace login handoff and gate readiness (#11671)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Managed worktree services run isolated Paperclip instances with cloned databases. > - A reachable service was reported as ready even when its database, runtime identity, or login path was not usable. > - The first candidate added verified database seeding and managed repair in #11665. > - This pull request consolidates that candidate with signed login handoff and a complete readiness contract. > - Post-QA fixes close five defects in repair identity, repair responses, UI retry, seed journal handling, and seed-source trust. > - The benefit is a workspace that either opens safely or reports one accurate recovery action. ## Linked Issues or Issue Description No public GitHub issue exists for this work, so the problem is described here. **What happened** Managed workspace URLs could return HTTP 200 and report ready while login failed. QA also found cases where repair used the wrong instance identity, returned a generic error, left the UI stuck, rejected a safe journal lag, or trusted a mutable workspace manifest. **Expected behavior** Opening a ready workspace signs the board user in to the correct isolated instance. Provisioning and repair use a registered source and report a structured recovery state. **Actual behavior** Entry depended on a password copied into the clone. Several failure paths could publish stale readiness, hide the repair precondition, or trust state that the workspace could modify. **Additional context** This pull request includes the commits first published in #11665. That pull request keeps the original base head for review history. This consolidated pull request is the merge candidate. Related open readiness work includes #11575 and #11621. ## What Changed - Adds a short-lived, signed, single-use login ticket. It binds the user, workspace, instance, and runtime origin. - Exchanges the ticket through Better Auth. It creates the session and cookie through the supported adapter path. - Adds protected workspace readiness fields for the database, clone data, login handoff, seed phase, and runtime identity. - Fails readiness closed when the guest has no company or execution-workspace binding. - Binds ticket issuance to the exact cloned user and active company membership selected for the handoff. - Verifies every current active board identity through the exact-user handoff before publication or reuse. - Gates managed runtime publication on the readiness contract and the recorded worktree instance identity. - Refreshes runtime work products from the live runtime row after a port change. - Adds one workspace access card with ready, degraded, repairing, and failed states. - Uses the runtime response identity for repair. It returns structured repair precondition errors. - Lets a valid source journal lag converge during provisioning. - Binds seed and repair manifests to a source registered outside the agent-writable worktree. - Clears recovered UI errors so a successful retry can open the workspace. - Makes runtime tests register canonical sources and avoid ports owned by live host listeners. - Keeps Vitest on source suites when compiled `dist` trees exist. - Isolates CLI and adapter tests from ambient AWS and runtime API environment variables. - Preserves a 404 response for cross-company workspace ID lookups before runtime authorization. - Makes concurrent single-flight coverage independent of path-canonicalization scheduling order. ## Verification The following checks passed on the integrated head: ```sh pnpm -r typecheck pnpm build pnpm check:token-gates pnpm --filter @paperclipai/db check:migrations ``` - The server source lane passed 420 files and 4,953 tests. Five tests were skipped. - The CLI lane passed 57 files and 385 tests. - The database lane passed 26 files and 97 tests. - The shared package passed 58 files and 506 tests. - The adapter utility lane passed 640 tests. Four tests were skipped. - The Claude adapter passed 220 tests. One test was skipped. - The Codex adapter passed 323 tests. - The OpenClaw adapter passed 13 tests. - The OpenCode adapter passed 42 tests. - The plugin SDK passed 45 tests. - The workspace runtime suite passed 124 tests. - The caller-scoped readiness and handoff suite passed 52 tests. - The workspace provisioning shell suite passed 7 tests. - The runtime exposure suite passed 17 tests while live host mappings occupied fixed test ports. - `git diff --check` passed and the worktree is clean. The serialized route lane will run in GitHub CI with its normal shards. No deployment or active-workspace migration was performed. ## Risks - This is a medium-risk authentication and runtime-readiness change. - The login ticket uses exact origin, workspace, instance, and user binding. It has a short expiry and a one-time nonce. - Runtime publication is stricter. A real readiness, identity, per-user handoff, or control-plane database disagreement now blocks publication. - This pull request supersedes #11665 as the merge candidate. Close #11665 after this pull request merges. - No new database migration is included. The lockfile and workflow files are unchanged. - Deployment and active-workspace migration are intentionally outside this pull request. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5 (`claude-opus-5[1m]`), 1M context, extended thinking, tool use, and code execution produced the main candidate. OpenAI GPT-5 (`gpt-5`) through Codex, with agentic reasoning, tool use, and code execution, integrated the post-QA fixes and hardened the test gates. The Codex context-window size was not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
6aaef2998f
commit
a2bf936f9a
59 files changed
+7754
-243
No files matched your search
@@ -0,0 +1,386 @@
|
||||
/**
|
||||
* Control-plane half of the workspace readiness/handoff contract (PAP-17572).
|
||||
*
|
||||
* Resolves the identity and derived key material for one managed workspace,
|
||||
* injects them into the guest process, and probes the guest's *protected*
|
||||
* readiness so `running / healthy` can never be published for a workspace whose
|
||||
* clone, identity, or login handoff disagrees with what was expected.
|
||||
*
|
||||
* Identity always comes from the worktree's own recorded instance pointer, never
|
||||
* from a branch name or basename heuristic, and an unresolvable identity fails
|
||||
* closed (no key injected, no readiness claimed).
|
||||
*/
|
||||
|
||||
import path from "node:path";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { authUsers, companyMemberships, type Db } from "@paperclipai/db";
|
||||
import type { WorkspaceReadiness, WorkspaceReadinessProbeResult } from "@paperclipai/shared";
|
||||
import {
|
||||
deriveWorkspaceHandoffKey,
|
||||
deriveWorkspaceReadinessToken,
|
||||
resolveWorkspaceHandoffRootSecret,
|
||||
WORKSPACE_EXECUTION_WORKSPACE_COMPANY_ID_ENV_KEY,
|
||||
WORKSPACE_EXECUTION_WORKSPACE_ID_ENV_KEY,
|
||||
WORKSPACE_HANDOFF_KEY_ENV_KEY,
|
||||
WORKSPACE_READINESS_TOKEN_ENV_KEY,
|
||||
WORKSPACE_READINESS_TOKEN_HEADER,
|
||||
WORKSPACE_READINESS_USER_EMAIL_HEADER,
|
||||
WORKSPACE_READINESS_USER_ID_HEADER,
|
||||
} from "../auth/workspace-login-handoff.js";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import {
|
||||
deriveWorktreeInstanceId,
|
||||
readWorktreeInstanceId,
|
||||
} from "./workspace-instance-cleanup.js";
|
||||
|
||||
/**
|
||||
* Per-probe budget. Matched to the semantic transport probe it replaces, so
|
||||
* upgrading a runtime health check to the readiness contract cannot make a
|
||||
* managed start or a reuse decision slower than it was.
|
||||
*/
|
||||
export const WORKSPACE_READINESS_PROBE_TIMEOUT_MS = 2_000;
|
||||
|
||||
export type ManagedWorkspaceIdentity = {
|
||||
instanceId: string;
|
||||
executionWorkspaceId: string;
|
||||
companyId: string;
|
||||
handoffKey: string;
|
||||
readinessToken: string;
|
||||
/** Whether the root secret was configured explicitly or derived. */
|
||||
secretSource: "dedicated" | "derived";
|
||||
};
|
||||
|
||||
/**
|
||||
* Resolve the isolated instance id a worktree actually runs as.
|
||||
*
|
||||
* The worktree's `.paperclip/.env` pointer is authoritative because it is what
|
||||
* the guest process itself loads. The path-derived id is only a fallback for a
|
||||
* worktree provisioned before the pointer existed, and the two agree by
|
||||
* construction for anything Paperclip provisioned.
|
||||
*/
|
||||
export function resolveManagedWorkspaceInstanceId(workspaceCwd: string): string | null {
|
||||
const recorded = readWorktreeInstanceId(workspaceCwd);
|
||||
if (recorded) return recorded;
|
||||
const derived = deriveWorktreeInstanceId(workspaceCwd);
|
||||
return derived || null;
|
||||
}
|
||||
|
||||
export function resolveManagedWorkspaceIdentity(input: {
|
||||
workspaceCwd: string | null | undefined;
|
||||
executionWorkspaceId: string | null | undefined;
|
||||
companyId: string | null | undefined;
|
||||
env?: NodeJS.ProcessEnv;
|
||||
}): ManagedWorkspaceIdentity | null {
|
||||
const workspaceCwd = input.workspaceCwd?.trim();
|
||||
const executionWorkspaceId = input.executionWorkspaceId?.trim();
|
||||
const companyId = input.companyId?.trim();
|
||||
if (!workspaceCwd || !executionWorkspaceId || !companyId) return null;
|
||||
|
||||
const root = resolveWorkspaceHandoffRootSecret(input.env ?? process.env);
|
||||
if (!root) return null;
|
||||
|
||||
const instanceId = resolveManagedWorkspaceInstanceId(path.resolve(workspaceCwd));
|
||||
if (!instanceId) return null;
|
||||
|
||||
return {
|
||||
instanceId,
|
||||
executionWorkspaceId,
|
||||
companyId,
|
||||
handoffKey: deriveWorkspaceHandoffKey({ rootSecret: root.secret, instanceId, executionWorkspaceId }),
|
||||
readinessToken: deriveWorkspaceReadinessToken({ rootSecret: root.secret, instanceId, executionWorkspaceId }),
|
||||
secretSource: root.source,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Env injected into a managed workspace's guest process.
|
||||
*
|
||||
* Only the *derived* per-workspace values cross the boundary: the guest never
|
||||
* receives the root secret, so a compromised workspace cannot mint a ticket for
|
||||
* a sibling workspace.
|
||||
*/
|
||||
export function buildManagedWorkspaceGuestEnv(identity: ManagedWorkspaceIdentity): Record<string, string> {
|
||||
return {
|
||||
[WORKSPACE_HANDOFF_KEY_ENV_KEY]: identity.handoffKey,
|
||||
[WORKSPACE_READINESS_TOKEN_ENV_KEY]: identity.readinessToken,
|
||||
[WORKSPACE_EXECUTION_WORKSPACE_ID_ENV_KEY]: identity.executionWorkspaceId,
|
||||
[WORKSPACE_EXECUTION_WORKSPACE_COMPANY_ID_ENV_KEY]: identity.companyId,
|
||||
};
|
||||
}
|
||||
|
||||
function isWorkspaceReadinessShape(value: unknown): value is WorkspaceReadiness {
|
||||
if (!value || typeof value !== "object") return false;
|
||||
const candidate = value as Record<string, unknown>;
|
||||
return (
|
||||
typeof candidate.state === "string"
|
||||
&& typeof candidate.databaseReady === "boolean"
|
||||
&& typeof candidate.cloneDataReady === "boolean"
|
||||
&& typeof candidate.authHandoffReady === "boolean"
|
||||
&& (typeof candidate.authHandoffUserId === "string" || candidate.authHandoffUserId === null)
|
||||
&& typeof candidate.seedState === "string"
|
||||
&& (typeof candidate.companyId === "string" || candidate.companyId === null)
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a managed workspace's protected readiness over loopback.
|
||||
*
|
||||
* Failure modes are distinguished so callers can tell "not reachable" from
|
||||
* "reachable but serving somebody else's clone" — the second is what made a
|
||||
* relocated port able to masquerade as healthy.
|
||||
*/
|
||||
export async function probeManagedWorkspaceReadiness(input: {
|
||||
healthUrl: string;
|
||||
identity: ManagedWorkspaceIdentity;
|
||||
fetchImpl?: typeof fetch;
|
||||
timeoutMs?: number;
|
||||
handoffSubject?: { userId: string; email: string } | null;
|
||||
}): Promise<WorkspaceReadinessProbeResult> {
|
||||
const fetchImpl = input.fetchImpl ?? fetch;
|
||||
let response: Response;
|
||||
try {
|
||||
response = await fetchImpl(input.healthUrl, {
|
||||
headers: {
|
||||
[WORKSPACE_READINESS_TOKEN_HEADER]: input.identity.readinessToken,
|
||||
...(input.handoffSubject
|
||||
? {
|
||||
[WORKSPACE_READINESS_USER_ID_HEADER]: input.handoffSubject.userId,
|
||||
[WORKSPACE_READINESS_USER_EMAIL_HEADER]: input.handoffSubject.email,
|
||||
}
|
||||
: {}),
|
||||
},
|
||||
signal: AbortSignal.timeout(input.timeoutMs ?? WORKSPACE_READINESS_PROBE_TIMEOUT_MS),
|
||||
});
|
||||
} catch (error) {
|
||||
return {
|
||||
ok: false,
|
||||
reason: "unreachable",
|
||||
readiness: null,
|
||||
detail: error instanceof Error ? error.message : String(error),
|
||||
};
|
||||
}
|
||||
|
||||
const payload = await response.json().catch(() => null) as
|
||||
| { status?: unknown; workspace?: unknown }
|
||||
| null;
|
||||
|
||||
if (!response.ok) {
|
||||
const readiness = isWorkspaceReadinessShape(payload?.workspace) ? payload.workspace : null;
|
||||
return { ok: false, reason: "http_error", readiness, detail: `HTTP ${response.status}` };
|
||||
}
|
||||
if (payload?.status !== "ok") {
|
||||
return { ok: false, reason: "unhealthy_payload", readiness: null, detail: String(payload?.status ?? "missing") };
|
||||
}
|
||||
if (
|
||||
payload.workspace
|
||||
&& typeof payload.workspace === "object"
|
||||
&& !("companyId" in payload.workspace)
|
||||
) {
|
||||
// A guest that implements readiness but predates the company binding is not
|
||||
// a legacy transport-only guest. Treat the partial contract as an identity
|
||||
// disagreement so auto compatibility mode cannot publish an unopenable clone.
|
||||
return {
|
||||
ok: false,
|
||||
reason: "identity_mismatch",
|
||||
readiness: null,
|
||||
detail: `expected company ${input.identity.companyId}, got missing`,
|
||||
};
|
||||
}
|
||||
if (
|
||||
input.handoffSubject
|
||||
&& payload.workspace
|
||||
&& typeof payload.workspace === "object"
|
||||
&& !("authHandoffUserId" in payload.workspace)
|
||||
) {
|
||||
return {
|
||||
ok: false,
|
||||
reason: "identity_mismatch",
|
||||
readiness: null,
|
||||
detail: `expected handoff user ${input.handoffSubject.userId}, got missing`,
|
||||
};
|
||||
}
|
||||
if (!isWorkspaceReadinessShape(payload.workspace)) {
|
||||
// Either the token was rejected or the guest predates this contract. Both
|
||||
// mean the control plane cannot prove user readiness, so neither may publish.
|
||||
return { ok: false, reason: "readiness_missing", readiness: null, detail: null };
|
||||
}
|
||||
|
||||
const readiness = payload.workspace;
|
||||
if (
|
||||
readiness.instanceId !== input.identity.instanceId
|
||||
|| readiness.executionWorkspaceId !== input.identity.executionWorkspaceId
|
||||
|| readiness.companyId !== input.identity.companyId
|
||||
) {
|
||||
return {
|
||||
ok: false,
|
||||
reason: "identity_mismatch",
|
||||
readiness,
|
||||
detail: `expected ${input.identity.instanceId}/${input.identity.executionWorkspaceId}/${input.identity.companyId}, got ${readiness.instanceId}/${readiness.executionWorkspaceId}/${readiness.companyId}`,
|
||||
};
|
||||
}
|
||||
if (!readiness.databaseReady || !readiness.cloneDataReady || !readiness.authHandoffReady) {
|
||||
return { ok: false, reason: "not_ready", readiness, detail: readiness.failurePhase };
|
||||
}
|
||||
if (input.handoffSubject && readiness.authHandoffUserId !== input.handoffSubject.userId) {
|
||||
return {
|
||||
ok: false,
|
||||
reason: "identity_mismatch",
|
||||
readiness,
|
||||
detail: `expected handoff user ${input.handoffSubject.userId}, got ${readiness.authHandoffUserId}`,
|
||||
};
|
||||
}
|
||||
|
||||
return { ok: true, readiness };
|
||||
}
|
||||
|
||||
/** Current board identities that must survive the clone for it to publish ready. */
|
||||
export async function listManagedWorkspaceHandoffSubjects(
|
||||
db: Db,
|
||||
companyId: string,
|
||||
): Promise<Array<{ userId: string; email: string }>> {
|
||||
const rows = await db
|
||||
.select({ userId: authUsers.id, email: authUsers.email })
|
||||
.from(authUsers)
|
||||
.innerJoin(
|
||||
companyMemberships,
|
||||
and(
|
||||
eq(companyMemberships.principalType, "user"),
|
||||
eq(companyMemberships.principalId, authUsers.id),
|
||||
eq(companyMemberships.companyId, companyId),
|
||||
eq(companyMemberships.status, "active"),
|
||||
),
|
||||
);
|
||||
return rows.flatMap((row) => {
|
||||
const userId = row.userId?.trim();
|
||||
const email = row.email?.trim();
|
||||
return userId && email ? [{ userId, email }] : [];
|
||||
});
|
||||
}
|
||||
|
||||
/** Prove that every current board member can use the advertised handoff. */
|
||||
export async function probeManagedWorkspaceHandoffSubjects(input: {
|
||||
db: Db;
|
||||
healthUrl: string;
|
||||
identity: ManagedWorkspaceIdentity;
|
||||
fetchImpl?: typeof fetch;
|
||||
}): Promise<WorkspaceReadinessProbeResult> {
|
||||
const subjects = await listManagedWorkspaceHandoffSubjects(input.db, input.identity.companyId);
|
||||
if (subjects.length === 0) {
|
||||
return {
|
||||
ok: false,
|
||||
reason: "not_ready",
|
||||
readiness: null,
|
||||
detail: "no active board user is eligible for workspace login handoff",
|
||||
};
|
||||
}
|
||||
const results = await Promise.all(subjects.map((handoffSubject) => probeManagedWorkspaceReadiness({
|
||||
healthUrl: input.healthUrl,
|
||||
identity: input.identity,
|
||||
fetchImpl: input.fetchImpl,
|
||||
handoffSubject,
|
||||
})));
|
||||
return results.find((result) => !result.ok) ?? results[0]!;
|
||||
}
|
||||
|
||||
/**
|
||||
* How strictly a managed start is gated on the guest's protected readiness.
|
||||
*
|
||||
* - `auto` (default): a guest that *reports* readiness must satisfy it, and a
|
||||
* guest serving a different instance/workspace is always rejected. A guest
|
||||
* that reports no readiness block at all — an older checkout, which a managed
|
||||
* worktree can legitimately be — falls back to the semantic transport check
|
||||
* and is logged. Failing that case closed would turn an upgrade lag into an
|
||||
* outage of every existing workspace, which is a worse reliability outcome
|
||||
* than the defect this gate exists to fix. Such a workspace is still not
|
||||
* silently "usable": it has no handoff plugin, so `Open workspace` reports the
|
||||
* snapshot-local credential fallback rather than pretending to be ready.
|
||||
* - `strict`: a missing readiness block also blocks publication. Intended for
|
||||
* deployments that have finished rolling the new guest build everywhere.
|
||||
*/
|
||||
export type WorkspaceReadinessGateMode = "auto" | "strict";
|
||||
|
||||
export function resolveWorkspaceReadinessGateMode(
|
||||
env: NodeJS.ProcessEnv = process.env,
|
||||
): WorkspaceReadinessGateMode {
|
||||
return env.PAPERCLIP_WORKSPACE_READINESS_GATE?.trim().toLowerCase() === "strict" ? "strict" : "auto";
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a rejected probe must stop the runtime from being published.
|
||||
*
|
||||
* Everything except "this guest does not implement the contract" is a real
|
||||
* disagreement about the clone and always blocks.
|
||||
*/
|
||||
export function shouldBlockPublicationOnReadiness(
|
||||
result: Extract<WorkspaceReadinessProbeResult, { ok: false }>,
|
||||
mode: WorkspaceReadinessGateMode = resolveWorkspaceReadinessGateMode(),
|
||||
): boolean {
|
||||
if (result.reason === "readiness_missing") return mode === "strict";
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* How long a freshly started workspace may take to satisfy readiness.
|
||||
*
|
||||
* Deliberately short. The transport readiness probe has already required a
|
||||
* `status: ok` response from this guest by the time the gate runs, so the only
|
||||
* thing left to absorb is a one-off blip — not a cold start. A longer budget
|
||||
* would add latency to every managed start to no benefit.
|
||||
*/
|
||||
export const WORKSPACE_READINESS_GATE_TIMEOUT_MS = 3_000;
|
||||
const WORKSPACE_READINESS_GATE_INTERVAL_MS = 250;
|
||||
|
||||
/**
|
||||
* Poll readiness until it passes or the gate budget runs out.
|
||||
*
|
||||
* `identity_mismatch` and `readiness_missing` are terminal and return
|
||||
* immediately: another instance owning the port, or a guest that does not
|
||||
* implement the contract, are not conditions that polling can change.
|
||||
*/
|
||||
export async function waitForManagedWorkspaceReadiness(input: {
|
||||
healthUrl: string;
|
||||
identity: ManagedWorkspaceIdentity;
|
||||
fetchImpl?: typeof fetch;
|
||||
timeoutMs?: number;
|
||||
now?: () => number;
|
||||
sleep?: (ms: number) => Promise<void>;
|
||||
}): Promise<WorkspaceReadinessProbeResult> {
|
||||
const now = input.now ?? Date.now;
|
||||
const sleep = input.sleep ?? ((ms: number) => new Promise<void>((resolve) => { setTimeout(resolve, ms); }));
|
||||
const deadline = now() + (input.timeoutMs ?? WORKSPACE_READINESS_GATE_TIMEOUT_MS);
|
||||
let last: WorkspaceReadinessProbeResult = {
|
||||
ok: false,
|
||||
reason: "unreachable",
|
||||
readiness: null,
|
||||
detail: "readiness gate did not run",
|
||||
};
|
||||
for (;;) {
|
||||
last = await probeManagedWorkspaceReadiness({
|
||||
healthUrl: input.healthUrl,
|
||||
identity: input.identity,
|
||||
fetchImpl: input.fetchImpl,
|
||||
});
|
||||
if (last.ok || last.reason === "identity_mismatch" || last.reason === "readiness_missing") return last;
|
||||
if (now() >= deadline) return last;
|
||||
await sleep(WORKSPACE_READINESS_GATE_INTERVAL_MS);
|
||||
}
|
||||
}
|
||||
|
||||
/** Log a rejected probe once, with the machine reason kept intact for operators. */
|
||||
export function logManagedWorkspaceReadinessRejection(input: {
|
||||
executionWorkspaceId: string;
|
||||
healthUrl: string;
|
||||
result: Extract<WorkspaceReadinessProbeResult, { ok: false }>;
|
||||
}) {
|
||||
logger.warn(
|
||||
{
|
||||
executionWorkspaceId: input.executionWorkspaceId,
|
||||
healthUrl: input.healthUrl,
|
||||
reason: input.result.reason,
|
||||
detail: input.result.detail,
|
||||
seedState: input.result.readiness?.seedState ?? null,
|
||||
failurePhase: input.result.readiness?.failurePhase ?? null,
|
||||
},
|
||||
"managed workspace readiness probe rejected the runtime",
|
||||
);
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
import { and, desc, eq } from "drizzle-orm";
|
||||
import { and, desc, eq, inArray } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { issueWorkProducts } from "@paperclipai/db";
|
||||
import { issueWorkProducts, workspaceRuntimeServices } from "@paperclipai/db";
|
||||
import type { IssueWorkProduct } from "@paperclipai/shared";
|
||||
import { insertRowsInChunks } from "./batch-insert.js";
|
||||
import type { ImportIssueWorkProductRow } from "./import-write-types.js";
|
||||
@@ -33,6 +33,44 @@ function toIssueWorkProduct(row: IssueWorkProductRow): IssueWorkProduct {
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Refresh runtime-service work products from the live runtime rows they point at
|
||||
* (PAP-17572).
|
||||
*
|
||||
* A runtime URL is only valid for as long as the process holds that port. A
|
||||
* managed restart can relocate it, which used to leave a user-facing preview link
|
||||
* that answers with somebody else's service or nothing at all. The runtime row is
|
||||
* the authoritative publication record, so it wins over the stored copy.
|
||||
*
|
||||
* Read-path only and deliberately non-destructive: a work product whose runtime
|
||||
* row is gone keeps its recorded URL and is reported unhealthy rather than
|
||||
* silently blanked, so the history of what was published survives.
|
||||
*/
|
||||
export function reconcileRuntimeServiceWorkProducts(
|
||||
products: IssueWorkProduct[],
|
||||
liveRuntimeServices: Array<{
|
||||
id: string;
|
||||
url: string | null;
|
||||
status: string;
|
||||
healthStatus: string;
|
||||
}>,
|
||||
): IssueWorkProduct[] {
|
||||
if (products.length === 0) return products;
|
||||
const liveById = new Map(liveRuntimeServices.map((service) => [service.id, service]));
|
||||
return products.map((product) => {
|
||||
if (product.type !== "runtime_service" || !product.runtimeServiceId) return product;
|
||||
const live = liveById.get(product.runtimeServiceId);
|
||||
if (!live) {
|
||||
return product.healthStatus === "unhealthy" ? product : { ...product, healthStatus: "unhealthy" };
|
||||
}
|
||||
const isServing = live.status === "running" && live.healthStatus === "healthy";
|
||||
const url = live.url ?? product.url;
|
||||
const healthStatus: IssueWorkProduct["healthStatus"] = isServing ? "healthy" : "unhealthy";
|
||||
if (product.url === url && product.healthStatus === healthStatus) return product;
|
||||
return { ...product, url, healthStatus };
|
||||
});
|
||||
}
|
||||
|
||||
export function workProductService(db: Db) {
|
||||
return {
|
||||
listForIssue: async (issueId: string) => {
|
||||
@@ -41,7 +79,21 @@ export function workProductService(db: Db) {
|
||||
.from(issueWorkProducts)
|
||||
.where(eq(issueWorkProducts.issueId, issueId))
|
||||
.orderBy(desc(issueWorkProducts.isPrimary), desc(issueWorkProducts.updatedAt));
|
||||
return rows.map(toIssueWorkProduct);
|
||||
const products = rows.map(toIssueWorkProduct);
|
||||
const runtimeServiceIds = products
|
||||
.map((product) => (product.type === "runtime_service" ? product.runtimeServiceId : null))
|
||||
.filter((value): value is string => Boolean(value));
|
||||
if (runtimeServiceIds.length === 0) return products;
|
||||
const liveRuntimeServices = await db
|
||||
.select({
|
||||
id: workspaceRuntimeServices.id,
|
||||
url: workspaceRuntimeServices.url,
|
||||
status: workspaceRuntimeServices.status,
|
||||
healthStatus: workspaceRuntimeServices.healthStatus,
|
||||
})
|
||||
.from(workspaceRuntimeServices)
|
||||
.where(inArray(workspaceRuntimeServices.id, [...new Set(runtimeServiceIds)]));
|
||||
return reconcileRuntimeServiceWorkProducts(products, liveRuntimeServices);
|
||||
},
|
||||
|
||||
getById: async (id: string) => {
|
||||
|
||||
@@ -174,10 +174,12 @@ describe("WorkspaceGitOperationScheduler", () => {
|
||||
await vi.waitFor(() => expect(scheduler.snapshot().totals.singleFlightJoins).toBe(1));
|
||||
expect(calls).toBe(1);
|
||||
gate.resolve();
|
||||
await expect(Promise.all([first, joined])).resolves.toEqual([
|
||||
expect.objectContaining({ stdout: "shared", singleFlightJoined: false }),
|
||||
expect.objectContaining({ stdout: "shared", singleFlightJoined: true }),
|
||||
const results = await Promise.all([first, joined]);
|
||||
expect(results).toEqual([
|
||||
expect.objectContaining({ stdout: "shared" }),
|
||||
expect.objectContaining({ stdout: "shared" }),
|
||||
]);
|
||||
expect(results.map((result) => result.singleFlightJoined).sort()).toEqual([false, true]);
|
||||
|
||||
shouldFail = true;
|
||||
await expect(scheduler.run(scanInput(workspace, "failure"))).rejects.toMatchObject({
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { execFile } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import { readFileSync } from "node:fs";
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
@@ -27,6 +28,27 @@ export function deriveWorktreeInstanceId(workspacePath: string): string {
|
||||
return `${prefix}-${pathHash}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* The instance id a seeded worktree actually runs as, read from the pointer the
|
||||
* guest process itself loads.
|
||||
*
|
||||
* Synchronous and null-on-anything-unexpected by design: callers use it to
|
||||
* decide whether they can prove workspace identity at all, and an unsafe or
|
||||
* unreadable pointer must fail closed rather than fall back to a guess.
|
||||
*/
|
||||
export function readWorktreeInstanceId(workspacePath: string): string | null {
|
||||
const envPath = path.join(path.resolve(workspacePath), ".paperclip", ".env");
|
||||
let contents: string;
|
||||
try {
|
||||
contents = readFileSync(envPath, "utf8");
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
const instanceId = parseEnvContents(contents).PAPERCLIP_INSTANCE_ID?.trim();
|
||||
if (!instanceId || !INSTANCE_ID_RE.test(instanceId)) return null;
|
||||
return instanceId;
|
||||
}
|
||||
|
||||
export type WorktreeInstancePointer = {
|
||||
envPath: string;
|
||||
envContents: string;
|
||||
|
||||
@@ -0,0 +1,215 @@
|
||||
/**
|
||||
* Issues signed workspace login handoff tickets (PAP-17572).
|
||||
*
|
||||
* Runs on the authenticated main control plane. Everything a ticket is bound to
|
||||
* is resolved here from server-side state — the caller's authenticated identity,
|
||||
* the live runtime row's published URL, and the worktree's recorded instance
|
||||
* pointer — so a client cannot influence the audience, workspace, or instance a
|
||||
* ticket is valid for.
|
||||
*
|
||||
* A ticket is only minted for a workspace that currently passes the protected
|
||||
* readiness probe. Handing out a ticket for a half-restored clone would turn a
|
||||
* diagnosable "clone is incomplete" state back into the opaque
|
||||
* "invalid email or password" that started this incident.
|
||||
*/
|
||||
|
||||
import { and, desc, eq } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { authUsers, instanceUserRoles, workspaceRuntimeServices } from "@paperclipai/db";
|
||||
import type { WorkspaceReadiness } from "@paperclipai/shared";
|
||||
import {
|
||||
buildWorkspaceHandoffExchangeUrl,
|
||||
issueWorkspaceHandoffTicket,
|
||||
normalizeWorkspaceHandoffOrigin,
|
||||
sanitizeWorkspaceHandoffRedirectPath,
|
||||
WORKSPACE_HANDOFF_TICKET_TTL_SECONDS,
|
||||
} from "../auth/workspace-login-handoff.js";
|
||||
import { resolvePaperclipInstanceId } from "../home-paths.js";
|
||||
import {
|
||||
probeManagedWorkspaceReadiness,
|
||||
resolveManagedWorkspaceIdentity,
|
||||
} from "./managed-workspace-identity.js";
|
||||
|
||||
export type WorkspaceLoginHandoffActor = {
|
||||
userId: string | null | undefined;
|
||||
userEmail: string | null | undefined;
|
||||
source: string | null | undefined;
|
||||
};
|
||||
|
||||
export type WorkspaceLoginHandoffFailure =
|
||||
| { reason: "no_board_identity" }
|
||||
| { reason: "handoff_not_configured" }
|
||||
| { reason: "runtime_not_running" }
|
||||
| { reason: "runtime_url_unusable"; detail: string }
|
||||
| { reason: "workspace_not_ready"; detail: string | null; readiness: WorkspaceReadiness | null };
|
||||
|
||||
export type WorkspaceLoginHandoffIssuance = {
|
||||
url: string;
|
||||
expiresAt: string;
|
||||
/** The live runtime origin the ticket was bound to. */
|
||||
origin: string;
|
||||
nonce: string;
|
||||
userId: string;
|
||||
instanceId: string;
|
||||
};
|
||||
|
||||
export type WorkspaceLoginHandoffResult =
|
||||
| { ok: true; issuance: WorkspaceLoginHandoffIssuance }
|
||||
| { ok: false; failure: WorkspaceLoginHandoffFailure };
|
||||
|
||||
/**
|
||||
* Resolve the board user a ticket will be minted for.
|
||||
*
|
||||
* In `authenticated` mode this is the caller's own session identity. The
|
||||
* `local_implicit` actor of a `local_trusted` instance has no auth row, so the
|
||||
* instance's oldest instance admin stands in: that mode already grants
|
||||
* unrestricted board access to this instance and its clones, so this widens
|
||||
* nothing, and it keeps local development password-independent too.
|
||||
*/
|
||||
export async function resolveWorkspaceHandoffBoardIdentity(
|
||||
db: Db,
|
||||
actor: WorkspaceLoginHandoffActor,
|
||||
): Promise<{ userId: string; email: string } | null> {
|
||||
const userId = actor.userId?.trim();
|
||||
const email = actor.userEmail?.trim();
|
||||
if (userId && email && actor.source !== "local_implicit") {
|
||||
return { userId, email };
|
||||
}
|
||||
if (actor.source !== "local_implicit") return null;
|
||||
|
||||
const admin = await db
|
||||
.select({ id: authUsers.id, email: authUsers.email, createdAt: authUsers.createdAt })
|
||||
.from(authUsers)
|
||||
.innerJoin(
|
||||
instanceUserRoles,
|
||||
and(eq(instanceUserRoles.userId, authUsers.id), eq(instanceUserRoles.role, "instance_admin")),
|
||||
)
|
||||
.orderBy(authUsers.createdAt, authUsers.id)
|
||||
.limit(1)
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (!admin?.email) return null;
|
||||
return { userId: admin.id, email: admin.email };
|
||||
}
|
||||
|
||||
/**
|
||||
* The live runtime row a workspace is currently served from.
|
||||
*
|
||||
* Reading the row (rather than a stored work-product URL) is what keeps the
|
||||
* handoff target correct after a port change: the row is the same source the
|
||||
* runtime publication path writes.
|
||||
*/
|
||||
export async function resolveLiveWorkspaceRuntimeOrigin(
|
||||
db: Db,
|
||||
input: { companyId: string; executionWorkspaceId: string },
|
||||
): Promise<{ url: string; runtimeServiceId: string } | null> {
|
||||
const row = await db
|
||||
.select({
|
||||
id: workspaceRuntimeServices.id,
|
||||
url: workspaceRuntimeServices.url,
|
||||
updatedAt: workspaceRuntimeServices.updatedAt,
|
||||
})
|
||||
.from(workspaceRuntimeServices)
|
||||
.where(
|
||||
and(
|
||||
eq(workspaceRuntimeServices.companyId, input.companyId),
|
||||
eq(workspaceRuntimeServices.executionWorkspaceId, input.executionWorkspaceId),
|
||||
eq(workspaceRuntimeServices.status, "running"),
|
||||
eq(workspaceRuntimeServices.healthStatus, "healthy"),
|
||||
),
|
||||
)
|
||||
.orderBy(desc(workspaceRuntimeServices.updatedAt))
|
||||
.then((rows) => rows.find((candidate) => Boolean(candidate.url)) ?? null);
|
||||
if (!row?.url) return null;
|
||||
return { url: row.url, runtimeServiceId: row.id };
|
||||
}
|
||||
|
||||
export async function issueWorkspaceLoginHandoff(input: {
|
||||
db: Db;
|
||||
companyId: string;
|
||||
executionWorkspace: { id: string; cwd: string | null };
|
||||
actor: WorkspaceLoginHandoffActor;
|
||||
next?: string | null;
|
||||
ttlSeconds?: number;
|
||||
now?: Date;
|
||||
probe?: typeof probeManagedWorkspaceReadiness;
|
||||
}): Promise<WorkspaceLoginHandoffResult> {
|
||||
const identity = resolveManagedWorkspaceIdentity({
|
||||
workspaceCwd: input.executionWorkspace.cwd,
|
||||
executionWorkspaceId: input.executionWorkspace.id,
|
||||
companyId: input.companyId,
|
||||
});
|
||||
if (!identity) return { ok: false, failure: { reason: "handoff_not_configured" } };
|
||||
|
||||
const boardIdentity = await resolveWorkspaceHandoffBoardIdentity(input.db, input.actor);
|
||||
if (!boardIdentity) return { ok: false, failure: { reason: "no_board_identity" } };
|
||||
|
||||
const live = await resolveLiveWorkspaceRuntimeOrigin(input.db, {
|
||||
companyId: input.companyId,
|
||||
executionWorkspaceId: input.executionWorkspace.id,
|
||||
});
|
||||
if (!live) return { ok: false, failure: { reason: "runtime_not_running" } };
|
||||
|
||||
const origin = normalizeWorkspaceHandoffOrigin(live.url);
|
||||
if (!origin) {
|
||||
return { ok: false, failure: { reason: "runtime_url_unusable", detail: live.url } };
|
||||
}
|
||||
|
||||
const probe = input.probe ?? probeManagedWorkspaceReadiness;
|
||||
const readinessResult = await probe({
|
||||
// Probe the origin the user will actually be sent to, so a stale row or a
|
||||
// reassigned port is caught before a ticket exists rather than after.
|
||||
healthUrl: new URL("/api/health", origin).toString(),
|
||||
identity,
|
||||
handoffSubject: boardIdentity,
|
||||
});
|
||||
if (!readinessResult.ok) {
|
||||
return {
|
||||
ok: false,
|
||||
failure: {
|
||||
reason: "workspace_not_ready",
|
||||
detail: readinessResult.detail ?? readinessResult.reason,
|
||||
readiness: readinessResult.readiness,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
const { ticket, payload } = issueWorkspaceHandoffTicket({
|
||||
key: identity.handoffKey,
|
||||
userId: boardIdentity.userId,
|
||||
email: boardIdentity.email,
|
||||
executionWorkspaceId: identity.executionWorkspaceId,
|
||||
companyId: identity.companyId,
|
||||
instanceId: identity.instanceId,
|
||||
origin,
|
||||
issuerInstanceId: resolvePaperclipInstanceId(),
|
||||
next: sanitizeWorkspaceHandoffRedirectPath(input.next),
|
||||
ttlSeconds: input.ttlSeconds ?? WORKSPACE_HANDOFF_TICKET_TTL_SECONDS,
|
||||
now: input.now,
|
||||
});
|
||||
|
||||
return {
|
||||
ok: true,
|
||||
issuance: {
|
||||
url: buildWorkspaceHandoffExchangeUrl({ origin, ticket }),
|
||||
expiresAt: new Date(payload.exp * 1000).toISOString(),
|
||||
origin,
|
||||
nonce: payload.jti,
|
||||
userId: boardIdentity.userId,
|
||||
instanceId: identity.instanceId,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/** HTTP status for a failed issuance. */
|
||||
export function workspaceLoginHandoffFailureStatus(failure: WorkspaceLoginHandoffFailure): number {
|
||||
switch (failure.reason) {
|
||||
case "no_board_identity":
|
||||
return 403;
|
||||
case "handoff_not_configured":
|
||||
return 501;
|
||||
case "runtime_not_running":
|
||||
case "runtime_url_unusable":
|
||||
case "workspace_not_ready":
|
||||
return 409;
|
||||
}
|
||||
}
|
||||
@@ -14,7 +14,7 @@ type WorkspaceOperationRow = typeof workspaceOperations.$inferSelect;
|
||||
* Managed runtime control actions. Every one of these mutates the runtime rows and
|
||||
* local listeners of a single execution workspace, so only one may be live at a time.
|
||||
*/
|
||||
const RUNTIME_CONTROL_ACTIONS = new Set(["start", "stop", "restart", "run"]);
|
||||
const RUNTIME_CONTROL_ACTIONS = new Set(["start", "stop", "restart", "repair", "run"]);
|
||||
|
||||
/**
|
||||
* Identity of this server process. A `running` runtime-control operation stamped with a
|
||||
@@ -234,7 +234,10 @@ export interface WorkspaceOperationRecorder {
|
||||
* timeout error is rethrown, so a hung provider can never leave an active operation.
|
||||
*/
|
||||
timeoutMs?: number | null;
|
||||
run: () => Promise<{
|
||||
run: (reportProgress: (input: {
|
||||
metadata?: Record<string, unknown> | null;
|
||||
system?: string | null;
|
||||
}) => Promise<void>) => Promise<{
|
||||
status?: WorkspaceOperationStatus;
|
||||
exitCode?: number | null;
|
||||
stdout?: string | null;
|
||||
@@ -484,7 +487,7 @@ export function workspaceOperationService(db: Db) {
|
||||
// Managed runtime controls get an ownership stamp so bounded recovery can tell a
|
||||
// slow-but-live operation from one abandoned by a dead request or server process.
|
||||
const runtimeControlAction = readRuntimeControlAction(recordInput.metadata);
|
||||
const insertedMetadata = runtimeControlAction
|
||||
let currentMetadata = runtimeControlAction
|
||||
? {
|
||||
...(recordInput.metadata ?? {}),
|
||||
runtimeControlOwner: {
|
||||
@@ -515,7 +518,7 @@ export function workspaceOperationService(db: Db) {
|
||||
logStore: handle.store,
|
||||
logRef: handle.logRef,
|
||||
metadata: redactCurrentUserValue(
|
||||
insertedMetadata,
|
||||
currentMetadata,
|
||||
currentUserRedactionOptions,
|
||||
) as Record<string, unknown> | null,
|
||||
startedAt,
|
||||
@@ -532,17 +535,21 @@ export function workspaceOperationService(db: Db) {
|
||||
if (runtimeControlAction) {
|
||||
heartbeatTimer = setInterval(() => {
|
||||
const heartbeatAt = new Date();
|
||||
currentMetadata = combineMetadata(currentMetadata, {
|
||||
runtimeControlOwner: {
|
||||
ownerId: RUNTIME_CONTROL_OWNER_ID,
|
||||
pid: process.pid,
|
||||
action: runtimeControlAction,
|
||||
heartbeatAt: heartbeatAt.toISOString(),
|
||||
} satisfies RuntimeControlOwnerStamp,
|
||||
});
|
||||
void db
|
||||
.update(workspaceOperations)
|
||||
.set({
|
||||
metadata: combineMetadata(insertedMetadata, {
|
||||
runtimeControlOwner: {
|
||||
ownerId: RUNTIME_CONTROL_OWNER_ID,
|
||||
pid: process.pid,
|
||||
action: runtimeControlAction,
|
||||
heartbeatAt: heartbeatAt.toISOString(),
|
||||
} satisfies RuntimeControlOwnerStamp,
|
||||
}),
|
||||
metadata: redactCurrentUserValue(
|
||||
currentMetadata,
|
||||
currentUserRedactionOptions,
|
||||
) as Record<string, unknown> | null,
|
||||
updatedAt: heartbeatAt,
|
||||
})
|
||||
.where(and(eq(workspaceOperations.id, id), eq(workspaceOperations.status, "running")))
|
||||
@@ -551,9 +558,29 @@ export function workspaceOperationService(db: Db) {
|
||||
heartbeatTimer.unref?.();
|
||||
}
|
||||
|
||||
const reportProgress = async (progress: {
|
||||
metadata?: Record<string, unknown> | null;
|
||||
system?: string | null;
|
||||
}) => {
|
||||
await append("system", progress.system ?? null);
|
||||
currentMetadata = combineMetadata(currentMetadata, progress.metadata);
|
||||
await db
|
||||
.update(workspaceOperations)
|
||||
.set({
|
||||
metadata: redactCurrentUserValue(
|
||||
currentMetadata,
|
||||
currentUserRedactionOptions,
|
||||
) as Record<string, unknown> | null,
|
||||
stdoutExcerpt: stdoutExcerpt || null,
|
||||
stderrExcerpt: stderrExcerpt || null,
|
||||
updatedAt: new Date(),
|
||||
})
|
||||
.where(and(eq(workspaceOperations.id, id), eq(workspaceOperations.status, "running")));
|
||||
};
|
||||
|
||||
const timeoutMs = recordInput.timeoutMs ?? defaultRuntimeControlTimeoutMs(runtimeControlAction);
|
||||
const settle = async () => {
|
||||
if (!timeoutMs || timeoutMs <= 0) return await recordInput.run();
|
||||
if (!timeoutMs || timeoutMs <= 0) return await recordInput.run(reportProgress);
|
||||
const timeout = new Promise<never>((_resolve, reject) => {
|
||||
timeoutTimer = setTimeout(
|
||||
() => reject(new WorkspaceOperationTimeoutError(timeoutMs, runtimeControlAction)),
|
||||
@@ -561,7 +588,7 @@ export function workspaceOperationService(db: Db) {
|
||||
);
|
||||
timeoutTimer.unref?.();
|
||||
});
|
||||
return await Promise.race([recordInput.run(), timeout]);
|
||||
return await Promise.race([recordInput.run(reportProgress), timeout]);
|
||||
};
|
||||
|
||||
try {
|
||||
@@ -583,7 +610,7 @@ export function workspaceOperationService(db: Db) {
|
||||
logSha256: finalized.sha256,
|
||||
logCompressed: finalized.compressed,
|
||||
metadata: redactCurrentUserValue(
|
||||
combineMetadata(insertedMetadata, result.metadata),
|
||||
combineMetadata(currentMetadata, result.metadata),
|
||||
currentUserRedactionOptions,
|
||||
) as Record<string, unknown> | null,
|
||||
finishedAt,
|
||||
@@ -613,7 +640,7 @@ export function workspaceOperationService(db: Db) {
|
||||
...(runtimeControlAction
|
||||
? {
|
||||
metadata: redactCurrentUserValue(
|
||||
combineMetadata(insertedMetadata, {
|
||||
combineMetadata(currentMetadata, {
|
||||
failureReason: error instanceof WorkspaceOperationTimeoutError
|
||||
? "runtime_control_timeout"
|
||||
: "runtime_control_error",
|
||||
|
||||
@@ -0,0 +1,296 @@
|
||||
/**
|
||||
* Protected workspace readiness (PAP-17572).
|
||||
*
|
||||
* `/api/health` answering 200 only proves a listener exists. This module answers
|
||||
* the question the control plane and the UI actually need — "can a board user
|
||||
* open this clone and see their data?" — from four independent signals: the
|
||||
* isolated database, the versioned seed manifest, representative cloned rows,
|
||||
* and the login-handoff configuration.
|
||||
*
|
||||
* Everything here is read-only and deliberately small: structural validation of
|
||||
* a clone happens once, during provisioning or repair, not on every probe.
|
||||
*
|
||||
* `repairing` is intentionally not derivable here. A guest process cannot know
|
||||
* that the control plane is running a managed repair against it; the UI layers
|
||||
* that state on from the live workspace-operation row.
|
||||
*/
|
||||
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { authUsers, companies, companyMemberships, issues } from "@paperclipai/db";
|
||||
import type {
|
||||
WorkspaceReadiness,
|
||||
WorkspaceReadinessState,
|
||||
WorkspaceSeedReadinessState,
|
||||
} from "@paperclipai/shared";
|
||||
import {
|
||||
resolveWorkspaceHandoffLocalCompanyId,
|
||||
resolveWorkspaceHandoffLocalKey,
|
||||
resolveWorkspaceHandoffLocalWorkspaceId,
|
||||
} from "../auth/workspace-login-handoff.js";
|
||||
import { resolvePaperclipInstanceId } from "../home-paths.js";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
|
||||
const WORKSPACE_SEED_MANIFEST_BASENAME = "seed-manifest.json";
|
||||
const LEGACY_SEED_PENDING_BASENAME = "seed-pending";
|
||||
const LEGACY_SEED_COMPLETE_BASENAME = "seed-complete";
|
||||
|
||||
type SeedManifestSummary = {
|
||||
state: WorkspaceSeedReadinessState;
|
||||
phase: string | null;
|
||||
mode: "minimal" | "full" | null;
|
||||
failurePhase: string | null;
|
||||
};
|
||||
|
||||
/**
|
||||
* Directory holding this instance's seed markers.
|
||||
*
|
||||
* `PAPERCLIP_CONFIG` is the authoritative pointer a seeded worktree is started
|
||||
* with; the cwd fallback covers a guest launched without it.
|
||||
*/
|
||||
export function resolveWorkspaceSeedMarkerDir(env: NodeJS.ProcessEnv = process.env): string {
|
||||
const configPath = env.PAPERCLIP_CONFIG?.trim();
|
||||
if (configPath) return path.dirname(path.resolve(configPath));
|
||||
return path.resolve(process.cwd(), ".paperclip");
|
||||
}
|
||||
|
||||
function readSeedManifestSummary(markerDir: string): SeedManifestSummary {
|
||||
const manifestPath = path.join(markerDir, WORKSPACE_SEED_MANIFEST_BASENAME);
|
||||
if (existsSync(manifestPath)) {
|
||||
let manifest: Record<string, unknown>;
|
||||
try {
|
||||
manifest = JSON.parse(readFileSync(manifestPath, "utf8")) as Record<string, unknown>;
|
||||
} catch {
|
||||
// A manifest we cannot parse is evidence of an interrupted write, which is
|
||||
// a failure — never a silent "assume seeded".
|
||||
return { state: "failed", phase: null, mode: null, failurePhase: "seed_manifest_unreadable" };
|
||||
}
|
||||
const rawState = typeof manifest.state === "string" ? manifest.state : null;
|
||||
const state: WorkspaceSeedReadinessState =
|
||||
rawState === "pending" || rawState === "running" || rawState === "verified" || rawState === "failed"
|
||||
? rawState
|
||||
: "unknown";
|
||||
const phase = typeof manifest.phase === "string" ? manifest.phase : null;
|
||||
const mode = manifest.seedMode === "minimal" || manifest.seedMode === "full" ? manifest.seedMode : null;
|
||||
return {
|
||||
state,
|
||||
phase,
|
||||
mode,
|
||||
failurePhase: state === "failed" ? phase ?? "unknown" : null,
|
||||
};
|
||||
}
|
||||
|
||||
// Legacy markers predate the versioned manifest. `seed-complete`, or the
|
||||
// absence of both markers on an already-running instance, means an adopted
|
||||
// legacy clone whose completeness this probe cannot prove.
|
||||
if (existsSync(path.join(markerDir, LEGACY_SEED_PENDING_BASENAME))) {
|
||||
return { state: "pending", phase: "pending", mode: null, failurePhase: null };
|
||||
}
|
||||
if (existsSync(path.join(markerDir, LEGACY_SEED_COMPLETE_BASENAME))) {
|
||||
return { state: "unknown", phase: "legacy_complete_marker", mode: null, failurePhase: null };
|
||||
}
|
||||
return { state: "absent", phase: null, mode: null, failurePhase: null };
|
||||
}
|
||||
|
||||
/**
|
||||
* Collapse the individual signals into one state.
|
||||
*
|
||||
* `degraded` is reserved for "the clone exists and was verified, but a signal
|
||||
* regressed", which is the case an operator can fix with one bounded repair.
|
||||
* A clone that never finished reports `provisioning`/`validating`/`failed`
|
||||
* instead, so the UI never offers a repair for work that is still in progress.
|
||||
*/
|
||||
export function resolveWorkspaceReadinessState(input: {
|
||||
databaseReady: boolean;
|
||||
cloneDataReady: boolean;
|
||||
authHandoffReady: boolean;
|
||||
seedState: WorkspaceSeedReadinessState;
|
||||
}): WorkspaceReadinessState {
|
||||
if (input.seedState === "failed") return "failed";
|
||||
if (input.seedState === "pending") return "provisioning";
|
||||
if (input.seedState === "running") return "provisioning";
|
||||
if (!input.databaseReady) return input.seedState === "verified" ? "degraded" : "provisioning";
|
||||
if (!input.cloneDataReady || !input.authHandoffReady) {
|
||||
return input.seedState === "verified" ? "degraded" : "validating";
|
||||
}
|
||||
// Cloned data and handoff both check out. A legacy or absent manifest is not
|
||||
// proof of a verified seed, so surface it as validating rather than ready.
|
||||
if (input.seedState !== "verified") return "validating";
|
||||
return "ready";
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether this process is a cloned workspace instance at all.
|
||||
*
|
||||
* The primary control plane has no seed manifest and no injected workspace
|
||||
* identity, and reporting a hollow readiness block for it would invite consumers
|
||||
* to treat "no clone" as "broken clone". Detection is by evidence on disk or
|
||||
* injected identity, never by a branch-name or path heuristic.
|
||||
*/
|
||||
let managedWorkspaceInstanceCache: { markerDir: string; value: boolean; checkedAtMs: number } | null = null;
|
||||
|
||||
/**
|
||||
* How long the marker-file answer is reused. `/api/health` is polled by the dev
|
||||
* runner, the control plane's readiness probe and the UI, so three `existsSync`
|
||||
* calls per request would be pure overhead on an instance whose answer is
|
||||
* effectively static. Short enough that a manifest appearing mid-provision is
|
||||
* picked up promptly.
|
||||
*/
|
||||
const MANAGED_WORKSPACE_DETECTION_TTL_MS = 5_000;
|
||||
|
||||
export function isManagedWorkspaceInstance(
|
||||
env: NodeJS.ProcessEnv = process.env,
|
||||
now: () => number = Date.now,
|
||||
): boolean {
|
||||
// Injected identity is authoritative and free to read, so it short-circuits
|
||||
// ahead of any filesystem work.
|
||||
if (resolveWorkspaceHandoffLocalKey(env)) return true;
|
||||
if (resolveWorkspaceHandoffLocalWorkspaceId(env)) return true;
|
||||
|
||||
const markerDir = resolveWorkspaceSeedMarkerDir(env);
|
||||
const cached = managedWorkspaceInstanceCache;
|
||||
if (
|
||||
cached
|
||||
&& cached.markerDir === markerDir
|
||||
&& now() - cached.checkedAtMs < MANAGED_WORKSPACE_DETECTION_TTL_MS
|
||||
) {
|
||||
return cached.value;
|
||||
}
|
||||
const value = existsSync(path.join(markerDir, WORKSPACE_SEED_MANIFEST_BASENAME))
|
||||
|| existsSync(path.join(markerDir, LEGACY_SEED_PENDING_BASENAME))
|
||||
|| existsSync(path.join(markerDir, LEGACY_SEED_COMPLETE_BASENAME));
|
||||
managedWorkspaceInstanceCache = { markerDir, value, checkedAtMs: now() };
|
||||
return value;
|
||||
}
|
||||
|
||||
/** Test-only seam so a suite can change marker files without waiting out the TTL. */
|
||||
export function resetManagedWorkspaceInstanceCacheForTests(): void {
|
||||
managedWorkspaceInstanceCache = null;
|
||||
}
|
||||
|
||||
export type WorkspaceReadinessDeps = {
|
||||
db: Db | null | undefined;
|
||||
env?: NodeJS.ProcessEnv;
|
||||
handoffSubject?: { userId: string; email: string } | null;
|
||||
};
|
||||
|
||||
/**
|
||||
* Assemble this instance's readiness. Never throws: a probe that cannot read the
|
||||
* database must report `databaseReady: false`, not fail the health endpoint.
|
||||
*/
|
||||
export async function resolveWorkspaceReadiness(deps: WorkspaceReadinessDeps): Promise<WorkspaceReadiness> {
|
||||
const env = deps.env ?? process.env;
|
||||
const seed = readSeedManifestSummary(resolveWorkspaceSeedMarkerDir(env));
|
||||
const instanceId = resolvePaperclipInstanceId();
|
||||
const executionWorkspaceId = resolveWorkspaceHandoffLocalWorkspaceId(env);
|
||||
// The company this workspace's board represents. Both product probes below are
|
||||
// scoped to it: "some company in the clone has issues" and "some user has some
|
||||
// membership" can both be true while the company the operator is opening is
|
||||
// missing or has no members, which is a workspace that must not report ready.
|
||||
const companyId = resolveWorkspaceHandoffLocalCompanyId(env);
|
||||
const handoffKeyPresent = Boolean(resolveWorkspaceHandoffLocalKey(env));
|
||||
const handoffUserId = deps.handoffSubject?.userId.trim() || null;
|
||||
const handoffUserEmail = deps.handoffSubject?.email.trim().toLowerCase() || null;
|
||||
|
||||
let databaseReady = false;
|
||||
let cloneDataReady = false;
|
||||
let clonedAdminPresent = false;
|
||||
let probeFailurePhase: string | null = null;
|
||||
|
||||
if (deps.db) {
|
||||
try {
|
||||
await deps.db.execute(sql`SELECT 1`);
|
||||
databaseReady = true;
|
||||
} catch (error) {
|
||||
logger.warn({ err: error }, "workspace readiness database probe failed");
|
||||
probeFailurePhase = "database_unreachable";
|
||||
}
|
||||
|
||||
if (databaseReady && companyId) {
|
||||
try {
|
||||
// One representative cloned company/issue pair proves the restore carried
|
||||
// product rows, not just an empty migrated schema.
|
||||
const clonedRows = await deps.db
|
||||
.select({ companyId: companies.id })
|
||||
.from(companies)
|
||||
.innerJoin(issues, eq(issues.companyId, companies.id))
|
||||
.where(eq(companies.id, companyId))
|
||||
.limit(1)
|
||||
.then((rows) => rows.length);
|
||||
cloneDataReady = clonedRows > 0;
|
||||
if (!cloneDataReady) probeFailurePhase ??= "clone_data_missing";
|
||||
} catch (error) {
|
||||
logger.warn({ err: error }, "workspace readiness clone-data probe failed");
|
||||
probeFailurePhase ??= "clone_data_unreadable";
|
||||
}
|
||||
|
||||
try {
|
||||
// The handoff can only sign in a user who survived the clone with an
|
||||
// active membership *in this workspace's company*, so readiness asserts
|
||||
// that identity exists here rather than discovering it at click time.
|
||||
// Existence, not a count: this runs on every protected health request and
|
||||
// counting the whole join would scale with instance size for an answer
|
||||
// that needs one row.
|
||||
const eligibleUsers = await deps.db
|
||||
.select({ userId: authUsers.id })
|
||||
.from(authUsers)
|
||||
.innerJoin(
|
||||
companyMemberships,
|
||||
and(
|
||||
eq(companyMemberships.principalType, "user"),
|
||||
eq(companyMemberships.principalId, authUsers.id),
|
||||
eq(companyMemberships.status, "active"),
|
||||
eq(companyMemberships.companyId, companyId),
|
||||
...(handoffUserId ? [eq(authUsers.id, handoffUserId)] : []),
|
||||
...(handoffUserEmail ? [sql`lower(${authUsers.email}) = ${handoffUserEmail}`] : []),
|
||||
),
|
||||
)
|
||||
.limit(1)
|
||||
.then((rows) => rows.length);
|
||||
clonedAdminPresent = eligibleUsers > 0;
|
||||
if (!clonedAdminPresent) probeFailurePhase ??= "cloned_membership_missing";
|
||||
} catch (error) {
|
||||
logger.warn({ err: error }, "workspace readiness identity probe failed");
|
||||
probeFailurePhase ??= "cloned_identity_unreadable";
|
||||
}
|
||||
}
|
||||
} else {
|
||||
probeFailurePhase = "database_not_configured";
|
||||
}
|
||||
|
||||
const authHandoffReady = Boolean(
|
||||
handoffKeyPresent
|
||||
&& executionWorkspaceId
|
||||
&& companyId
|
||||
&& clonedAdminPresent,
|
||||
);
|
||||
if (!companyId) probeFailurePhase ??= "workspace_company_not_configured";
|
||||
if (!executionWorkspaceId) probeFailurePhase ??= "workspace_identity_not_configured";
|
||||
if (!handoffKeyPresent) probeFailurePhase ??= "auth_handoff_not_configured";
|
||||
|
||||
const state = resolveWorkspaceReadinessState({
|
||||
databaseReady,
|
||||
cloneDataReady,
|
||||
authHandoffReady,
|
||||
seedState: seed.state,
|
||||
});
|
||||
|
||||
return {
|
||||
state,
|
||||
databaseReady,
|
||||
cloneDataReady,
|
||||
authHandoffReady,
|
||||
authHandoffUserId: handoffUserId,
|
||||
seedState: seed.state,
|
||||
seedPhase: seed.phase,
|
||||
seedMode: seed.mode,
|
||||
instanceId,
|
||||
executionWorkspaceId,
|
||||
companyId,
|
||||
// A seed-recorded failure phase is more specific than anything this probe
|
||||
// can infer, so it wins.
|
||||
failurePhase: seed.failurePhase ?? (state === "ready" ? null : probeFailurePhase),
|
||||
};
|
||||
}
|
||||
@@ -6,7 +6,10 @@ import path from "node:path";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from "vitest";
|
||||
|
||||
import type { BrokerClient, BrokerListenerRequest } from "./runtime-exposure/broker-client.js";
|
||||
import { diagnoseRuntimeListenerBinds } from "./runtime-exposure/loopback-listener.js";
|
||||
import {
|
||||
diagnoseRuntimeListenerBinds,
|
||||
readListenerBindFacts,
|
||||
} from "./runtime-exposure/loopback-listener.js";
|
||||
import {
|
||||
resetRuntimeServicesForTests,
|
||||
setWorkspaceRuntimeExposureDepsForTests,
|
||||
@@ -35,7 +38,9 @@ afterEach(async () => {
|
||||
});
|
||||
|
||||
function serviceCommand() {
|
||||
return `node -e 'const http=require("http");const p=Number(process.env.PORT);for(const q of [p,p+10000])http.createServer((_,r)=>{r.statusCode=200;r.end("ok")}).listen(q,"127.0.0.1");setInterval(()=>{},1000)'`;
|
||||
// Answers `/api/health` the way a real Paperclip dev runtime does: managed
|
||||
// publication requires semantic health, not just a 200 (PAP-17572).
|
||||
return `node -e 'const http=require("http");const p=Number(process.env.PORT);for(const q of [p,p+10000])http.createServer((rq,r)=>{if(rq.url==="/api/health"){r.setHeader("content-type","application/json");r.end(JSON.stringify({status:"ok"}));return}r.statusCode=200;r.end("ok")}).listen(q,"127.0.0.1");setInterval(()=>{},1000)'`;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -70,8 +75,11 @@ const host = mode === "custom" ? (valueOf("--bind-host") ?? "127.0.0.1")
|
||||
: mode === "lan" ? "0.0.0.0"
|
||||
: "127.0.0.1";
|
||||
const p = Number(process.env.PORT);
|
||||
// Even a pre-exposure checkout answered /api/health semantically; these guests
|
||||
// model bind behaviour, not health behaviour.
|
||||
const health = (rq, r) => { if (rq.url === "/api/health") { r.setHeader("content-type", "application/json"); r.end(JSON.stringify({ status: "ok" })); return true; } return false; };
|
||||
for (const q of [p, p + 10000]) {
|
||||
http.createServer((_, r) => { r.statusCode = 200; r.end("ok"); }).listen(q, host);
|
||||
http.createServer((rq, r) => { if (health(rq, r)) return; r.statusCode = 200; r.end("ok"); }).listen(q, host);
|
||||
}
|
||||
setInterval(() => {}, 1000);
|
||||
`;
|
||||
@@ -89,7 +97,8 @@ const argv = process.argv.slice(2);
|
||||
const at = argv.indexOf("--bind-host");
|
||||
const host = at >= 0 ? argv[at + 1] : "127.0.0.1";
|
||||
const p = Number(process.env.PORT);
|
||||
http.createServer((_, r) => { r.statusCode = 200; r.end("ok"); }).listen(p, host);
|
||||
const health = (rq, r) => { if (rq.url === "/api/health") { r.setHeader("content-type", "application/json"); r.end(JSON.stringify({ status: "ok" })); return true; } return false; };
|
||||
http.createServer((rq, r) => { if (health(rq, r)) return; r.statusCode = 200; r.end("ok"); }).listen(p, host);
|
||||
// No host argument: Vite's own HMR listener lands on the wildcard.
|
||||
http.createServer((_, r) => { r.statusCode = 426; r.end(); }).listen(p + 10000);
|
||||
setInterval(() => {}, 1000);
|
||||
@@ -99,8 +108,9 @@ setInterval(() => {}, 1000);
|
||||
const ALWAYS_WILDCARD_GUEST = `
|
||||
import http from "node:http";
|
||||
const p = Number(process.env.PORT);
|
||||
const health = (rq, r) => { if (rq.url === "/api/health") { r.setHeader("content-type", "application/json"); r.end(JSON.stringify({ status: "ok" })); return true; } return false; };
|
||||
for (const q of [p, p + 10000]) {
|
||||
http.createServer((_, r) => { r.statusCode = 200; r.end("ok"); }).listen(q, "0.0.0.0");
|
||||
http.createServer((rq, r) => { if (health(rq, r)) return; r.statusCode = 200; r.end("ok"); }).listen(q, "0.0.0.0");
|
||||
}
|
||||
setInterval(() => {}, 1000);
|
||||
`;
|
||||
@@ -155,6 +165,9 @@ function createBroker() {
|
||||
* "make exposure fixture honor occupied ports").
|
||||
*/
|
||||
async function isLoopbackPortFree(port: number): Promise<boolean> {
|
||||
const facts = await readListenerBindFacts(port);
|
||||
if (facts?.present) return false;
|
||||
|
||||
return await new Promise<boolean>((resolve) => {
|
||||
const probe = net.createServer();
|
||||
probe.unref();
|
||||
|
||||
@@ -28,7 +28,7 @@ const TERMINAL_HEARTBEAT_RUN_STATUSES = new Set([
|
||||
]);
|
||||
|
||||
/** Runtime control actions that mutate the workspace and therefore need the lease. */
|
||||
export const LEASED_WORKSPACE_RUNTIME_ACTIONS: readonly string[] = ["start", "stop", "restart"];
|
||||
export const LEASED_WORKSPACE_RUNTIME_ACTIONS: readonly string[] = ["start", "stop", "restart", "repair"];
|
||||
|
||||
/**
|
||||
* Upper bound on how long a lease survives without the owner touching it. Recovery
|
||||
|
||||
@@ -36,6 +36,16 @@ import { and, desc, eq, gte, inArray, isNull, lte, ne, or } from "drizzle-orm";
|
||||
import { asNumber, asString, parseObject, renderTemplate } from "../adapters/utils.js";
|
||||
import { conflict } from "../errors.js";
|
||||
import { resolveHomeAwarePath } from "../home-paths.js";
|
||||
import { hasVerifiedWorktreeSeedManifest } from "../worktree-seed-manifest.js";
|
||||
import {
|
||||
buildManagedWorkspaceGuestEnv,
|
||||
logManagedWorkspaceReadinessRejection,
|
||||
probeManagedWorkspaceHandoffSubjects,
|
||||
probeManagedWorkspaceReadiness,
|
||||
resolveManagedWorkspaceIdentity,
|
||||
shouldBlockPublicationOnReadiness,
|
||||
waitForManagedWorkspaceReadiness,
|
||||
} from "./managed-workspace-identity.js";
|
||||
import {
|
||||
createLocalServiceKey,
|
||||
findLocalServiceRegistryRecordByRuntimeServiceId,
|
||||
@@ -4728,14 +4738,74 @@ function resolveRuntimeServiceHealthUrl(
|
||||
return url;
|
||||
}
|
||||
|
||||
type RuntimeServiceHealthProbeInput = {
|
||||
db?: Db;
|
||||
serviceName?: string | null;
|
||||
command?: string | null;
|
||||
provider?: string | null;
|
||||
port?: number | null;
|
||||
/**
|
||||
* Workspace identity, when the caller knows it. Supplying all three upgrades
|
||||
* the probe from "the port answered with status ok" to the full protected
|
||||
* readiness contract, which is what stops a relocated port or a half-restored
|
||||
* clone from masquerading as healthy (PAP-17572).
|
||||
*/
|
||||
cwd?: string | null;
|
||||
executionWorkspaceId?: string | null;
|
||||
companyId?: string | null;
|
||||
};
|
||||
|
||||
/**
|
||||
* Whether a managed workspace runtime satisfies the *user* readiness contract.
|
||||
*
|
||||
* Returns null when this service is not an identity-resolvable managed workspace
|
||||
* runtime, so non-workspace services keep their existing behavior.
|
||||
*
|
||||
* For a workspace runtime this *replaces* the semantic transport check rather
|
||||
* than adding to it. The probe reads the same `/api/health` response the legacy
|
||||
* check read, so every legacy verdict is already implied: reaching
|
||||
* `readiness_missing` means the response was `200` with `status: ok` (exactly
|
||||
* what the legacy check asserted), and every other rejection means it was not.
|
||||
* Stacking a second request on top would double the latency of every reuse
|
||||
* decision for no extra information.
|
||||
*/
|
||||
async function probeManagedWorkspaceRuntimeReadiness(
|
||||
healthUrl: string,
|
||||
input: RuntimeServiceHealthProbeInput,
|
||||
): Promise<boolean | null> {
|
||||
if (!isPaperclipDevRuntimeService(input)) return null;
|
||||
const identity = resolveManagedWorkspaceIdentity({
|
||||
workspaceCwd: input.cwd ?? null,
|
||||
executionWorkspaceId: input.executionWorkspaceId ?? null,
|
||||
companyId: input.companyId ?? null,
|
||||
});
|
||||
if (!identity) return null;
|
||||
|
||||
const result = await probeManagedWorkspaceReadiness({ healthUrl, identity });
|
||||
const verified = !result.ok
|
||||
? result
|
||||
: input.db
|
||||
? await probeManagedWorkspaceHandoffSubjects({ db: input.db, healthUrl, identity })
|
||||
: {
|
||||
ok: false as const,
|
||||
reason: "not_ready" as const,
|
||||
readiness: result.readiness,
|
||||
detail: "control-plane database is unavailable for board identity verification",
|
||||
};
|
||||
if (verified.ok) return true;
|
||||
logManagedWorkspaceReadinessRejection({
|
||||
executionWorkspaceId: identity.executionWorkspaceId,
|
||||
healthUrl,
|
||||
result: verified,
|
||||
});
|
||||
// A guest that does not implement the readiness contract yet is not evidence of
|
||||
// an unhealthy clone; it is only evidence that the contract cannot be checked.
|
||||
return !shouldBlockPublicationOnReadiness(verified);
|
||||
}
|
||||
|
||||
async function isRuntimeServiceUrlHealthy(
|
||||
url: string | null,
|
||||
input?: {
|
||||
serviceName?: string | null;
|
||||
command?: string | null;
|
||||
provider?: string | null;
|
||||
port?: number | null;
|
||||
},
|
||||
input?: RuntimeServiceHealthProbeInput,
|
||||
) {
|
||||
const localProbeUrl = input?.provider === "local_process" && input.port && isPaperclipDevRuntimeService(input)
|
||||
? `http://127.0.0.1:${input.port}`
|
||||
@@ -4744,6 +4814,10 @@ async function isRuntimeServiceUrlHealthy(
|
||||
if (!probeUrl) return true;
|
||||
const healthUrl = resolveRuntimeServiceHealthUrl(probeUrl, input);
|
||||
if (!healthUrl) return false;
|
||||
|
||||
const readiness = await probeManagedWorkspaceRuntimeReadiness(healthUrl, input ?? {});
|
||||
if (readiness !== null) return readiness;
|
||||
|
||||
try {
|
||||
const response = await fetch(healthUrl, { signal: AbortSignal.timeout(2_000) });
|
||||
if (!response.ok) return false;
|
||||
@@ -5015,6 +5089,7 @@ export function resolveRuntimeProvisionCommand(input: {
|
||||
if (input.workspace.strategy !== "git_worktree") return "";
|
||||
|
||||
const stateDir = path.join(input.workspace.cwd, ".paperclip");
|
||||
const manifestPath = path.join(stateDir, "seed-manifest.json");
|
||||
const pendingMarker = path.join(stateDir, "seed-pending");
|
||||
const completeMarker = path.join(stateDir, "seed-complete");
|
||||
const provisionScript = path.join(
|
||||
@@ -5022,11 +5097,11 @@ export function resolveRuntimeProvisionCommand(input: {
|
||||
"scripts",
|
||||
"provision-worktree-runtime.sh",
|
||||
);
|
||||
if (
|
||||
!existsSync(pendingMarker)
|
||||
|| existsSync(completeMarker)
|
||||
|| !existsSync(provisionScript)
|
||||
) {
|
||||
let needsSeed = existsSync(pendingMarker) && !existsSync(completeMarker);
|
||||
if (existsSync(manifestPath)) {
|
||||
needsSeed = !hasVerifiedWorktreeSeedManifest(manifestPath);
|
||||
}
|
||||
if (!needsSeed || !existsSync(provisionScript)) {
|
||||
return "";
|
||||
}
|
||||
|
||||
@@ -5371,6 +5446,21 @@ async function spawnLocalRuntimeService(input: StartLocalRuntimeServiceInput): P
|
||||
env[portEnvKey] = String(port);
|
||||
}
|
||||
|
||||
// Per-workspace handoff key, readiness token, and workspace id. Injected for
|
||||
// the Paperclip dev runtime whether or not it is HTTPS-exposed, because the
|
||||
// password-independent login handoff and the protected readiness probe are
|
||||
// both needed for a plain-HTTP loopback workspace too (PAP-17572).
|
||||
const managedWorkspaceIdentity = isPaperclipDevRuntimeService({ serviceName, command })
|
||||
? resolveManagedWorkspaceIdentity({
|
||||
workspaceCwd: input.workspace.cwd,
|
||||
executionWorkspaceId: input.executionWorkspaceId ?? null,
|
||||
companyId: input.agent.companyId,
|
||||
})
|
||||
: null;
|
||||
if (managedWorkspaceIdentity) {
|
||||
Object.assign(env, buildManagedWorkspaceGuestEnv(managedWorkspaceIdentity));
|
||||
}
|
||||
|
||||
if (exposureConfig) {
|
||||
// Paperclip dev-runtime-specific hardening. Other managed processes are
|
||||
// still rejected by the broker unless /proc proves loopback-only listeners.
|
||||
@@ -5426,7 +5516,14 @@ async function spawnLocalRuntimeService(input: StartLocalRuntimeServiceInput): P
|
||||
});
|
||||
if (adoptedRecord) {
|
||||
const adoptedUrl = adoptedRecord.url ?? backendUrl;
|
||||
if (!(await isRuntimeServiceUrlHealthy(adoptedUrl, { serviceName, command }))) {
|
||||
if (!(await isRuntimeServiceUrlHealthy(adoptedUrl, {
|
||||
db: input.db,
|
||||
serviceName,
|
||||
command,
|
||||
cwd: input.workspace.cwd,
|
||||
executionWorkspaceId: input.executionWorkspaceId ?? null,
|
||||
companyId: input.agent.companyId,
|
||||
}))) {
|
||||
await terminateLocalService(adoptedRecord);
|
||||
await removeLocalServiceRegistryRecord(adoptedRecord.serviceKey);
|
||||
} else {
|
||||
@@ -5698,6 +5795,45 @@ async function spawnLocalRuntimeService(input: StartLocalRuntimeServiceInput): P
|
||||
);
|
||||
}
|
||||
}
|
||||
// Transport readiness only proves a listener answered. A managed workspace
|
||||
// must additionally satisfy the protected readiness contract — own database,
|
||||
// cloned rows, login handoff, and matching instance/workspace identity —
|
||||
// before it may be published as running/healthy (PAP-17572).
|
||||
if (managedWorkspaceIdentity) {
|
||||
const publishHealthUrl = resolveRuntimeServiceHealthUrl(
|
||||
record.port ? `http://127.0.0.1:${record.port}` : rewriteUrlHostToLoopback(record.url ?? backendUrl),
|
||||
{ serviceName, command },
|
||||
);
|
||||
if (!publishHealthUrl) {
|
||||
throw new Error("Managed workspace readiness gate could not resolve a health URL");
|
||||
}
|
||||
let gate = await waitForManagedWorkspaceReadiness({
|
||||
healthUrl: publishHealthUrl,
|
||||
identity: managedWorkspaceIdentity,
|
||||
});
|
||||
if (gate.ok) {
|
||||
if (!record.db) {
|
||||
throw new Error("Managed workspace readiness gate could not resolve the control-plane database");
|
||||
}
|
||||
gate = await probeManagedWorkspaceHandoffSubjects({
|
||||
db: record.db,
|
||||
healthUrl: publishHealthUrl,
|
||||
identity: managedWorkspaceIdentity,
|
||||
});
|
||||
}
|
||||
if (!gate.ok) {
|
||||
logManagedWorkspaceReadinessRejection({
|
||||
executionWorkspaceId: managedWorkspaceIdentity.executionWorkspaceId,
|
||||
healthUrl: publishHealthUrl,
|
||||
result: gate,
|
||||
});
|
||||
if (shouldBlockPublicationOnReadiness(gate)) {
|
||||
throw new Error(
|
||||
`Workspace is not ready to publish (${gate.reason}${gate.detail ? `: ${gate.detail}` : ""})`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
record.status = "running";
|
||||
record.healthStatus = "healthy";
|
||||
record.lastUsedAt = new Date().toISOString();
|
||||
@@ -5952,10 +6088,14 @@ async function findHealthyRunningRuntimeService(reuseKey: string | null) {
|
||||
const existing = existingId ? runtimeServicesById.get(existingId) : null;
|
||||
if (!existing || existing.status !== "running") return null;
|
||||
const healthInput = {
|
||||
db: existing.db,
|
||||
serviceName: existing.serviceName,
|
||||
command: existing.command,
|
||||
provider: existing.provider,
|
||||
port: existing.port,
|
||||
cwd: existing.cwd,
|
||||
executionWorkspaceId: existing.executionWorkspaceId,
|
||||
companyId: existing.companyId,
|
||||
};
|
||||
let healthy = await isRuntimeServiceUrlHealthy(existing.url, healthInput);
|
||||
if (!healthy) {
|
||||
@@ -7231,6 +7371,9 @@ export async function refreshPersistedRuntimeServiceHealth(input: {
|
||||
port: workspaceRuntimeServices.port,
|
||||
url: workspaceRuntimeServices.url,
|
||||
healthStatus: workspaceRuntimeServices.healthStatus,
|
||||
cwd: workspaceRuntimeServices.cwd,
|
||||
executionWorkspaceId: workspaceRuntimeServices.executionWorkspaceId,
|
||||
companyId: workspaceRuntimeServices.companyId,
|
||||
})
|
||||
.from(workspaceRuntimeServices)
|
||||
.where(and(
|
||||
@@ -7241,7 +7384,9 @@ export async function refreshPersistedRuntimeServiceHealth(input: {
|
||||
));
|
||||
const results = await Promise.all(rows.map(async (row) => ({
|
||||
row,
|
||||
healthStatus: await isRuntimeServiceUrlHealthy(row.url, row) ? "healthy" as const : "unhealthy" as const,
|
||||
healthStatus: await isRuntimeServiceUrlHealthy(row.url, { ...row, db: input.db })
|
||||
? "healthy" as const
|
||||
: "unhealthy" as const,
|
||||
})));
|
||||
await Promise.all(results.map(async ({ row, healthStatus }) => {
|
||||
const liveRecord = runtimeServicesById.get(row.id);
|
||||
@@ -7442,10 +7587,14 @@ export async function reconcilePersistedRuntimeServicesOnStartup(db: Db) {
|
||||
backfillDecision.action === "reprovision"
|
||||
|| !exposureHealthMatches
|
||||
|| !(await isRuntimeServiceUrlHealthy(adoptedUrl, {
|
||||
db,
|
||||
serviceName: row.serviceName,
|
||||
command: row.command,
|
||||
provider: "local_process",
|
||||
port: adoptedRecord.port ?? row.port,
|
||||
cwd: row.cwd,
|
||||
executionWorkspaceId: row.executionWorkspaceId ?? null,
|
||||
companyId: row.companyId,
|
||||
}))
|
||||
) {
|
||||
if (backfillDecision.action === "reprovision") backfilled += 1;
|
||||
|
||||
Reference in new issue
Block a user