mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-01 06:42:06 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs can accept a result before workspace copy-back finishes. > - The periodic orphan pass repeatedly tries this recovery, including cases that require operator action. > - Copy-back needs exact ownership evidence. A generic timer cannot grant that authority. > - This pull request wakes workspace recovery from committed changes and local owner completion. > - Saved retries use the existing scheduler. Empty work, completed barriers, and operator holds become quiet. ## Linked Issues or Issue Description Refs #17717. This is slice 52 of the recurring-server sleep stack, targeting master after parent #17717 merged at `4c56702e06c698cc10d4a86514669aba0f4d9a9b`. Related work: #17709 (status reassessment), #17687 (unsafe export recovery), and #16478 (workspace repair holds). This PR covers native workspace copy-back and validation. Result commit/projection and resource settlement remain separate recovery duties. It does not claim full-instance sleep. ## What Changed - Register a company/run/task-scoped workspace worker. Reconstruct accepted, unfinished work from existing records. Use saved retry and lease deadlines. - Notify from both accepted-result writers (control-plane and PRP coordinator), workspace receipts, coordinator changes, operator confirmation, retained workspace repair, and relevant task updates. Notifications follow the outer transaction. - Distinguish active local ownership, busy external ownership, pending Cloud evidence, operator-required evidence, and changed bindings. Only concrete external ownership/evidence waits receive a scoped retry. - Preserve advisory locks, signed stop proof, exact binding checks, physical callback joins, and existing failure policy. Join physical operations during shutdown. - Respect the existing heartbeat execution-suppression policy before discovery and execution; wake suppressed work on settings activation. - Skip physical workspace recovery in the enabled periodic orphan pass. Keep direct/startup reconciliation and the remaining result/cleanup recovery paths. - Keep routine company-only finalization hints quiet. Ordinary work requires issue/run scope; explicit settings and owner-release events keep their company-wide contract. - Preserve the parent's entire Runner/package tree, profile 80 and all historical identities. Add the dated obligation, producer, lifecycle and limitation audit. ## Verification - Current local source: `4059b24e3ea5c623be54ade18ec9b6ab78eb0289`, prepared by a normal merge of the current parent. The sole manual conflict keeps ordinary workspace fanout inside the existing issue/run guard. All 1,198 parent-only, 15 owned-only and 10 automatic shared paths are preserved. The complete package tree matches the parent byte-for-byte. - Controlled regression: the original workspace-only unscoped fanout performs three candidate SELECTs for routine company hints. The same zero-query assertion passes with scoped fanout. A real accepted result then produces one successful workspace settlement and returns to quiet. The failed baseline remains in the audit. - The first complete local group passes **1,693 cases in 14 files**, including all 19 workspace-worker cases. It covers both accepted-result producers, physical ownership, export retry/failure, finalization recovery, arbiter corpus, issue recovery actions, control-plane port, workspace recovery, heartbeat recovery, startup, retained cleanup and native execution. - Separate process-recovery and external-chat response files qualify the changed recovery/settlement paths without sharing their general test database with other files. Both pass in full: 410 process-recovery cases in 162.79 seconds and 273 external-chat cases in 53.79 seconds. The total is **2,376 distinct passes in 16 complete files**, with no skips. Normal merge `5c46c52873158529736b8349e4d44658865108ee` adopts the parent squash with this entire tested child tree unchanged. - Production server typechecking, build, module boundaries, migration order and source-profile checks pass. Thirteen dedicated PostgreSQL 17 fixtures use the unchanged migration 413 schema. Embedded-only paths use the existing recovery helper with fresh per-call fixtures. All temporary imports are restored byte-exact. No test assertion or timeout is relaxed. - Runner validation belongs to the unchanged parent package tree: 130 passing installation/compatibility/transport cases, 22 installed ACPX/Grok contract cases and 26 native-provider Rust cases. One existing opt-in closed Pi snapshot case is skipped because its distribution root is not configured. These are source-scoped checks, with no live provider qualification claim. - The owner audit at `doc/plans/2026-10-26-native-workspace-finalization-sleep-audit.md` preserves earlier failures and their disposition. Current-head acceptance: all 57 checks are clean, including protected CI and security. Greptile gives 5/5 on `7df5c9c44ea9e171be3c61b4430d3727b10f3337` with no actionable findings. Earlier scores do not substitute for this result. No Ponytail verdict is present under the authorized missing-review policy. - No claim of a full local repository suite, deployed app/database sleep, expanded Cloud wake coverage or measured cost savings is made. ## Risks - Supported wake producers are in-process application services. This adds no cross-replica or independent SQL-writer transport. - A real unresolved export still needs recovery. Busy owners and pending Cloud stop evidence retain a 30-second scoped check. No timeout authorizes copy-back. - Result projection and resource release still use their existing recovery. Some existing cleanup failures remain best effort; this PR does not infer cleanup success from a workspace barrier. - Cloud alarm coverage and host-sleep safety remain unchanged. No deployed suspension or cost savings have been measured. - Parent #17717 is merged. Current base is master at `4c56702e06c698cc10d4a86514669aba0f4d9a9b`. Fresh current-head CI and review must pass before this slice merges. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The exact serving model ID and context-window size are not exposed in this session. No subagent was used for this slice. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>
74 lines
4.7 KiB
TypeScript
74 lines
4.7 KiB
TypeScript
import { verify } from "node:crypto";
|
|
import { cloudRuntimePublicKeyForKid } from "../cloud-runtime-identity.js";
|
|
import { getCloudStackContext } from "../cloud-instance.js";
|
|
|
|
/** Deployment identity covers the controller and every orphaned tar/Git child. */
|
|
export type WorkspaceWorker = {
|
|
provider: "railway"; projectId: string; environmentId: string; serviceId: string; deploymentId: string;
|
|
};
|
|
export type WorkspaceStopBinding = { runId: string; ownerToken: string; worker: WorkspaceWorker };
|
|
export type WorkspaceWorkerStopProof = WorkspaceStopBinding & {
|
|
v: 1; iss: "paperclip-cloud"; aud: "paperclip-workspace-worker-stop/v1"; sub: string;
|
|
stopped: true; iat: number; exp: number;
|
|
};
|
|
function validWorker(worker: unknown): worker is WorkspaceWorker {
|
|
if (!worker || typeof worker !== "object" || Array.isArray(worker)) return false;
|
|
const value = worker as Record<string, unknown>;
|
|
return value.provider === "railway" && ["projectId", "environmentId", "serviceId", "deploymentId"].every(key =>
|
|
typeof value[key] === "string" && value[key].length > 0 && value[key].length <= 200);
|
|
}
|
|
export function currentWorkspaceWorker(env: NodeJS.ProcessEnv = process.env): WorkspaceWorker | undefined {
|
|
const ids = [env.RAILWAY_PROJECT_ID, env.RAILWAY_ENVIRONMENT_ID, env.RAILWAY_SERVICE_ID, env.RAILWAY_DEPLOYMENT_ID];
|
|
if (ids.some(id => !id?.trim())) return undefined;
|
|
return { provider: "railway", projectId: ids[0]!, environmentId: ids[1]!, serviceId: ids[2]!, deploymentId: ids[3]! };
|
|
}
|
|
export function verifyWorkspaceWorkerStopReceipt(
|
|
receipt: string, binding: WorkspaceStopBinding, env: NodeJS.ProcessEnv = process.env, now = new Date(),
|
|
): WorkspaceWorkerStopProof | null {
|
|
try {
|
|
if (receipt.length > 16_000 || !validWorker(binding.worker)) return null;
|
|
const parts = receipt.split(".");
|
|
if (parts.length !== 3 || parts.some(part => !/^[A-Za-z0-9_-]+$/.test(part))) return null;
|
|
const header = JSON.parse(Buffer.from(parts[0], "base64url").toString("utf8"));
|
|
if (header.alg !== "EdDSA" || header.typ !== "paperclip-workspace-worker-stop+jwt" || typeof header.kid !== "string") return null;
|
|
if (!verify(null, Buffer.from(`${parts[0]}.${parts[1]}`, "ascii"), cloudRuntimePublicKeyForKid(env, header.kid), Buffer.from(parts[2], "base64url"))) return null;
|
|
const proof = JSON.parse(Buffer.from(parts[1], "base64url").toString("utf8")) as WorkspaceWorkerStopProof;
|
|
const seconds = Math.floor(now.getTime()/1000);
|
|
if (proof.v !== 1 || proof.iss !== "paperclip-cloud" || proof.aud !== "paperclip-workspace-worker-stop/v1"
|
|
|| !env.PAPERCLIP_CLOUD_STACK_ID || proof.sub !== env.PAPERCLIP_CLOUD_STACK_ID || proof.stopped !== true
|
|
|| proof.runId !== binding.runId || proof.ownerToken !== binding.ownerToken
|
|
|| !Number.isInteger(proof.iat) || !Number.isInteger(proof.exp) || proof.iat > seconds+30
|
|
|| proof.exp <= seconds || proof.exp <= proof.iat || proof.exp-proof.iat > 60
|
|
|| !validWorker(proof.worker) || Object.keys(binding.worker).some(key => proof.worker[key as keyof WorkspaceWorker] !== binding.worker[key as keyof WorkspaceWorker])) return null;
|
|
return proof;
|
|
} catch { return null; }
|
|
}
|
|
export type WorkspaceWorkerStopCheck =
|
|
| { status: "verified"; proof: WorkspaceWorkerStopProof }
|
|
| { status: "pending" }
|
|
| { status: "unsupported" };
|
|
|
|
|
|
export async function requestWorkspaceWorkerStopProof(binding: WorkspaceStopBinding, options: {
|
|
env?: NodeJS.ProcessEnv; fetchImpl?: typeof fetch; now?: Date;
|
|
} = {}): Promise<WorkspaceWorkerStopCheck> {
|
|
const env = options.env ?? process.env;
|
|
const context = getCloudStackContext(env);
|
|
const token = env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN?.trim();
|
|
if (!context?.stackId || !context.cloudOrigin || !token || !validWorker(binding.worker)) return { status: "unsupported" };
|
|
let origin: URL;
|
|
try { origin = new URL(context.cloudOrigin); } catch { return { status: "unsupported" }; }
|
|
if (origin.protocol !== "https:" || origin.username || origin.password) return { status: "unsupported" };
|
|
try {
|
|
const response = await (options.fetchImpl ?? fetch)(`${origin.origin}/v1/tenant/workspace-worker-stop`, {
|
|
method: "POST", redirect: "error", headers: { authorization: `Bearer ${token}`, "x-paperclip-cloud-stack-id": context.stackId, "content-type": "application/json" },
|
|
body: JSON.stringify(binding), signal: AbortSignal.timeout(10_000),
|
|
});
|
|
if (!response.ok) return { status: "pending" };
|
|
const payload = await response.json() as { receipt?: unknown };
|
|
const proof = typeof payload?.receipt === "string"
|
|
? verifyWorkspaceWorkerStopReceipt(payload.receipt, binding, env, options.now) : null;
|
|
return proof ? { status: "verified", proof } : { status: "pending" };
|
|
} catch { return { status: "pending" }; }
|
|
}
|