Files
PaperClipAI/server/src/services/native-runtime/native-workspace-platform-proof.ts
T
a019d00a39 fix(server): wake native workspace finalization from committed work (#17734)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs can accept a result before workspace copy-back finishes.
> - The periodic orphan pass repeatedly tries this recovery, including
cases that require operator action.
> - Copy-back needs exact ownership evidence. A generic timer cannot
grant that authority.
> - This pull request wakes workspace recovery from committed changes
and local owner completion.
> - Saved retries use the existing scheduler. Empty work, completed
barriers, and operator holds become quiet.

## Linked Issues or Issue Description

Refs #17717. This is slice 52 of the recurring-server sleep stack,
targeting master after parent #17717 merged at
`4c56702e06c698cc10d4a86514669aba0f4d9a9b`. Related work: #17709 (status
reassessment), #17687 (unsafe export recovery), and #16478 (workspace
repair holds).

This PR covers native workspace copy-back and validation. Result
commit/projection and resource settlement remain separate recovery
duties. It does not claim full-instance sleep.

## What Changed

- Register a company/run/task-scoped workspace worker. Reconstruct
accepted, unfinished work from existing records. Use saved retry and
lease deadlines.
- Notify from both accepted-result writers (control-plane and PRP
coordinator), workspace receipts, coordinator changes, operator
confirmation, retained workspace repair, and relevant task updates.
Notifications follow the outer transaction.
- Distinguish active local ownership, busy external ownership, pending
Cloud evidence, operator-required evidence, and changed bindings. Only
concrete external ownership/evidence waits receive a scoped retry.
- Preserve advisory locks, signed stop proof, exact binding checks,
physical callback joins, and existing failure policy. Join physical
operations during shutdown.
- Respect the existing heartbeat execution-suppression policy before
discovery and execution; wake suppressed work on settings activation.
- Skip physical workspace recovery in the enabled periodic orphan pass.
Keep direct/startup reconciliation and the remaining result/cleanup
recovery paths.
- Keep routine company-only finalization hints quiet. Ordinary work
requires issue/run scope; explicit settings and owner-release events
keep their company-wide contract.
- Preserve the parent's entire Runner/package tree, profile 80 and all
historical identities. Add the dated obligation, producer, lifecycle and
limitation audit.

## Verification

- Current local source: `4059b24e3ea5c623be54ade18ec9b6ab78eb0289`,
prepared by a normal merge of the current parent. The sole manual
conflict keeps ordinary workspace fanout inside the existing issue/run
guard. All 1,198 parent-only, 15 owned-only and 10 automatic shared
paths are preserved. The complete package tree matches the parent
byte-for-byte.
- Controlled regression: the original workspace-only unscoped fanout
performs three candidate SELECTs for routine company hints. The same
zero-query assertion passes with scoped fanout. A real accepted result
then produces one successful workspace settlement and returns to quiet.
The failed baseline remains in the audit.
- The first complete local group passes **1,693 cases in 14 files**,
including all 19 workspace-worker cases. It covers both accepted-result
producers, physical ownership, export retry/failure, finalization
recovery, arbiter corpus, issue recovery actions, control-plane port,
workspace recovery, heartbeat recovery, startup, retained cleanup and
native execution.
- Separate process-recovery and external-chat response files qualify the
changed recovery/settlement paths without sharing their general test
database with other files. Both pass in full: 410 process-recovery cases
in 162.79 seconds and 273 external-chat cases in 53.79 seconds. The
total is **2,376 distinct passes in 16 complete files**, with no skips.
Normal merge `5c46c52873158529736b8349e4d44658865108ee` adopts the
parent squash with this entire tested child tree unchanged.
- Production server typechecking, build, module boundaries, migration
order and source-profile checks pass. Thirteen dedicated PostgreSQL 17
fixtures use the unchanged migration 413 schema. Embedded-only paths use
the existing recovery helper with fresh per-call fixtures. All temporary
imports are restored byte-exact. No test assertion or timeout is
relaxed.
- Runner validation belongs to the unchanged parent package tree: 130
passing installation/compatibility/transport cases, 22 installed
ACPX/Grok contract cases and 26 native-provider Rust cases. One existing
opt-in closed Pi snapshot case is skipped because its distribution root
is not configured. These are source-scoped checks, with no live provider
qualification claim.
- The owner audit at
`doc/plans/2026-10-26-native-workspace-finalization-sleep-audit.md`
preserves earlier failures and their disposition. Current-head
acceptance: all 57 checks are clean, including protected CI and
security. Greptile gives 5/5 on
`7df5c9c44ea9e171be3c61b4430d3727b10f3337` with no actionable findings.
Earlier scores do not substitute for this result. No Ponytail verdict is
present under the authorized missing-review policy.
- No claim of a full local repository suite, deployed app/database
sleep, expanded Cloud wake coverage or measured cost savings is made.

## Risks

- Supported wake producers are in-process application services. This
adds no cross-replica or independent SQL-writer transport.
- A real unresolved export still needs recovery. Busy owners and pending
Cloud stop evidence retain a 30-second scoped check. No timeout
authorizes copy-back.
- Result projection and resource release still use their existing
recovery. Some existing cleanup failures remain best effort; this PR
does not infer cleanup success from a workspace barrier.
- Cloud alarm coverage and host-sleep safety remain unchanged. No
deployed suspension or cost savings have been measured.
- Parent #17717 is merged. Current base is master at
`4c56702e06c698cc10d4a86514669aba0f4d9a9b`. Fresh current-head CI and
review must pass before this slice merges.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The exact serving model ID and context-window size are not exposed in
this session. No subagent was used for this slice.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 04:40:01 -05:00

74 lines
4.7 KiB
TypeScript

import { verify } from "node:crypto";
import { cloudRuntimePublicKeyForKid } from "../cloud-runtime-identity.js";
import { getCloudStackContext } from "../cloud-instance.js";
/** Deployment identity covers the controller and every orphaned tar/Git child. */
export type WorkspaceWorker = {
provider: "railway"; projectId: string; environmentId: string; serviceId: string; deploymentId: string;
};
export type WorkspaceStopBinding = { runId: string; ownerToken: string; worker: WorkspaceWorker };
export type WorkspaceWorkerStopProof = WorkspaceStopBinding & {
v: 1; iss: "paperclip-cloud"; aud: "paperclip-workspace-worker-stop/v1"; sub: string;
stopped: true; iat: number; exp: number;
};
function validWorker(worker: unknown): worker is WorkspaceWorker {
if (!worker || typeof worker !== "object" || Array.isArray(worker)) return false;
const value = worker as Record<string, unknown>;
return value.provider === "railway" && ["projectId", "environmentId", "serviceId", "deploymentId"].every(key =>
typeof value[key] === "string" && value[key].length > 0 && value[key].length <= 200);
}
export function currentWorkspaceWorker(env: NodeJS.ProcessEnv = process.env): WorkspaceWorker | undefined {
const ids = [env.RAILWAY_PROJECT_ID, env.RAILWAY_ENVIRONMENT_ID, env.RAILWAY_SERVICE_ID, env.RAILWAY_DEPLOYMENT_ID];
if (ids.some(id => !id?.trim())) return undefined;
return { provider: "railway", projectId: ids[0]!, environmentId: ids[1]!, serviceId: ids[2]!, deploymentId: ids[3]! };
}
export function verifyWorkspaceWorkerStopReceipt(
receipt: string, binding: WorkspaceStopBinding, env: NodeJS.ProcessEnv = process.env, now = new Date(),
): WorkspaceWorkerStopProof | null {
try {
if (receipt.length > 16_000 || !validWorker(binding.worker)) return null;
const parts = receipt.split(".");
if (parts.length !== 3 || parts.some(part => !/^[A-Za-z0-9_-]+$/.test(part))) return null;
const header = JSON.parse(Buffer.from(parts[0], "base64url").toString("utf8"));
if (header.alg !== "EdDSA" || header.typ !== "paperclip-workspace-worker-stop+jwt" || typeof header.kid !== "string") return null;
if (!verify(null, Buffer.from(`${parts[0]}.${parts[1]}`, "ascii"), cloudRuntimePublicKeyForKid(env, header.kid), Buffer.from(parts[2], "base64url"))) return null;
const proof = JSON.parse(Buffer.from(parts[1], "base64url").toString("utf8")) as WorkspaceWorkerStopProof;
const seconds = Math.floor(now.getTime()/1000);
if (proof.v !== 1 || proof.iss !== "paperclip-cloud" || proof.aud !== "paperclip-workspace-worker-stop/v1"
|| !env.PAPERCLIP_CLOUD_STACK_ID || proof.sub !== env.PAPERCLIP_CLOUD_STACK_ID || proof.stopped !== true
|| proof.runId !== binding.runId || proof.ownerToken !== binding.ownerToken
|| !Number.isInteger(proof.iat) || !Number.isInteger(proof.exp) || proof.iat > seconds+30
|| proof.exp <= seconds || proof.exp <= proof.iat || proof.exp-proof.iat > 60
|| !validWorker(proof.worker) || Object.keys(binding.worker).some(key => proof.worker[key as keyof WorkspaceWorker] !== binding.worker[key as keyof WorkspaceWorker])) return null;
return proof;
} catch { return null; }
}
export type WorkspaceWorkerStopCheck =
| { status: "verified"; proof: WorkspaceWorkerStopProof }
| { status: "pending" }
| { status: "unsupported" };
export async function requestWorkspaceWorkerStopProof(binding: WorkspaceStopBinding, options: {
env?: NodeJS.ProcessEnv; fetchImpl?: typeof fetch; now?: Date;
} = {}): Promise<WorkspaceWorkerStopCheck> {
const env = options.env ?? process.env;
const context = getCloudStackContext(env);
const token = env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN?.trim();
if (!context?.stackId || !context.cloudOrigin || !token || !validWorker(binding.worker)) return { status: "unsupported" };
let origin: URL;
try { origin = new URL(context.cloudOrigin); } catch { return { status: "unsupported" }; }
if (origin.protocol !== "https:" || origin.username || origin.password) return { status: "unsupported" };
try {
const response = await (options.fetchImpl ?? fetch)(`${origin.origin}/v1/tenant/workspace-worker-stop`, {
method: "POST", redirect: "error", headers: { authorization: `Bearer ${token}`, "x-paperclip-cloud-stack-id": context.stackId, "content-type": "application/json" },
body: JSON.stringify(binding), signal: AbortSignal.timeout(10_000),
});
if (!response.ok) return { status: "pending" };
const payload = await response.json() as { receipt?: unknown };
const proof = typeof payload?.receipt === "string"
? verifyWorkspaceWorkerStopReceipt(payload.receipt, binding, env, options.now) : null;
return proof ? { status: "verified", proof } : { status: "pending" };
} catch { return { status: "pending" }; }
}