mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native execution saves an accepted result before it finishes
workspace export and status projection.
> - The upgrade report currently recognizes only a live execution in the
observed phase.
> - A durable accepted result can therefore block an upgrade after its
execution lease is released.
> - Existing finalization owners already resume that saved work with
task binding, physical worker stop proof, and an atomic owner claim.
> - This pull request lets the report inspect those existing contracts
without taking authority.
> - Qualified remote work can continue through a normal server
replacement.
## Linked Issues or Issue Description
Refs #18037 and #19596.
**What happened?**
An accepted remote native result remains in result acceptance, workspace
finalization, retry, or committed projection. The upgrade report marks
it blocking because it is no longer a result-less observed execution
with a live execution lease.
**Expected behavior**
Report a saved result as recoverable only when the existing replacement
owners can resume it. Preserve all task, result, contract, remote
workspace, provider lease, physical owner, decision, and workspace
barrier checks. Keep unsupported or unknown work blocking.
**Steps to reproduce**
Run the new PostgreSQL post-result readiness fixture. It persists a real
encrypted Runner header and an accepted terminal result with a bound
remote workspace. The unchanged report returns one blocker.
**Paperclip version or commit**
Regression base: ee6c101854. Related shutdown recovery is in #19596.
## What Changed
- Delegate saved-result eligibility to the native finalization owner.
- Require the accepted terminal result and contract, current task
binding, remote authority, retained provider lease or successful
workspace receipt, and a supported physical worker identity.
- Require an applied decision and assessment for committed projection.
Keep finalizer leases beyond the supported outage budget blocking.
- Reuse the existing result recovery binding with selected metadata. Do
not load complete run or result payloads for the new observation.
- Keep real signed stop verification, owner compare-and-swap, retry
deadlines, and actual finalization with their existing owners.
- Share a two-second scheduling budget across saved-result inspections,
leaving unchecked candidates blocking after the budget expires.
- Add PostgreSQL, encrypted-header, signed stop-proof, competing-owner,
projection, and independent-work cases. Clarify the operator contract.
## Verification
- The valid accepted-result fixture reproduced the original blocking
report, then passed with this change.
- The initial eight post-result cases passed against PostgreSQL. They
include a real Ed25519 stop receipt, the existing verifier, a competing
PostgreSQL ownership claim, one copyback, and receipt reuse.
- Seventeen focused existing PostgreSQL shutdown and workspace ownership
cases passed.
- A broader five-file run passed 91 cases. Its one failure was an
invalid negative fixture: PostgreSQL correctly refused a contract change
protected by the accepted-result foreign key. The test now asserts that
refusal.
- A subsequent standalone run timed out in PostgreSQL bootstrap before
tests ran. The final run used the existing shared PostgreSQL command and
isolated per-suite template databases: all 92 tests passed across five
files (159.95 seconds). It covers both readiness suites, platform stop
proof, workspace finalization, and graceful shutdown. That scope passed
on `ce4e3995b6`. A follow-up restricts provider leases to the active
status and adds released/unknown-status controls. The new fixture now
uses the existing PostgreSQL bootstrap timeout constant. The affected
eight-case rerun passed on `850859b44d`. The explicit configured-stack
guard also passed its six platform-proof cases. Broad local source
typecheck still has stale borrowed dependency declarations; fresh CI
supplies that coverage.
- Review's locked-query regression reproduced RED: three saved-result
candidates took 6.081 seconds. With the shared scheduling budget, both
readiness suites passed all 16 cases (49.36 seconds) on final code
committed as `70b0b7b8f7`. The lock case leaves all three candidates
blocking, releases its connection, and observes all three as recoverable
after the lock is released.
- Greptile reviewed final `70b0b7b8f7` at 5/5 with no open threads.
Current-head CI typecheck, build, tests, and native review are pending.
No hosted replacement or enrollment acceptance is claimed.
## Risks
A false positive could select an unsafe upgrade. Missing evidence and
read errors remain blocking. Readiness does not stop admission, detach
an agent, acquire a lease, request a stop receipt, or authorize
takeover. The replacement still verifies physical stop and rechecks
current task ownership before copyback. Saved results retaining a
previous controller boot remain conservatively blocking, including
finalizers resumed by a later boot. This PR does not change enrollment
policy or migrate data.
## Model Used
OpenAI GPT-6.1, model ID `gpt-6.1-sol`, with reasoning, tool use, and
code execution. The host did not expose the context-window size.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
97 lines
6.3 KiB
TypeScript
97 lines
6.3 KiB
TypeScript
import { verify } from "node:crypto";
|
|
import { cloudRuntimePublicKeyForKid } from "../cloud-runtime-identity.js";
|
|
import { getCloudStackContext } from "../cloud-instance.js";
|
|
|
|
/** Deployment identity covers the controller and every orphaned tar/Git child. */
|
|
export type WorkspaceWorker = {
|
|
provider: "railway"; projectId: string; environmentId: string; serviceId: string; deploymentId: string;
|
|
};
|
|
export type WorkspaceStopBinding = { runId: string; ownerToken: string; worker: WorkspaceWorker };
|
|
export type WorkspaceWorkerStopProof = WorkspaceStopBinding & {
|
|
v: 1; iss: "paperclip-cloud"; aud: "paperclip-workspace-worker-stop/v1"; sub: string;
|
|
stopped: true; iat: number; exp: number;
|
|
};
|
|
function validWorker(worker: unknown): worker is WorkspaceWorker {
|
|
if (!worker || typeof worker !== "object" || Array.isArray(worker)) return false;
|
|
const value = worker as Record<string, unknown>;
|
|
return value.provider === "railway" && ["projectId", "environmentId", "serviceId", "deploymentId"].every(key =>
|
|
typeof value[key] === "string" && value[key].length > 0 && value[key].length <= 200);
|
|
}
|
|
export function isRetainedWorkspaceWorkerStopProof(value: unknown, runId: string): value is WorkspaceWorkerStopProof {
|
|
if (!value || typeof value !== "object" || Array.isArray(value)) return false;
|
|
const proof = value as Record<string, unknown>;
|
|
return proof.v === 1 && proof.iss === "paperclip-cloud" && proof.aud === "paperclip-workspace-worker-stop/v1"
|
|
&& proof.runId === runId && proof.stopped === true
|
|
&& typeof proof.sub === "string" && proof.sub.length > 0 && proof.sub.length <= 200
|
|
&& typeof proof.ownerToken === "string" && proof.ownerToken.length > 0 && proof.ownerToken.length <= 200
|
|
&& Number.isInteger(proof.iat) && Number.isInteger(proof.exp)
|
|
&& Number(proof.exp) > Number(proof.iat) && Number(proof.exp) - Number(proof.iat) <= 60
|
|
&& validWorker(proof.worker)
|
|
&& Object.keys(proof.worker).every(key => ["provider", "projectId", "environmentId", "serviceId", "deploymentId"].includes(key))
|
|
&& Object.keys(proof).every(key => ["v", "iss", "aud", "sub", "stopped", "iat", "exp", "runId", "ownerToken", "worker"].includes(key));
|
|
}
|
|
|
|
export function currentWorkspaceWorker(env: NodeJS.ProcessEnv = process.env): WorkspaceWorker | undefined {
|
|
const ids = [env.RAILWAY_PROJECT_ID, env.RAILWAY_ENVIRONMENT_ID, env.RAILWAY_SERVICE_ID, env.RAILWAY_DEPLOYMENT_ID];
|
|
if (ids.some(id => !id?.trim())) return undefined;
|
|
return { provider: "railway", projectId: ids[0]!, environmentId: ids[1]!, serviceId: ids[2]!, deploymentId: ids[3]! };
|
|
}
|
|
export function verifyWorkspaceWorkerStopReceipt(
|
|
receipt: string, binding: WorkspaceStopBinding, env: NodeJS.ProcessEnv = process.env, now = new Date(),
|
|
): WorkspaceWorkerStopProof | null {
|
|
try {
|
|
if (receipt.length > 16_000 || !validWorker(binding.worker)) return null;
|
|
const parts = receipt.split(".");
|
|
if (parts.length !== 3 || parts.some(part => !/^[A-Za-z0-9_-]+$/.test(part))) return null;
|
|
const header = JSON.parse(Buffer.from(parts[0], "base64url").toString("utf8"));
|
|
if (header.alg !== "EdDSA" || header.typ !== "paperclip-workspace-worker-stop+jwt" || typeof header.kid !== "string") return null;
|
|
if (!verify(null, Buffer.from(`${parts[0]}.${parts[1]}`, "ascii"), cloudRuntimePublicKeyForKid(env, header.kid), Buffer.from(parts[2], "base64url"))) return null;
|
|
const proof = JSON.parse(Buffer.from(parts[1], "base64url").toString("utf8")) as WorkspaceWorkerStopProof;
|
|
const seconds = Math.floor(now.getTime()/1000);
|
|
if (proof.v !== 1 || proof.iss !== "paperclip-cloud" || proof.aud !== "paperclip-workspace-worker-stop/v1"
|
|
|| !env.PAPERCLIP_CLOUD_STACK_ID || proof.sub !== env.PAPERCLIP_CLOUD_STACK_ID || proof.stopped !== true
|
|
|| proof.runId !== binding.runId || proof.ownerToken !== binding.ownerToken
|
|
|| !Number.isInteger(proof.iat) || !Number.isInteger(proof.exp) || proof.iat > seconds+30
|
|
|| proof.exp <= seconds || proof.exp <= proof.iat || proof.exp-proof.iat > 60
|
|
|| !validWorker(proof.worker) || Object.keys(binding.worker).some(key => proof.worker[key as keyof WorkspaceWorker] !== binding.worker[key as keyof WorkspaceWorker])) return null;
|
|
return proof;
|
|
} catch { return null; }
|
|
}
|
|
export type WorkspaceWorkerStopCheck =
|
|
| { status: "verified"; proof: WorkspaceWorkerStopProof }
|
|
| { status: "pending" }
|
|
| { status: "unsupported" };
|
|
|
|
export function workspaceWorkerStopProofSupported(worker: WorkspaceWorker | undefined, env: NodeJS.ProcessEnv = process.env): boolean {
|
|
const context = getCloudStackContext(env);
|
|
if (!context?.stackId || !context.cloudOrigin || !env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN?.trim() || !validWorker(worker)) return false;
|
|
try {
|
|
const origin = new URL(context.cloudOrigin);
|
|
return origin.protocol === "https:" && !origin.username && !origin.password;
|
|
} catch { return false; }
|
|
}
|
|
|
|
|
|
export async function requestWorkspaceWorkerStopProof(binding: WorkspaceStopBinding, options: {
|
|
env?: NodeJS.ProcessEnv; fetchImpl?: typeof fetch; now?: Date;
|
|
} = {}): Promise<WorkspaceWorkerStopCheck> {
|
|
const env = options.env ?? process.env;
|
|
const context = getCloudStackContext(env);
|
|
const token = env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN?.trim();
|
|
if (!workspaceWorkerStopProofSupported(binding.worker, env) || !context?.stackId || !context.cloudOrigin || !token) return { status: "unsupported" };
|
|
let origin: URL;
|
|
try { origin = new URL(context.cloudOrigin); } catch { return { status: "unsupported" }; }
|
|
if (origin.protocol !== "https:" || origin.username || origin.password) return { status: "unsupported" };
|
|
try {
|
|
const response = await (options.fetchImpl ?? fetch)(`${origin.origin}/v1/tenant/workspace-worker-stop`, {
|
|
method: "POST", redirect: "error", headers: { authorization: `Bearer ${token}`, "x-paperclip-cloud-stack-id": context.stackId, "content-type": "application/json" },
|
|
body: JSON.stringify(binding), signal: AbortSignal.timeout(10_000),
|
|
});
|
|
if (!response.ok) return { status: "pending" };
|
|
const payload = await response.json() as { receipt?: unknown };
|
|
const proof = typeof payload?.receipt === "string"
|
|
? verifyWorkspaceWorkerStopReceipt(payload.receipt, binding, env, options.now) : null;
|
|
return proof ? { status: "verified", proof } : { status: "pending" };
|
|
} catch { return { status: "pending" }; }
|
|
}
|