Files
PaperClipAI/server/src/services/native-runtime/native-workspace-platform-proof.test.ts
T
DottaandPaperclip c09fb60ca6 fix(recovery): retain worker-stop audit in preparation retries (#19612)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Recovery must preserve saved files before it admits another
execution.
> - A failed preparation can need a verified stop of its previous
workspace worker.
> - Recovery keeps that verified stop receipt in the run profile for
audit.
> - The preparation retry guard treats the retained receipt as unknown
execution evidence.
> - This pull request recognizes a valid same-run audit receipt while it
retains all current execution and file checks.
> - A saved Retry can then continue once after recovery completes.

## Linked Issues or Issue Description

**What happened?**

A native preparation failed before provider dispatch. The recovery
worker verified that the previous workspace worker had stopped and saved
its files. The same HTTP Retry stayed deferred because the run profile
retained `nativeWorkspaceOwnerStopProof`.

**Expected behavior**

The saved Retry should continue once after all preparation, stop, file,
ownership, and authorization checks pass. The old run and its audit
receipt should remain unchanged.

**Steps to reproduce**

1. Fail native preparation with a retained remote workspace and no
provider activity.
2. Let the recovery worker acquire the old workspace owner through
verified platform stop evidence.
3. Submit one HTTP Retry and let preservation complete.
4. Observe that the old guard retains the retry hold even after
saved-file verification passes.

**Paperclip version or commit**

Reproduced against `d728be8fbf`. Related work: #18908, #19021, #19026,
and #19466. This change covers the retained workspace-worker stop audit
left by a later repair.

## What Changed

- Add a typed reader for a host-persisted same-run worker-stop audit
receipt. Keep signature and expiry validation at ownership acquisition.
- Recognize only that validated audit field in failed-preparation
settlement. Keep unknown profiles and active owners blocked.
- Exercise real preservation, HTTP Retry delivery, native dispatch,
saved files, accepted results, and task completion. Add malformed,
foreign-run, unstopped-worker, and active-owner cases.
- Document the distinction between retained audit evidence and current
execution authority.

## Verification

- Failing before the fix: the current-worker preservation case passed;
the stopped-worker case produced zero successor runs after the same
Retry, where one was expected.
- Passing after the fix: 47 recovery cases, 10 native
dispatch/preservation cases, and 6 platform-proof tests. The two
preservation success cases overlap between the first two groups.
- Server typecheck passed all seven layers after the final test changes.
The Runner and SDK dependencies were built in this isolated worktree.
- Final comment-free commit `44f8f33f8d917dddb51852283a479786102eab4d`:
a fresh combined run passed all 61 affected cases in 67.92s. Fresh
server typecheck passed all seven layers.
- The earlier `95d0376c` combined run passed 60 cases and hit one
`socket hang up` in the existing `catalog_provider_warning` HTTP test.
That case passed in a bounded isolated run (12.48s), and the final
61-case combined run passed. The socket failure remains recorded as an
intermittent HTTP harness limitation. No retry or weaker assertion was
added.
- Final commit `44f8f33f8d917dddb51852283a479786102eab4d` completed all
required CI gates: 57 checks succeeded, 3 intentional skips, and the
advisory Paperclip review is neutral with a substantive 5/5 verdict.
Greptile is 5/5 and there are no unresolved review threads. [Final CI
run](https://github.com/paperclipai/paperclip/actions/runs/42293625696).
The follow-up commit only removes six new code-comment lines to follow
CONTRIBUTING.
- The native provider and signed platform API are controlled test
boundaries. The tests use real PostgreSQL, recovery, ownership, file
persistence, retry admission, dispatch, and finalization services.
Deployed recovery remains unverified until normal rollout.

## Risks

- The main risk is mistaking an audit receipt for current stop
authority. The reader checks its exact shape and run binding. The
receipt cannot replace stopped leases, saved-file receipts, settled
preparation, current authorization, or the absence of an active owner or
provider evidence.
- The negative matrix runs with the audit field present. Provider
activity, unknown usage, bad file receipts, Stop, actor changes, and
unsettled leases still block continuation.
- No schema or migration change is required.

## Model Used

OpenAI GPT-6 in Codex. The exact deployment model ID and context-window
size are not exposed in this session. Used reasoning, code editing,
shell execution, and test tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 15:36:23 -06:00

68 lines
5.3 KiB
TypeScript

import { generateKeyPairSync, sign } from "node:crypto";
import { describe, expect, it, vi } from "vitest";
import { verifyWorkspaceWorkerStopReceipt, requestWorkspaceWorkerStopProof, currentWorkspaceWorker, isRetainedWorkspaceWorkerStopProof } from "./native-workspace-platform-proof.js";
const { privateKey, publicKey } = generateKeyPairSync("ed25519");
const env = { PAPERCLIP_CLOUD_STACK_ID: "stack", PAPERCLIP_CLOUD_API_ORIGIN: "https://cloud.example", PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN: "fixture-token", PAPERCLIP_CLOUD_RUNTIME_IDENTITY_JWKS: JSON.stringify({ keys: [{ ...publicKey.export({ format: "jwk" }), kid: "test", alg: "EdDSA", use: "sig" }] }) };
const worker = { provider: "railway" as const, projectId: "project", environmentId: "environment", serviceId: "service", deploymentId: "old-deployment" };
const binding = { runId: "run", ownerToken: "owner", worker };
const now = new Date("2099-01-01T00:00:00Z");
function receipt(overrides = {}) {
const header = Buffer.from(JSON.stringify({ alg: "EdDSA", kid: "test", typ: "paperclip-workspace-worker-stop+jwt" })).toString("base64url");
const payload = Buffer.from(JSON.stringify({ v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1", sub: "stack", ...binding, stopped: true, iat: now.getTime()/1000, exp: now.getTime()/1000+60, ...overrides })).toString("base64url");
return `${header}.${payload}.${sign(null, Buffer.from(`${header}.${payload}`), privateKey).toString("base64url")}`;
}
describe("workspace platform stop proof", () => {
it("accepts only signed proof for the exact stack, owner, run and full stopped deployment", () => {
expect(verifyWorkspaceWorkerStopReceipt(receipt(), binding, env, now)).toMatchObject({ stopped: true, worker });
for (const invalid of [{ sub: "other-stack" }, { runId: "other-run" }, { ownerToken: "other-owner" }, { stopped: false }, { exp: now.getTime()/1000-1 }, { worker: { ...worker, deploymentId: "other" } }, { aud: "paperclip-cloud-control/v1" }]) {
expect(verifyWorkspaceWorkerStopReceipt(receipt(invalid), binding, env, now)).toBeNull();
}
expect(verifyWorkspaceWorkerStopReceipt(receipt().slice(0,-5)+"wrong", binding, env, now)).toBeNull();
});
it("retains verified same-run audit evidence after acquisition expiry without granting new stop authority", () => {
const proof = verifyWorkspaceWorkerStopReceipt(receipt(), binding, env, now)!;
const later = new Date(now.getTime() + 120_000);
expect(verifyWorkspaceWorkerStopReceipt(receipt(), binding, env, later)).toBeNull();
expect(isRetainedWorkspaceWorkerStopProof(proof, binding.runId)).toBe(true);
for (const invalid of [null, [], { stopped: true }, { ...proof, runId: "other-run" },
{ ...proof, stopped: false }, { ...proof, exp: proof.iat }, { ...proof, exp: proof.iat + 61 },
{ ...proof, iat: "100" }, { ...proof, ownerToken: "" }, { ...proof, sub: "" },
{ ...proof, worker: { ...worker, deploymentId: "" } }, { ...proof, worker: { ...worker, provider: "other" } },
{ ...proof, worker: { ...worker, providerStarted: true } }, { ...proof, providerStarted: true }]) {
expect(isRetainedWorkspaceWorkerStopProof(invalid, binding.runId)).toBe(false);
}
});
it("fails closed on live, unknown, unreachable and unsigned platform responses", async () => {
for (const response of [{ stopped: false }, { stopped: true }, { receipt: "forged" }]) {
const fetchImpl = vi.fn().mockResolvedValue(Response.json(response));
expect(await requestWorkspaceWorkerStopProof(binding, { env, now, fetchImpl })).toEqual({ status: "pending" });
}
expect(await requestWorkspaceWorkerStopProof(binding, { env, now, fetchImpl: vi.fn().mockRejectedValue(new Error("offline")) })).toEqual({ status: "pending" });
expect(await requestWorkspaceWorkerStopProof(binding, { env, now, fetchImpl: vi.fn().mockResolvedValue(Response.json({ receipt: receipt() })) })).toMatchObject({ status: "verified", proof: { stopped: true } });
});
it("does not schedule platform verification without a supported authenticated endpoint", async () => {
for (const unavailable of [
{},
{ ...env, PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN: "" },
{ ...env, PAPERCLIP_CLOUD_API_ORIGIN: "http://cloud.example" },
{ ...env, PAPERCLIP_CLOUD_API_ORIGIN: "https://user:password@cloud.example" },
{ ...env, PAPERCLIP_CLOUD_API_ORIGIN: "not a URL" },
]) {
const fetchImpl = vi.fn();
expect(await requestWorkspaceWorkerStopProof(binding, { env: unavailable, now, fetchImpl }))
.toEqual({ status: "unsupported" });
expect(fetchImpl).not.toHaveBeenCalled();
}
});
it("keeps an authenticated but temporarily unavailable verification pending", async () => {
const fetchImpl = vi.fn().mockResolvedValue(new Response(null, { status: 503 }));
expect(await requestWorkspaceWorkerStopProof(binding, { env, now, fetchImpl }))
.toEqual({ status: "pending" });
expect(fetchImpl).toHaveBeenCalledOnce();
});
it("records a complete platform identity, and does not guess legacy ownership", () => {
expect(currentWorkspaceWorker({ RAILWAY_PROJECT_ID: "project", RAILWAY_ENVIRONMENT_ID: "environment", RAILWAY_SERVICE_ID: "service", RAILWAY_DEPLOYMENT_ID: "old-deployment" })).toEqual(worker);
expect(currentWorkspaceWorker({ RAILWAY_DEPLOYMENT_ID: "old-deployment" })).toBeUndefined();
});
});