Files
PaperClipAI/server/src/services/remote-workspace-snapshot-storage.ts
T
Devin FoleyandPaperclip ee556a266d perf(storage): upload workspace snapshots in parallel verified parts (#19917)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - With remote (sandbox) execution, every agent turn ends by saving the
workspace: capture an archive, encrypt it, upload it to object storage,
and verify it.
> - On a busy S3-backed instance, the upload step is the slowest part of
a save. A 1.5 GB workspace can hold a run open for minutes after the
agent finished.
> - The upload step reads the archive several times, uploads through one
stream with 4 parts in flight, and then downloads the whole object again
to verify it.
> - Parts that use the stream path share one process-wide 32 MiB buffer
window, so saves that run at the same time slow each other down.
> - This pull request uploads parts in parallel straight from the
encrypted file, lets the backend verify every part with Content-MD5, and
replaces the full read-back with a size check for those verified
uploads.
> - The benefit is faster saves, a smaller run-finalization window, and
no interference between concurrent saves.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Remote workspace snapshot upload (`uploadWorkspaceSnapshot`) to
S3-compatible storage.

**Subsystem affected**
Server storage (`server/src/storage/*`) and remote workspace snapshot
storage.

**Current behavior**
For one save, the upload step:
1. hashes the archive;
2. reads the archive again to encrypt it into a temporary file;
3. hashes the encrypted file;
4. uploads it as one stream with 4 parts in flight;
5. downloads the whole object to verify it.

On one hosted instance over 7 days (724 saves, median 731 MB), the
upload step ran at a median of 6.5 MB/s. With no other save running it
reached 11–14 MB/s. With six or more saves at the same time it fell to
4.4 MB/s. From the same region and with the same part settings, a plain
client reached 31 MB/s to the same kind of bucket.

**Proposed behavior**
Upload parts in parallel from byte ranges of the encrypted file, with
Content-MD5 on each part. Confirm only the stored size after a verified
upload. Digest the archive in the same pass that encrypts it.

**Reason and benefit**
Saves finish faster and hold their run open for less time. Concurrent
saves no longer serialize on the shared buffer window. A transient part
failure is retried from disk instead of failing the whole save.

**Breaking changes**
None. `StorageProvider.putObject` may now return a receipt. Providers
that return nothing keep the current behavior, including the full
read-back.

Related: none found.

## What Changed

- `storage/types.ts`, `storage/service.ts`: an upload can name its
`sourceFile`, a private file whose bytes are exactly the body.
`putObject` may return `{ integrity: "server_verified_md5" }`, and
`putFile` passes it through as `PutFileResult.integrity`.
- `storage/s3-multipart.ts`: new `putS3MultipartFile`.
- Each part reads its own byte range of the file. Up to 8 parts upload
at once, with no shared buffer window.
  - Each part carries the Content-MD5 of its range.
  - A failed part is resent from disk, up to 3 attempts.
- The first part that runs out of attempts stops its siblings. The
multipart upload is aborted only after every part settles.
- A source file whose size differs from the declared length is refused
before any request.
- `storage/s3-provider.ts`:
- A `sourceFile` larger than 16 MiB uses `putS3MultipartFile`. A smaller
one uses one PUT with Content-MD5.
- These requests use a second client with `requestChecksumCalculation:
"WHEN_REQUIRED"`, so bodies are plain and the backend verifies
Content-MD5 directly. The default client wraps bodies in aws-chunked
framing with a CRC32 trailer.
  - Other uploads are unchanged.
- `services/remote-workspace-snapshot-storage.ts`:
- Encryption digests the plaintext in the same read. An unchanged file
identity (dev, inode, size, mtime, ctime) across that read replaces the
separate digest pass. A change still fails with `archive_changed` before
any upload.
- The encrypted file is still digested from disk, so the recorded hash
covers the exact bytes that the parts read.
- The upload passes `sourceFile`. When the provider reports
`server_verified_md5`, a HEAD confirms the stored size instead of
downloading the object.
  - The full read-back stays for:
    - other providers (local disk);
    - recovery of earlier attempts;
    - the caller that keeps a verified archive file.

## Verification

- `pnpm --filter @paperclipai/server typecheck`: exit 0.
- `vitest run src/__tests__/storage-s3-multipart.test.ts
src/services/remote-workspace-snapshot-storage.test.ts`: 59 passed. New
tests:
- every part carries the Content-MD5 of its range, and 8 parts are in
flight;
  - a failed part is resent and the upload completes;
- a part that keeps failing aborts the upload only after in-flight
siblings settle;
  - a size mismatch is refused before any request;
- a small file goes as one PUT with Content-MD5, through a real HTTP
server;
- the read-back is skipped only for a server-verified upload, and the
snapshot still restores;
  - the read-back stays when the caller keeps the read-back file;
  - a stored size mismatch is rejected;
  - an archive that changes during encryption is rejected before upload.
- With the source change reverted, the three new snapshot tests fail.
- Real S3-compatible bucket (a Railway bucket, run from a workstation):
- A 3 MiB single PUT stored exactly: HEAD size, ETag and read-back
SHA-256 all matched.
  - A 96 MiB, 13-part upload stored exactly, with the same checks.
  - A part with a wrong Content-MD5 was rejected with `BadDigest`.
- The same backend stored a wrong `x-amz-checksum-sha256` without
checking it. That is why this change uses Content-MD5.
- Measured from a container in the app's region, 1 GiB with the app's
settings (8 MiB parts, 4 in flight), to the same kind of bucket: 31
MB/s. With 16 in flight: 69 MB/s.
- 77 storage-related suites: the only failures are the same 19 that fail
on unmodified `master` in this checkout. They are in 4 files: missing
bundled connector skills and an unbuilt local runner daemon.

## Risks

- Integrity now depends on the backend enforcing Content-MD5 for file
uploads. AWS S3 and the tested S3-compatible backend enforce it. A
backend that ignores Content-MD5 would accept corrupted parts without a
read-back. A later restore still fails closed: AES-GCM authentication
and the plaintext SHA-256 are checked on load.
- Up to 8 parallel part requests per save, instead of 4 shared ones.
Memory per part is a 64 KiB stream buffer.
- Archive-change detection now uses file identity instead of a second
digest. The archive lives in a private temporary directory that the
capture step creates.
- Medium risk overall: the change is limited to file-sourced S3 uploads
and the snapshot verify step.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs, S3 API probes). AI-assisted
investigation, change and tests, with human direction.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
docs describe this path)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-07 14:04:49 -08:00

367 lines
24 KiB
TypeScript

import { createCipheriv, createDecipheriv, createHash, randomBytes, randomUUID } from "node:crypto";
import { constants, createReadStream, createWriteStream } from "node:fs";
import { link, lstat, mkdtemp, open, realpath, rm } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { pipeline } from "node:stream/promises";
import { Transform } from "node:stream";
import { isDeepStrictEqual } from "node:util";
import { and, eq } from "drizzle-orm";
import { executionWorkspaceSnapshots, type Db } from "@paperclipai/db";
import { localEncryptedProvider } from "../secrets/local-encrypted-provider.js";
import type { StoredSecretVersionMaterial } from "../secrets/types.js";
import type { StorageService } from "../storage/types.js";
import { notifyRemoteWorkspaceSnapshots, REMOTE_WORKSPACE_SNAPSHOT_GRACE_MS } from "./remote-workspace-snapshot-retention.js";
import { WORKSPACE_ARCHIVE_IO_BUFFER_BYTES } from "./remote-workspace-archive-io.js";
const SCHEMA = "paperclip.workspace-snapshot.v1";
const KEY_SCHEMA = "paperclip.workspace-snapshot-key.v1";
const MAGIC = Buffer.from("PCWSE1\n");
const FORMAT = "paperclip.remote-workspace-archive.v1";
const DOWNLOAD_IDLE_TIMEOUT_MS = 5 * 60_000;
type SnapshotRow = typeof executionWorkspaceSnapshots.$inferSelect;
export interface WorkspaceSnapshotRef {
schema: typeof SCHEMA;
id: string;
companyId: string;
workspaceId: string;
generation: number;
objectKey: string;
sha256: string;
byteSize: number;
key: StoredSecretVersionMaterial;
manifestDigest: string;
}
type UploadAttempt = Omit<WorkspaceSnapshotRef, "schema" | "companyId" | "workspaceId" | "generation" | "id">;
function failure(reason: string): never { throw new Error(`remote_workspace_snapshot_${reason}`); }
const aad = (ref: Pick<WorkspaceSnapshotRef, "companyId" | "workspaceId" | "id" | "generation">) =>
Buffer.from(JSON.stringify([SCHEMA, ref.companyId, ref.workspaceId, ref.id, ref.generation]));
const scope = (input: { companyId: string; workspaceId: string; snapshotId: string }) => and(
eq(executionWorkspaceSnapshots.id, input.snapshotId), eq(executionWorkspaceSnapshots.companyId, input.companyId),
eq(executionWorkspaceSnapshots.executionWorkspaceId, input.workspaceId),
);
function reference(row: SnapshotRow): WorkspaceSnapshotRef {
if (!row.objectKey || !row.sha256 || !row.byteSize || !row.key || !row.manifestDigest) failure("not_uploaded");
const value = { schema: SCHEMA, id: row.id, companyId: row.companyId, workspaceId: row.executionWorkspaceId,
generation: row.generation, objectKey: row.objectKey!, sha256: row.sha256!, byteSize: row.byteSize!, key: row.key!, manifestDigest: row.manifestDigest! } as WorkspaceSnapshotRef;
validateReference(value);
return value;
}
function validateReference(ref: WorkspaceSnapshotRef) {
if (ref.schema !== SCHEMA || !Number.isSafeInteger(ref.generation) || ref.generation < 1
|| !ref.objectKey.startsWith(`${ref.companyId}/workspace-snapshots/${ref.workspaceId}/${ref.id}/`)
|| !/^[a-f0-9]{64}$/.test(ref.sha256) || !/^[a-f0-9]{64}$/.test(ref.manifestDigest)
|| !Number.isSafeInteger(ref.byteSize) || ref.byteSize < MAGIC.length + 12 + 16
|| !ref.key || typeof ref.key !== "object") failure("reference_invalid");
}
async function digestFile(file: string, signal?: AbortSignal) {
const digest = createHash("sha256"); let byteSize = 0;
for await (const part of createReadStream(file, { signal, highWaterMark: WORKSPACE_ARCHIVE_IO_BUFFER_BYTES })) { digest.update(part); byteSize += part.length; }
return { sha256: digest.digest("hex"), byteSize };
}
async function verifyStoredSize(storage: StorageService, ref: WorkspaceSnapshotRef, signal?: AbortSignal) {
const head = await storage.headObject(ref.companyId, ref.objectKey, { signal });
if (!head.exists || head.contentLength !== ref.byteSize) failure("object_size_mismatch");
}
async function verifyObject(storage: StorageService, ref: WorkspaceSnapshotRef, signal?: AbortSignal) {
const object = await storage.getObject(ref.companyId, ref.objectKey, { signal });
const digest = createHash("sha256"); let size = 0;
try {
for await (const chunk of object.stream) {
signal?.throwIfAborted(); size += chunk.length;
if (size > ref.byteSize) failure("object_size_mismatch");
digest.update(chunk);
}
if (size !== ref.byteSize || digest.digest("hex") !== ref.sha256) failure("object_hash_mismatch");
} finally { object.stream.destroy(); }
}
export async function beginWorkspaceSnapshot(db: Db, input: {
companyId: string; workspaceId: string; operationId: string; runId?: string | null;
parentSnapshotId?: string | null; generation: number; epoch: number;
}) {
if (!Number.isSafeInteger(input.generation) || input.generation < 1 || !Number.isSafeInteger(input.epoch) || input.epoch < 0) failure("generation_invalid");
await db.transaction(async tx => {
await notifyRemoteWorkspaceSnapshots(tx, `workspace:${input.workspaceId}`);
await tx.insert(executionWorkspaceSnapshots).values({ companyId: input.companyId, executionWorkspaceId: input.workspaceId,
sourceOperationId: input.operationId, sourceRunId: input.runId ?? null, parentSnapshotId: input.parentSnapshotId ?? null,
generation: input.generation, epoch: input.epoch, format: FORMAT,
deleteAfter: new Date(Date.now() + REMOTE_WORKSPACE_SNAPSHOT_GRACE_MS) }).onConflictDoNothing();
});
const [row] = await db.select().from(executionWorkspaceSnapshots).where(and(
eq(executionWorkspaceSnapshots.companyId, input.companyId), eq(executionWorkspaceSnapshots.executionWorkspaceId, input.workspaceId),
eq(executionWorkspaceSnapshots.sourceOperationId, input.operationId),
));
if (!row || row.generation !== input.generation || row.epoch !== input.epoch || row.parentSnapshotId !== (input.parentSnapshotId ?? null)
|| row.sourceRunId !== (input.runId ?? null)) failure("operation_binding_changed");
return row!;
}
export async function getWorkspaceSnapshot(db: Db, input: { companyId: string; workspaceId: string; snapshotId: string }) {
const [row] = await db.select().from(executionWorkspaceSnapshots).where(scope(input));
return row && ["uploaded", "committed", "superseded"].includes(row.status) ? reference(row) : null;
}
export async function recoverWorkspaceSnapshotArchive(input: {
db: Db; storage: StorageService; snapshotId: string; companyId: string; workspaceId: string;
targetPath: string; signal?: AbortSignal;
}): Promise<WorkspaceSnapshotRef | null> {
const [row] = await input.db.select().from(executionWorkspaceSnapshots).where(scope(input));
if (!row) failure("missing");
if (["uploaded", "committed", "superseded"].includes(row!.status)) {
const ref = reference(row!);
await loadWorkspaceSnapshotArchive({ ...input, reference: ref });
return ref;
}
if (row!.status !== "pending") failure("not_writable");
const attempts = Array.isArray(row!.metadata?.uploads) ? row!.metadata!.uploads as UploadAttempt[] : [];
for (const attempt of attempts) {
input.signal?.throwIfAborted();
const ref: WorkspaceSnapshotRef = { schema: SCHEMA, ...attempt, id: row!.id, companyId: row!.companyId,
workspaceId: row!.executionWorkspaceId, generation: row!.generation };
validateReference(ref);
if (!(await input.storage.headObject(ref.companyId, ref.objectKey, { signal: input.signal })).exists) continue;
await loadWorkspaceSnapshotArchive({ ...input, reference: ref });
return ref;
}
return null;
}
type WorkspaceSnapshotUpload = {
db: Db; storage: StorageService; snapshotId: string; companyId: string; workspaceId: string;
archivePath: string; manifestDigest: string; signal?: AbortSignal; verifiedArchivePath?: string;
// Only an owner with an immutable logical source may replace an unavailable,
// unpublished capture. A recoverable or concurrently published attempt wins,
// and its returned manifest may differ from the new physical capture.
unpublishedRecapture?: { sourceOperationId: string };
};
/** Upload attempts are journaled before remote I/O. The caller owns publication. */
export async function uploadWorkspaceSnapshot(input: WorkspaceSnapshotUpload): Promise<WorkspaceSnapshotRef> {
if (input.verifiedArchivePath === undefined)
return uploadVerifiedWorkspaceSnapshot(input, ref => verifyObject(input.storage, ref, input.signal), { acceptServerVerifiedUpload: true });
input.signal?.throwIfAborted();
const target = input.verifiedArchivePath, parent = path.dirname(target);
if (!path.isAbsolute(target) || path.normalize(target) !== target) failure("readback_parent_invalid");
const owner = await lstat(parent);
const assertParent = async () => {
const current = await lstat(parent);
if (!current.isDirectory() || current.isSymbolicLink() || (current.mode & 0o777) !== 0o700
|| typeof process.getuid === "function" && current.uid !== process.getuid()
|| current.dev !== owner.dev || current.ino !== owner.ino || await realpath(parent) !== parent) failure("readback_parent_invalid");
};
await assertParent();
await lstat(target).then(() => failure("readback_target_exists"), (error: NodeJS.ErrnoException) => {
if (error.code !== "ENOENT") throw error;
});
const scratch = await mkdtemp(path.join(parent, ".workspace-upload-readback-"));
const verified: Array<{ ref: WorkspaceSnapshotRef; file: string }> = [];
const verify = async (ref: WorkspaceSnapshotRef) => {
if (verified.some(saved => isDeepStrictEqual(saved.ref, ref))) return;
const file = path.join(scratch, `${randomUUID()}.pcws`);
await loadWorkspaceSnapshotArchive({ storage: input.storage, reference: ref, targetPath: file,
signal: input.signal, temporaryParent: scratch });
verified.push({ ref: structuredClone(ref), file });
};
try {
// This caller keeps the read-back file, so every upload is read back.
const ref = await uploadVerifiedWorkspaceSnapshot(input, verify, { acceptServerVerifiedUpload: false });
await verify(ref);
const file = verified.find(saved => isDeepStrictEqual(saved.ref, ref))!.file;
const wrapped = await snapshotKey(ref), handle = await open(file, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK);
try {
const before = await handle.stat(), digest = createHash("sha256"); let bytes = 0;
if (!before.isFile() || (before.mode & 0o777) !== 0o600 || before.nlink !== 1
|| typeof process.getuid === "function" && before.uid !== process.getuid()) failure("readback_file_changed");
for await (const chunk of createReadStream(file, { fd: handle.fd, autoClose: false, signal: input.signal, highWaterMark: WORKSPACE_ARCHIVE_IO_BUFFER_BYTES })) {
digest.update(chunk); bytes += chunk.length;
}
const after = await handle.stat(), named = await lstat(file);
if (!named.isFile() || named.isSymbolicLink() || named.dev !== before.dev || named.ino !== before.ino
|| named.nlink !== 1 || named.mode !== before.mode || after.size !== before.size
|| after.mtimeMs !== before.mtimeMs || after.ctimeMs !== before.ctimeMs
|| bytes !== wrapped.plaintextBytes || digest.digest("hex") !== wrapped.plaintextSha256) failure("readback_file_changed");
await assertParent(); input.signal?.throwIfAborted();
await link(file, target);
} finally { await handle.close(); }
return ref;
} finally { await rm(scratch, { recursive: true, force: true }); }
}
async function uploadVerifiedWorkspaceSnapshot(input: WorkspaceSnapshotUpload, verify: (ref: WorkspaceSnapshotRef) => Promise<void>,
options: { acceptServerVerifiedUpload: boolean }): Promise<WorkspaceSnapshotRef> {
const [row] = await input.db.select().from(executionWorkspaceSnapshots).where(scope(input));
if (!row) failure("missing");
if (input.unpublishedRecapture && row!.sourceOperationId !== input.unpublishedRecapture.sourceOperationId) failure("operation_binding_changed");
if (["uploaded", "committed", "superseded"].includes(row!.status)) {
const saved = reference(row!);
if (saved.manifestDigest !== input.manifestDigest) failure("operation_content_changed");
await verify(saved); return saved;
}
if (row!.status !== "pending" || !/^[a-f0-9]{64}$/.test(input.manifestDigest)) failure("not_writable");
const recapture = Boolean(input.unpublishedRecapture);
const requireUnpublished = (current: SnapshotRow) => {
if (current.sourceOperationId !== input.unpublishedRecapture!.sourceOperationId
|| current.objectKey !== null || current.sha256 !== null || current.byteSize !== null
|| current.key !== null || current.manifestDigest !== null) failure("not_unpublished");
};
if (recapture) requireUnpublished(row!);
const identity = { id: row!.id, companyId: row!.companyId, workspaceId: row!.executionWorkspaceId, generation: row!.generation };
const publish = async (attempt: UploadAttempt) => {
const updated = await input.db.transaction(async tx => {
await notifyRemoteWorkspaceSnapshots(tx, `workspace:${input.workspaceId}`);
return (await tx.update(executionWorkspaceSnapshots).set({ ...attempt, status: "uploaded", updatedAt: new Date() })
.where(and(scope(input), eq(executionWorkspaceSnapshots.status, "pending"))).returning())[0];
});
const winning = updated ?? (await input.db.select().from(executionWorkspaceSnapshots).where(scope(input)))[0];
if (!winning || !["uploaded", "committed", "superseded"].includes(winning.status)) failure("publication_changed");
const saved = reference(winning!);
if (!recapture && saved.manifestDigest !== input.manifestDigest) failure("operation_content_changed");
if (saved.objectKey !== attempt.objectKey) await verify(saved);
return saved;
};
// Recover a completed upload even when the previous process lost its reply
// and its scratch directory. Failed attempts remain referenced for cleanup.
const availableAttempt = async (attempts: UploadAttempt[]) => {
for (const attempt of attempts) {
input.signal?.throwIfAborted();
if (!recapture && attempt.manifestDigest !== input.manifestDigest) failure("operation_content_changed");
const ref = { schema: SCHEMA, ...identity, ...attempt } as WorkspaceSnapshotRef;
validateReference(ref);
if ((await input.storage.headObject(ref.companyId, ref.objectKey, { signal: input.signal })).exists) {
await verify(ref); return attempt;
}
}
return null;
};
const attempts = Array.isArray(row!.metadata?.uploads) ? row!.metadata!.uploads as UploadAttempt[] : [];
const available = await availableAttempt(attempts);
if (available) return publish(available);
const scratch = await mkdtemp(path.join(os.tmpdir(), "pc-workspace-encrypt-"));
try {
const encrypted = path.join(scratch, "archive.encrypted"), dataKey = randomBytes(32), iv = randomBytes(12);
const cipher = createCipheriv("aes-256-gcm", dataKey, iv); cipher.setAAD(aad(identity));
const header = await open(encrypted, "wx", 0o600);
try { await header.writeFile(Buffer.concat([MAGIC, iv])); } finally { await header.close(); }
// One read of the archive both digests and encrypts it. An unchanged file
// identity across that read replaces the separate digest pass.
const before = await lstat(input.archivePath);
const incomingDigest = createHash("sha256"); let incomingBytes = 0;
const observed = new Transform({ transform(bytes, _encoding, callback) { incomingDigest.update(bytes); incomingBytes += bytes.length; callback(null, bytes); } });
await pipeline(createReadStream(input.archivePath, { signal: input.signal, highWaterMark: WORKSPACE_ARCHIVE_IO_BUFFER_BYTES }), observed, cipher, createWriteStream(encrypted, { flags: "a", mode: 0o600, highWaterMark: WORKSPACE_ARCHIVE_IO_BUFFER_BYTES }), { signal: input.signal });
const after = await lstat(input.archivePath);
if (!before.isFile() || before.isSymbolicLink() || incomingBytes !== before.size || after.dev !== before.dev || after.ino !== before.ino
|| after.size !== before.size || after.mtimeMs !== before.mtimeMs || after.ctimeMs !== before.ctimeMs) failure("archive_changed");
const plaintext = { sha256: incomingDigest.digest("hex"), byteSize: incomingBytes };
const footer = await open(encrypted, "a"); try { await footer.writeFile(cipher.getAuthTag()); } finally { await footer.close(); }
const { material } = await localEncryptedProvider.createVersion({ value: JSON.stringify({ schema: KEY_SCHEMA,
...identity, dataKey: dataKey.toString("base64"), plaintextSha256: plaintext.sha256, plaintextBytes: plaintext.byteSize }) });
dataKey.fill(0);
// Digest the encrypted file as stored: upload parts are read from these bytes.
const content = await digestFile(encrypted, input.signal);
const attempt: UploadAttempt = { objectKey: `${identity.companyId}/workspace-snapshots/${identity.workspaceId}/${identity.id}/${randomUUID()}.pcws`,
...content, key: material, manifestDigest: input.manifestDigest };
// Recheck after encryption, then bind admission to the unchanged ledger.
// Remote I/O stays outside the row lock. An older upload completing after
// this check can still win the pending-to-uploaded publication below.
if (recapture) {
const recovered = await availableAttempt(attempts);
if (recovered) return publish(recovered);
}
const recorded = await input.db.transaction(async tx => {
const [current] = await tx.select().from(executionWorkspaceSnapshots).where(scope(input)).for("update");
if (!current || current.status !== "pending") return null;
const previous = Array.isArray(current.metadata?.uploads) ? current.metadata.uploads as UploadAttempt[] : [];
if (recapture) {
requireUnpublished(current);
if (!isDeepStrictEqual(previous, attempts)) failure("upload_attempts_changed");
} else if (previous.some(saved => saved.manifestDigest !== input.manifestDigest)) failure("operation_content_changed");
await notifyRemoteWorkspaceSnapshots(tx, `workspace:${input.workspaceId}`);
await tx.update(executionWorkspaceSnapshots).set({ metadata: { ...current.metadata, uploads: [...previous, attempt] }, updatedAt: new Date() }).where(scope(input));
return true;
});
if (!recorded) {
const saved = await getWorkspaceSnapshot(input.db, input);
if (!saved || (!recapture && saved.manifestDigest !== input.manifestDigest)) failure("publication_changed");
await verify(saved); return saved;
}
// The object key is unique per attempt, so a late losing uploader cannot
// overwrite an object already used by a committed generation.
const stored = await input.storage.putFile({ companyId: identity.companyId, namespace: "workspace-snapshots", objectKey: attempt.objectKey,
originalFilename: null, contentType: "application/octet-stream", body: createReadStream(encrypted, { highWaterMark: WORKSPACE_ARCHIVE_IO_BUFFER_BYTES }),
byteSize: attempt.byteSize, sha256: attempt.sha256, sourceFile: encrypted, signal: input.signal });
const uploaded = { schema: SCHEMA, ...identity, ...attempt } as WorkspaceSnapshotRef;
// The backend checked every part of this upload against digests of these
// exact bytes, so only its size needs confirming. Any other provider, and
// a caller that needs the bytes back, still reads the object back.
if (options.acceptServerVerifiedUpload && stored.integrity === "server_verified_md5") await verifyStoredSize(input.storage, uploaded, input.signal);
else await verify(uploaded);
if (recapture) {
const recovered = await availableAttempt(attempts);
if (recovered) return publish(recovered);
}
return await publish(attempt);
} finally { await rm(scratch, { recursive: true, force: true }); }
}
async function snapshotKey(ref: WorkspaceSnapshotRef) {
validateReference(ref);
const wrapped = JSON.parse(await localEncryptedProvider.resolveVersion({ material: ref.key, externalRef: null }));
if (wrapped.schema !== KEY_SCHEMA || wrapped.companyId !== ref.companyId || wrapped.workspaceId !== ref.workspaceId
|| wrapped.id !== ref.id || wrapped.generation !== ref.generation || !/^[a-f0-9]{64}$/.test(wrapped.plaintextSha256 ?? "")
|| !Number.isSafeInteger(wrapped.plaintextBytes) || wrapped.plaintextBytes < 0) failure("key_binding_invalid");
return wrapped;
}
/** Authentication finishes in private scratch before any plaintext is published. */
export async function loadWorkspaceSnapshotArchive(input: {
storage: StorageService; reference: WorkspaceSnapshotRef; targetPath: string; signal?: AbortSignal; temporaryParent?: string;
}): Promise<void> {
input.signal?.throwIfAborted();
const ref = input.reference, wrapped = await snapshotKey(ref);
const scratch = await mkdtemp(path.join(input.temporaryParent ?? os.tmpdir(), "pc-workspace-decrypt-"));
try {
input.signal?.throwIfAborted();
const ciphertext = path.join(scratch, "ciphertext"), plaintext = path.join(scratch, "plaintext");
let received = 0; const digest = createHash("sha256");
const stalled = new DOMException("remote_workspace_snapshot_download_stalled", "TimeoutError");
const idle = new AbortController();
const signal = input.signal ? AbortSignal.any([input.signal, idle.signal]) : idle.signal;
const timer = setTimeout(() => idle.abort(stalled), DOWNLOAD_IDLE_TIMEOUT_MS);
timer.unref();
try {
const object = await input.storage.getObject(ref.companyId, ref.objectKey, { signal });
const bounded = new Transform({ transform(bytes, _encoding, callback) {
if (bytes.length > 0) timer.refresh();
received += bytes.length; digest.update(bytes);
callback(received > ref.byteSize ? new Error("remote_workspace_snapshot_object_size_mismatch") : null, bytes);
} });
try { await pipeline(object.stream, bounded, createWriteStream(ciphertext, { flags: "wx", mode: 0o600, highWaterMark: WORKSPACE_ARCHIVE_IO_BUFFER_BYTES }), { signal }); }
finally { object.stream.destroy(); }
} catch (error) {
if (signal.aborted && signal.reason === stalled) throw stalled;
throw error;
} finally { clearTimeout(timer); }
if (received !== ref.byteSize || digest.digest("hex") !== ref.sha256) failure("object_hash_mismatch");
const file = await open(ciphertext, "r"), header = Buffer.alloc(MAGIC.length + 12), tag = Buffer.alloc(16);
try { await file.read(header, 0, header.length, 0); await file.read(tag, 0, tag.length, ref.byteSize - tag.length); } finally { await file.close(); }
if (!header.subarray(0, MAGIC.length).equals(MAGIC)) failure("ciphertext_header_invalid");
const key = Buffer.from(wrapped.dataKey, "base64"); if (key.length !== 32) failure("key_invalid");
const decipher = createDecipheriv("aes-256-gcm", key, header.subarray(MAGIC.length));
decipher.setAAD(aad(ref)); decipher.setAuthTag(tag);
try {
await pipeline(createReadStream(ciphertext, { start: header.length, end: ref.byteSize - tag.length - 1, signal: input.signal, highWaterMark: WORKSPACE_ARCHIVE_IO_BUFFER_BYTES }),
decipher, createWriteStream(plaintext, { flags: "wx", mode: 0o600, highWaterMark: WORKSPACE_ARCHIVE_IO_BUFFER_BYTES }), { signal: input.signal });
} finally { key.fill(0); }
const actual = await digestFile(plaintext, input.signal);
if (actual.sha256 !== wrapped.plaintextSha256 || actual.byteSize !== wrapped.plaintextBytes) failure("plaintext_hash_mismatch");
// Publish a complete file without replacing an existing destination. A
// failed/cancelled relay cannot leave a plaintext cache that looks complete.
const staged = path.join(path.dirname(input.targetPath), `.workspace-snapshot-${randomUUID()}.tmp`);
try {
await pipeline(createReadStream(plaintext, { signal: input.signal, highWaterMark: WORKSPACE_ARCHIVE_IO_BUFFER_BYTES }), createWriteStream(staged, { flags: "wx", mode: 0o600, highWaterMark: WORKSPACE_ARCHIVE_IO_BUFFER_BYTES }), { signal: input.signal });
input.signal?.throwIfAborted(); await link(staged, input.targetPath);
} finally { await rm(staged, { force: true }); }
} finally { await rm(scratch, { recursive: true, force: true }); }
}