Files
PaperClipAI/server/src/secrets/local-encrypted-provider.ts
T
af0fe8fe81 Keep personal MCP connections functional while idle tenants sleep (#19427)
## Thinking Path

> - Paperclip manages work for AI agents.
> - Personal MCP connections should remain enabled while an idle tenant
sleeps.
> - Unfinished OAuth, transfer and delivery work must retain its owner
and deadline.
> - Cloud can receive bounded metadata and incoming work while the
tenant remains the company-scoped authorizer.
> - Current identity, revision and delivery acknowledgments establish
when ownership has transferred.
> - This change preserves useful wake, retries and revocation while
making settled ownership inspectable without writes.

## Linked Issues or Issue Description

Refs: #19377 and #19750. Hosted companion:
[paperclip-cloud#1217](https://github.com/paperclipai/paperclip-cloud/pull/1217).

**What happened?** Enabled personal MCP grants and subscriptions could
keep an otherwise idle tenant awake. Ignoring those records would lose
discovery, authorization, delivery and revocation obligations.

**Expected behavior** Unconsumed authorization codes, device requests,
file handoffs and delivery work retain their original deadlines. Settled
connections remain enabled and become quiet after the current external
owner acknowledges responsibility. Incoming useful work resumes normal
authorization and delivery; revocation still takes effect for the
current company.

**Steps to reproduce** Complete normal PKCE/device consent and company
selection, synchronize client/device and revocation state, retain the
connection while idle, then exercise useful tools/events and revocation.
Sleep eligibility requires current incoming/deadline ownership
throughout.

## What Changed

- Add bounded signed client/device metadata and revocation
synchronization with exact revision, identity and ACK checks, startup
recovery and finite legacy-device cutover.
- Preserve company selection, authorization and operation identity.
Register grant work before its INSERT in the same transaction; retain
rollback, expiry and retry behavior.
- Require current managed identity/resource before hosted publication.
Receipt-cleared re-enable must obtain a fresh source ACK before device
creation.
- Reuse read-only settings and inspection budgets; connect MCP owners to
normal startup, drain and shutdown.
- Bound metadata pages and the actual signed request body while
preserving the default limits of other operations.
- Generate migration 0440 after the merged Legacy migration. Preserve
all 438 existing executable migrations and the journal prefix; retain
the normal five newest snapshots.
- Preserve the merged transaction-owner forwarding and regression
coverage from #19750 through normal ancestry integration.

## Verification

- Current own Shared/SDK, database, eval-kernel and Runner builds
passed, together with all seven normal application TypeScript programs.
Missing generated package exports/workspace links were restored through
normal setup; the original setup failures were retained. No source,
compiler option, test assertion or timeout was weakened.
- Normal migration generation changed only the five intended MCP tables;
its SQL is byte-identical to the reviewed MCP delta. Fresh reference
databases and both retained master/Muse histories upgraded through 0440,
preserving historical hashes, row fields, schema and idempotence. Direct
assertions cover the Legacy null default and MCP null/zero defaults. Two
selected history cases passed; the other 21 database cases were
unselected.
- All six current MCP lifecycle cases passed with zero skips: normal
PKCE approval intent, ACK recovery after outage, mutation during
publication, disable/re-enable with original expiry, company-deletion
rollback/commit and key rotation.
- All 81 checks in five current whole files passed with zero skips:
OpenAPI 34, read-only settings 2, diagnostic catalog 25, PostgreSQL
diagnostics 7 and dependency scheduling 13. PostgreSQL fixtures ran
normal migrations before test imports.
- Earlier canonical proof at App `cd1c8c6` / Cloud `9f67531` includes
real 60-second rate and three 15-second quiet windows, useful work and
three revokes. Later paired proof at App `43af514` / Cloud `914eb0f`
plus its reviewed candidate includes normal DCR, PostgreSQL metadata, a
144,017-byte signed publication, actual ACK, cap/default-413 controls
and a real 15-second quiet window. The relevant producer/signing/wire
inputs remain byte-identical in this integration; shared changes are
qualified above. These remain source-scoped observations, not a fresh
full canonical run on the integrated tree.
- All known test owners, process groups and OS sockets closed; an
independent final PostgreSQL audit found no remaining test sessions. One
dependency-test trace recorded 46 opens and 45 close events. Its cause
is unknown; this is not claimed as proof of graceful closure.
- Fresh exact-head remote checks passed: 59 successful and 4 skipped
contexts, with a complete current 5/5 Greptile review and all seven
review threads resolved. Targeted local checks do not constitute a new
whole-repository test run.

## Risks

App and Cloud must agree on identity, revision, namespace, deadlines and
revocation. Partial rollout, unavailable authority or missing ACK must
retain awake debt. Shared startup/shutdown and settings changed since
the earlier canonical profile, so historical observations carry only
within their stated source scope. No deployed Railway/Neon suspension,
Cloud alarm, savings or complete hosted Dot/Muse acceptance is claimed.

## Model Used

OpenAI `gpt-6.1-sol` produced code and verification, with a Codex
coordinator reviewing and integrating the changes. The coordinator's
exact runtime model ID and context-window sizes were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-11-07 02:12:48 -06:00

341 lines
12 KiB
TypeScript

import { createCipheriv, createDecipheriv, createHash, hkdfSync, randomBytes } from "node:crypto";
import { chmodSync, existsSync, linkSync, mkdirSync, readFileSync, statSync, unlinkSync, writeFileSync } from "node:fs";
import path from "node:path";
import { resolveDefaultSecretsKeyFilePath } from "../home-paths.js";
import type {
PreparedSecretVersion,
SecretProviderHealthCheck,
SecretProviderModule,
SecretProviderValidationResult,
StoredSecretVersionMaterial,
} from "./types.js";
import { badRequest } from "../errors.js";
import { SecretProviderClientError } from "./types.js";
interface LocalEncryptedMaterial extends StoredSecretVersionMaterial {
scheme: "local_encrypted_v1";
iv: string;
tag: string;
ciphertext: string;
}
export type LocalEncryptedKeyEnv = Partial<Record<
"PAPERCLIP_SECRETS_MASTER_KEY" | "PAPERCLIP_SECRETS_MASTER_KEY_FILE", string | undefined
>>;
function resolveMasterKeyFilePath(env: LocalEncryptedKeyEnv = process.env) {
const fromEnv = env.PAPERCLIP_SECRETS_MASTER_KEY_FILE;
if (fromEnv && fromEnv.trim().length > 0) return path.resolve(fromEnv.trim());
return resolveDefaultSecretsKeyFilePath();
}
function decodeMasterKey(raw: string): Buffer | null {
const trimmed = raw.trim();
if (!trimmed) return null;
if (/^[A-Fa-f0-9]{64}$/.test(trimmed)) {
return Buffer.from(trimmed, "hex");
}
try {
const decoded = Buffer.from(trimmed, "base64");
if (decoded.length === 32) return decoded;
} catch {
// ignored
}
if (Buffer.byteLength(trimmed, "utf8") === 32) {
return Buffer.from(trimmed, "utf8");
}
return null;
}
/** Check the configured key without reading, creating, or changing key files. */
export function isValidLocalEncryptedMasterKey(raw: string): boolean {
return decodeMasterKey(raw) !== null;
}
function loadOrCreateMasterKey(allowCreate = true, env: LocalEncryptedKeyEnv = process.env, options:{readOnly?:boolean}={}): Buffer {
const envKeyRaw = env.PAPERCLIP_SECRETS_MASTER_KEY;
if (envKeyRaw && envKeyRaw.trim().length > 0) {
const fromEnv = decodeMasterKey(envKeyRaw);
if (!fromEnv) {
throw badRequest(
"Invalid PAPERCLIP_SECRETS_MASTER_KEY (expected 32-byte base64, 64-char hex, or raw 32-char string)",
);
}
return fromEnv;
}
const keyPath = resolveMasterKeyFilePath(env);
if (existsSync(keyPath)) {
if(options.readOnly){
if((statSync(keyPath).mode&0o077)!==0)throw new Error("Secrets master key permissions are unsafe for inspection");
}else enforceKeyFilePermissionsBestEffort(keyPath);
const raw = readFileSync(keyPath, "utf8");
const decoded = decodeMasterKey(raw);
if (!decoded) {
throw badRequest(`Invalid secrets master key at ${keyPath}`);
}
return decoded;
}
if (!allowCreate||options.readOnly) throw new Error(`Secrets master key file is missing: ${keyPath}`);
const dir = path.dirname(keyPath);
mkdirSync(dir, { recursive: true });
const generated = randomBytes(32);
const temporaryPath = `${keyPath}.${process.pid}.${randomBytes(12).toString("hex")}.tmp`;
writeFileSync(temporaryPath, generated.toString("base64"), { encoding: "utf8", mode: 0o600, flag: "wx" });
try {
// A hard link publishes a complete file without overwriting a competitor's
// key. Opening the destination with wx would expose an empty/partial file.
try {
linkSync(temporaryPath, keyPath);
} catch (err) {
if ((err as NodeJS.ErrnoException).code !== "EEXIST") throw err;
}
} finally {
unlinkSync(temporaryPath);
}
return loadOrCreateMasterKey(false, env);
}
/** Derive a purpose-specific instance secret without exposing or reusing the vault key. */
export function deriveLocalEncryptedInstanceSecret(purpose: string, env: LocalEncryptedKeyEnv = process.env, options: { allowCreate?: boolean;readOnly?:boolean } = {}): string {
return Buffer.from(hkdfSync("sha256", loadOrCreateMasterKey(options.readOnly?false:options.allowCreate ?? true, env,{readOnly:options.readOnly}), Buffer.alloc(0),
`paperclip:instance-secret:v1\0${purpose}`, 32)).toString("base64url");
}
function enforceKeyFilePermissionsBestEffort(keyPath: string) {
try {
const mode = statSync(keyPath).mode & 0o777;
if ((mode & 0o077) !== 0) {
chmodSync(keyPath, 0o600);
}
} catch {
// best effort only; health checks surface persistent permission problems.
}
}
function sha256Hex(value: string): string {
return createHash("sha256").update(value).digest("hex");
}
function prepareManagedVersion(value: string): PreparedSecretVersion {
const masterKey = loadOrCreateMasterKey();
const valueSha256 = sha256Hex(value);
return {
material: encryptValue(masterKey, value),
valueSha256,
fingerprintSha256: valueSha256,
externalRef: null,
};
}
async function inspectLocalEncryptedHealth(): Promise<SecretProviderHealthCheck> {
const envKeyRaw = process.env.PAPERCLIP_SECRETS_MASTER_KEY;
if (envKeyRaw && envKeyRaw.trim().length > 0) {
if (!decodeMasterKey(envKeyRaw)) {
return {
provider: "local_encrypted",
status: "error",
message:
"PAPERCLIP_SECRETS_MASTER_KEY is invalid; expected 32-byte base64, 64-char hex, or raw 32-char string",
};
}
return {
provider: "local_encrypted",
status: "ok",
message: "Local encrypted provider is using PAPERCLIP_SECRETS_MASTER_KEY",
backupGuidance: [
"Back up the configured master key separately from the database.",
"A restore needs both the database metadata and the same master key.",
],
details: { keySource: "env" },
};
}
const keyPath = resolveMasterKeyFilePath();
if (!existsSync(keyPath)) {
return {
provider: "local_encrypted",
status: "warn",
message: `Secrets key file does not exist yet: ${keyPath}`,
warnings: ["The first managed secret write will create this key file with 0600 permissions."],
backupGuidance: [
"Back up the key file together with database backups.",
"The database alone cannot restore local encrypted secret values.",
],
details: { keySource: "file", keyFilePath: keyPath },
};
}
let mode: number | null = null;
try {
mode = statSync(keyPath).mode & 0o777;
} catch (err) {
return {
provider: "local_encrypted",
status: "error",
message: `Could not stat secrets key file: ${err instanceof Error ? err.message : String(err)}`,
details: { keySource: "file", keyFilePath: keyPath },
};
}
try {
const raw = readFileSync(keyPath, "utf8");
if (!decodeMasterKey(raw)) {
return {
provider: "local_encrypted",
status: "error",
message: `Invalid key material in ${keyPath}`,
details: { keySource: "file", keyFilePath: keyPath },
};
}
} catch (err) {
return {
provider: "local_encrypted",
status: "error",
message: `Could not read secrets key file: ${err instanceof Error ? err.message : String(err)}`,
details: { keySource: "file", keyFilePath: keyPath },
};
}
// Windows has no POSIX permission bits: stat() reports 0o666 for any writable
// file and chmod cannot restrict it, so this check would always warn there.
const warnings =
process.platform !== "win32" && mode !== null && (mode & 0o077) !== 0
? [`Secrets key file permissions are ${mode.toString(8)}; run chmod 600 ${keyPath}`]
: [];
return {
provider: "local_encrypted",
status: warnings.length > 0 ? "warn" : "ok",
message: `Local encrypted provider configured with key file ${keyPath}`,
warnings,
backupGuidance: [
"Back up the key file together with database backups.",
"The database alone cannot restore local encrypted secret values.",
],
details: { keySource: "file", keyFilePath: keyPath },
};
}
function encryptValue(masterKey: Buffer, value: string): LocalEncryptedMaterial {
const iv = randomBytes(12);
const cipher = createCipheriv("aes-256-gcm", masterKey, iv);
const ciphertext = Buffer.concat([cipher.update(value, "utf8"), cipher.final()]);
const tag = cipher.getAuthTag();
return {
scheme: "local_encrypted_v1",
iv: iv.toString("base64"),
tag: tag.toString("base64"),
ciphertext: ciphertext.toString("base64"),
};
}
function decryptValue(masterKey: Buffer, material: LocalEncryptedMaterial): string {
const iv = Buffer.from(material.iv, "base64");
const tag = Buffer.from(material.tag, "base64");
const ciphertext = Buffer.from(material.ciphertext, "base64");
const decipher = createDecipheriv("aes-256-gcm", masterKey, iv);
decipher.setAuthTag(tag);
try {
const plain = Buffer.concat([decipher.update(ciphertext), decipher.final()]);
return plain.toString("utf8");
} catch (err) {
const keyFingerprint = createHash("sha256").update(masterKey).digest("hex").slice(0, 12);
throw new Error(
`Secret decryption failed (master key fingerprint: ${keyFingerprint}). ` +
`This usually means the master key does not match the key used to encrypt the secret. ` +
`Original error: ${err instanceof Error ? err.message : String(err)}`,
);
}
}
function asLocalEncryptedMaterial(value: StoredSecretVersionMaterial): LocalEncryptedMaterial {
if (
value &&
typeof value === "object" &&
value.scheme === "local_encrypted_v1" &&
typeof value.iv === "string" &&
typeof value.tag === "string" &&
typeof value.ciphertext === "string"
) {
return value as LocalEncryptedMaterial;
}
throw badRequest("Invalid local_encrypted secret material");
}
function resolveLocalEncryptedValue<T>(operation: () => T): T {
try {
return operation();
} catch (cause) {
// This boundary contains only local key I/O and decryption, never database work.
throw new SecretProviderClientError({
code: "provider_unavailable", provider: "local_encrypted", operation: "resolveVersion",
message: cause instanceof Error ? cause.message : "Local encrypted secret could not be resolved",
cause,
});
}
}
/** Reuse one resolved key within a single batch; reload it for the next operation. */
export function createLocalEncryptedCipher(mode: "read" | "write",options:{readOnly?:boolean}={}) {
if(mode==="write"&&options.readOnly)throw badRequest("Read-only inspection cannot create an encryption writer");
const masterKey = mode === "write"
? loadOrCreateMasterKey()
: resolveLocalEncryptedValue(() => loadOrCreateMasterKey(false,process.env,options));
return {
encrypt(value: string): StoredSecretVersionMaterial {
return encryptValue(masterKey, value);
},
decrypt(material: StoredSecretVersionMaterial): string {
return resolveLocalEncryptedValue(() => decryptValue(masterKey, asLocalEncryptedMaterial(material)));
},
};
}
export const localEncryptedProvider: SecretProviderModule = {
id: "local_encrypted",
descriptor() {
return {
id: "local_encrypted",
label: "Local encrypted (default)",
requiresExternalRef: false,
supportsManagedValues: true,
supportsExternalReferences: false,
configured: true,
};
},
async validateConfig(input): Promise<SecretProviderValidationResult> {
const warnings: string[] = [];
if (input?.deploymentMode === "authenticated" && input.strictMode !== true) {
warnings.push("Strict secret mode should be enabled for authenticated deployments");
}
const health = await inspectLocalEncryptedHealth();
if (health.status === "error") {
throw badRequest(health.message);
}
warnings.push(...(health.warnings ?? []));
return { ok: true, warnings };
},
async createSecret(input) {
return prepareManagedVersion(input.value);
},
async createVersion(input) {
return prepareManagedVersion(input.value);
},
async linkExternalSecret() {
throw badRequest("local_encrypted does not support external reference secrets");
},
async resolveVersion(input) {
return createLocalEncryptedCipher("read").decrypt(input.material);
},
async deleteOrArchive() {
// Secret metadata deletion is handled in Paperclip DB; the local key is shared and must remain.
},
async healthCheck() {
return inspectLocalEncryptedHealth();
},
};