mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip manages work for AI agents. > - Personal MCP connections should remain enabled while an idle tenant sleeps. > - Unfinished OAuth, transfer and delivery work must retain its owner and deadline. > - Cloud can receive bounded metadata and incoming work while the tenant remains the company-scoped authorizer. > - Current identity, revision and delivery acknowledgments establish when ownership has transferred. > - This change preserves useful wake, retries and revocation while making settled ownership inspectable without writes. ## Linked Issues or Issue Description Refs: #19377 and #19750. Hosted companion: [paperclip-cloud#1217](https://github.com/paperclipai/paperclip-cloud/pull/1217). **What happened?** Enabled personal MCP grants and subscriptions could keep an otherwise idle tenant awake. Ignoring those records would lose discovery, authorization, delivery and revocation obligations. **Expected behavior** Unconsumed authorization codes, device requests, file handoffs and delivery work retain their original deadlines. Settled connections remain enabled and become quiet after the current external owner acknowledges responsibility. Incoming useful work resumes normal authorization and delivery; revocation still takes effect for the current company. **Steps to reproduce** Complete normal PKCE/device consent and company selection, synchronize client/device and revocation state, retain the connection while idle, then exercise useful tools/events and revocation. Sleep eligibility requires current incoming/deadline ownership throughout. ## What Changed - Add bounded signed client/device metadata and revocation synchronization with exact revision, identity and ACK checks, startup recovery and finite legacy-device cutover. - Preserve company selection, authorization and operation identity. Register grant work before its INSERT in the same transaction; retain rollback, expiry and retry behavior. - Require current managed identity/resource before hosted publication. Receipt-cleared re-enable must obtain a fresh source ACK before device creation. - Reuse read-only settings and inspection budgets; connect MCP owners to normal startup, drain and shutdown. - Bound metadata pages and the actual signed request body while preserving the default limits of other operations. - Generate migration 0440 after the merged Legacy migration. Preserve all 438 existing executable migrations and the journal prefix; retain the normal five newest snapshots. - Preserve the merged transaction-owner forwarding and regression coverage from #19750 through normal ancestry integration. ## Verification - Current own Shared/SDK, database, eval-kernel and Runner builds passed, together with all seven normal application TypeScript programs. Missing generated package exports/workspace links were restored through normal setup; the original setup failures were retained. No source, compiler option, test assertion or timeout was weakened. - Normal migration generation changed only the five intended MCP tables; its SQL is byte-identical to the reviewed MCP delta. Fresh reference databases and both retained master/Muse histories upgraded through 0440, preserving historical hashes, row fields, schema and idempotence. Direct assertions cover the Legacy null default and MCP null/zero defaults. Two selected history cases passed; the other 21 database cases were unselected. - All six current MCP lifecycle cases passed with zero skips: normal PKCE approval intent, ACK recovery after outage, mutation during publication, disable/re-enable with original expiry, company-deletion rollback/commit and key rotation. - All 81 checks in five current whole files passed with zero skips: OpenAPI 34, read-only settings 2, diagnostic catalog 25, PostgreSQL diagnostics 7 and dependency scheduling 13. PostgreSQL fixtures ran normal migrations before test imports. - Earlier canonical proof at App `cd1c8c6` / Cloud `9f67531` includes real 60-second rate and three 15-second quiet windows, useful work and three revokes. Later paired proof at App `43af514` / Cloud `914eb0f` plus its reviewed candidate includes normal DCR, PostgreSQL metadata, a 144,017-byte signed publication, actual ACK, cap/default-413 controls and a real 15-second quiet window. The relevant producer/signing/wire inputs remain byte-identical in this integration; shared changes are qualified above. These remain source-scoped observations, not a fresh full canonical run on the integrated tree. - All known test owners, process groups and OS sockets closed; an independent final PostgreSQL audit found no remaining test sessions. One dependency-test trace recorded 46 opens and 45 close events. Its cause is unknown; this is not claimed as proof of graceful closure. - Fresh exact-head remote checks passed: 59 successful and 4 skipped contexts, with a complete current 5/5 Greptile review and all seven review threads resolved. Targeted local checks do not constitute a new whole-repository test run. ## Risks App and Cloud must agree on identity, revision, namespace, deadlines and revocation. Partial rollout, unavailable authority or missing ACK must retain awake debt. Shared startup/shutdown and settings changed since the earlier canonical profile, so historical observations carry only within their stated source scope. No deployed Railway/Neon suspension, Cloud alarm, savings or complete hosted Dot/Muse acceptance is claimed. ## Model Used OpenAI `gpt-6.1-sol` produced code and verification, with a Codex coordinator reviewing and integrating the changes. The coordinator's exact runtime model ID and context-window sizes were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>
341 lines
12 KiB
TypeScript
341 lines
12 KiB
TypeScript
import { createCipheriv, createDecipheriv, createHash, hkdfSync, randomBytes } from "node:crypto";
|
|
import { chmodSync, existsSync, linkSync, mkdirSync, readFileSync, statSync, unlinkSync, writeFileSync } from "node:fs";
|
|
import path from "node:path";
|
|
import { resolveDefaultSecretsKeyFilePath } from "../home-paths.js";
|
|
import type {
|
|
PreparedSecretVersion,
|
|
SecretProviderHealthCheck,
|
|
SecretProviderModule,
|
|
SecretProviderValidationResult,
|
|
StoredSecretVersionMaterial,
|
|
} from "./types.js";
|
|
import { badRequest } from "../errors.js";
|
|
import { SecretProviderClientError } from "./types.js";
|
|
|
|
interface LocalEncryptedMaterial extends StoredSecretVersionMaterial {
|
|
scheme: "local_encrypted_v1";
|
|
iv: string;
|
|
tag: string;
|
|
ciphertext: string;
|
|
}
|
|
|
|
export type LocalEncryptedKeyEnv = Partial<Record<
|
|
"PAPERCLIP_SECRETS_MASTER_KEY" | "PAPERCLIP_SECRETS_MASTER_KEY_FILE", string | undefined
|
|
>>;
|
|
|
|
function resolveMasterKeyFilePath(env: LocalEncryptedKeyEnv = process.env) {
|
|
const fromEnv = env.PAPERCLIP_SECRETS_MASTER_KEY_FILE;
|
|
if (fromEnv && fromEnv.trim().length > 0) return path.resolve(fromEnv.trim());
|
|
return resolveDefaultSecretsKeyFilePath();
|
|
}
|
|
|
|
function decodeMasterKey(raw: string): Buffer | null {
|
|
const trimmed = raw.trim();
|
|
if (!trimmed) return null;
|
|
|
|
if (/^[A-Fa-f0-9]{64}$/.test(trimmed)) {
|
|
return Buffer.from(trimmed, "hex");
|
|
}
|
|
|
|
try {
|
|
const decoded = Buffer.from(trimmed, "base64");
|
|
if (decoded.length === 32) return decoded;
|
|
} catch {
|
|
// ignored
|
|
}
|
|
|
|
if (Buffer.byteLength(trimmed, "utf8") === 32) {
|
|
return Buffer.from(trimmed, "utf8");
|
|
}
|
|
return null;
|
|
}
|
|
|
|
/** Check the configured key without reading, creating, or changing key files. */
|
|
export function isValidLocalEncryptedMasterKey(raw: string): boolean {
|
|
return decodeMasterKey(raw) !== null;
|
|
}
|
|
|
|
function loadOrCreateMasterKey(allowCreate = true, env: LocalEncryptedKeyEnv = process.env, options:{readOnly?:boolean}={}): Buffer {
|
|
const envKeyRaw = env.PAPERCLIP_SECRETS_MASTER_KEY;
|
|
if (envKeyRaw && envKeyRaw.trim().length > 0) {
|
|
const fromEnv = decodeMasterKey(envKeyRaw);
|
|
if (!fromEnv) {
|
|
throw badRequest(
|
|
"Invalid PAPERCLIP_SECRETS_MASTER_KEY (expected 32-byte base64, 64-char hex, or raw 32-char string)",
|
|
);
|
|
}
|
|
return fromEnv;
|
|
}
|
|
|
|
const keyPath = resolveMasterKeyFilePath(env);
|
|
if (existsSync(keyPath)) {
|
|
if(options.readOnly){
|
|
if((statSync(keyPath).mode&0o077)!==0)throw new Error("Secrets master key permissions are unsafe for inspection");
|
|
}else enforceKeyFilePermissionsBestEffort(keyPath);
|
|
const raw = readFileSync(keyPath, "utf8");
|
|
const decoded = decodeMasterKey(raw);
|
|
if (!decoded) {
|
|
throw badRequest(`Invalid secrets master key at ${keyPath}`);
|
|
}
|
|
return decoded;
|
|
}
|
|
|
|
if (!allowCreate||options.readOnly) throw new Error(`Secrets master key file is missing: ${keyPath}`);
|
|
const dir = path.dirname(keyPath);
|
|
mkdirSync(dir, { recursive: true });
|
|
const generated = randomBytes(32);
|
|
const temporaryPath = `${keyPath}.${process.pid}.${randomBytes(12).toString("hex")}.tmp`;
|
|
writeFileSync(temporaryPath, generated.toString("base64"), { encoding: "utf8", mode: 0o600, flag: "wx" });
|
|
try {
|
|
// A hard link publishes a complete file without overwriting a competitor's
|
|
// key. Opening the destination with wx would expose an empty/partial file.
|
|
try {
|
|
linkSync(temporaryPath, keyPath);
|
|
} catch (err) {
|
|
if ((err as NodeJS.ErrnoException).code !== "EEXIST") throw err;
|
|
}
|
|
} finally {
|
|
unlinkSync(temporaryPath);
|
|
}
|
|
return loadOrCreateMasterKey(false, env);
|
|
}
|
|
|
|
/** Derive a purpose-specific instance secret without exposing or reusing the vault key. */
|
|
export function deriveLocalEncryptedInstanceSecret(purpose: string, env: LocalEncryptedKeyEnv = process.env, options: { allowCreate?: boolean;readOnly?:boolean } = {}): string {
|
|
return Buffer.from(hkdfSync("sha256", loadOrCreateMasterKey(options.readOnly?false:options.allowCreate ?? true, env,{readOnly:options.readOnly}), Buffer.alloc(0),
|
|
`paperclip:instance-secret:v1\0${purpose}`, 32)).toString("base64url");
|
|
}
|
|
|
|
function enforceKeyFilePermissionsBestEffort(keyPath: string) {
|
|
try {
|
|
const mode = statSync(keyPath).mode & 0o777;
|
|
if ((mode & 0o077) !== 0) {
|
|
chmodSync(keyPath, 0o600);
|
|
}
|
|
} catch {
|
|
// best effort only; health checks surface persistent permission problems.
|
|
}
|
|
}
|
|
|
|
function sha256Hex(value: string): string {
|
|
return createHash("sha256").update(value).digest("hex");
|
|
}
|
|
|
|
function prepareManagedVersion(value: string): PreparedSecretVersion {
|
|
const masterKey = loadOrCreateMasterKey();
|
|
const valueSha256 = sha256Hex(value);
|
|
return {
|
|
material: encryptValue(masterKey, value),
|
|
valueSha256,
|
|
fingerprintSha256: valueSha256,
|
|
externalRef: null,
|
|
};
|
|
}
|
|
|
|
async function inspectLocalEncryptedHealth(): Promise<SecretProviderHealthCheck> {
|
|
const envKeyRaw = process.env.PAPERCLIP_SECRETS_MASTER_KEY;
|
|
if (envKeyRaw && envKeyRaw.trim().length > 0) {
|
|
if (!decodeMasterKey(envKeyRaw)) {
|
|
return {
|
|
provider: "local_encrypted",
|
|
status: "error",
|
|
message:
|
|
"PAPERCLIP_SECRETS_MASTER_KEY is invalid; expected 32-byte base64, 64-char hex, or raw 32-char string",
|
|
};
|
|
}
|
|
return {
|
|
provider: "local_encrypted",
|
|
status: "ok",
|
|
message: "Local encrypted provider is using PAPERCLIP_SECRETS_MASTER_KEY",
|
|
backupGuidance: [
|
|
"Back up the configured master key separately from the database.",
|
|
"A restore needs both the database metadata and the same master key.",
|
|
],
|
|
details: { keySource: "env" },
|
|
};
|
|
}
|
|
|
|
const keyPath = resolveMasterKeyFilePath();
|
|
if (!existsSync(keyPath)) {
|
|
return {
|
|
provider: "local_encrypted",
|
|
status: "warn",
|
|
message: `Secrets key file does not exist yet: ${keyPath}`,
|
|
warnings: ["The first managed secret write will create this key file with 0600 permissions."],
|
|
backupGuidance: [
|
|
"Back up the key file together with database backups.",
|
|
"The database alone cannot restore local encrypted secret values.",
|
|
],
|
|
details: { keySource: "file", keyFilePath: keyPath },
|
|
};
|
|
}
|
|
|
|
let mode: number | null = null;
|
|
try {
|
|
mode = statSync(keyPath).mode & 0o777;
|
|
} catch (err) {
|
|
return {
|
|
provider: "local_encrypted",
|
|
status: "error",
|
|
message: `Could not stat secrets key file: ${err instanceof Error ? err.message : String(err)}`,
|
|
details: { keySource: "file", keyFilePath: keyPath },
|
|
};
|
|
}
|
|
|
|
try {
|
|
const raw = readFileSync(keyPath, "utf8");
|
|
if (!decodeMasterKey(raw)) {
|
|
return {
|
|
provider: "local_encrypted",
|
|
status: "error",
|
|
message: `Invalid key material in ${keyPath}`,
|
|
details: { keySource: "file", keyFilePath: keyPath },
|
|
};
|
|
}
|
|
} catch (err) {
|
|
return {
|
|
provider: "local_encrypted",
|
|
status: "error",
|
|
message: `Could not read secrets key file: ${err instanceof Error ? err.message : String(err)}`,
|
|
details: { keySource: "file", keyFilePath: keyPath },
|
|
};
|
|
}
|
|
|
|
// Windows has no POSIX permission bits: stat() reports 0o666 for any writable
|
|
// file and chmod cannot restrict it, so this check would always warn there.
|
|
const warnings =
|
|
process.platform !== "win32" && mode !== null && (mode & 0o077) !== 0
|
|
? [`Secrets key file permissions are ${mode.toString(8)}; run chmod 600 ${keyPath}`]
|
|
: [];
|
|
return {
|
|
provider: "local_encrypted",
|
|
status: warnings.length > 0 ? "warn" : "ok",
|
|
message: `Local encrypted provider configured with key file ${keyPath}`,
|
|
warnings,
|
|
backupGuidance: [
|
|
"Back up the key file together with database backups.",
|
|
"The database alone cannot restore local encrypted secret values.",
|
|
],
|
|
details: { keySource: "file", keyFilePath: keyPath },
|
|
};
|
|
}
|
|
|
|
function encryptValue(masterKey: Buffer, value: string): LocalEncryptedMaterial {
|
|
const iv = randomBytes(12);
|
|
const cipher = createCipheriv("aes-256-gcm", masterKey, iv);
|
|
const ciphertext = Buffer.concat([cipher.update(value, "utf8"), cipher.final()]);
|
|
const tag = cipher.getAuthTag();
|
|
return {
|
|
scheme: "local_encrypted_v1",
|
|
iv: iv.toString("base64"),
|
|
tag: tag.toString("base64"),
|
|
ciphertext: ciphertext.toString("base64"),
|
|
};
|
|
}
|
|
|
|
function decryptValue(masterKey: Buffer, material: LocalEncryptedMaterial): string {
|
|
const iv = Buffer.from(material.iv, "base64");
|
|
const tag = Buffer.from(material.tag, "base64");
|
|
const ciphertext = Buffer.from(material.ciphertext, "base64");
|
|
const decipher = createDecipheriv("aes-256-gcm", masterKey, iv);
|
|
decipher.setAuthTag(tag);
|
|
try {
|
|
const plain = Buffer.concat([decipher.update(ciphertext), decipher.final()]);
|
|
return plain.toString("utf8");
|
|
} catch (err) {
|
|
const keyFingerprint = createHash("sha256").update(masterKey).digest("hex").slice(0, 12);
|
|
throw new Error(
|
|
`Secret decryption failed (master key fingerprint: ${keyFingerprint}). ` +
|
|
`This usually means the master key does not match the key used to encrypt the secret. ` +
|
|
`Original error: ${err instanceof Error ? err.message : String(err)}`,
|
|
);
|
|
}
|
|
}
|
|
|
|
function asLocalEncryptedMaterial(value: StoredSecretVersionMaterial): LocalEncryptedMaterial {
|
|
if (
|
|
value &&
|
|
typeof value === "object" &&
|
|
value.scheme === "local_encrypted_v1" &&
|
|
typeof value.iv === "string" &&
|
|
typeof value.tag === "string" &&
|
|
typeof value.ciphertext === "string"
|
|
) {
|
|
return value as LocalEncryptedMaterial;
|
|
}
|
|
throw badRequest("Invalid local_encrypted secret material");
|
|
}
|
|
|
|
function resolveLocalEncryptedValue<T>(operation: () => T): T {
|
|
try {
|
|
return operation();
|
|
} catch (cause) {
|
|
// This boundary contains only local key I/O and decryption, never database work.
|
|
throw new SecretProviderClientError({
|
|
code: "provider_unavailable", provider: "local_encrypted", operation: "resolveVersion",
|
|
message: cause instanceof Error ? cause.message : "Local encrypted secret could not be resolved",
|
|
cause,
|
|
});
|
|
}
|
|
}
|
|
|
|
/** Reuse one resolved key within a single batch; reload it for the next operation. */
|
|
export function createLocalEncryptedCipher(mode: "read" | "write",options:{readOnly?:boolean}={}) {
|
|
if(mode==="write"&&options.readOnly)throw badRequest("Read-only inspection cannot create an encryption writer");
|
|
const masterKey = mode === "write"
|
|
? loadOrCreateMasterKey()
|
|
: resolveLocalEncryptedValue(() => loadOrCreateMasterKey(false,process.env,options));
|
|
return {
|
|
encrypt(value: string): StoredSecretVersionMaterial {
|
|
return encryptValue(masterKey, value);
|
|
},
|
|
decrypt(material: StoredSecretVersionMaterial): string {
|
|
return resolveLocalEncryptedValue(() => decryptValue(masterKey, asLocalEncryptedMaterial(material)));
|
|
},
|
|
};
|
|
}
|
|
|
|
export const localEncryptedProvider: SecretProviderModule = {
|
|
id: "local_encrypted",
|
|
descriptor() {
|
|
return {
|
|
id: "local_encrypted",
|
|
label: "Local encrypted (default)",
|
|
requiresExternalRef: false,
|
|
supportsManagedValues: true,
|
|
supportsExternalReferences: false,
|
|
configured: true,
|
|
};
|
|
},
|
|
async validateConfig(input): Promise<SecretProviderValidationResult> {
|
|
const warnings: string[] = [];
|
|
if (input?.deploymentMode === "authenticated" && input.strictMode !== true) {
|
|
warnings.push("Strict secret mode should be enabled for authenticated deployments");
|
|
}
|
|
const health = await inspectLocalEncryptedHealth();
|
|
if (health.status === "error") {
|
|
throw badRequest(health.message);
|
|
}
|
|
warnings.push(...(health.warnings ?? []));
|
|
return { ok: true, warnings };
|
|
},
|
|
async createSecret(input) {
|
|
return prepareManagedVersion(input.value);
|
|
},
|
|
async createVersion(input) {
|
|
return prepareManagedVersion(input.value);
|
|
},
|
|
async linkExternalSecret() {
|
|
throw badRequest("local_encrypted does not support external reference secrets");
|
|
},
|
|
async resolveVersion(input) {
|
|
return createLocalEncryptedCipher("read").decrypt(input.material);
|
|
},
|
|
async deleteOrArchive() {
|
|
// Secret metadata deletion is handled in Paperclip DB; the local key is shared and must remain.
|
|
},
|
|
async healthCheck() {
|
|
return inspectLocalEncryptedHealth();
|
|
},
|
|
};
|