mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner sandbox. A server upgrade
detaches each running turn, and the new server must reconnect to its
Runner (restart recovery).
> - A reconnect runs the full turn preparation before it contacts the
Runner. On a new server, skill preparation downloads every saved skill
version again, under one lock for the whole company.
> - After an upgrade, many reconnects wait behind each other's
downloads. The Runner stops answering after about 2 minutes, so a late
reconnect loses the turn.
> - This pull request lets a reconnect to the same run's live remote
Runner use the exact saved skill versions, without downloading them and
without the lock.
> - The benefit is that skill preparation no longer adds to the
reconnect delay, and it does not grow with the number of reconnects.
Normal turn starts do not change.
## Linked Issues or Issue Description
**What happened?**
After an automatic staging upgrade, runs that the new server reached
late failed. The measured time from detach to reconnect was 114–152 s
for the failed runs. Turn preparation took 30–130 s, and skill
preparation alone took 46 s on one run.
**Expected behavior**
A reconnect to a live Runner reaches it quickly. Its speed must not
depend on how many other runs prepare skills at the same time.
**Steps to reproduce**
1. Run a remote Runner turn for each of many tasks in one company.
2. Upgrade the server while those turns run, so the new server starts
with an empty skill cache.
3. See that each reconnect waits in
`heartbeat.prepare.configuration_skills` for the company skill cache
lock and downloads the skills again.
**Paperclip version or commit**
master at 4228bcf89c
**Additional context**
Root cause: `withRuntimeSkillEntries` holds the company-wide skill cache
lock (a Postgres advisory lock) while it materializes each saved skill
version. A new server has an empty skill cache, so each reconnect
downloads its skills again, one run at a time. The Runner already has
these files, and the reconnect path does not read them.
## What Changed
- `company-skills.ts`: new `admittedVersionsOnly` option. It still
checks that each saved version exists and that the agent can read the
skill. It does not materialize the files or take the company skill cache
lock.
- `heartbeat/live-runner-adoption.ts`: `adoptsLiveRemoteRunner` is true
only for a `reattach_remote_runner` claim for this run, with the claim's
own lease owner, on a native run.
- `heartbeat/execution-configuration.ts`: uses the option only when all
of these are true: the reconnect gate is true, the run has its saved
native input with a runtime context, and the run has its saved skill
versions. In that case it times the phase as
`configuration_skills_adopted` and adds one lifecycle run event
(`skillPreparation: "reused_admitted"`).
- `heartbeat.ts`: passes the gate to execution configuration.
- `native-runtime/runtime-context.ts` and `native-session-executor.ts`:
on the same reconnect to a remote Runner filesystem, the Runner
controller describes the Runner's skills by their admitted digests. It
rehydrates the pinned skill files (which takes the same lock) only if it
must launch a Runner process and stage them. All other starts rehydrate
as before.
- `doc/execution-semantics.md`: describes the new behavior.
Not in this PR (by design): a separate "dispatch-only" reconnect path,
and a new startup scheduling lane. The native executor needs about 40
values from preparation (runner environment, MCP token, managed AI
credentials, the reattached execution target, run output). Connector and
instruction preparation also feed the session fingerprint. That cut is
not mechanical, so this PR removes the largest measured single cost
only.
Other `heartbeat.prepare.*` costs that stay on the reconnect path (use
post-upgrade timings to choose the next cut):
- `configuration_secrets`, `configuration_connector_assignments`,
`configuration_connection_instructions`,
`configuration_connector_skills`/`_delivery`,
`configuration_agent_identity`, `configuration_ai_runtime`
- `environment_selection`, `workspace` (retained lease resume and
realization), `session`, `task_context`, `run_identity`
- `github_environment`, `github_launchers`, `run_output`,
`runtime_services`, `instructions`, `runtime_selection` (including
`runtime_selection.skill_assets` and `runtime_selection.workspace_sync`)
- The wait from detach to "run started" (queue and controller-preparing
waits) is outside `heartbeat.prepare.*`.
## Verification
- New test `reconnects to a live remote Runner with its admitted skills
without re-downloading them or waiting on the cache`
(`server/src/__tests__/heartbeat-execution-configuration.test.ts`):
- It admits a run, deletes its skill cache (like a new server), and
holds the company cache lock.
- The reconnect preparation finishes while the lock is held. It returns
the same key, runtime name and version. The session fingerprint is
identical. The cache is not created again.
- Normal preparation of the same saved selection still waits for the
lock and materializes the files ("normal start unchanged").
- Red on master source: `A live Runner reattach must not queue behind
skill downloads: expected null not to be null`. Green with this change.
- New test in `managed-skill-content.test.ts`: with the cache lock held
and an empty cache, `prepareRuntimeContextSource` for a reconnect
returns at once with the admitted context. Its staging step and a normal
start still wait for the lock and then rehydrate the same digests.
- New `server/src/services/heartbeat/live-runner-adoption.test.ts`: the
gate is false for a normal start, a scheduled resume, a dead-Runner
resume, an incomplete bootstrap, another run's claim, a different lease
owner, and a legacy run.
- `npx vitest run
src/__tests__/heartbeat-execution-configuration.test.ts
src/__tests__/skill-version-cache.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/heartbeat/live-runner-adoption.test.ts`: 289 passed, 14
skipped. After the controller change: restart-recovery, diskless
restart, runtime skill preservation, managed skill content, session
executor and execution configuration suites: 1686 passed, 14 skipped.
- `tsc --noEmit -p server/tsconfig.json`: pass.
- `pnpm verify` asks for broader coverage (exit 2). Required CI supplies
it.
After deploy, a reconnect run shows
`heartbeat.prepare.configuration_skills_adopted` and the
`reused_admitted` event, and no `skills.cache.acquire` or
`skills.version.materialize` phases.
## Risks
- Low. The option applies only to a same-run reconnect of a retained
remote Runner that has a saved runtime context and saved skill versions.
Every other start keeps the current path.
- Access revocation and deleted versions still fail the reconnect, as
before. A failure to download the files on this server no longer fails a
live reconnect, because the Runner already holds them.
- Nothing new is saved, so rows from older releases, two live servers,
and a deploy during a reconnect behave as before.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (code search, editing, shell, tests).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
415 lines
26 KiB
TypeScript
415 lines
26 KiB
TypeScript
import { skillExecutionContent } from "../skill-execution-content.js";
|
||
import { withRuntimeAssetUse, removeRuntimeAssetTree } from "./runtime-asset-retention.js";
|
||
import { cachedSkillAsset, rememberSkillAsset } from "./skill-asset-cache.js";
|
||
import { verifyFileManifest, type FileManifestEntry } from "../file-manifest-cache.js";
|
||
import { measureRunPhase } from "../execution-preparation-timing.js";
|
||
import { spekoToolsForSession } from "../voice/speko-agent-tools.js";
|
||
import { githubBotConnectionIdsForRun } from "../chat-github-tools.js";
|
||
import { boundRunIssueIdSql } from "../run-task-binding.js";
|
||
import { isBrowserUseConnection } from "../browser-use-client.js";
|
||
import { restConnectorFor } from "../connectors/rest-registry.js";
|
||
import { createHash, randomUUID } from "node:crypto";
|
||
import fs from "node:fs/promises";
|
||
import path from "node:path";
|
||
import { and, desc, eq } from "drizzle-orm";
|
||
import { agentInstructionRevisions, agents, companySkillVersions, heartbeatRuns, type Db } from "@paperclipai/db";
|
||
import type { PaperclipSkillEntry } from "@paperclipai/adapter-utils/server-utils";
|
||
import { isToolConnectionAttentionHealth } from "@paperclipai/shared";
|
||
import {
|
||
PAPERCLIP_OPERATIONAL_SKILL_KEY,
|
||
resolvePaperclipDesiredSkillNames,
|
||
} from "@paperclipai/adapter-utils/server-utils";
|
||
import {
|
||
NATIVE_RUNTIME_ASSET_SCHEMA,
|
||
PAPERCLIP_EXECUTION_PROMPT,
|
||
PAPERCLIP_EXECUTION_PROMPT_REVISION,
|
||
canonicalNativeRuntimeContextDigest,
|
||
nativeRuntimePromptDigest,
|
||
parseNativeRuntimeContext,
|
||
type NativeRuntimeAssetReference,
|
||
type NativeRuntimeContextSnapshot,
|
||
} from "../../vendor/paperclip-runner/index.js";
|
||
import { resolvePaperclipInstanceRoot } from "../../home-paths.js";
|
||
import { agentInstructionsService, agentInstructionsBundleMode } from "../agent-instructions.js";
|
||
import { agentInstructionRevisionService } from "../agent-instruction-revisions.js";
|
||
import { toolAccessService } from "../tool-access.js";
|
||
import { filterResolvedGitHubConnectionsForRun } from "../git-credentials.js";
|
||
import { rehydratePreservedRuntimeSkill } from "./runtime-skill-preservation.js";
|
||
|
||
const MAX_ASSET_FILES = 10_000;
|
||
const MAX_ASSET_BYTES = 64 * 1024 * 1024;
|
||
type RuntimeAgent = { id: string; companyId: string; name: string; adapterType?: string | null; adapterConfig: unknown };
|
||
type AssetFile = { path: string; content: Buffer; mode: number };
|
||
|
||
const sha256 = (value: string | Buffer) => createHash("sha256").update(value).digest("hex");
|
||
const safeRelativePath = (value: string, label: string) => {
|
||
const normalized = value.replaceAll("\\", "/");
|
||
if (!normalized || normalized.includes("\0") || normalized.startsWith("/") || normalized.split("/").some((part) => !part || part === "." || part === "..")) {
|
||
throw new Error(`${label} must stay within its bundle root`);
|
||
}
|
||
return normalized;
|
||
};
|
||
|
||
async function collectDirectoryFiles(sourceRoot: string): Promise<AssetFile[]> {
|
||
const root = path.resolve(sourceRoot);
|
||
const rootStat = await fs.lstat(root).catch(() => null);
|
||
if (!rootStat?.isDirectory() || rootStat.isSymbolicLink()) throw new Error(`runtime context source is not a safe directory: ${root}`);
|
||
const files: AssetFile[] = [];
|
||
let totalBytes = 0;
|
||
async function visit(relativeDirectory: string): Promise<void> {
|
||
const entries = await fs.readdir(relativeDirectory ? path.join(root, relativeDirectory) : root, { withFileTypes: true });
|
||
entries.sort((a, b) => a.name.localeCompare(b.name));
|
||
for (const entry of entries) {
|
||
const relativePath = safeRelativePath(relativeDirectory ? `${relativeDirectory}/${entry.name}` : entry.name, "runtime context path");
|
||
const absolutePath = path.join(root, relativePath);
|
||
const stat = await fs.lstat(absolutePath);
|
||
if (stat.isSymbolicLink()) throw new Error(`runtime context bundles do not permit symlinks: ${relativePath}`);
|
||
if (stat.isDirectory()) { await visit(relativePath); continue; }
|
||
if (!stat.isFile()) throw new Error(`runtime context bundle contains an unsupported file: ${relativePath}`);
|
||
const content = await fs.readFile(absolutePath);
|
||
totalBytes += content.byteLength;
|
||
if (files.length + 1 > MAX_ASSET_FILES || totalBytes > MAX_ASSET_BYTES) throw new Error("runtime context bundle exceeds its bound");
|
||
files.push({ path: relativePath, content, mode: stat.mode & 0o555 });
|
||
}
|
||
}
|
||
await visit("");
|
||
if (!files.length) throw new Error(`runtime context source is empty: ${root}`);
|
||
return files;
|
||
}
|
||
|
||
async function makeDirectoriesReadOnly(directory: string): Promise<void> {
|
||
for (const entry of await fs.readdir(directory, { withFileTypes: true })) {
|
||
if (entry.isDirectory()) await makeDirectoriesReadOnly(path.join(directory, entry.name));
|
||
}
|
||
await fs.chmod(directory, 0o555);
|
||
}
|
||
|
||
async function verifyMaterializedAsset(
|
||
rootPath: string,
|
||
manifestFiles: Array<{ path: string; sha256: string; mode: number; size: number }>,
|
||
): Promise<void> {
|
||
const expected = manifestFiles.map(file => ({ ...file, executable: Boolean(file.mode & 0o111) }));
|
||
if (!await measureRunPhase("runtime_context.asset_read", () => verifyFileManifest(rootPath, expected))) {
|
||
throw new Error("runtime context asset digest mismatch");
|
||
}
|
||
}
|
||
|
||
export async function materializeAsset(files: AssetFile[]): Promise<NativeRuntimeAssetReference> {
|
||
const { sorted, manifestFiles, totalBytes, assetDigest, manifestText, manifestDigest } = await measureRunPhase("runtime_context.asset_manifest", () => {
|
||
const sorted = [...files].sort((a, b) => a.path.localeCompare(b.path));
|
||
const manifestFiles = sorted.map((file) => ({ path: safeRelativePath(file.path, "runtime context path"), sha256: sha256(file.content), mode: file.mode & 0o555, size: file.content.byteLength }));
|
||
const totalBytes = manifestFiles.reduce((sum, file) => sum + file.size, 0);
|
||
if (!manifestFiles.length || manifestFiles.length > MAX_ASSET_FILES || totalBytes > MAX_ASSET_BYTES) throw new Error("runtime context bundle is empty or exceeds its bound");
|
||
const assetDigest = sha256(JSON.stringify(manifestFiles));
|
||
const manifestText = `${JSON.stringify({ schema: "paperclip.runtime-asset-manifest.v1", digest: assetDigest, fileCount: manifestFiles.length, totalBytes, files: manifestFiles })}\n`;
|
||
const manifestDigest = sha256(manifestText);
|
||
return { sorted, manifestFiles, totalBytes, assetDigest, manifestText, manifestDigest };
|
||
});
|
||
return withRuntimeAssetUse(assetDigest, async function publishOrReuse(locked): Promise<NativeRuntimeAssetReference> {
|
||
const assetsRoot = path.join(resolvePaperclipInstanceRoot(), "runtime-context-assets");
|
||
const rootPath = path.join(assetsRoot, "bundles", assetDigest);
|
||
const manifestPath = path.join(assetsRoot, "manifests", `${assetDigest}.json`);
|
||
const existing = await measureRunPhase("runtime_context.asset_cache", () => fs.readFile(manifestPath, "utf8").catch(() => null));
|
||
const rootExists = (await fs.stat(rootPath).catch(() => null))?.isDirectory();
|
||
if (existing !== null && rootExists && sha256(existing) === manifestDigest
|
||
&& await verifyFileManifest(rootPath, manifestFiles.map(file => ({ ...file, executable: Boolean(file.mode & 0o111) })))) {
|
||
return { schema: NATIVE_RUNTIME_ASSET_SCHEMA, digest: assetDigest, manifestDigest, rootPath, fileCount: manifestFiles.length, totalBytes };
|
||
}
|
||
if (!locked) return withRuntimeAssetUse(assetDigest, publishOrReuse, true);
|
||
// The caller supplied the exact authoritative bytes for this digest. A missing
|
||
// or damaged disposable copy can be rebuilt under the publication/GC lock.
|
||
if (rootExists) await removeRuntimeAssetTree(rootPath);
|
||
await fs.rm(manifestPath, { force: true });
|
||
await measureRunPhase("runtime_context.asset_materialize", async () => {
|
||
const stagingRoot = path.join(assetsRoot, ".staging", randomUUID());
|
||
await fs.mkdir(stagingRoot, { recursive: true, mode: 0o700 });
|
||
try {
|
||
for (const file of sorted) {
|
||
const target = path.join(stagingRoot, file.path);
|
||
await fs.mkdir(path.dirname(target), { recursive: true, mode: 0o700 });
|
||
await fs.writeFile(target, file.content, { mode: file.mode & 0o555 });
|
||
await fs.chmod(target, file.mode & 0o555);
|
||
}
|
||
await fs.mkdir(path.dirname(rootPath), { recursive: true, mode: 0o700 });
|
||
let publishedRoot = false;
|
||
await fs.rename(stagingRoot, rootPath).then(() => {
|
||
publishedRoot = true;
|
||
}).catch(async (error: NodeJS.ErrnoException) => {
|
||
// Concurrent publishers can surface EACCES on macOS when the winning
|
||
// destination has already been made read-only. Treat that as a race only
|
||
// when the content-addressed destination now exists; a genuine permission
|
||
// failure with no published destination must still fail loudly.
|
||
const publishedByPeer = (await fs.stat(rootPath).catch(() => null))?.isDirectory() === true;
|
||
if (error.code !== "EEXIST" && error.code !== "ENOTEMPTY" && !publishedByPeer) throw error;
|
||
await fs.rm(stagingRoot, { recursive: true, force: true });
|
||
});
|
||
// Publish the writable staging directory first. Renaming a read-only
|
||
// directory can fail with EACCES on macOS even when both parents are owned
|
||
// and writable. Files are already immutable; directories become immutable
|
||
// immediately after this process wins publication.
|
||
if (publishedRoot) await makeDirectoriesReadOnly(rootPath);
|
||
await fs.mkdir(path.dirname(manifestPath), { recursive: true, mode: 0o700 });
|
||
await fs.writeFile(manifestPath, manifestText, { flag: "wx", mode: 0o400 }).catch(async (error: NodeJS.ErrnoException) => {
|
||
if (error.code !== "EEXIST") throw error;
|
||
if (sha256(await measureRunPhase("runtime_context.asset_cache", () => fs.readFile(manifestPath, "utf8"))) !== manifestDigest) throw new Error(`runtime context manifest race mismatch: ${assetDigest}`);
|
||
});
|
||
await measureRunPhase("runtime_context.asset_verify", () => verifyMaterializedAsset(rootPath, manifestFiles), { fileCount: manifestFiles.length, bytes: totalBytes });
|
||
} catch (error) {
|
||
await fs.rm(stagingRoot, { recursive: true, force: true }).catch(() => undefined);
|
||
throw error;
|
||
}
|
||
}, { fileCount: manifestFiles.length, bytes: totalBytes });
|
||
return { schema: NATIVE_RUNTIME_ASSET_SCHEMA, digest: assetDigest, manifestDigest, rootPath, fileCount: manifestFiles.length, totalBytes };
|
||
});
|
||
}
|
||
|
||
async function materializeInstructionBundle(db: Db, agent: RuntimeAgent, agentFiles = false) {
|
||
if (agentFiles) {
|
||
const current = await agentInstructionRevisionService(db).readCommittedForRuntime({ companyId: agent.companyId, agentId: agent.id });
|
||
if (!current) throw new Error("Configured instruction entry is missing");
|
||
const entryPath = safeRelativePath(current.revision.entryFile, "instruction entry path");
|
||
return { entryPath, bundle: await materializeAsset([{ path: entryPath, content: Buffer.from(current.content, "utf8"), mode: 0o444 }]) };
|
||
}
|
||
const exported = await agentInstructionsService(db).exportFiles(agent, { rejectSymlinks: true });
|
||
if (agentInstructionsBundleMode(agent) === "managed") {
|
||
// The database head remains authoritative when a saved revision could not
|
||
// update its disk projection. Overlay the exact committed snapshot directly;
|
||
// a repair followed by a separate export can race another pending commit.
|
||
const current = await agentInstructionRevisionService(db).readCommittedForRuntime({ companyId: agent.companyId, agentId: agent.id });
|
||
if (current) {
|
||
exported.entryFile = current.revision.entryFile;
|
||
exported.files[current.revision.entryFile] = current.content;
|
||
}
|
||
}
|
||
const entryPath = safeRelativePath(exported.entryFile, "instruction entry path");
|
||
if (!(entryPath in exported.files)) throw new Error(`configured instruction entry is missing: ${entryPath}`);
|
||
const files = Object.entries(exported.files).map(([relativePath, content]) => ({ path: safeRelativePath(relativePath, "instruction path"), content: Buffer.from(content, "utf8"), mode: 0o444 }));
|
||
return { entryPath, bundle: await materializeAsset(files) };
|
||
}
|
||
|
||
async function materializeSkillAsset(source: string) {
|
||
let bundle = await measureRunPhase("runtime_context.asset_verify", () => cachedSkillAsset(source));
|
||
if (!bundle) {
|
||
const files = await measureRunPhase("runtime_context.asset_read", () => collectDirectoryFiles(source));
|
||
bundle = await materializeAsset(files);
|
||
const manifest: FileManifestEntry[] = files.map(file => ({ path: file.path, size: file.content.length,
|
||
sha256: sha256(file.content), mode: file.mode & 0o555, executable: Boolean(file.mode & 0o111) }));
|
||
if (await verifyFileManifest(source, manifest)) rememberSkillAsset(source, manifest, bundle);
|
||
}
|
||
return bundle;
|
||
}
|
||
|
||
async function localRuntimeAssetPresent(asset: NativeRuntimeAssetReference) {
|
||
const root = path.join(resolvePaperclipInstanceRoot(), "runtime-context-assets");
|
||
const manifest = await fs.readFile(path.join(root, "manifests", `${asset.digest}.json`), "utf8").catch(() => null);
|
||
return manifest !== null && sha256(manifest) === asset.manifestDigest
|
||
&& (await fs.lstat(path.join(root, "bundles", asset.digest)).catch(() => null))?.isDirectory() === true;
|
||
}
|
||
|
||
function singleFileAssetDigest(entryPath: string, content: Buffer) {
|
||
return sha256(JSON.stringify([{ path: entryPath, sha256: sha256(content), mode: 0o444, size: content.byteLength }]));
|
||
}
|
||
|
||
export async function rehydratePinnedInstructionAsset(
|
||
db: Db,
|
||
binding: { companyId: string; agentId: string },
|
||
context: NativeRuntimeContextSnapshot,
|
||
): Promise<NativeRuntimeContextSnapshot> {
|
||
const pinned = context.instructions.bundle;
|
||
if (pinned.rootPath !== path.join(resolvePaperclipInstanceRoot(), "runtime-context-assets", "bundles", pinned.digest)) return context;
|
||
if (await localRuntimeAssetPresent(pinned)) return context;
|
||
const matches = (bundle: NativeRuntimeAssetReference) => bundle.digest === pinned.digest
|
||
&& bundle.manifestDigest === pinned.manifestDigest && bundle.fileCount === pinned.fileCount && bundle.totalBytes === pinned.totalBytes;
|
||
if (pinned.fileCount === 1) {
|
||
const rows = await db.select({ contentBase64: agentInstructionRevisions.contentBase64 }).from(agentInstructionRevisions)
|
||
.where(and(eq(agentInstructionRevisions.companyId, binding.companyId), eq(agentInstructionRevisions.agentId, binding.agentId),
|
||
eq(agentInstructionRevisions.entryFile, context.instructions.entryPath)))
|
||
.orderBy(desc(agentInstructionRevisions.createdAt)).limit(500);
|
||
for (const row of rows) {
|
||
const content = Buffer.from(row.contentBase64, "base64");
|
||
if (singleFileAssetDigest(context.instructions.entryPath, content) !== pinned.digest) continue;
|
||
const bundle = await materializeAsset([{ path: context.instructions.entryPath, content, mode: 0o444 }]);
|
||
if (matches(bundle)) return { ...context, instructions: { ...context.instructions, bundle } };
|
||
}
|
||
}
|
||
const [agent] = await db.select().from(agents).where(and(eq(agents.companyId, binding.companyId), eq(agents.id, binding.agentId)));
|
||
if (agent) {
|
||
for (const agentFiles of [true, false]) {
|
||
const rebuilt = await materializeInstructionBundle(db, agent, agentFiles).catch(() => null);
|
||
if (rebuilt && rebuilt.entryPath === context.instructions.entryPath && matches(rebuilt.bundle)) {
|
||
return { ...context, instructions: { ...context.instructions, bundle: rebuilt.bundle } };
|
||
}
|
||
}
|
||
}
|
||
return context;
|
||
}
|
||
|
||
export async function rehydratePinnedSkillAssets(
|
||
db: Db,
|
||
binding: { companyId: string; runId: string; agentId: string },
|
||
context: NativeRuntimeContextSnapshot,
|
||
): Promise<NativeRuntimeContextSnapshot> {
|
||
if (!context.skills.length) return context;
|
||
const [run] = await db.select({ responsibleUserId: heartbeatRuns.responsibleUserId,
|
||
id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,
|
||
nativeIssueId: heartbeatRuns.nativeIssueId, runnerProfileJson: heartbeatRuns.runnerProfileJson,
|
||
contextSnapshot: heartbeatRuns.contextSnapshot }).from(heartbeatRuns).where(and(
|
||
eq(heartbeatRuns.companyId, binding.companyId), eq(heartbeatRuns.id, binding.runId), eq(heartbeatRuns.agentId, binding.agentId)));
|
||
if (!run) throw new Error("Skill recovery has no matching admitted run");
|
||
const actor = { type: "agent" as const, companyId: binding.companyId, agentId: binding.agentId,
|
||
runId: binding.runId, source: "agent_jwt" as const,
|
||
onBehalfOfUserId: run.responsibleUserId === "local-board" ? null : run.responsibleUserId };
|
||
const delivery = skillExecutionContent(db, { actor });
|
||
const pins = run.contextSnapshot?.skillVersions;
|
||
const skills: NativeRuntimeContextSnapshot["skills"] = [];
|
||
for (const skill of context.skills) {
|
||
if (!skill.versionId) {
|
||
// Newly admitted local assets remain usable during an ordinary rolling
|
||
// upgrade. A migrated asset uses its exact durable receipt on cold hosts.
|
||
const preserved = run.runnerProfileJson?.nativeRuntimeSkillPreservation;
|
||
if (!preserved) { skills.push(skill); continue; }
|
||
const bundle = await rehydratePreservedRuntimeSkill(db, run, skill, materializeSkillAsset);
|
||
skills.push({ ...skill, bundle });
|
||
continue;
|
||
}
|
||
if (pins !== undefined && (!pins || typeof pins !== "object" || Array.isArray(pins)
|
||
|| (pins as Record<string, unknown>)[skill.key] !== skill.versionId)) {
|
||
throw new Error("Skill recovery differs from the admitted version selection");
|
||
}
|
||
const [version] = await db.select({ skillId: companySkillVersions.companySkillId }).from(companySkillVersions).where(and(
|
||
eq(companySkillVersions.companyId, binding.companyId), eq(companySkillVersions.id, skill.versionId)));
|
||
if (!version) throw new Error("Pinned skill version is unavailable for recovery");
|
||
const bundle = await delivery.withVersion({ companyId: binding.companyId, skillId: version.skillId, versionId: skill.versionId },
|
||
source => materializeSkillAsset(source));
|
||
if (bundle.digest !== skill.bundle.digest || bundle.manifestDigest !== skill.bundle.manifestDigest
|
||
|| bundle.fileCount !== skill.bundle.fileCount || bundle.totalBytes !== skill.bundle.totalBytes) {
|
||
throw new Error("Recovered skill projection does not match its admitted digest");
|
||
}
|
||
skills.push({ ...skill, bundle });
|
||
}
|
||
return { ...context, skills };
|
||
}
|
||
|
||
/**
|
||
* The admitted context names every skill by digest, so the Runner's copy is
|
||
* described without local files. Pinned skill files are rehydrated on this
|
||
* server now, or, when reconnecting to a live remote Runner that already holds
|
||
* them, only if a launch must stage them again.
|
||
*/
|
||
export async function prepareRuntimeContextSource(
|
||
db: Db,
|
||
binding: { companyId: string; runId: string; agentId: string },
|
||
context: NativeRuntimeContextSnapshot,
|
||
options: { deferUntilStaged: boolean },
|
||
) {
|
||
// Defer only when preparation already checked access to every admitted skill:
|
||
// each one is pinned to its exact version in the run's saved selection.
|
||
const pinned = options.deferUntilStaged && context.skills.length > 0
|
||
? await db.select({ contextSnapshot: heartbeatRuns.contextSnapshot }).from(heartbeatRuns).where(and(
|
||
eq(heartbeatRuns.companyId, binding.companyId), eq(heartbeatRuns.id, binding.runId), eq(heartbeatRuns.agentId, binding.agentId)))
|
||
.then(([run]) => {
|
||
const pins = run?.contextSnapshot?.skillVersions;
|
||
return Boolean(pins && typeof pins === "object" && !Array.isArray(pins)
|
||
&& context.skills.every(skill => skill.versionId && (pins as Record<string, unknown>)[skill.key] === skill.versionId));
|
||
})
|
||
: options.deferUntilStaged;
|
||
let staged = pinned ? null : await rehydratePinnedSkillAssets(db, binding, context);
|
||
return {
|
||
described: staged ?? context,
|
||
forStaging: async () => (staged ??= await rehydratePinnedSkillAssets(db, binding, context)),
|
||
};
|
||
}
|
||
|
||
export async function materializeSelectedSkills(runtimeConfig: Record<string, unknown>, entries: PaperclipSkillEntry[], omitLegacy: boolean) {
|
||
const desiredKeys = resolvePaperclipDesiredSkillNames(runtimeConfig, entries).filter(
|
||
(key) => !omitLegacy || key !== PAPERCLIP_OPERATIONAL_SKILL_KEY,
|
||
);
|
||
const byKey = new Map(entries.map((entry) => [entry.key, entry]));
|
||
return Promise.all(desiredKeys.sort().map(async (key) => {
|
||
const entry = byKey.get(key);
|
||
if (!entry) throw new Error(`assigned runtime skill is missing from the company library: ${key}`);
|
||
if (entry.sourceStatus === "missing") throw new Error(entry.missingDetail ?? `assigned runtime skill source is missing: ${key}`);
|
||
const bundle = await materializeSkillAsset(entry.source);
|
||
return { key: entry.key, runtimeName: safeRelativePath(entry.runtimeName, `runtime name for ${key}`), versionId: entry.versionId ?? entry.currentVersionId ?? null, bundle };
|
||
}));
|
||
}
|
||
|
||
export async function resolveNativeRuntimeMcpSnapshot(input: { db: Db; agent: Pick<RuntimeAgent, "id" | "companyId">; runId: string }) {
|
||
const effective = await toolAccessService(input.db).getEffectiveToolAssignmentsForAgent(input.agent.companyId, input.agent.id);
|
||
const permitted = new Set([...effective.entries.filter((entry) => entry.effect === "include" && entry.connectionId).map((entry) => entry.connectionId!), ...effective.allowedTools.map((tool) => tool.connectionId)]);
|
||
const hasGitHubConnection = effective.installedConnections.some((connection) => {
|
||
const config = connection.config && typeof connection.config === "object"
|
||
? connection.config as Record<string, unknown>
|
||
: {};
|
||
const transportConfig = connection.transportConfig && typeof connection.transportConfig === "object"
|
||
? connection.transportConfig as Record<string, unknown>
|
||
: {};
|
||
return config.sourceTemplateKey === "github" || transportConfig.sourceTemplateKey === "github";
|
||
});
|
||
const hasVoiceConnection = effective.installedConnections.some(connection => connection.transport === "voice");
|
||
const [runIdentity] = hasGitHubConnection || hasVoiceConnection
|
||
? await input.db.select({ issueId: boundRunIssueIdSql(), responsibleUserId: heartbeatRuns.responsibleUserId, activeIdentityContextId: heartbeatRuns.activeIdentityContextId })
|
||
.from(heartbeatRuns)
|
||
.where(and(
|
||
eq(heartbeatRuns.id, input.runId),
|
||
eq(heartbeatRuns.companyId, input.agent.companyId),
|
||
eq(heartbeatRuns.agentId, input.agent.id),
|
||
))
|
||
.limit(1)
|
||
: [];
|
||
// Broker runs pin the permitted tool catalog, not one person’s credential.
|
||
// Selection happens at each operation and can change through steering.
|
||
const resolvedInstalledConnections = runIdentity?.activeIdentityContextId
|
||
? effective.installedConnections : await filterResolvedGitHubConnectionsForRun({
|
||
db: input.db,
|
||
companyId: input.agent.companyId,
|
||
agentId: input.agent.id,
|
||
responsibleUserId: runIdentity?.responsibleUserId ?? null,
|
||
connections: effective.installedConnections,
|
||
});
|
||
// App access is optional runtime context. Keep usable assignments pinned, but
|
||
// do not stop unrelated work because an assigned app needs attention.
|
||
const githubBotConnectionIds = await githubBotConnectionIdsForRun(input.db, input.agent.companyId, input.agent.id, input.runId);
|
||
const spekoConnectionIds = hasVoiceConnection ? new Set((await spekoToolsForSession(input.db, { companyId: input.agent.companyId, agentId: input.agent.id, runId: input.runId, issueId: runIdentity?.issueId ?? null, identityContextId: runIdentity?.activeIdentityContextId })).map(tool => tool.connectionId)) : new Set<string>();
|
||
const availableConnectionIds = new Set(resolvedInstalledConnections.filter((connection) =>
|
||
permitted.has(connection.id)
|
||
&& connection.status === "active"
|
||
&& connection.enabled
|
||
&& (Boolean(runIdentity?.activeIdentityContextId) && (connection.config?.sourceTemplateKey === "github" || connection.transportConfig?.sourceTemplateKey === "github")
|
||
|| connection.credentialPolicy === "per_user"
|
||
|| !isToolConnectionAttentionHealth(connection.healthStatus))
|
||
&& (["mcp_remote", "local_stdio"].includes(connection.transport) || isBrowserUseConnection(connection) || restConnectorFor(connection) || githubBotConnectionIds.has(connection.id) || spekoConnectionIds.has(connection.id))
|
||
).map((connection) => connection.id));
|
||
const assignment = {
|
||
version: 1,
|
||
agentId: input.agent.id,
|
||
connections: [...availableConnectionIds].sort(),
|
||
tools: effective.allowedTools
|
||
.filter((tool) => availableConnectionIds.has(tool.connectionId))
|
||
.map((tool) => tool.id)
|
||
.sort(),
|
||
};
|
||
const assignmentDigest = sha256(JSON.stringify(assignment));
|
||
return { assignmentSetId: `sha256:${assignmentDigest}`, digest: assignmentDigest, bindingId: assignment.connections.length ? `native-mcp:${input.runId}` : null };
|
||
}
|
||
|
||
export async function buildNativeRuntimeContext(input: { db: Db; agent: RuntimeAgent; runId: string; runtimeConfig: Record<string, unknown>; runtimeSkillEntries: PaperclipSkillEntry[]; instructionWorkingCopy?: { rootPath: string; entryPath: string; kind?: "agent_files" } }): Promise<NativeRuntimeContextSnapshot> {
|
||
const preconfigured = input.runtimeConfig.homeMode === "preconfigured";
|
||
const emptyMcpDigest = sha256("preconfigured");
|
||
const [instructions, skills, mcp] = await Promise.all([
|
||
measureRunPhase("runtime_context.instructions", () => materializeInstructionBundle(input.db, input.agent, input.instructionWorkingCopy?.kind === "agent_files")),
|
||
measureRunPhase("runtime_context.skills", () => preconfigured ? [] : materializeSelectedSkills(input.runtimeConfig, input.runtimeSkillEntries, input.agent.adapterType === "paperclip_runner")),
|
||
measureRunPhase("runtime_context.mcp", () => preconfigured ? { assignmentSetId: `sha256:${emptyMcpDigest}`, digest: emptyMcpDigest, bindingId: null } : resolveNativeRuntimeMcpSnapshot({ db: input.db, agent: input.agent, runId: input.runId })),
|
||
]);
|
||
const snapshot = {
|
||
prompt: { revision: PAPERCLIP_EXECUTION_PROMPT_REVISION, text: PAPERCLIP_EXECUTION_PROMPT, digest: nativeRuntimePromptDigest() },
|
||
instructions: { ...instructions, ...(input.instructionWorkingCopy ? { workingCopy: input.instructionWorkingCopy } : {}) },
|
||
skills,
|
||
mcp,
|
||
...(input.runtimeConfig.paperclipConnectionInstructions ? {
|
||
connectionInstructions: input.runtimeConfig.paperclipConnectionInstructions as { text: string; digest: string },
|
||
} : {}),
|
||
};
|
||
return parseNativeRuntimeContext({ ...snapshot, aggregateDigest: canonicalNativeRuntimeContextDigest(snapshot) });
|
||
}
|