mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 07:10:34 +01:00
## Thinking Path > - Paperclip manages AI agents and their work. > - The experimental Runner owns provider processes and durable sessions. > - Pi needs working task execution and human controls. > - The five-PR stack must preserve changes already on master. > - Each layer now carries the complete integrated source for a safe sequential fallback. > - This PR belongs to native GitHub stack #15602, ending at #14956. ## Linked Issues or Issue Description Refs #14436, #14631, #14743 and #14956. Ship Pi 1.0 through the experimental Paperclip Runner. The five PRs are #14921, #14922, #14923, #14924 and #14956. The user authorized the complete merge after checks pass. Existing `pi_local` execution is unchanged. Accounting and wider provider/platform qualification remain deferred. ## What Changed - Recover missing final replies after workspace finalization changes owners, using accepted-turn evidence without rerunning work or granting external-chat publication. - Preserve the admitted Pi instruction root across warm runs, while retaining changed-root rejection. - Give Pi a bounded 15-second default shutdown grace so stop, drain acknowledgement and durable suspension can complete. Explicit deadlines and other providers retain their existing behavior. - Integrate the Pi 1.0 runtime and master contracts. - Use Pi profile 22. Preserve explicit caller-selected models and exact native thinking levels. Keep Pi's wrapper, helper, extension and question/control behavior unchanged from the qualified profile-19 runtime. - Preserve master's Dot lifecycle and consent fields, configured task environment, status guards and current Codex/Claude dependency versions. Cursor stays qualified. Copilot stays pending; profile 17 binds the changed shared protocol validation sources. - Exclude general AWS IAM credentials from Pi static/custom provider bindings and selected task projections; preserve the provider-scoped Bedrock bearer key. Profile 21 is retained as historical provenance. Rust and cloud install probes use the current declaration. - Patch bundled brace-expansion 5.0.9 to the exact official 5.0.12 payload. Pin the patch and complete runtime closures. Include the patch in normal installed setup tooling. Keep the upstream Pi shrinkwrap as provenance and permit only this exact security correction. - Include current attestation files in the Docker build context. Keep the repository lockfile unchanged from master. CI and private image builds resolve manifest changes before their frozen installation. ## Verification - Full local `pnpm -r typecheck` passes, including Runner Rust, server and UI. Focused integration checks pass: 194 Runner admission/environment tests, 63 profile/credential tests with one expected skip, 152 Dot/UI configuration tests, and Pi transcript/notice tests. - Full local `pnpm build` passes on the final source. - Fresh final-source checks pass: all 698 Rust workspace tests (32 binaries), 156 credential/profile/controller tests with one expected skip, Runner TypeScript typecheck, and 20 package/setup/sandbox tests. - The profile-21 Pi materializer passes on the native host with the official pinned Node 24.21.0 and its npm. It verifies all 150 locked packages, the patched dependency and the exact closure. Setup/package bundle tests and UI token gates pass. - The old hashes were reproduced for all three supported targets before calculating the patched graph. New closure hashes are darwin-arm64 `282022db10150c6632b3444df421342e7d534bdf5d5fb1097a2e79d0625a2bcf`, darwin-x64 `64e251e19009f755c0b04f73ce2138246faab71a961b0f13d75ebfcc34bef12e`, and linux-x64 `713b1fdff42fb56a1518bdc084f181d70bee8ebadc3e4b1d76321ed9108c8410`. Independent native platform execution is separate from graph identity reproduction. - Historical cloud qualification remains unchanged: all seven core cases pass on shipping source `10dc43c9ec65d88c2f782d62afb296d09494f215`, harness `1a4408a48cfb5a1f094a311141c257c92cd7a893`, image `sha256:5b3a775b383591bda1b0c1889e509acc70ce7f37c53f09733c81d59037f02280`, and accepted Sonnet 4.6/low fixture. All 215 canonical files and all seven cleanup checks pass independent verification. These are profile-19 results and are not relabeled as fresh profile-22 runs. - Current Pi digest: `sha256:e92078bee3c23bec4100aa589013a44613d054cd686826534025d8019e9f39a9`. [The readiness plan](https://github.com/paperclipai/paperclip/blob/codex/pi-production-readiness/doc/plans/2026-10-02-pi-production-readiness.md) preserves campaign and failed-attempt provenance. - Merge only after every PR's current-head CI and fresh review pass. Linux CI covers the full suites, build and browser tests. The local embedded Postgres API-authority suite cannot start on this macOS/Node 26 host, so Linux CI must confirm that suite. ### Fresh profile-22 core qualification — 2026-10-08 All seven accepted core cases pass canonically on Pi profile 22, with `openrouter/anthropic/claude-sonnet-4.6` and native-confirmed low thinking. This model is a fixture; production accepts the caller's explicit Pi provider/model. Runtime/install source: `3241a992f2a7703e59e97ed0fd3e5d6405de4401`. Frozen accepted harness: `1a4408a48cfb5a1f094a311141c257c92cd7a893`. Immutable cloud image: `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:506f22db7edd78f37c0c40bec1cc084af1850455026dbf467194bfbb8fcef141`. Pi digest: `sha256:e92078bee3c23bec4100aa589013a44613d054cd686826534025d8019e9f39a9`. [Hosted Linux image and clean-install verification](https://github.com/paperclipai/paperclip/actions/runs/37868328023) passes, including all 20 source-bound archives, normal CLI/Pi setup, companion import and the production pack reader. This exact installation source includes the latest master integration and the corrected Pi warm instruction-root fence. Full local typecheck/build and current-head hosted CI verify the final stack. All 13 focused real-root regressions pass. The full local executor suite passed 662 tests; 15 database tests could not start the Mac embedded PostgreSQL service. Hosted Linux CI passes the full required verification and E2E checks. These fresh results keep their own source identity; profile-19 results remain historical. | Core path | Canonical campaign | Retained archive SHA-256 | | --- | --- | --- | | File edit, validation, download and Done | `pi-core22-replyfix-0-1791511228` | 23 files; `a473e8603a3dd4737863291f8d3d1e392391f0b16d433c3e0e0e9d8baf7a97b0` | | Pending question and controller restart | `pi-core22-replyfix-1-1791511376` | 33 files; `6b829c4eb74e1f32a89c692a4ae7130dbfc1c6d3cf13915effe2103d9e242c8e` | | Three-turn session/process/workspace continuity | `pi-core22-replyfix-2-1791511587` | 23 files; `7a87021f8f9a3fdd3c58bb4467f8d82c635e3ea4795d6e75f144d9aa14818df8` | | Four typed questions and browser reconnects | `pi-core22-replyfix-3-1791511881` | 42 files; `9e31755252be1f4f9cb0626c984c142d4d1ae5f5bee3a7af08444db8d12c280a` | | Plan approval and completion | `pi-core22-replyfix-4-1791512031` | 22 files; `a0383ce1aab38e7b5a25ce0e9dd3bebea5c037ebd96ae6b29dae19015da2ae2c` | | Same-turn steering and permission denial | `pi-core22-replyfix-5-1791512261` | 39 files; `c929b8c7070f0b66aedc17e65ca46e6beab1e363926ac9f7e2a75fb250f05949` | | Stop during pending permission | `pi-core22-replyfix-6-1791512390` | 33 files; `7f0a58ae0f4d5bfc76149435f4e322537089c5bd16e7ffe9b5ad71f10a621a07` | All 215 canonical files (28714587 bytes) are independently hash-verified. All seven cleanup grades pass, with no owned runtime process or temporary root after each case. Automatic retries are zero. The owned cloud host stopped normally after retention. The prior profile-22 warm attempt remains failed and separately retained: archive SHA-256 `1e54eba5ec72b50cee1534b23d1d1d4f21a090006b8a64501ba70db972abfde5`. Its original canonical classification is preserved. Diagnosis reproduced a product bug comparing an agent-files root against an unset checkpoint-only field. The fix stores the admitted physical root separately from the adopted per-run collection capability. The real-root regression fails before the fix and passes afterward, including rejection of a changed physical root. Fixture, grader, model and all seven accepted case IDs are unchanged; this fresh campaign tests final-reply publication after file registration first. The intermediate restart attempt also remains failed and retained: archive SHA-256 `5dcaefdf1d17cf4cd54fd4cf810f45e736667392339b8ce7caf08bb4e225277f`. Its original canonical classification is preserved. Pi resumed, wrote the verified answer and completed its task; exact runner suspension was proven, but idle stop consumed about 5.2s and left under 3s for the drain acknowledgement. The Pi-only default shutdown grace is now 15s, preserving a full 5s drain round trip and a finite suspension reserve. Explicit caller deadlines, other provider defaults, literal drain receipts and exact suspension identity checks remain unchanged. The timing regression fails before this correction and passes afterward; all 18 focused settlement tests and Runner typecheck pass. The final-source file attempt is also preserved as failed (`candidate_failure`), archive SHA-256 `db6767b6773ea618997927ac77bdb005a5ac81492c7b9c0ffbc900449f829bc9`. Native edit, validation, exact downloadable artifact and Done/succeeded all passed, and the exact final reply was durably recorded. A workspace recovery owner completed before the live heartbeat reached presentation, leaving that reply absent from task chat. Recovery now materializes only a completed final reply from the accepted turn of an ordinary internal Done task, preserving issue/run/contract binding, suppression, external-chat authorization and same-run deduplication. The database regression covers the generated file-preparation receipt, suppression, unapproved external continuation and replay. Server typecheck and all 49 response-selection tests pass; hosted Linux verifies the database regression because embedded PostgreSQL cannot start on this Mac. The delayed-final-answer database regression passes on [the final root-source Linux server shard](https://github.com/paperclipai/paperclip/actions/runs/37868262553/job/113628594152), alongside 1,108 passing tests. The first root Runner shard had one unchanged durable-resume test exceed its 5-second timeout; the identical top-source shard and the isolated exact test passed. One rerun of that failed job and its required aggregate passed without source or test changes. The original failed job log and the single-rerun receipt remain retained. ### October 9 merge verification Current merge head: `5a8fe63512a7166aaef5cf50065a25008aa8b44b`. All current-head checks pass, including `ci / verify` and `ci / e2e`; exact-head Greptile review is 5/5 with no unresolved threads. Current master conflicts are resolved. The user authorized the maintainer override of the code-owner review gate after these checks. The seven retained live core cases remain bound to source `3241a992f2a7703e59e97ed0fd3e5d6405de4401` and its recorded cloud image. ## Risks - The security correction changes the dependency closure and profile identity. Old sessions must reopen on the new profile. Exact identities and credential bindings fail closed. - The runner remains experimental and requires explicit selection. Legacy Pi Local is unchanged. Caller model IDs pass through; the E2E model is a fixture. - Accounting and the broad platform/provider matrix remain deferred. This merge does not publish a release or deploy a service. ## Model Used OpenAI GPT-6 through Codex assisted with reasoning, repository inspection, editing and tool use. The exact serving ID and context window are not exposed in this session. Final live qualification uses Pi 1.0.0 with `openrouter/anthropic/claude-sonnet-4.6` and native-confirmed low thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
427 lines
32 KiB
TypeScript
427 lines
32 KiB
TypeScript
import { agentDirectoryWorkingCopyService, isAgentDirectoryCopy } from "./agent-directory-working-copies.js";
|
|
import { createHash } from "node:crypto";
|
|
import { execFile as execFileCallback } from "node:child_process";
|
|
import { promisify } from "node:util";
|
|
import fs from "node:fs/promises";
|
|
import path from "node:path";
|
|
import { and, asc, desc, eq, inArray, lte, or, isNull, isNotNull, sql } from "drizzle-orm";
|
|
import { agents, heartbeatRuns, agentInstructionWorkingCopies as copies, type Db } from "@paperclipai/db";
|
|
import {
|
|
prepareAdapterExecutionTargetRuntime,
|
|
runAdapterExecutionTargetShellCommand,
|
|
type AdapterExecutionTarget,
|
|
} from "@paperclipai/adapter-utils/execution-target";
|
|
import { conflict, notFound } from "../errors.js";
|
|
import { agentInstructionsService, agentInstructionsBundleMode } from "./agent-instructions.js";
|
|
import { agentInstructionRevisionService } from "./agent-instruction-revisions.js";
|
|
import { resolveInstructionActor } from "./agent-instruction-authorization.js";
|
|
import { assertInstructionPathSafe, instructionBytes, instructionPath, materializeInstructionBytes, MAX_INSTRUCTION_BYTES, readInstructionBytes, instructionGitExcludeProgram } from "./agent-instruction-files.js";
|
|
import { hasNativeLocalProcessStop } from "./native-local-process-stop.js";
|
|
import { remoteExecutionHasStopped } from "./remote-execution-termination.js";
|
|
import type { AuthorizationActor } from "./authorization.js";
|
|
import type { EnvironmentRuntimeService } from "./environment-runtime.js";
|
|
import { logger } from "../middleware/logger.js";
|
|
|
|
const execFile = promisify(execFileCallback);
|
|
type Copy = typeof copies.$inferSelect;
|
|
const hash = (bytes: Uint8Array) => createHash("sha256").update(bytes).digest("hex");
|
|
const quote = (value: string) => `'${value.replaceAll("'", `'"'"'`)}'`;
|
|
const completed = new Set(["saved", "unchanged", "resolved"]);
|
|
const MAX_COLLECTION_ATTEMPTS = 3;
|
|
// Only orchestration can register a live target; tools and HTTP callers cannot
|
|
// supply one. These handles are optional: durable captured bytes survive restart.
|
|
const liveTargets = new Map<string, AdapterExecutionTarget>();
|
|
const targetKey = (companyId: string, runId: string) => `${companyId}:${runId}`;
|
|
|
|
/** Finish only bounded immediate collection work; deferred ownership remains pending. */
|
|
export async function collectStoppedInstructionCopyWithRetries<T extends Pick<Copy, "state" | "attempts" | "nextAttemptAt">>(collect: () => Promise<T | null>): Promise<T | null> {
|
|
let saved: T | null = null;
|
|
let previousAttempts = -1;
|
|
for (let calls = 0; calls < MAX_COLLECTION_ATTEMPTS; calls++) {
|
|
saved = await collect();
|
|
if (!saved || saved.state !== "pending_collection" || saved.nextAttemptAt === null
|
|
|| saved.attempts >= MAX_COLLECTION_ATTEMPTS || saved.attempts <= previousAttempts) break;
|
|
previousAttempts = saved.attempts;
|
|
}
|
|
return saved;
|
|
}
|
|
|
|
export function instructionWorkingCopyGuidance(copy: Pick<Copy, "executionRoot" | "entryFile" | "receipt">) {
|
|
if (isAgentDirectoryCopy(copy)) return `Your persistent agent directory is ${copy.executionRoot} (AGENT_HOME). Your instruction entry is ${copy.executionRoot}/${copy.entryFile}. Read and write your own files and subfolders there. This directory belongs to this agent across tasks and sessions; task files belong in the task working directory. Paperclip restores this directory before execution and saves validated changes at turn boundaries. A warm native Codex session keeps the same writable directory between turns. Other sessions collect after the provider stops. Regular files, including binary files, persist; symlinks and special files are unsupported. Check the agent-files save receipt before claiming persistence; Only files you change or delete are synchronized. If another run changes the same file, the last completed synchronization wins. Temporary copies are removed when the owning session stops; there is no per-run file history. Storage allows 256 MiB per file, 2 GiB total, and 100,000 entries; the instruction entry must remain UTF-8 and at most 1 MiB. Reaching a storage limit never prevents this or future tasks from running. Remove or shrink files to free space; changes that exceed the limits will not be saved.${typeof copy.receipt?.storageWarning === "string" ? `\n\n${copy.receipt.storageWarning}` : ""}`;
|
|
return `Your editable agent instruction file is ${copy.executionRoot}/${copy.entryFile}. Edit this registered private copy normally. After this run stops, Paperclip saves changed content as a persistent revision if your responsible user still has permission and the baseline has not changed. Check the run's instruction-save receipt before claiming persistence. Use read_agent_instructions, update_agent_instructions, get_agent_instruction_history, and restore_agent_instructions for immediate saves and history. Read first and pin the returned revision. Preserve conflicts; never silently retry against a newer head. Repository instructions, skills, and the loaded prompt are separate and are not collected.`;
|
|
}
|
|
|
|
/** Remote reads use the registered entry only; never scan for files called AGENTS.md. */
|
|
export function instructionCollectionScript(root: string, entry: string) {
|
|
instructionPath(entry);
|
|
return `node -e ${quote([
|
|
"const fs=require('node:fs'),path=require('node:path')",
|
|
"const root=process.argv[1],entry=process.argv[2],max=Number(process.argv[3])",
|
|
"const absolute=path.resolve(root,entry),parts=absolute.split('/').filter(Boolean)",
|
|
"let current='/'",
|
|
"for(let i=0;i<parts.length;i++){current=path.join(current,parts[i]);let s;try{s=fs.lstatSync(current)}catch(e){if(e.code==='ENOENT'){process.stdout.write(JSON.stringify({missing:true}));process.exit(0)}throw e}if(s.isSymbolicLink()||(i<parts.length-1?!s.isDirectory():!s.isFile()))throw Error('INSTRUCTION_PATH_INVALID')}",
|
|
"const fd=fs.openSync(absolute,fs.constants.O_RDONLY|fs.constants.O_NOFOLLOW)",
|
|
"try{const s=fs.fstatSync(fd);if(!s.isFile()||s.size>max)throw Error('INSTRUCTION_CONTENT_INVALID');const bytes=Buffer.alloc(max+1);let n=0;while(n<bytes.length){const count=fs.readSync(fd,bytes,n,bytes.length-n,n);if(!count)break;n+=count}if(n>max)throw Error('INSTRUCTION_CONTENT_INVALID');process.stdout.write(JSON.stringify({contentBase64:bytes.subarray(0,n).toString('base64')}))}finally{fs.closeSync(fd)}",
|
|
].join(";"))} ${quote(root)} ${quote(entry)} ${MAX_INSTRUCTION_BYTES}`;
|
|
}
|
|
|
|
|
|
export function agentInstructionWorkingCopyService(db: Db, options: { environmentRuntime?: EnvironmentRuntimeService } = {}) {
|
|
const revisions = agentInstructionRevisionService(db);
|
|
const scope = (companyId: string, runId: string) => and(eq(copies.companyId, companyId), eq(copies.runId, runId));
|
|
async function get(companyId: string, runId: string) {
|
|
const [row] = await db.select().from(copies).where(scope(companyId, runId));
|
|
return row ?? null;
|
|
}
|
|
async function patch(row: Copy, values: Partial<typeof copies.$inferInsert>) {
|
|
const [updated] = await db.update(copies).set({ ...values, updatedAt: new Date() }).where(and(
|
|
scope(row.companyId, row.runId), eq(copies.state, row.state), eq(copies.attempts, row.attempts), eq(copies.baseHash, row.baseHash),
|
|
sql`${copies.receipt}->>'retainedByRunId' IS NOT DISTINCT FROM ${typeof row.receipt?.retainedByRunId === "string" ? row.receipt.retainedByRunId : null}`,
|
|
)).returning();
|
|
// A late cleanup must not overwrite an explicit resolution or a newer
|
|
// baseline. Canonical content has its own independent head CAS.
|
|
return updated ?? (await get(row.companyId, row.runId))!;
|
|
}
|
|
function actorFor(row: Copy): AuthorizationActor {
|
|
return { type: "agent", companyId: row.companyId, agentId: row.agentId, runId: row.runId, onBehalfOfUserId: row.responsibleUserId };
|
|
}
|
|
const directories = agentDirectoryWorkingCopyService(db, get, patch, options.environmentRuntime);
|
|
async function recoverDirectoryCleanup(row: Copy, cleanup: () => Promise<unknown>) {
|
|
// Reserve a later retry before I/O, including lock waits. A blocked owner
|
|
// must not occupy every sweep or prevent other copies from being reclaimed.
|
|
try {
|
|
await patch(row, { nextAttemptAt: new Date(Date.now() + 30_000) });
|
|
await cleanup();
|
|
}
|
|
catch (error) {
|
|
try { await patch(row, { nextAttemptAt: new Date(Date.now() + 30_000) }); }
|
|
catch (retryError) {
|
|
logger.warn({ err: retryError, runId: row.runId }, "Agent file cleanup retry could not be scheduled");
|
|
}
|
|
logger.warn({ err: error, runId: row.runId }, "Agent file cleanup deferred; save receipt unchanged");
|
|
}
|
|
}
|
|
async function prepare(input: { companyId: string; agentId: string; runId: string; target?: AdapterExecutionTarget | null; cwd: string; legacy?: boolean; warm?: boolean; reuseRunId?: string; onWarmHandoff?: (copy: Copy) => void }) {
|
|
const existing = await get(input.companyId, input.runId);
|
|
if (isAgentDirectoryCopy(existing) || (!existing && !input.legacy)) return directories.prepare(input);
|
|
let refreshStoppedCopy = false;
|
|
const workspace = await fs.realpath(input.cwd);
|
|
const expectedLocalRoot = path.join(workspace, ".paperclip-runtime", `instruction-edits-${input.runId}`, "instructions");
|
|
if (existing && existing.localRoot !== expectedLocalRoot) throw conflict("The registered instruction copy belongs to a different run workspace");
|
|
const location = input.target?.kind === "remote" ? `remote:${input.target.environmentId ?? ""}` : "local";
|
|
if (existing && (existing.agentId !== input.agentId || existing.location !== location)) {
|
|
throw conflict("The registered instruction copy belongs to a different run environment");
|
|
}
|
|
// Preserve in-flight bytes and their CAS baseline. A completed, stopped copy
|
|
// can start a fresh lifecycle only after its recorded bytes are accounted for.
|
|
if (existing && existing.state !== "preparing") {
|
|
if (["conflict", "pending_commit", "unavailable"].includes(existing.state)) {
|
|
throw conflict("Resolve this run's preserved instruction candidate before editing its working copy again");
|
|
}
|
|
if (input.target) liveTargets.set(targetKey(input.companyId, input.runId), input.target);
|
|
if (completed.has(existing.state) && existing.processStoppedAt) {
|
|
const privateBytes = await readCandidate(existing, input.target);
|
|
// Refresh only bytes whose previous lifecycle is complete. A changed
|
|
// private file must keep its old CAS fence; never silently rebase edits.
|
|
refreshStoppedCopy = privateBytes === null || hash(privateBytes) === (existing.candidateHash ?? existing.baseHash);
|
|
if (!refreshStoppedCopy && privateBytes !== null) {
|
|
// A failed staging attempt can already have replaced the file before
|
|
// the completed row advances. Exact current canonical bytes are also
|
|
// accounted for; any other private edit still keeps its old fence.
|
|
const current = await revisions.readCommittedForRuntime({ companyId: input.companyId, agentId: input.agentId });
|
|
refreshStoppedCopy = current?.revision.entryFile === existing.entryFile
|
|
&& current.revision.contentHash === hash(privateBytes);
|
|
}
|
|
}
|
|
if (!refreshStoppedCopy) {
|
|
if (completed.has(existing.state) || existing.state === "unchanged_turn") {
|
|
// unchanged_turn is a live warm-owner observation, not stop proof.
|
|
return patch(existing, { state: "prepared", candidateBase64: null, candidateHash: null,
|
|
receipt: null, processStoppedAt: null, attempts: 0, nextAttemptAt: null });
|
|
}
|
|
return existing;
|
|
}
|
|
}
|
|
const [agent] = await db.select().from(agents).where(and(eq(agents.id, input.agentId), eq(agents.companyId, input.companyId)));
|
|
if (!agent) throw notFound("Agent not found");
|
|
if (agentInstructionsBundleMode(agent) !== "managed") return null;
|
|
const bound = await resolveInstructionActor(db, { type: "agent", companyId: input.companyId, agentId: input.agentId, runId: input.runId });
|
|
const baseline = existing?.baseRevisionId && !refreshStoppedCopy
|
|
? await revisions.readRevision({ companyId: input.companyId, agentId: input.agentId, entryFile: existing.entryFile, revisionId: existing.baseRevisionId }, bound).catch(async (error) => {
|
|
if ((error as { status?: number }).status !== 404) throw error;
|
|
const current = await revisions.readCurrent({ companyId: input.companyId, agentId: input.agentId }, bound);
|
|
if (current?.revision.id !== existing.baseRevisionId) throw conflict("The instruction baseline changed during preparation");
|
|
return current;
|
|
})
|
|
: await revisions.readCurrent({ companyId: input.companyId, agentId: input.agentId }, bound);
|
|
if (!baseline) return null;
|
|
const exported = await agentInstructionsService().exportFiles(agent, { rejectSymlinks: true });
|
|
const entryFile = instructionPath(baseline?.revision.entryFile ?? exported.entryFile);
|
|
const content = instructionBytes(baseline?.content ?? exported.files[entryFile] ?? "");
|
|
const localRoot = existing?.localRoot ?? expectedLocalRoot;
|
|
const target = input.target?.kind === "remote" ? input.target : null;
|
|
const executionRoot = target
|
|
? path.posix.join(target.remoteCwd, ".paperclip-runtime", `instruction-edits-${input.runId}`, "instructions")
|
|
: localRoot;
|
|
if (!existing) {
|
|
await db.insert(copies).values({
|
|
runId: input.runId, companyId: input.companyId, agentId: input.agentId,
|
|
responsibleUserId: bound.onBehalfOfUserId!, entryFile,
|
|
baseRevisionId: baseline?.revision.id ?? null, baseHash: hash(content),
|
|
localRoot, executionRoot, location, state: "preparing",
|
|
});
|
|
}
|
|
const row = (await get(input.companyId, input.runId))!;
|
|
await assertInstructionPathSafe(localRoot, entryFile);
|
|
await execFile(process.execPath, ["-e", instructionGitExcludeProgram, workspace], { timeout: 15_000 });
|
|
await fs.mkdir(localRoot, { recursive: true, mode: 0o700 });
|
|
for (const [name, text] of Object.entries({ ...exported.files, [entryFile]: content.toString("utf8") })) {
|
|
const relative = instructionPath(name);
|
|
const filename = await assertInstructionPathSafe(localRoot, relative);
|
|
await fs.mkdir(path.dirname(filename), { recursive: true, mode: 0o700 });
|
|
// Atomic replacement also allows interrupted preparation to restage an
|
|
// already-read-only sibling without writing through its existing inode.
|
|
await materializeInstructionBytes(localRoot, relative, instructionBytes(text));
|
|
await fs.chmod(filename, name === entryFile ? 0o600 : 0o400);
|
|
}
|
|
if (target) {
|
|
const staged = await prepareAdapterExecutionTargetRuntime({
|
|
target, runId: input.runId, adapterKey: `instruction-edits-${input.runId}`,
|
|
workspaceLocalDir: input.cwd, syncWorkspace: false,
|
|
assets: [{ key: "instructions", localDir: localRoot, followSymlinks: false }],
|
|
});
|
|
if (staged.assetDirs.instructions !== executionRoot) throw new Error("Instruction copy staging path changed");
|
|
const excluded = await runAdapterExecutionTargetShellCommand(input.runId, target,
|
|
`node -e ${quote(instructionGitExcludeProgram)} ${quote(target.remoteCwd)}`, { cwd: target.remoteCwd, env: {}, timeoutSec: 15 });
|
|
if (excluded.exitCode !== 0 || excluded.timedOut) throw new Error("Could not exclude private instructions from Git staging");
|
|
}
|
|
// A refresh does not supersede the completed turn until every local and
|
|
// remote staging step succeeds. Keep its receipt and stop evidence intact
|
|
// if preparation throws or the controller stops midway through staging.
|
|
const prepared = await patch(row, { state: "prepared", ...(refreshStoppedCopy ? {
|
|
entryFile, baseRevisionId: baseline.revision.id, baseHash: hash(content),
|
|
candidateBase64: null, candidateHash: null, receipt: null, processStoppedAt: null,
|
|
attempts: 0, nextAttemptAt: null, errorCode: null, errorMessage: null,
|
|
} : {}) });
|
|
if (input.target) liveTargets.set(targetKey(input.companyId, input.runId), input.target);
|
|
return prepared;
|
|
}
|
|
|
|
async function readCandidate(row: Copy, target?: AdapterExecutionTarget | null) {
|
|
if (row.location === "local") return readInstructionBytes(row.localRoot, row.entryFile);
|
|
if (target?.kind !== "remote") throw new Error("The original execution environment is needed to retrieve this instruction copy");
|
|
if (row.location !== `remote:${target.environmentId ?? ""}`) throw new Error("Instruction copy execution environment changed");
|
|
const expected = path.posix.join(target.remoteCwd, ".paperclip-runtime", `instruction-edits-${row.runId}`, "instructions");
|
|
if (expected !== row.executionRoot) throw new Error("Instruction copy execution environment changed");
|
|
const result = await runAdapterExecutionTargetShellCommand(row.runId, target, instructionCollectionScript(row.executionRoot, row.entryFile), { cwd: target.remoteCwd, env: {}, timeoutSec: 30 });
|
|
if (result.exitCode !== 0 || result.timedOut) throw new Error("Could not safely retrieve the stopped run's instruction file");
|
|
const payload = JSON.parse(result.stdout) as { missing?: boolean; contentBase64?: string };
|
|
if (payload.missing) return null;
|
|
if (typeof payload.contentBase64 !== "string" || payload.contentBase64.length > Math.ceil(MAX_INSTRUCTION_BYTES / 3) * 4) throw new Error("Instruction response exceeds the collection bound");
|
|
return instructionBytes(Buffer.from(payload.contentBase64, "base64"));
|
|
}
|
|
|
|
/** A terminal-turn probe is not a save: changed/invalid copies require a
|
|
* stopped provider before capture. Unchanged warm providers remain reusable. */
|
|
async function hasChanges(input: { companyId: string; runId: string; target?: AdapterExecutionTarget | null }) {
|
|
const row = await get(input.companyId, input.runId);
|
|
if (!row || completed.has(row.state)) return false;
|
|
if (isAgentDirectoryCopy(row)) return directories.hasChanges(row, input.target);
|
|
try {
|
|
const content = await readCandidate(row, input.target);
|
|
if (content !== null && hash(content) === row.baseHash) {
|
|
// This is an unchanged turn receipt, not stopped-process collection.
|
|
// No candidate bytes or new revision are persisted from a live owner.
|
|
await patch(row, { state: "unchanged_turn", nextAttemptAt: null });
|
|
return false;
|
|
}
|
|
return true;
|
|
} catch { return true; }
|
|
}
|
|
async function checkpointWarm(input: { companyId: string; runId: string; target?: AdapterExecutionTarget | null }) {
|
|
const row = await get(input.companyId, input.runId);
|
|
return row && isAgentDirectoryCopy(row) ? directories.checkpointWarm(row, input.target) : row;
|
|
}
|
|
|
|
async function acknowledgeExplicitSave(input: { companyId: string; agentId: string; runId: string; entryFile: string; revisionId: string; contentHash: string }) {
|
|
const row = await get(input.companyId, input.runId);
|
|
if (!row || isAgentDirectoryCopy(row) || row.agentId !== input.agentId || row.entryFile !== input.entryFile || row.state !== "prepared") return;
|
|
const target = liveTargets.get(targetKey(row.companyId, row.runId));
|
|
const content = await readCandidate(row, target).catch(() => null);
|
|
// Never advance a stale local edit past an unrelated explicit update.
|
|
if (content === null || hash(content) !== input.contentHash) return;
|
|
await db.update(copies).set({ baseRevisionId: input.revisionId, baseHash: input.contentHash, updatedAt: new Date() })
|
|
.where(and(scope(row.companyId, row.runId), eq(copies.state, "prepared"), eq(copies.baseHash, row.baseHash)));
|
|
}
|
|
|
|
async function commitCandidate(row: Copy) {
|
|
if (isAgentDirectoryCopy(row)) return directories.collectStopped(row);
|
|
if (row.candidateBase64 === null || completed.has(row.state) || row.state === "conflict") return row;
|
|
try {
|
|
const receipt = await revisions.commit({
|
|
companyId: row.companyId, agentId: row.agentId, entryFile: row.entryFile,
|
|
baseRevisionId: row.baseRevisionId, content: Buffer.from(row.candidateBase64, "base64"), source: "cleanup",
|
|
}, actorFor(row));
|
|
return patch(row, { state: "saved", receipt: { ...receipt }, errorCode: null, errorMessage: null, nextAttemptAt: null });
|
|
} catch (error) {
|
|
const detail = error as { status?: number; details?: { code?: string }; message?: string };
|
|
const code = detail.details?.code ?? "INSTRUCTION_SAVE_FAILED";
|
|
const retryable = !detail.status || detail.status >= 500;
|
|
const message = detail.status === 403
|
|
? `${detail.message ?? "Current permissions do not allow this instruction save."} The candidate was preserved.`
|
|
: detail.status === 409
|
|
? "Instructions changed after this run started. Review the preserved candidate against the current revision."
|
|
: "Instruction edits were preserved but not saved. Review the candidate before retrying.";
|
|
return patch(row, { state: retryable ? "pending_commit" : "conflict", errorCode: code, errorMessage: message, nextAttemptAt: retryable && row.attempts < MAX_COLLECTION_ATTEMPTS ? new Date(Date.now() + 30_000) : null });
|
|
}
|
|
}
|
|
|
|
/** Caller must have joined the owned provider process before granting this proof. */
|
|
async function collectStopped(input: { companyId: string; runId: string; target?: AdapterExecutionTarget | null }) {
|
|
let row = await get(input.companyId, input.runId);
|
|
if (!row) return row;
|
|
if (isAgentDirectoryCopy(row)) return directories.collectStopped(row, input.target);
|
|
if (completed.has(row.state) || row.state === "conflict") return row;
|
|
row = await patch(row, { processStoppedAt: row.processStoppedAt ?? new Date(), attempts: row.attempts + 1 });
|
|
if (completed.has(row.state) || row.state === "conflict") return row;
|
|
if (row.candidateBase64 !== null) return commitCandidate(row);
|
|
try {
|
|
const candidate = await readCandidate(row, input.target);
|
|
if (candidate === null) {
|
|
return patch(row, { state: "unavailable", errorCode: "INSTRUCTION_FILE_MISSING", errorMessage: "The registered instruction file was removed. Canonical instructions were kept; no edits were saved.", nextAttemptAt: null });
|
|
}
|
|
if (hash(candidate) === row.baseHash) return patch(row, { state: "unchanged", nextAttemptAt: null });
|
|
// Save the candidate durably before attempting authorization/CAS or releasing the environment.
|
|
row = await patch(row, { state: "pending_commit", candidateBase64: candidate.toString("base64"), candidateHash: hash(candidate), nextAttemptAt: new Date() });
|
|
return commitCandidate(row);
|
|
} catch (error) {
|
|
return patch(row, { state: row.attempts < MAX_COLLECTION_ATTEMPTS ? "pending_collection" : "unavailable", errorCode: "INSTRUCTION_COLLECTION_FAILED", errorMessage: "The registered instruction copy could not be read safely. Canonical instructions were kept; no save receipt exists.", nextAttemptAt: row.attempts < MAX_COLLECTION_ATTEMPTS ? new Date(Date.now() + 30_000) : null });
|
|
}
|
|
}
|
|
|
|
/** Restarts can retry persisted bytes without touching a live filesystem or launching a model. */
|
|
async function recoverCaptured() {
|
|
const pending = await db.select().from(copies).where(and(
|
|
eq(copies.state, "pending_commit"),
|
|
or(isNull(copies.nextAttemptAt), lte(copies.nextAttemptAt, new Date())),
|
|
lte(copies.attempts, MAX_COLLECTION_ATTEMPTS - 1),
|
|
)).limit(20);
|
|
for (const row of pending) {
|
|
const result = await commitCandidate(await patch(row, { attempts: row.attempts + 1 }));
|
|
if (isAgentDirectoryCopy(result) && completed.has(result.state)) await directories.release(result);
|
|
}
|
|
// A crash between a durable save and release must not retain run snapshots
|
|
// forever. Only discard copies with recorded stop proof and a terminal receipt.
|
|
const cleanup = await db.select().from(copies).where(and(
|
|
inArray(copies.state, [...completed, "unavailable"]), isNotNull(copies.processStoppedAt),
|
|
sql`${copies.receipt}->>'schema' = 'paperclip.agent-files.v1'`,
|
|
or(sql`${copies.receipt} ? 'baseline'`, sql`${copies.receipt}->>'cleanupPending' = 'true'`),
|
|
or(isNull(copies.nextAttemptAt), lte(copies.nextAttemptAt, new Date())),
|
|
)).orderBy(asc(copies.updatedAt)).limit(20);
|
|
for (const row of cleanup) await recoverDirectoryCleanup(row, () => directories.release(row));
|
|
return pending.length;
|
|
}
|
|
|
|
/** Recover only the registered copy after independently recorded process-stop
|
|
* proof. Terminal run status is never a substitute for that proof. Remote
|
|
* receipts mean the environment is stopped; do not restart it just to read. */
|
|
async function recoverStopped() {
|
|
const pending = await db.select({ copy: copies, runtimeMode: heartbeatRuns.runtimeMode }).from(copies)
|
|
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, copies.companyId), eq(heartbeatRuns.id, copies.runId)))
|
|
.where(and(or(and(or(inArray(copies.state, ["prepared", "pending_collection", "unchanged_turn", "warm_saved"]),
|
|
and(eq(copies.state, "preparing"), sql`${copies.receipt}->>'schema' = 'paperclip.agent-files.v1'`)),
|
|
lte(copies.attempts, MAX_COLLECTION_ATTEMPTS - 1)),
|
|
and(eq(copies.state, "unavailable"), isNull(copies.processStoppedAt),
|
|
sql`${copies.location} like 'remote:%'`,
|
|
sql`${copies.receipt}->>'schema' = 'paperclip.agent-files.v1'`)),
|
|
sql`NOT (coalesce(${copies.receipt}, '{}'::jsonb) ? 'retainedByRunId')`,
|
|
inArray(heartbeatRuns.status, ["succeeded", "failed", "cancelled", "timed_out", "interrupted"]),
|
|
or(isNull(copies.nextAttemptAt), lte(copies.nextAttemptAt, new Date())))).orderBy(asc(copies.updatedAt)).limit(20);
|
|
for (const { copy: row, runtimeMode } of pending) {
|
|
if (row.state === "unavailable" && isAgentDirectoryCopy(row)) {
|
|
await recoverDirectoryCleanup(row, () => directories.recoverUnavailable(row));
|
|
} else if (row.state === "preparing" && isAgentDirectoryCopy(row)) {
|
|
// A provider cannot launch until preparation records "prepared". With
|
|
// the owning run terminal, this is an interrupted staging copy only.
|
|
await directories.release(await patch(row, { state: "unavailable", processStoppedAt: new Date(),
|
|
errorCode: "AGENT_FILES_PREPARE_FAILED", errorMessage: "Agent-file preparation was interrupted. The temporary copy was discarded.", nextAttemptAt: null }));
|
|
} else if (row.location === "local" && (row.processStoppedAt || runtimeMode === "native" &&
|
|
await hasNativeLocalProcessStop(db, row.companyId, row.runId))) {
|
|
const result = await collectStopped({ companyId: row.companyId, runId: row.runId });
|
|
if (result && isAgentDirectoryCopy(result) && completed.has(result.state)) await directories.release(result);
|
|
} else if (row.location !== "local" && await remoteExecutionHasStopped(db, row.companyId, row.runId)) {
|
|
if (row.receipt?.warm === true) {
|
|
const unavailable = row.receipt?.warm === true
|
|
? await patch(row, { state: "unavailable", errorCode: "AGENT_FILES_FINAL_COLLECTION_UNAVAILABLE",
|
|
errorMessage: "The last completed checkpoint remains saved. Files changed afterwards could not be collected after remote termination.", nextAttemptAt: null })
|
|
: await reportUnavailable(row.companyId, row.runId);
|
|
if (unavailable && isAgentDirectoryCopy(unavailable)) {
|
|
await directories.release(await patch(unavailable, { processStoppedAt: unavailable.processStoppedAt ?? new Date() }));
|
|
}
|
|
} else {
|
|
if (isAgentDirectoryCopy(row)) await directories.recoverStoppedRemote(row);
|
|
else await reportUnavailable(row.companyId, row.runId);
|
|
}
|
|
} else if (row.state === "prepared") {
|
|
await patch(row, { state: "pending_collection", errorCode: "INSTRUCTION_STOP_UNCONFIRMED",
|
|
errorMessage: "The provider's stop has not been confirmed. Instruction collection is pending; no save is claimed.", nextAttemptAt: new Date(Date.now() + 30_000) });
|
|
} else if (["warm_saved", "pending_collection", "unchanged_turn"].includes(row.state)) {
|
|
// Live retained sessions must not occupy every batch and starve stopped
|
|
// copies from other agents. This does not permit reads without stop proof.
|
|
await patch(row, { nextAttemptAt: new Date(Date.now() + 30_000) });
|
|
}
|
|
}
|
|
return pending.length;
|
|
}
|
|
|
|
async function list(companyId: string, agentId: string, actor: AuthorizationActor) {
|
|
await revisions.readCurrent({ companyId, agentId }, actor);
|
|
const rows = await db.select().from(copies).where(and(eq(copies.companyId, companyId), eq(copies.agentId, agentId), inArray(copies.state, ["conflict", "pending_collection", "pending_commit", "unavailable"]))).orderBy(desc(copies.createdAt)).limit(50);
|
|
return rows.map(row => ({ runId: row.runId, entryFile: row.entryFile, baseRevisionId: row.baseRevisionId,
|
|
baseHash: row.baseHash, state: row.state as "conflict" | "pending_collection" | "pending_commit" | "unavailable",
|
|
candidateHash: row.candidateHash, contract: isAgentDirectoryCopy(row) ? "agent_files" : "legacy",
|
|
content: row.candidateBase64 === null ? null : Buffer.from(row.candidateBase64, "base64").toString("utf8"),
|
|
errorCode: row.errorCode, errorMessage: row.errorMessage, createdAt: row.createdAt.toISOString(), updatedAt: row.updatedAt.toISOString() }));
|
|
}
|
|
|
|
async function resolve(input: { companyId: string; agentId: string; runId: string; baseRevisionId: string | null; content: string }, actor: AuthorizationActor) {
|
|
const row = await get(input.companyId, input.runId);
|
|
if (!row || row.agentId !== input.agentId) throw notFound("Instruction candidate not found");
|
|
if (completed.has(row.state)) throw conflict("This instruction candidate has already been resolved");
|
|
if (isAgentDirectoryCopy(row)) throw conflict("Agent-file synchronization failures have no preserved candidate");
|
|
const receipt = await revisions.commit({ companyId: input.companyId, agentId: input.agentId, entryFile: row.entryFile, content: input.content, baseRevisionId: input.baseRevisionId, source: "api" }, actor);
|
|
await patch(row, { state: "resolved", receipt: { ...receipt }, nextAttemptAt: null });
|
|
return receipt;
|
|
}
|
|
|
|
/** A controller may report loss before disposal, never fabricate a save or
|
|
* read a possibly live provider. Already captured candidates remain intact. */
|
|
async function reportUnavailable(companyId: string, runId: string) {
|
|
const row = await get(companyId, runId);
|
|
if (!row || completed.has(row.state) || ["unchanged_turn", "warm_saved", "superseded"].includes(row.state) || row.candidateBase64 !== null || (isAgentDirectoryCopy(row) && row.candidateHash !== null) || row.state === "conflict") return row;
|
|
// A failed close or deferred lease observation grants no stop proof. Keep
|
|
// that row recoverable through generic heartbeat cleanup.
|
|
if (isAgentDirectoryCopy(row) && (row.state === "unavailable"
|
|
|| row.state === "pending_collection" && !row.processStoppedAt)) return row;
|
|
return patch(row, { state: "unavailable", errorCode: "INSTRUCTION_COLLECTION_UNAVAILABLE",
|
|
errorMessage: "The registered instruction copy could not be retrieved safely before environment release. No instruction save is claimed.", nextAttemptAt: null });
|
|
}
|
|
|
|
async function release(companyId: string, runId: string) { liveTargets.delete(targetKey(companyId, runId)); const row = await get(companyId, runId); if (row && isAgentDirectoryCopy(row)) await directories.release(row); }
|
|
async function reportRetirementUnconfirmed(companyId: string, runId: string) {
|
|
const row = await get(companyId, runId);
|
|
if (!row || !isAgentDirectoryCopy(row) || row.receipt?.retainedByRunId || row.processStoppedAt) return row;
|
|
return patch(row, { state: "pending_collection", errorCode: "INSTRUCTION_STOP_UNCONFIRMED",
|
|
errorMessage: "The retained provider did not confirm retirement. Its registered agent directory is preserved; no save is claimed.", nextAttemptAt: null });
|
|
}
|
|
async function adopt(input: Parameters<typeof directories.adopt>[0]) {
|
|
const row = await directories.adopt(input);
|
|
if (row) {
|
|
liveTargets.delete(targetKey(input.companyId, input.previousRunId));
|
|
if (input.target) liveTargets.set(targetKey(input.companyId, input.runId), input.target);
|
|
}
|
|
return row;
|
|
}
|
|
return { prepare, checkpointWarm, canReuseWarm: directories.canReuse, adopt, reportRetirementUnconfirmed, get, hasChanges, acknowledgeExplicitSave, collectStopped, recoverCaptured, recoverStopped, list, resolve, reportUnavailable, release };
|
|
}
|