Files
PaperClipAI/server/src/services/agent-instruction-working-copies.ts
T
DottaandPaperclip 57e977be72 feat: integrate Pi 1.0 into the experimental Runner (#14921)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - The experimental Runner owns provider processes and durable
sessions.
> - Pi needs working task execution and human controls.
> - The five-PR stack must preserve changes already on master.
> - Each layer now carries the complete integrated source for a safe
sequential fallback.
> - This PR belongs to native GitHub stack #15602, ending at #14956.

## Linked Issues or Issue Description

Refs #14436, #14631, #14743 and #14956.

Ship Pi 1.0 through the experimental Paperclip Runner. The five PRs are
#14921, #14922, #14923, #14924 and #14956. The user authorized the
complete merge after checks pass. Existing `pi_local` execution is
unchanged. Accounting and wider provider/platform qualification remain
deferred.

## What Changed

- Recover missing final replies after workspace finalization changes
owners, using accepted-turn evidence without rerunning work or granting
external-chat publication.
- Preserve the admitted Pi instruction root across warm runs, while
retaining changed-root rejection.
- Give Pi a bounded 15-second default shutdown grace so stop, drain
acknowledgement and durable suspension can complete. Explicit deadlines
and other providers retain their existing behavior.
- Integrate the Pi 1.0 runtime and master contracts.
- Use Pi profile 22. Preserve explicit caller-selected models and exact
native thinking levels. Keep Pi's wrapper, helper, extension and
question/control behavior unchanged from the qualified profile-19
runtime.
- Preserve master's Dot lifecycle and consent fields, configured task
environment, status guards and current Codex/Claude dependency versions.
Cursor stays qualified. Copilot stays pending; profile 17 binds the
changed shared protocol validation sources.
- Exclude general AWS IAM credentials from Pi static/custom provider
bindings and selected task projections; preserve the provider-scoped
Bedrock bearer key. Profile 21 is retained as historical provenance.
Rust and cloud install probes use the current declaration.
- Patch bundled brace-expansion 5.0.9 to the exact official 5.0.12
payload. Pin the patch and complete runtime closures. Include the patch
in normal installed setup tooling. Keep the upstream Pi shrinkwrap as
provenance and permit only this exact security correction.
- Include current attestation files in the Docker build context. Keep
the repository lockfile unchanged from master. CI and private image
builds resolve manifest changes before their frozen installation.

## Verification

- Full local `pnpm -r typecheck` passes, including Runner Rust, server
and UI. Focused integration checks pass: 194 Runner
admission/environment tests, 63 profile/credential tests with one
expected skip, 152 Dot/UI configuration tests, and Pi transcript/notice
tests.
- Full local `pnpm build` passes on the final source.
- Fresh final-source checks pass: all 698 Rust workspace tests (32
binaries), 156 credential/profile/controller tests with one expected
skip, Runner TypeScript typecheck, and 20 package/setup/sandbox tests.
- The profile-21 Pi materializer passes on the native host with the
official pinned Node 24.21.0 and its npm. It verifies all 150 locked
packages, the patched dependency and the exact closure. Setup/package
bundle tests and UI token gates pass.
- The old hashes were reproduced for all three supported targets before
calculating the patched graph. New closure hashes are darwin-arm64
`282022db10150c6632b3444df421342e7d534bdf5d5fb1097a2e79d0625a2bcf`,
darwin-x64
`64e251e19009f755c0b04f73ce2138246faab71a961b0f13d75ebfcc34bef12e`, and
linux-x64
`713b1fdff42fb56a1518bdc084f181d70bee8ebadc3e4b1d76321ed9108c8410`.
Independent native platform execution is separate from graph identity
reproduction.
- Historical cloud qualification remains unchanged: all seven core cases
pass on shipping source `10dc43c9ec65d88c2f782d62afb296d09494f215`,
harness `1a4408a48cfb5a1f094a311141c257c92cd7a893`, image
`sha256:5b3a775b383591bda1b0c1889e509acc70ce7f37c53f09733c81d59037f02280`,
and accepted Sonnet 4.6/low fixture. All 215 canonical files and all
seven cleanup checks pass independent verification. These are profile-19
results and are not relabeled as fresh profile-22 runs.
- Current Pi digest:
`sha256:e92078bee3c23bec4100aa589013a44613d054cd686826534025d8019e9f39a9`.
[The readiness
plan](https://github.com/paperclipai/paperclip/blob/codex/pi-production-readiness/doc/plans/2026-10-02-pi-production-readiness.md)
preserves campaign and failed-attempt provenance.
- Merge only after every PR's current-head CI and fresh review pass.
Linux CI covers the full suites, build and browser tests. The local
embedded Postgres API-authority suite cannot start on this macOS/Node 26
host, so Linux CI must confirm that suite.

### Fresh profile-22 core qualification — 2026-10-08

All seven accepted core cases pass canonically on Pi profile 22, with
`openrouter/anthropic/claude-sonnet-4.6` and native-confirmed low
thinking. This model is a fixture; production accepts the caller's
explicit Pi provider/model.

Runtime/install source: `3241a992f2a7703e59e97ed0fd3e5d6405de4401`.
Frozen accepted harness: `1a4408a48cfb5a1f094a311141c257c92cd7a893`.
Immutable cloud image:
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:506f22db7edd78f37c0c40bec1cc084af1850455026dbf467194bfbb8fcef141`.
Pi digest:
`sha256:e92078bee3c23bec4100aa589013a44613d054cd686826534025d8019e9f39a9`.

[Hosted Linux image and clean-install
verification](https://github.com/paperclipai/paperclip/actions/runs/37868328023)
passes, including all 20 source-bound archives, normal CLI/Pi setup,
companion import and the production pack reader. This exact installation
source includes the latest master integration and the corrected Pi warm
instruction-root fence. Full local typecheck/build and current-head
hosted CI verify the final stack. All 13 focused real-root regressions
pass. The full local executor suite passed 662 tests; 15 database tests
could not start the Mac embedded PostgreSQL service. Hosted Linux CI
passes the full required verification and E2E checks. These fresh
results keep their own source identity; profile-19 results remain
historical.

| Core path | Canonical campaign | Retained archive SHA-256 |
| --- | --- | --- |
| File edit, validation, download and Done |
`pi-core22-replyfix-0-1791511228` | 23 files;
`a473e8603a3dd4737863291f8d3d1e392391f0b16d433c3e0e0e9d8baf7a97b0` |
| Pending question and controller restart |
`pi-core22-replyfix-1-1791511376` | 33 files;
`6b829c4eb74e1f32a89c692a4ae7130dbfc1c6d3cf13915effe2103d9e242c8e` |
| Three-turn session/process/workspace continuity |
`pi-core22-replyfix-2-1791511587` | 23 files;
`7a87021f8f9a3fdd3c58bb4467f8d82c635e3ea4795d6e75f144d9aa14818df8` |
| Four typed questions and browser reconnects |
`pi-core22-replyfix-3-1791511881` | 42 files;
`9e31755252be1f4f9cb0626c984c142d4d1ae5f5bee3a7af08444db8d12c280a` |
| Plan approval and completion | `pi-core22-replyfix-4-1791512031` | 22
files;
`a0383ce1aab38e7b5a25ce0e9dd3bebea5c037ebd96ae6b29dae19015da2ae2c` |
| Same-turn steering and permission denial |
`pi-core22-replyfix-5-1791512261` | 39 files;
`c929b8c7070f0b66aedc17e65ca46e6beab1e363926ac9f7e2a75fb250f05949` |
| Stop during pending permission | `pi-core22-replyfix-6-1791512390` |
33 files;
`7f0a58ae0f4d5bfc76149435f4e322537089c5bd16e7ffe9b5ad71f10a621a07` |

All 215 canonical files (28714587 bytes) are independently
hash-verified. All seven cleanup grades pass, with no owned runtime
process or temporary root after each case. Automatic retries are zero.
The owned cloud host stopped normally after retention. The prior
profile-22 warm attempt remains failed and separately retained: archive
SHA-256
`1e54eba5ec72b50cee1534b23d1d1d4f21a090006b8a64501ba70db972abfde5`. Its
original canonical classification is preserved. Diagnosis reproduced a
product bug comparing an agent-files root against an unset
checkpoint-only field. The fix stores the admitted physical root
separately from the adopted per-run collection capability. The real-root
regression fails before the fix and passes afterward, including
rejection of a changed physical root. Fixture, grader, model and all
seven accepted case IDs are unchanged; this fresh campaign tests
final-reply publication after file registration first. The intermediate
restart attempt also remains failed and retained: archive SHA-256
`5dcaefdf1d17cf4cd54fd4cf810f45e736667392339b8ce7caf08bb4e225277f`. Its
original canonical classification is preserved. Pi resumed, wrote the
verified answer and completed its task; exact runner suspension was
proven, but idle stop consumed about 5.2s and left under 3s for the
drain acknowledgement. The Pi-only default shutdown grace is now 15s,
preserving a full 5s drain round trip and a finite suspension reserve.
Explicit caller deadlines, other provider defaults, literal drain
receipts and exact suspension identity checks remain unchanged. The
timing regression fails before this correction and passes afterward; all
18 focused settlement tests and Runner typecheck pass. The final-source
file attempt is also preserved as failed (`candidate_failure`), archive
SHA-256
`db6767b6773ea618997927ac77bdb005a5ac81492c7b9c0ffbc900449f829bc9`.
Native edit, validation, exact downloadable artifact and Done/succeeded
all passed, and the exact final reply was durably recorded. A workspace
recovery owner completed before the live heartbeat reached presentation,
leaving that reply absent from task chat. Recovery now materializes only
a completed final reply from the accepted turn of an ordinary internal
Done task, preserving issue/run/contract binding, suppression,
external-chat authorization and same-run deduplication. The database
regression covers the generated file-preparation receipt, suppression,
unapproved external continuation and replay. Server typecheck and all 49
response-selection tests pass; hosted Linux verifies the database
regression because embedded PostgreSQL cannot start on this Mac.

The delayed-final-answer database regression passes on [the final
root-source Linux server
shard](https://github.com/paperclipai/paperclip/actions/runs/37868262553/job/113628594152),
alongside 1,108 passing tests. The first root Runner shard had one
unchanged durable-resume test exceed its 5-second timeout; the identical
top-source shard and the isolated exact test passed. One rerun of that
failed job and its required aggregate passed without source or test
changes. The original failed job log and the single-rerun receipt remain
retained.

### October 9 merge verification

Current merge head: `5a8fe63512a7166aaef5cf50065a25008aa8b44b`. All
current-head checks pass, including `ci / verify` and `ci / e2e`;
exact-head Greptile review is 5/5 with no unresolved threads. Current
master conflicts are resolved. The user authorized the maintainer
override of the code-owner review gate after these checks. The seven
retained live core cases remain bound to source
`3241a992f2a7703e59e97ed0fd3e5d6405de4401` and its recorded cloud image.

## Risks

- The security correction changes the dependency closure and profile
identity. Old sessions must reopen on the new profile. Exact identities
and credential bindings fail closed.
- The runner remains experimental and requires explicit selection.
Legacy Pi Local is unchanged. Caller model IDs pass through; the E2E
model is a fixture.
- Accounting and the broad platform/provider matrix remain deferred.
This merge does not publish a release or deploy a service.

## Model Used

OpenAI GPT-6 through Codex assisted with reasoning, repository
inspection, editing and tool use. The exact serving ID and context
window are not exposed in this session. Final live qualification uses Pi
1.0.0 with `openrouter/anthropic/claude-sonnet-4.6` and native-confirmed
low thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-09 08:56:41 -05:00

427 lines
32 KiB
TypeScript

import { agentDirectoryWorkingCopyService, isAgentDirectoryCopy } from "./agent-directory-working-copies.js";
import { createHash } from "node:crypto";
import { execFile as execFileCallback } from "node:child_process";
import { promisify } from "node:util";
import fs from "node:fs/promises";
import path from "node:path";
import { and, asc, desc, eq, inArray, lte, or, isNull, isNotNull, sql } from "drizzle-orm";
import { agents, heartbeatRuns, agentInstructionWorkingCopies as copies, type Db } from "@paperclipai/db";
import {
prepareAdapterExecutionTargetRuntime,
runAdapterExecutionTargetShellCommand,
type AdapterExecutionTarget,
} from "@paperclipai/adapter-utils/execution-target";
import { conflict, notFound } from "../errors.js";
import { agentInstructionsService, agentInstructionsBundleMode } from "./agent-instructions.js";
import { agentInstructionRevisionService } from "./agent-instruction-revisions.js";
import { resolveInstructionActor } from "./agent-instruction-authorization.js";
import { assertInstructionPathSafe, instructionBytes, instructionPath, materializeInstructionBytes, MAX_INSTRUCTION_BYTES, readInstructionBytes, instructionGitExcludeProgram } from "./agent-instruction-files.js";
import { hasNativeLocalProcessStop } from "./native-local-process-stop.js";
import { remoteExecutionHasStopped } from "./remote-execution-termination.js";
import type { AuthorizationActor } from "./authorization.js";
import type { EnvironmentRuntimeService } from "./environment-runtime.js";
import { logger } from "../middleware/logger.js";
const execFile = promisify(execFileCallback);
type Copy = typeof copies.$inferSelect;
const hash = (bytes: Uint8Array) => createHash("sha256").update(bytes).digest("hex");
const quote = (value: string) => `'${value.replaceAll("'", `'"'"'`)}'`;
const completed = new Set(["saved", "unchanged", "resolved"]);
const MAX_COLLECTION_ATTEMPTS = 3;
// Only orchestration can register a live target; tools and HTTP callers cannot
// supply one. These handles are optional: durable captured bytes survive restart.
const liveTargets = new Map<string, AdapterExecutionTarget>();
const targetKey = (companyId: string, runId: string) => `${companyId}:${runId}`;
/** Finish only bounded immediate collection work; deferred ownership remains pending. */
export async function collectStoppedInstructionCopyWithRetries<T extends Pick<Copy, "state" | "attempts" | "nextAttemptAt">>(collect: () => Promise<T | null>): Promise<T | null> {
let saved: T | null = null;
let previousAttempts = -1;
for (let calls = 0; calls < MAX_COLLECTION_ATTEMPTS; calls++) {
saved = await collect();
if (!saved || saved.state !== "pending_collection" || saved.nextAttemptAt === null
|| saved.attempts >= MAX_COLLECTION_ATTEMPTS || saved.attempts <= previousAttempts) break;
previousAttempts = saved.attempts;
}
return saved;
}
export function instructionWorkingCopyGuidance(copy: Pick<Copy, "executionRoot" | "entryFile" | "receipt">) {
if (isAgentDirectoryCopy(copy)) return `Your persistent agent directory is ${copy.executionRoot} (AGENT_HOME). Your instruction entry is ${copy.executionRoot}/${copy.entryFile}. Read and write your own files and subfolders there. This directory belongs to this agent across tasks and sessions; task files belong in the task working directory. Paperclip restores this directory before execution and saves validated changes at turn boundaries. A warm native Codex session keeps the same writable directory between turns. Other sessions collect after the provider stops. Regular files, including binary files, persist; symlinks and special files are unsupported. Check the agent-files save receipt before claiming persistence; Only files you change or delete are synchronized. If another run changes the same file, the last completed synchronization wins. Temporary copies are removed when the owning session stops; there is no per-run file history. Storage allows 256 MiB per file, 2 GiB total, and 100,000 entries; the instruction entry must remain UTF-8 and at most 1 MiB. Reaching a storage limit never prevents this or future tasks from running. Remove or shrink files to free space; changes that exceed the limits will not be saved.${typeof copy.receipt?.storageWarning === "string" ? `\n\n${copy.receipt.storageWarning}` : ""}`;
return `Your editable agent instruction file is ${copy.executionRoot}/${copy.entryFile}. Edit this registered private copy normally. After this run stops, Paperclip saves changed content as a persistent revision if your responsible user still has permission and the baseline has not changed. Check the run's instruction-save receipt before claiming persistence. Use read_agent_instructions, update_agent_instructions, get_agent_instruction_history, and restore_agent_instructions for immediate saves and history. Read first and pin the returned revision. Preserve conflicts; never silently retry against a newer head. Repository instructions, skills, and the loaded prompt are separate and are not collected.`;
}
/** Remote reads use the registered entry only; never scan for files called AGENTS.md. */
export function instructionCollectionScript(root: string, entry: string) {
instructionPath(entry);
return `node -e ${quote([
"const fs=require('node:fs'),path=require('node:path')",
"const root=process.argv[1],entry=process.argv[2],max=Number(process.argv[3])",
"const absolute=path.resolve(root,entry),parts=absolute.split('/').filter(Boolean)",
"let current='/'",
"for(let i=0;i<parts.length;i++){current=path.join(current,parts[i]);let s;try{s=fs.lstatSync(current)}catch(e){if(e.code==='ENOENT'){process.stdout.write(JSON.stringify({missing:true}));process.exit(0)}throw e}if(s.isSymbolicLink()||(i<parts.length-1?!s.isDirectory():!s.isFile()))throw Error('INSTRUCTION_PATH_INVALID')}",
"const fd=fs.openSync(absolute,fs.constants.O_RDONLY|fs.constants.O_NOFOLLOW)",
"try{const s=fs.fstatSync(fd);if(!s.isFile()||s.size>max)throw Error('INSTRUCTION_CONTENT_INVALID');const bytes=Buffer.alloc(max+1);let n=0;while(n<bytes.length){const count=fs.readSync(fd,bytes,n,bytes.length-n,n);if(!count)break;n+=count}if(n>max)throw Error('INSTRUCTION_CONTENT_INVALID');process.stdout.write(JSON.stringify({contentBase64:bytes.subarray(0,n).toString('base64')}))}finally{fs.closeSync(fd)}",
].join(";"))} ${quote(root)} ${quote(entry)} ${MAX_INSTRUCTION_BYTES}`;
}
export function agentInstructionWorkingCopyService(db: Db, options: { environmentRuntime?: EnvironmentRuntimeService } = {}) {
const revisions = agentInstructionRevisionService(db);
const scope = (companyId: string, runId: string) => and(eq(copies.companyId, companyId), eq(copies.runId, runId));
async function get(companyId: string, runId: string) {
const [row] = await db.select().from(copies).where(scope(companyId, runId));
return row ?? null;
}
async function patch(row: Copy, values: Partial<typeof copies.$inferInsert>) {
const [updated] = await db.update(copies).set({ ...values, updatedAt: new Date() }).where(and(
scope(row.companyId, row.runId), eq(copies.state, row.state), eq(copies.attempts, row.attempts), eq(copies.baseHash, row.baseHash),
sql`${copies.receipt}->>'retainedByRunId' IS NOT DISTINCT FROM ${typeof row.receipt?.retainedByRunId === "string" ? row.receipt.retainedByRunId : null}`,
)).returning();
// A late cleanup must not overwrite an explicit resolution or a newer
// baseline. Canonical content has its own independent head CAS.
return updated ?? (await get(row.companyId, row.runId))!;
}
function actorFor(row: Copy): AuthorizationActor {
return { type: "agent", companyId: row.companyId, agentId: row.agentId, runId: row.runId, onBehalfOfUserId: row.responsibleUserId };
}
const directories = agentDirectoryWorkingCopyService(db, get, patch, options.environmentRuntime);
async function recoverDirectoryCleanup(row: Copy, cleanup: () => Promise<unknown>) {
// Reserve a later retry before I/O, including lock waits. A blocked owner
// must not occupy every sweep or prevent other copies from being reclaimed.
try {
await patch(row, { nextAttemptAt: new Date(Date.now() + 30_000) });
await cleanup();
}
catch (error) {
try { await patch(row, { nextAttemptAt: new Date(Date.now() + 30_000) }); }
catch (retryError) {
logger.warn({ err: retryError, runId: row.runId }, "Agent file cleanup retry could not be scheduled");
}
logger.warn({ err: error, runId: row.runId }, "Agent file cleanup deferred; save receipt unchanged");
}
}
async function prepare(input: { companyId: string; agentId: string; runId: string; target?: AdapterExecutionTarget | null; cwd: string; legacy?: boolean; warm?: boolean; reuseRunId?: string; onWarmHandoff?: (copy: Copy) => void }) {
const existing = await get(input.companyId, input.runId);
if (isAgentDirectoryCopy(existing) || (!existing && !input.legacy)) return directories.prepare(input);
let refreshStoppedCopy = false;
const workspace = await fs.realpath(input.cwd);
const expectedLocalRoot = path.join(workspace, ".paperclip-runtime", `instruction-edits-${input.runId}`, "instructions");
if (existing && existing.localRoot !== expectedLocalRoot) throw conflict("The registered instruction copy belongs to a different run workspace");
const location = input.target?.kind === "remote" ? `remote:${input.target.environmentId ?? ""}` : "local";
if (existing && (existing.agentId !== input.agentId || existing.location !== location)) {
throw conflict("The registered instruction copy belongs to a different run environment");
}
// Preserve in-flight bytes and their CAS baseline. A completed, stopped copy
// can start a fresh lifecycle only after its recorded bytes are accounted for.
if (existing && existing.state !== "preparing") {
if (["conflict", "pending_commit", "unavailable"].includes(existing.state)) {
throw conflict("Resolve this run's preserved instruction candidate before editing its working copy again");
}
if (input.target) liveTargets.set(targetKey(input.companyId, input.runId), input.target);
if (completed.has(existing.state) && existing.processStoppedAt) {
const privateBytes = await readCandidate(existing, input.target);
// Refresh only bytes whose previous lifecycle is complete. A changed
// private file must keep its old CAS fence; never silently rebase edits.
refreshStoppedCopy = privateBytes === null || hash(privateBytes) === (existing.candidateHash ?? existing.baseHash);
if (!refreshStoppedCopy && privateBytes !== null) {
// A failed staging attempt can already have replaced the file before
// the completed row advances. Exact current canonical bytes are also
// accounted for; any other private edit still keeps its old fence.
const current = await revisions.readCommittedForRuntime({ companyId: input.companyId, agentId: input.agentId });
refreshStoppedCopy = current?.revision.entryFile === existing.entryFile
&& current.revision.contentHash === hash(privateBytes);
}
}
if (!refreshStoppedCopy) {
if (completed.has(existing.state) || existing.state === "unchanged_turn") {
// unchanged_turn is a live warm-owner observation, not stop proof.
return patch(existing, { state: "prepared", candidateBase64: null, candidateHash: null,
receipt: null, processStoppedAt: null, attempts: 0, nextAttemptAt: null });
}
return existing;
}
}
const [agent] = await db.select().from(agents).where(and(eq(agents.id, input.agentId), eq(agents.companyId, input.companyId)));
if (!agent) throw notFound("Agent not found");
if (agentInstructionsBundleMode(agent) !== "managed") return null;
const bound = await resolveInstructionActor(db, { type: "agent", companyId: input.companyId, agentId: input.agentId, runId: input.runId });
const baseline = existing?.baseRevisionId && !refreshStoppedCopy
? await revisions.readRevision({ companyId: input.companyId, agentId: input.agentId, entryFile: existing.entryFile, revisionId: existing.baseRevisionId }, bound).catch(async (error) => {
if ((error as { status?: number }).status !== 404) throw error;
const current = await revisions.readCurrent({ companyId: input.companyId, agentId: input.agentId }, bound);
if (current?.revision.id !== existing.baseRevisionId) throw conflict("The instruction baseline changed during preparation");
return current;
})
: await revisions.readCurrent({ companyId: input.companyId, agentId: input.agentId }, bound);
if (!baseline) return null;
const exported = await agentInstructionsService().exportFiles(agent, { rejectSymlinks: true });
const entryFile = instructionPath(baseline?.revision.entryFile ?? exported.entryFile);
const content = instructionBytes(baseline?.content ?? exported.files[entryFile] ?? "");
const localRoot = existing?.localRoot ?? expectedLocalRoot;
const target = input.target?.kind === "remote" ? input.target : null;
const executionRoot = target
? path.posix.join(target.remoteCwd, ".paperclip-runtime", `instruction-edits-${input.runId}`, "instructions")
: localRoot;
if (!existing) {
await db.insert(copies).values({
runId: input.runId, companyId: input.companyId, agentId: input.agentId,
responsibleUserId: bound.onBehalfOfUserId!, entryFile,
baseRevisionId: baseline?.revision.id ?? null, baseHash: hash(content),
localRoot, executionRoot, location, state: "preparing",
});
}
const row = (await get(input.companyId, input.runId))!;
await assertInstructionPathSafe(localRoot, entryFile);
await execFile(process.execPath, ["-e", instructionGitExcludeProgram, workspace], { timeout: 15_000 });
await fs.mkdir(localRoot, { recursive: true, mode: 0o700 });
for (const [name, text] of Object.entries({ ...exported.files, [entryFile]: content.toString("utf8") })) {
const relative = instructionPath(name);
const filename = await assertInstructionPathSafe(localRoot, relative);
await fs.mkdir(path.dirname(filename), { recursive: true, mode: 0o700 });
// Atomic replacement also allows interrupted preparation to restage an
// already-read-only sibling without writing through its existing inode.
await materializeInstructionBytes(localRoot, relative, instructionBytes(text));
await fs.chmod(filename, name === entryFile ? 0o600 : 0o400);
}
if (target) {
const staged = await prepareAdapterExecutionTargetRuntime({
target, runId: input.runId, adapterKey: `instruction-edits-${input.runId}`,
workspaceLocalDir: input.cwd, syncWorkspace: false,
assets: [{ key: "instructions", localDir: localRoot, followSymlinks: false }],
});
if (staged.assetDirs.instructions !== executionRoot) throw new Error("Instruction copy staging path changed");
const excluded = await runAdapterExecutionTargetShellCommand(input.runId, target,
`node -e ${quote(instructionGitExcludeProgram)} ${quote(target.remoteCwd)}`, { cwd: target.remoteCwd, env: {}, timeoutSec: 15 });
if (excluded.exitCode !== 0 || excluded.timedOut) throw new Error("Could not exclude private instructions from Git staging");
}
// A refresh does not supersede the completed turn until every local and
// remote staging step succeeds. Keep its receipt and stop evidence intact
// if preparation throws or the controller stops midway through staging.
const prepared = await patch(row, { state: "prepared", ...(refreshStoppedCopy ? {
entryFile, baseRevisionId: baseline.revision.id, baseHash: hash(content),
candidateBase64: null, candidateHash: null, receipt: null, processStoppedAt: null,
attempts: 0, nextAttemptAt: null, errorCode: null, errorMessage: null,
} : {}) });
if (input.target) liveTargets.set(targetKey(input.companyId, input.runId), input.target);
return prepared;
}
async function readCandidate(row: Copy, target?: AdapterExecutionTarget | null) {
if (row.location === "local") return readInstructionBytes(row.localRoot, row.entryFile);
if (target?.kind !== "remote") throw new Error("The original execution environment is needed to retrieve this instruction copy");
if (row.location !== `remote:${target.environmentId ?? ""}`) throw new Error("Instruction copy execution environment changed");
const expected = path.posix.join(target.remoteCwd, ".paperclip-runtime", `instruction-edits-${row.runId}`, "instructions");
if (expected !== row.executionRoot) throw new Error("Instruction copy execution environment changed");
const result = await runAdapterExecutionTargetShellCommand(row.runId, target, instructionCollectionScript(row.executionRoot, row.entryFile), { cwd: target.remoteCwd, env: {}, timeoutSec: 30 });
if (result.exitCode !== 0 || result.timedOut) throw new Error("Could not safely retrieve the stopped run's instruction file");
const payload = JSON.parse(result.stdout) as { missing?: boolean; contentBase64?: string };
if (payload.missing) return null;
if (typeof payload.contentBase64 !== "string" || payload.contentBase64.length > Math.ceil(MAX_INSTRUCTION_BYTES / 3) * 4) throw new Error("Instruction response exceeds the collection bound");
return instructionBytes(Buffer.from(payload.contentBase64, "base64"));
}
/** A terminal-turn probe is not a save: changed/invalid copies require a
* stopped provider before capture. Unchanged warm providers remain reusable. */
async function hasChanges(input: { companyId: string; runId: string; target?: AdapterExecutionTarget | null }) {
const row = await get(input.companyId, input.runId);
if (!row || completed.has(row.state)) return false;
if (isAgentDirectoryCopy(row)) return directories.hasChanges(row, input.target);
try {
const content = await readCandidate(row, input.target);
if (content !== null && hash(content) === row.baseHash) {
// This is an unchanged turn receipt, not stopped-process collection.
// No candidate bytes or new revision are persisted from a live owner.
await patch(row, { state: "unchanged_turn", nextAttemptAt: null });
return false;
}
return true;
} catch { return true; }
}
async function checkpointWarm(input: { companyId: string; runId: string; target?: AdapterExecutionTarget | null }) {
const row = await get(input.companyId, input.runId);
return row && isAgentDirectoryCopy(row) ? directories.checkpointWarm(row, input.target) : row;
}
async function acknowledgeExplicitSave(input: { companyId: string; agentId: string; runId: string; entryFile: string; revisionId: string; contentHash: string }) {
const row = await get(input.companyId, input.runId);
if (!row || isAgentDirectoryCopy(row) || row.agentId !== input.agentId || row.entryFile !== input.entryFile || row.state !== "prepared") return;
const target = liveTargets.get(targetKey(row.companyId, row.runId));
const content = await readCandidate(row, target).catch(() => null);
// Never advance a stale local edit past an unrelated explicit update.
if (content === null || hash(content) !== input.contentHash) return;
await db.update(copies).set({ baseRevisionId: input.revisionId, baseHash: input.contentHash, updatedAt: new Date() })
.where(and(scope(row.companyId, row.runId), eq(copies.state, "prepared"), eq(copies.baseHash, row.baseHash)));
}
async function commitCandidate(row: Copy) {
if (isAgentDirectoryCopy(row)) return directories.collectStopped(row);
if (row.candidateBase64 === null || completed.has(row.state) || row.state === "conflict") return row;
try {
const receipt = await revisions.commit({
companyId: row.companyId, agentId: row.agentId, entryFile: row.entryFile,
baseRevisionId: row.baseRevisionId, content: Buffer.from(row.candidateBase64, "base64"), source: "cleanup",
}, actorFor(row));
return patch(row, { state: "saved", receipt: { ...receipt }, errorCode: null, errorMessage: null, nextAttemptAt: null });
} catch (error) {
const detail = error as { status?: number; details?: { code?: string }; message?: string };
const code = detail.details?.code ?? "INSTRUCTION_SAVE_FAILED";
const retryable = !detail.status || detail.status >= 500;
const message = detail.status === 403
? `${detail.message ?? "Current permissions do not allow this instruction save."} The candidate was preserved.`
: detail.status === 409
? "Instructions changed after this run started. Review the preserved candidate against the current revision."
: "Instruction edits were preserved but not saved. Review the candidate before retrying.";
return patch(row, { state: retryable ? "pending_commit" : "conflict", errorCode: code, errorMessage: message, nextAttemptAt: retryable && row.attempts < MAX_COLLECTION_ATTEMPTS ? new Date(Date.now() + 30_000) : null });
}
}
/** Caller must have joined the owned provider process before granting this proof. */
async function collectStopped(input: { companyId: string; runId: string; target?: AdapterExecutionTarget | null }) {
let row = await get(input.companyId, input.runId);
if (!row) return row;
if (isAgentDirectoryCopy(row)) return directories.collectStopped(row, input.target);
if (completed.has(row.state) || row.state === "conflict") return row;
row = await patch(row, { processStoppedAt: row.processStoppedAt ?? new Date(), attempts: row.attempts + 1 });
if (completed.has(row.state) || row.state === "conflict") return row;
if (row.candidateBase64 !== null) return commitCandidate(row);
try {
const candidate = await readCandidate(row, input.target);
if (candidate === null) {
return patch(row, { state: "unavailable", errorCode: "INSTRUCTION_FILE_MISSING", errorMessage: "The registered instruction file was removed. Canonical instructions were kept; no edits were saved.", nextAttemptAt: null });
}
if (hash(candidate) === row.baseHash) return patch(row, { state: "unchanged", nextAttemptAt: null });
// Save the candidate durably before attempting authorization/CAS or releasing the environment.
row = await patch(row, { state: "pending_commit", candidateBase64: candidate.toString("base64"), candidateHash: hash(candidate), nextAttemptAt: new Date() });
return commitCandidate(row);
} catch (error) {
return patch(row, { state: row.attempts < MAX_COLLECTION_ATTEMPTS ? "pending_collection" : "unavailable", errorCode: "INSTRUCTION_COLLECTION_FAILED", errorMessage: "The registered instruction copy could not be read safely. Canonical instructions were kept; no save receipt exists.", nextAttemptAt: row.attempts < MAX_COLLECTION_ATTEMPTS ? new Date(Date.now() + 30_000) : null });
}
}
/** Restarts can retry persisted bytes without touching a live filesystem or launching a model. */
async function recoverCaptured() {
const pending = await db.select().from(copies).where(and(
eq(copies.state, "pending_commit"),
or(isNull(copies.nextAttemptAt), lte(copies.nextAttemptAt, new Date())),
lte(copies.attempts, MAX_COLLECTION_ATTEMPTS - 1),
)).limit(20);
for (const row of pending) {
const result = await commitCandidate(await patch(row, { attempts: row.attempts + 1 }));
if (isAgentDirectoryCopy(result) && completed.has(result.state)) await directories.release(result);
}
// A crash between a durable save and release must not retain run snapshots
// forever. Only discard copies with recorded stop proof and a terminal receipt.
const cleanup = await db.select().from(copies).where(and(
inArray(copies.state, [...completed, "unavailable"]), isNotNull(copies.processStoppedAt),
sql`${copies.receipt}->>'schema' = 'paperclip.agent-files.v1'`,
or(sql`${copies.receipt} ? 'baseline'`, sql`${copies.receipt}->>'cleanupPending' = 'true'`),
or(isNull(copies.nextAttemptAt), lte(copies.nextAttemptAt, new Date())),
)).orderBy(asc(copies.updatedAt)).limit(20);
for (const row of cleanup) await recoverDirectoryCleanup(row, () => directories.release(row));
return pending.length;
}
/** Recover only the registered copy after independently recorded process-stop
* proof. Terminal run status is never a substitute for that proof. Remote
* receipts mean the environment is stopped; do not restart it just to read. */
async function recoverStopped() {
const pending = await db.select({ copy: copies, runtimeMode: heartbeatRuns.runtimeMode }).from(copies)
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, copies.companyId), eq(heartbeatRuns.id, copies.runId)))
.where(and(or(and(or(inArray(copies.state, ["prepared", "pending_collection", "unchanged_turn", "warm_saved"]),
and(eq(copies.state, "preparing"), sql`${copies.receipt}->>'schema' = 'paperclip.agent-files.v1'`)),
lte(copies.attempts, MAX_COLLECTION_ATTEMPTS - 1)),
and(eq(copies.state, "unavailable"), isNull(copies.processStoppedAt),
sql`${copies.location} like 'remote:%'`,
sql`${copies.receipt}->>'schema' = 'paperclip.agent-files.v1'`)),
sql`NOT (coalesce(${copies.receipt}, '{}'::jsonb) ? 'retainedByRunId')`,
inArray(heartbeatRuns.status, ["succeeded", "failed", "cancelled", "timed_out", "interrupted"]),
or(isNull(copies.nextAttemptAt), lte(copies.nextAttemptAt, new Date())))).orderBy(asc(copies.updatedAt)).limit(20);
for (const { copy: row, runtimeMode } of pending) {
if (row.state === "unavailable" && isAgentDirectoryCopy(row)) {
await recoverDirectoryCleanup(row, () => directories.recoverUnavailable(row));
} else if (row.state === "preparing" && isAgentDirectoryCopy(row)) {
// A provider cannot launch until preparation records "prepared". With
// the owning run terminal, this is an interrupted staging copy only.
await directories.release(await patch(row, { state: "unavailable", processStoppedAt: new Date(),
errorCode: "AGENT_FILES_PREPARE_FAILED", errorMessage: "Agent-file preparation was interrupted. The temporary copy was discarded.", nextAttemptAt: null }));
} else if (row.location === "local" && (row.processStoppedAt || runtimeMode === "native" &&
await hasNativeLocalProcessStop(db, row.companyId, row.runId))) {
const result = await collectStopped({ companyId: row.companyId, runId: row.runId });
if (result && isAgentDirectoryCopy(result) && completed.has(result.state)) await directories.release(result);
} else if (row.location !== "local" && await remoteExecutionHasStopped(db, row.companyId, row.runId)) {
if (row.receipt?.warm === true) {
const unavailable = row.receipt?.warm === true
? await patch(row, { state: "unavailable", errorCode: "AGENT_FILES_FINAL_COLLECTION_UNAVAILABLE",
errorMessage: "The last completed checkpoint remains saved. Files changed afterwards could not be collected after remote termination.", nextAttemptAt: null })
: await reportUnavailable(row.companyId, row.runId);
if (unavailable && isAgentDirectoryCopy(unavailable)) {
await directories.release(await patch(unavailable, { processStoppedAt: unavailable.processStoppedAt ?? new Date() }));
}
} else {
if (isAgentDirectoryCopy(row)) await directories.recoverStoppedRemote(row);
else await reportUnavailable(row.companyId, row.runId);
}
} else if (row.state === "prepared") {
await patch(row, { state: "pending_collection", errorCode: "INSTRUCTION_STOP_UNCONFIRMED",
errorMessage: "The provider's stop has not been confirmed. Instruction collection is pending; no save is claimed.", nextAttemptAt: new Date(Date.now() + 30_000) });
} else if (["warm_saved", "pending_collection", "unchanged_turn"].includes(row.state)) {
// Live retained sessions must not occupy every batch and starve stopped
// copies from other agents. This does not permit reads without stop proof.
await patch(row, { nextAttemptAt: new Date(Date.now() + 30_000) });
}
}
return pending.length;
}
async function list(companyId: string, agentId: string, actor: AuthorizationActor) {
await revisions.readCurrent({ companyId, agentId }, actor);
const rows = await db.select().from(copies).where(and(eq(copies.companyId, companyId), eq(copies.agentId, agentId), inArray(copies.state, ["conflict", "pending_collection", "pending_commit", "unavailable"]))).orderBy(desc(copies.createdAt)).limit(50);
return rows.map(row => ({ runId: row.runId, entryFile: row.entryFile, baseRevisionId: row.baseRevisionId,
baseHash: row.baseHash, state: row.state as "conflict" | "pending_collection" | "pending_commit" | "unavailable",
candidateHash: row.candidateHash, contract: isAgentDirectoryCopy(row) ? "agent_files" : "legacy",
content: row.candidateBase64 === null ? null : Buffer.from(row.candidateBase64, "base64").toString("utf8"),
errorCode: row.errorCode, errorMessage: row.errorMessage, createdAt: row.createdAt.toISOString(), updatedAt: row.updatedAt.toISOString() }));
}
async function resolve(input: { companyId: string; agentId: string; runId: string; baseRevisionId: string | null; content: string }, actor: AuthorizationActor) {
const row = await get(input.companyId, input.runId);
if (!row || row.agentId !== input.agentId) throw notFound("Instruction candidate not found");
if (completed.has(row.state)) throw conflict("This instruction candidate has already been resolved");
if (isAgentDirectoryCopy(row)) throw conflict("Agent-file synchronization failures have no preserved candidate");
const receipt = await revisions.commit({ companyId: input.companyId, agentId: input.agentId, entryFile: row.entryFile, content: input.content, baseRevisionId: input.baseRevisionId, source: "api" }, actor);
await patch(row, { state: "resolved", receipt: { ...receipt }, nextAttemptAt: null });
return receipt;
}
/** A controller may report loss before disposal, never fabricate a save or
* read a possibly live provider. Already captured candidates remain intact. */
async function reportUnavailable(companyId: string, runId: string) {
const row = await get(companyId, runId);
if (!row || completed.has(row.state) || ["unchanged_turn", "warm_saved", "superseded"].includes(row.state) || row.candidateBase64 !== null || (isAgentDirectoryCopy(row) && row.candidateHash !== null) || row.state === "conflict") return row;
// A failed close or deferred lease observation grants no stop proof. Keep
// that row recoverable through generic heartbeat cleanup.
if (isAgentDirectoryCopy(row) && (row.state === "unavailable"
|| row.state === "pending_collection" && !row.processStoppedAt)) return row;
return patch(row, { state: "unavailable", errorCode: "INSTRUCTION_COLLECTION_UNAVAILABLE",
errorMessage: "The registered instruction copy could not be retrieved safely before environment release. No instruction save is claimed.", nextAttemptAt: null });
}
async function release(companyId: string, runId: string) { liveTargets.delete(targetKey(companyId, runId)); const row = await get(companyId, runId); if (row && isAgentDirectoryCopy(row)) await directories.release(row); }
async function reportRetirementUnconfirmed(companyId: string, runId: string) {
const row = await get(companyId, runId);
if (!row || !isAgentDirectoryCopy(row) || row.receipt?.retainedByRunId || row.processStoppedAt) return row;
return patch(row, { state: "pending_collection", errorCode: "INSTRUCTION_STOP_UNCONFIRMED",
errorMessage: "The retained provider did not confirm retirement. Its registered agent directory is preserved; no save is claimed.", nextAttemptAt: null });
}
async function adopt(input: Parameters<typeof directories.adopt>[0]) {
const row = await directories.adopt(input);
if (row) {
liveTargets.delete(targetKey(input.companyId, input.previousRunId));
if (input.target) liveTargets.set(targetKey(input.companyId, input.runId), input.target);
}
return row;
}
return { prepare, checkpointWarm, canReuseWarm: directories.canReuse, adopt, reportRetirementUnconfirmed, get, hasChanges, acknowledgeExplicitSave, collectStopped, recoverCaptured, recoverStopped, list, resolve, reportUnavailable, release };
}