mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-30 05:16:50 +01:00
feat: integrate Pi 1.0 into the experimental Runner (#14921)
## Thinking Path > - Paperclip manages AI agents and their work. > - The experimental Runner owns provider processes and durable sessions. > - Pi needs working task execution and human controls. > - The five-PR stack must preserve changes already on master. > - Each layer now carries the complete integrated source for a safe sequential fallback. > - This PR belongs to native GitHub stack #15602, ending at #14956. ## Linked Issues or Issue Description Refs #14436, #14631, #14743 and #14956. Ship Pi 1.0 through the experimental Paperclip Runner. The five PRs are #14921, #14922, #14923, #14924 and #14956. The user authorized the complete merge after checks pass. Existing `pi_local` execution is unchanged. Accounting and wider provider/platform qualification remain deferred. ## What Changed - Recover missing final replies after workspace finalization changes owners, using accepted-turn evidence without rerunning work or granting external-chat publication. - Preserve the admitted Pi instruction root across warm runs, while retaining changed-root rejection. - Give Pi a bounded 15-second default shutdown grace so stop, drain acknowledgement and durable suspension can complete. Explicit deadlines and other providers retain their existing behavior. - Integrate the Pi 1.0 runtime and master contracts. - Use Pi profile 22. Preserve explicit caller-selected models and exact native thinking levels. Keep Pi's wrapper, helper, extension and question/control behavior unchanged from the qualified profile-19 runtime. - Preserve master's Dot lifecycle and consent fields, configured task environment, status guards and current Codex/Claude dependency versions. Cursor stays qualified. Copilot stays pending; profile 17 binds the changed shared protocol validation sources. - Exclude general AWS IAM credentials from Pi static/custom provider bindings and selected task projections; preserve the provider-scoped Bedrock bearer key. Profile 21 is retained as historical provenance. Rust and cloud install probes use the current declaration. - Patch bundled brace-expansion 5.0.9 to the exact official 5.0.12 payload. Pin the patch and complete runtime closures. Include the patch in normal installed setup tooling. Keep the upstream Pi shrinkwrap as provenance and permit only this exact security correction. - Include current attestation files in the Docker build context. Keep the repository lockfile unchanged from master. CI and private image builds resolve manifest changes before their frozen installation. ## Verification - Full local `pnpm -r typecheck` passes, including Runner Rust, server and UI. Focused integration checks pass: 194 Runner admission/environment tests, 63 profile/credential tests with one expected skip, 152 Dot/UI configuration tests, and Pi transcript/notice tests. - Full local `pnpm build` passes on the final source. - Fresh final-source checks pass: all 698 Rust workspace tests (32 binaries), 156 credential/profile/controller tests with one expected skip, Runner TypeScript typecheck, and 20 package/setup/sandbox tests. - The profile-21 Pi materializer passes on the native host with the official pinned Node 24.21.0 and its npm. It verifies all 150 locked packages, the patched dependency and the exact closure. Setup/package bundle tests and UI token gates pass. - The old hashes were reproduced for all three supported targets before calculating the patched graph. New closure hashes are darwin-arm64 `282022db10150c6632b3444df421342e7d534bdf5d5fb1097a2e79d0625a2bcf`, darwin-x64 `64e251e19009f755c0b04f73ce2138246faab71a961b0f13d75ebfcc34bef12e`, and linux-x64 `713b1fdff42fb56a1518bdc084f181d70bee8ebadc3e4b1d76321ed9108c8410`. Independent native platform execution is separate from graph identity reproduction. - Historical cloud qualification remains unchanged: all seven core cases pass on shipping source `10dc43c9ec65d88c2f782d62afb296d09494f215`, harness `1a4408a48cfb5a1f094a311141c257c92cd7a893`, image `sha256:5b3a775b383591bda1b0c1889e509acc70ce7f37c53f09733c81d59037f02280`, and accepted Sonnet 4.6/low fixture. All 215 canonical files and all seven cleanup checks pass independent verification. These are profile-19 results and are not relabeled as fresh profile-22 runs. - Current Pi digest: `sha256:e92078bee3c23bec4100aa589013a44613d054cd686826534025d8019e9f39a9`. [The readiness plan](https://github.com/paperclipai/paperclip/blob/codex/pi-production-readiness/doc/plans/2026-10-02-pi-production-readiness.md) preserves campaign and failed-attempt provenance. - Merge only after every PR's current-head CI and fresh review pass. Linux CI covers the full suites, build and browser tests. The local embedded Postgres API-authority suite cannot start on this macOS/Node 26 host, so Linux CI must confirm that suite. ### Fresh profile-22 core qualification — 2026-10-08 All seven accepted core cases pass canonically on Pi profile 22, with `openrouter/anthropic/claude-sonnet-4.6` and native-confirmed low thinking. This model is a fixture; production accepts the caller's explicit Pi provider/model. Runtime/install source: `3241a992f2a7703e59e97ed0fd3e5d6405de4401`. Frozen accepted harness: `1a4408a48cfb5a1f094a311141c257c92cd7a893`. Immutable cloud image: `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:506f22db7edd78f37c0c40bec1cc084af1850455026dbf467194bfbb8fcef141`. Pi digest: `sha256:e92078bee3c23bec4100aa589013a44613d054cd686826534025d8019e9f39a9`. [Hosted Linux image and clean-install verification](https://github.com/paperclipai/paperclip/actions/runs/37868328023) passes, including all 20 source-bound archives, normal CLI/Pi setup, companion import and the production pack reader. This exact installation source includes the latest master integration and the corrected Pi warm instruction-root fence. Full local typecheck/build and current-head hosted CI verify the final stack. All 13 focused real-root regressions pass. The full local executor suite passed 662 tests; 15 database tests could not start the Mac embedded PostgreSQL service. Hosted Linux CI passes the full required verification and E2E checks. These fresh results keep their own source identity; profile-19 results remain historical. | Core path | Canonical campaign | Retained archive SHA-256 | | --- | --- | --- | | File edit, validation, download and Done | `pi-core22-replyfix-0-1791511228` | 23 files; `a473e8603a3dd4737863291f8d3d1e392391f0b16d433c3e0e0e9d8baf7a97b0` | | Pending question and controller restart | `pi-core22-replyfix-1-1791511376` | 33 files; `6b829c4eb74e1f32a89c692a4ae7130dbfc1c6d3cf13915effe2103d9e242c8e` | | Three-turn session/process/workspace continuity | `pi-core22-replyfix-2-1791511587` | 23 files; `7a87021f8f9a3fdd3c58bb4467f8d82c635e3ea4795d6e75f144d9aa14818df8` | | Four typed questions and browser reconnects | `pi-core22-replyfix-3-1791511881` | 42 files; `9e31755252be1f4f9cb0626c984c142d4d1ae5f5bee3a7af08444db8d12c280a` | | Plan approval and completion | `pi-core22-replyfix-4-1791512031` | 22 files; `a0383ce1aab38e7b5a25ce0e9dd3bebea5c037ebd96ae6b29dae19015da2ae2c` | | Same-turn steering and permission denial | `pi-core22-replyfix-5-1791512261` | 39 files; `c929b8c7070f0b66aedc17e65ca46e6beab1e363926ac9f7e2a75fb250f05949` | | Stop during pending permission | `pi-core22-replyfix-6-1791512390` | 33 files; `7f0a58ae0f4d5bfc76149435f4e322537089c5bd16e7ffe9b5ad71f10a621a07` | All 215 canonical files (28714587 bytes) are independently hash-verified. All seven cleanup grades pass, with no owned runtime process or temporary root after each case. Automatic retries are zero. The owned cloud host stopped normally after retention. The prior profile-22 warm attempt remains failed and separately retained: archive SHA-256 `1e54eba5ec72b50cee1534b23d1d1d4f21a090006b8a64501ba70db972abfde5`. Its original canonical classification is preserved. Diagnosis reproduced a product bug comparing an agent-files root against an unset checkpoint-only field. The fix stores the admitted physical root separately from the adopted per-run collection capability. The real-root regression fails before the fix and passes afterward, including rejection of a changed physical root. Fixture, grader, model and all seven accepted case IDs are unchanged; this fresh campaign tests final-reply publication after file registration first. The intermediate restart attempt also remains failed and retained: archive SHA-256 `5dcaefdf1d17cf4cd54fd4cf810f45e736667392339b8ce7caf08bb4e225277f`. Its original canonical classification is preserved. Pi resumed, wrote the verified answer and completed its task; exact runner suspension was proven, but idle stop consumed about 5.2s and left under 3s for the drain acknowledgement. The Pi-only default shutdown grace is now 15s, preserving a full 5s drain round trip and a finite suspension reserve. Explicit caller deadlines, other provider defaults, literal drain receipts and exact suspension identity checks remain unchanged. The timing regression fails before this correction and passes afterward; all 18 focused settlement tests and Runner typecheck pass. The final-source file attempt is also preserved as failed (`candidate_failure`), archive SHA-256 `db6767b6773ea618997927ac77bdb005a5ac81492c7b9c0ffbc900449f829bc9`. Native edit, validation, exact downloadable artifact and Done/succeeded all passed, and the exact final reply was durably recorded. A workspace recovery owner completed before the live heartbeat reached presentation, leaving that reply absent from task chat. Recovery now materializes only a completed final reply from the accepted turn of an ordinary internal Done task, preserving issue/run/contract binding, suppression, external-chat authorization and same-run deduplication. The database regression covers the generated file-preparation receipt, suppression, unapproved external continuation and replay. Server typecheck and all 49 response-selection tests pass; hosted Linux verifies the database regression because embedded PostgreSQL cannot start on this Mac. The delayed-final-answer database regression passes on [the final root-source Linux server shard](https://github.com/paperclipai/paperclip/actions/runs/37868262553/job/113628594152), alongside 1,108 passing tests. The first root Runner shard had one unchanged durable-resume test exceed its 5-second timeout; the identical top-source shard and the isolated exact test passed. One rerun of that failed job and its required aggregate passed without source or test changes. The original failed job log and the single-rerun receipt remain retained. ### October 9 merge verification Current merge head: `5a8fe63512a7166aaef5cf50065a25008aa8b44b`. All current-head checks pass, including `ci / verify` and `ci / e2e`; exact-head Greptile review is 5/5 with no unresolved threads. Current master conflicts are resolved. The user authorized the maintainer override of the code-owner review gate after these checks. The seven retained live core cases remain bound to source `3241a992f2a7703e59e97ed0fd3e5d6405de4401` and its recorded cloud image. ## Risks - The security correction changes the dependency closure and profile identity. Old sessions must reopen on the new profile. Exact identities and credential bindings fail closed. - The runner remains experimental and requires explicit selection. Legacy Pi Local is unchanged. Caller model IDs pass through; the E2E model is a fixture. - Accounting and the broad platform/provider matrix remain deferred. This merge does not publish a release or deploy a service. ## Model Used OpenAI GPT-6 through Codex assisted with reasoning, repository inspection, editing and tool use. The exact serving ID and context window are not exposed in this session. Final live qualification uses Pi 1.0.0 with `openrouter/anthropic/claude-sonnet-4.6` and native-confirmed low thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
4fb1ab53a8
commit
57e977be72
517 files changed
+61737
-1440
No files matched your search
@@ -1,4 +1,4 @@
|
||||
import { probeAcpxClaudeInstallation, probeAcpxCursorInstallation } from "@paperclipai/paperclip-runner/live";
|
||||
import { probeAcpxClaudeInstallation, probeAcpxPiInstallation } from "../vendor/paperclip-runner/live/index.js";
|
||||
import { describe, expect, it, beforeEach, afterEach, vi } from "vitest";
|
||||
import { buildSandboxNpmInstallCommand } from "@paperclipai/adapter-utils";
|
||||
import type { ServerAdapterModule } from "../adapters/index.js";
|
||||
@@ -17,9 +17,10 @@ import {
|
||||
setOverridePaused,
|
||||
} from "../adapters/registry.js";
|
||||
|
||||
vi.mock("@paperclipai/paperclip-runner/live", () => ({
|
||||
vi.mock("../vendor/paperclip-runner/live/index.js", () => ({
|
||||
probeAcpxClaudeInstallation: vi.fn(async () => undefined),
|
||||
probeAcpxGrokInstallation: vi.fn(async () => undefined),
|
||||
probeAcpxPiInstallation: vi.fn(async () => undefined),
|
||||
probeAcpxCursorInstallation: vi.fn(async () => undefined),
|
||||
}));
|
||||
|
||||
@@ -327,23 +328,10 @@ describe("server adapter registry", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it.each([true, false])("checks ordinary Cursor runtime readiness (%s)", async (ready) => {
|
||||
const probe = vi.mocked(probeAcpxCursorInstallation);
|
||||
it.each([true, false])("probes the exact Pi installation without qualification opt-in (ready=%s)", async (ready) => {
|
||||
const probe = vi.mocked(probeAcpxPiInstallation);
|
||||
if (ready) probe.mockResolvedValueOnce(undefined);
|
||||
else probe.mockRejectedValueOnce(new Error("Run paperclipai runtime setup cursor"));
|
||||
const model = "gpt-5.6-luna[context=272k,reasoning=medium,fast=false]";
|
||||
const result = await requireServerAdapter("paperclip_runner").testEnvironment({
|
||||
companyId: "company-1", adapterType: "paperclip_runner",
|
||||
config: { provider: "acpx", acpxAgent: "cursor", model },
|
||||
});
|
||||
expect(probe).toHaveBeenLastCalledWith(model);
|
||||
expect(result).toMatchObject({
|
||||
status: ready ? "pass" : "fail",
|
||||
checks: [expect.objectContaining({ code: ready ? "acpx_runtime_ready" : "acpx_runtime_unavailable" })],
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps the ACPX Pi profile unavailable", async () => {
|
||||
else probe.mockRejectedValueOnce(new Error("Pi closure unavailable"));
|
||||
const result = await requireServerAdapter("paperclip_runner").testEnvironment({
|
||||
companyId: "company-1",
|
||||
adapterType: "paperclip_runner",
|
||||
@@ -355,9 +343,10 @@ describe("server adapter registry", () => {
|
||||
});
|
||||
|
||||
expect(result).toMatchObject({
|
||||
status: "fail",
|
||||
checks: [{ code: "paperclip_runner_acpx_agent_unavailable" }],
|
||||
status: ready ? "pass" : "fail",
|
||||
checks: [{ code: ready ? "acpx_runtime_ready" : "acpx_runtime_unavailable" }],
|
||||
});
|
||||
expect(probe).toHaveBeenLastCalledWith("openrouter/deepseek/deepseek-v4-flash-0731");
|
||||
});
|
||||
it("reports qualification-only readiness for an exact host-authorized candidate", async () => {
|
||||
const key = "PAPERCLIP_RUNNER_ACPX_QUALIFICATION";
|
||||
|
||||
@@ -0,0 +1,221 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import fs from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { createRequire } from "node:module";
|
||||
import childProcess, { execFileSync } from "node:child_process";
|
||||
import { prepareAdapterExecutionTargetRuntime } from "@paperclipai/adapter-utils/execution-target";
|
||||
import { captureDirectorySnapshot } from "@paperclipai/adapter-utils/workspace-restore-merge";
|
||||
import { prepareSandboxManagedRuntime } from "@paperclipai/adapter-utils/sandbox-managed-runtime";
|
||||
import { agentDirectoryBaselineDigest, agentDirectoryProbeProgram, observeLocalAgentDirectory, probeAgentDirectory, retireAgentDirectoryTransferScratch, agentDirectoryTransferCleanupProgram } from "../services/agent-directory-probe.js";
|
||||
|
||||
describe("stable live agent directory observation", () => {
|
||||
let root: string;
|
||||
beforeEach(() => {
|
||||
root = fs.realpathSync(fs.mkdtempSync(join(tmpdir(), "agent-directory-probe-")));
|
||||
fs.mkdirSync(join(root, "notes"));
|
||||
fs.writeFileSync(join(root, "AGENTS.md"), "Instructions\n");
|
||||
fs.writeFileSync(join(root, "notes", "memory.bin"), Buffer.from([0, 1, 255]));
|
||||
});
|
||||
afterEach(() => { vi.restoreAllMocks(); fs.rmSync(root, { recursive: true, force: true }); });
|
||||
function productionPrograms(): { probe: string; cleanup: string } {
|
||||
// Product/dev starts the server with tsx. Vitest's own transform does not
|
||||
// reproduce the lexical helpers injected into function.toString() there.
|
||||
const loader = createRequire(import.meta.url).resolve("tsx");
|
||||
const source = new URL("../services/agent-directory-probe.ts", import.meta.url).href;
|
||||
const program = `import { agentDirectoryProbeProgram, agentDirectoryTransferCleanupProgram } from ${JSON.stringify(source)}; process.stdout.write(JSON.stringify({probe:agentDirectoryProbeProgram,cleanup:agentDirectoryTransferCleanupProgram}));`;
|
||||
return JSON.parse(execFileSync(process.execPath, ["--import", loader, "--input-type=module", "--eval", program], {
|
||||
cwd: root, env: { TSX_DISABLE_CACHE: "1", HOME: root, TMPDIR: root }, encoding: "utf8", timeout: 10_000, maxBuffer: 64 * 1024,
|
||||
}));
|
||||
}
|
||||
it("runs a self-contained live probe after the production tsx transform", () => {
|
||||
const { probe } = productionPrograms();
|
||||
const observe = () => JSON.parse(execFileSync(process.execPath, ["-e", probe, root], {
|
||||
env: {}, encoding: "utf8", timeout: 10_000, maxBuffer: 64 * 1024, stdio: "pipe",
|
||||
}));
|
||||
const before = probeAgentDirectory(root);
|
||||
expect(observe()).toEqual(before);
|
||||
fs.writeFileSync(join(root, "AGENTS.md"), "Changed instructions\n");
|
||||
expect(observe()).toEqual(probeAgentDirectory(root));
|
||||
expect(observe().digest).not.toBe(before.digest);
|
||||
fs.symlinkSync(join(root, "AGENTS.md"), join(root, "unsafe"));
|
||||
expect(observe).toThrow();
|
||||
});
|
||||
it("runs bounded transfer cleanup after the production tsx transform", () => {
|
||||
const { cleanup } = productionPrograms();
|
||||
const scratch = join(root, ".paperclip-runtime", "agent-files");
|
||||
const retire = () => execFileSync(process.execPath, ["-e", cleanup, root], {
|
||||
env: {}, encoding: "utf8", timeout: 10_000, maxBuffer: 64 * 1024, stdio: "pipe",
|
||||
});
|
||||
fs.mkdirSync(scratch, { recursive: true });
|
||||
fs.writeFileSync(join(scratch, "unexpected"), "retain");
|
||||
expect(retire).toThrow();
|
||||
expect(fs.readFileSync(join(scratch, "unexpected"), "utf8")).toBe("retain");
|
||||
fs.unlinkSync(join(scratch, "unexpected"));
|
||||
retire();
|
||||
expect(fs.existsSync(join(root, ".paperclip-runtime"))).toBe(false);
|
||||
expect(fs.readFileSync(join(root, "AGENTS.md"), "utf8")).toBe("Instructions\n");
|
||||
expect(fs.readFileSync(join(root, "notes", "memory.bin"))).toEqual(Buffer.from([0, 1, 255]));
|
||||
});
|
||||
it("matches the complete materialized baseline and the actual remote Node program", async () => {
|
||||
const baseline = await captureDirectorySnapshot(root);
|
||||
const observed = probeAgentDirectory(root);
|
||||
expect(observed.digest).toBe(agentDirectoryBaselineDigest(baseline));
|
||||
const remote = JSON.parse(execFileSync(process.execPath, ["-e", agentDirectoryProbeProgram, root], { encoding: "utf8", env: {}, timeout: 10_000 }));
|
||||
expect(remote).toEqual(observed);
|
||||
});
|
||||
it.runIf(process.platform === "darwin")("accepts only the observing Darwin host's fixed system aliases", () => {
|
||||
const alias = root.replace(/^\/private(?=\/(tmp|var|etc)\/)/, "");
|
||||
expect(alias).not.toBe(root);
|
||||
const observed = probeAgentDirectory(root);
|
||||
expect(probeAgentDirectory(alias)).toEqual(observed);
|
||||
const remote = JSON.parse(execFileSync(process.execPath, ["-e", agentDirectoryProbeProgram, alias], { encoding: "utf8", env: {}, timeout: 10_000 }));
|
||||
expect(remote).toEqual(observed);
|
||||
// A Linux remote cannot inherit the controller's Darwin exception.
|
||||
expect(() => execFileSync(process.execPath, ["-e", agentDirectoryProbeProgram.replace("platform:process.platform", "platform:'linux'"), alias],
|
||||
{ encoding: "utf8", env: {}, timeout: 10_000, stdio: "pipe" })).toThrow();
|
||||
fs.symlinkSync(root, join(root, "redirect"));
|
||||
expect(() => probeAgentDirectory(`${alias}/redirect`)).toThrow();
|
||||
});
|
||||
it("observes the complete agent directory after the actual remote transfer", async () => {
|
||||
const remote = `${root}-remote`;
|
||||
fs.mkdirSync(remote);
|
||||
const target = { kind: "remote" as const, transport: "sandbox" as const, remoteCwd: remote,
|
||||
runner: { execute: async (input: Parameters<import("@paperclipai/adapter-utils/command-managed-runtime").CommandManagedRuntimeRunner["execute"]>[0]) => ({
|
||||
stdout: execFileSync(input.command, input.args ?? [], { cwd: input.cwd, input: input.stdin, encoding: "utf8",
|
||||
env: { PATH: process.env.PATH, ...input.env }, timeout: input.timeoutMs, maxBuffer: 32 * 1024 * 1024 }),
|
||||
stderr: "", exitCode: 0, signal: null, timedOut: false, pid: null, startedAt: new Date().toISOString(),
|
||||
}) } };
|
||||
let runtime: Awaited<ReturnType<typeof prepareAdapterExecutionTargetRuntime>> | undefined;
|
||||
try {
|
||||
const baseline = await captureDirectorySnapshot(root);
|
||||
runtime = await prepareAdapterExecutionTargetRuntime({ target, runId: "probe-transfer", adapterKey: "agent-files",
|
||||
workspaceLocalDir: root, workspaceRemoteDir: remote, syncWorkspace: true, workspaceBaseline: baseline,
|
||||
workspaceGitSnapshot: null, workspaceFileMode: "all", workspaceExclude: [".paperclip-runtime", ".paperclip-runtime/**"] });
|
||||
expect(fs.readdirSync(`${remote}/.paperclip-runtime`)).toEqual(["agent-files"]);
|
||||
expect(fs.readdirSync(`${remote}/.paperclip-runtime/agent-files`)).toEqual([]);
|
||||
execFileSync(process.execPath, ["-e", agentDirectoryTransferCleanupProgram, remote], { env: {}, timeout: 10_000 });
|
||||
expect(() => probeAgentDirectory(remote)).not.toThrow();
|
||||
expect(probeAgentDirectory(remote).digest).toBe(agentDirectoryBaselineDigest(baseline));
|
||||
} finally { await runtime?.cleanupWorkspaceSnapshot?.(); fs.rmSync(remote, { recursive: true, force: true }); }
|
||||
});
|
||||
it("keeps restart fallback transfer scratch outside the native memory directory", async () => {
|
||||
const remoteCwd = `${root}-remote`;
|
||||
const agentHome = join(remoteCwd, ".paperclip-runtime", "agent-files", "agent", "original-run");
|
||||
const scratch = join(remoteCwd, ".paperclip-runtime", "paperclip-runner", "agent-file-transfers", "agent", "original-run");
|
||||
fs.mkdirSync(agentHome, { recursive: true });
|
||||
fs.cpSync(root, agentHome, { recursive: true });
|
||||
const target = { kind: "remote" as const, transport: "sandbox" as const, remoteCwd,
|
||||
// A newly bound controller target may have no native sync hooks. Exercise
|
||||
// the production command/base64 fallback rather than a transport mock.
|
||||
runner: { execute: async (input: Parameters<import("@paperclipai/adapter-utils/command-managed-runtime").CommandManagedRuntimeRunner["execute"]>[0]) => ({
|
||||
stdout: execFileSync(input.command, input.args ?? [], { cwd: input.cwd, input: input.stdin, encoding: "utf8",
|
||||
env: { PATH: process.env.PATH, ...input.env }, timeout: input.timeoutMs, maxBuffer: 32 * 1024 * 1024 }),
|
||||
stderr: "", exitCode: 0, signal: null, timedOut: false, pid: null, startedAt: new Date().toISOString(),
|
||||
}) } };
|
||||
let runtime: Awaited<ReturnType<typeof prepareAdapterExecutionTargetRuntime>> | undefined;
|
||||
try {
|
||||
const baseline = await captureDirectorySnapshot(root);
|
||||
runtime = await prepareAdapterExecutionTargetRuntime({ target, runId: "restart-transfer", adapterKey: "agent-files",
|
||||
workspaceLocalDir: root, workspaceRemoteDir: agentHome, runtimeRootDir: scratch,
|
||||
syncWorkspace: true, workspaceInboundMode: "adopt_remote", workspaceBaseline: baseline,
|
||||
workspaceGitSnapshot: null, workspaceFileMode: "all", workspaceExclude: [".paperclip-runtime", ".paperclip-runtime/**"] });
|
||||
const memory = "0123456789abcdef0123456789abcdef\n";
|
||||
fs.writeFileSync(join(agentHome, "notes", "memory.txt"), memory);
|
||||
await runtime.restoreWorkspace();
|
||||
expect(fs.readFileSync(join(root, "notes", "memory.txt"), "utf8")).toBe(memory);
|
||||
// A transfer must not introduce a reserved path that the product's own
|
||||
// full agent-directory probe rejects, including after controller recovery.
|
||||
expect(() => probeAgentDirectory(agentHome)).not.toThrow();
|
||||
expect(fs.existsSync(join(agentHome, ".paperclip-runtime"))).toBe(false);
|
||||
expect(runtime.runtimeRootDir).toBe(scratch);
|
||||
expect(fs.readdirSync(scratch)).toEqual([]);
|
||||
} finally { await runtime?.cleanupWorkspaceSnapshot?.(); fs.rmSync(remoteCwd, { recursive: true, force: true }); }
|
||||
});
|
||||
it.each(["outside", "reserved-root", "relative", "traversal", "trailing-slash", "nul"])("rejects %s transfer scratch before filesystem work", async kind => {
|
||||
const reserved = join(root, ".paperclip-runtime");
|
||||
const invalid = {
|
||||
outside: `${root}-foreign/scratch`, "reserved-root": reserved, relative: ".paperclip-runtime/scratch",
|
||||
traversal: `${reserved}/../../scratch`, "trailing-slash": `${reserved}/scratch/`, nul: `${reserved}/scratch\0`,
|
||||
}[kind]!;
|
||||
await expect(prepareSandboxManagedRuntime({
|
||||
spec: { transport: "sandbox", provider: "fixture", sandboxId: "owned", remoteCwd: root, apiKey: null },
|
||||
adapterKey: "agent-files", client: {} as never, workspaceLocalDir: root, runtimeRootDir: invalid,
|
||||
})).rejects.toThrow("Transfer scratch must remain within the lease runtime tree");
|
||||
expect(fs.existsSync(reserved)).toBe(false);
|
||||
});
|
||||
it.each(["file", "extra-directory", "symlink", "race"])("fails closed on %s in transfer-owned scratch without deleting contents", kind => {
|
||||
const parent = join(root, ".paperclip-runtime"), scratch = join(parent, "agent-files");
|
||||
fs.mkdirSync(scratch, { recursive: true });
|
||||
const retained = join(parent, "retained.txt");
|
||||
if (kind === "file") fs.writeFileSync(join(scratch, "retained.txt"), "keep");
|
||||
if (kind === "extra-directory") fs.mkdirSync(join(parent, "unknown"));
|
||||
if (kind === "symlink") { fs.rmdirSync(scratch); fs.symlinkSync(join(root, "notes"), scratch); }
|
||||
if (kind === "race") {
|
||||
const rmdir = fs.rmdirSync;
|
||||
vi.spyOn(fs, "rmdirSync").mockImplementation(((directory: fs.PathLike) => {
|
||||
if (directory === scratch) fs.writeFileSync(retained, "concurrent bytes");
|
||||
return rmdir(directory);
|
||||
}) as typeof rmdir);
|
||||
}
|
||||
expect(() => retireAgentDirectoryTransferScratch(root)).toThrow();
|
||||
expect(fs.existsSync(parent)).toBe(true);
|
||||
if (kind === "file") expect(fs.readFileSync(join(scratch, "retained.txt"), "utf8")).toBe("keep");
|
||||
if (kind === "race") expect(fs.readFileSync(retained, "utf8")).toBe("concurrent bytes");
|
||||
expect(fs.existsSync(join(root, "notes", "memory.bin"))).toBe(true);
|
||||
});
|
||||
it("runs the local observation in its bounded credential-free child", async () => {
|
||||
expect(await observeLocalAgentDirectory(root)).toEqual(probeAgentDirectory(root));
|
||||
});
|
||||
it.each(["timeout", "oversized", "malformed"])("rejects %s local-child observations", async kind => {
|
||||
const stub = vi.spyOn(childProcess, "execFile").mockImplementation(((file: string, args: string[], options: object, callback: (error: Error | null, stdout: string, stderr: string) => void) => {
|
||||
expect(file).toBe(process.execPath);
|
||||
expect(args).toEqual(["-e", agentDirectoryProbeProgram, root]);
|
||||
expect(options).toMatchObject({ cwd: root, env: {}, timeout: 10_000, maxBuffer: 1024, killSignal: "SIGKILL" });
|
||||
callback(kind === "timeout" ? Object.assign(new Error("timed out"), { killed: true, signal: "SIGKILL" }) : null,
|
||||
kind === "oversized" ? "x".repeat(1025) : "{}", "");
|
||||
return {} as childProcess.ChildProcess;
|
||||
}) as typeof childProcess.execFile);
|
||||
await expect(observeLocalAgentDirectory(root)).rejects.toThrow();
|
||||
expect(stub).toHaveBeenCalledOnce();
|
||||
});
|
||||
it.each(["edit", "add", "remove", "mode"])("detects a legitimate %s", change => {
|
||||
const before = probeAgentDirectory(root);
|
||||
if (change === "edit") fs.writeFileSync(join(root, "AGENTS.md"), "New instructions\n");
|
||||
if (change === "add") fs.writeFileSync(join(root, "new.txt"), "New");
|
||||
if (change === "remove") fs.unlinkSync(join(root, "notes", "memory.bin"));
|
||||
if (change === "mode") fs.chmodSync(join(root, "AGENTS.md"), 0o700);
|
||||
expect(probeAgentDirectory(root).digest).not.toBe(before.digest);
|
||||
});
|
||||
it.each(["symlink", "hardlink", "reserved"])("rejects %s entries", kind => {
|
||||
if (kind === "symlink") fs.symlinkSync(join(root, "AGENTS.md"), join(root, "unsafe"));
|
||||
if (kind === "hardlink") fs.linkSync(join(root, "AGENTS.md"), join(root, "unsafe"));
|
||||
if (kind === "reserved") fs.mkdirSync(join(root, ".paperclip-runtime"));
|
||||
expect(() => probeAgentDirectory(root)).toThrow();
|
||||
});
|
||||
it("detects file mutation during a bounded read", () => {
|
||||
const read = fs.readSync;
|
||||
let changed = false;
|
||||
vi.spyOn(fs, "readSync").mockImplementation(((...args: Parameters<typeof read>) => {
|
||||
const result = Reflect.apply(read, fs, args);
|
||||
if (!changed) { changed = true; fs.appendFileSync(join(root, "AGENTS.md"), "Concurrent change"); }
|
||||
return result;
|
||||
}) as typeof read);
|
||||
expect(() => probeAgentDirectory(root)).toThrow(/changed|grew/);
|
||||
});
|
||||
it("fails on unreadable observations instead of reporting unchanged", () => {
|
||||
vi.spyOn(fs, "openSync").mockImplementation(() => { throw Object.assign(new Error("denied"), { code: "EACCES" }); });
|
||||
expect(() => probeAgentDirectory(root)).toThrow("denied");
|
||||
});
|
||||
it("does not let matching replacement bytes preserve the pinned root identity", () => {
|
||||
const before = probeAgentDirectory(root);
|
||||
const retired = `${root}-retired`;
|
||||
fs.renameSync(root, retired);
|
||||
try { fs.cpSync(retired, root, { recursive: true }); expect(probeAgentDirectory(root).identity).not.toBe(before.identity); }
|
||||
finally { fs.rmSync(retired, { recursive: true, force: true }); }
|
||||
});
|
||||
it("rejects excluded/incomplete baselines", async () => {
|
||||
const baseline = await captureDirectorySnapshot(root);
|
||||
expect(() => agentDirectoryBaselineDigest({ ...baseline, exclude: ["notes"] })).toThrow("Incomplete");
|
||||
});
|
||||
});
|
||||
@@ -74,6 +74,183 @@ describe("persistent agent directories", () => {
|
||||
await fs.writeFile(path.join(root, entryFile), initial);
|
||||
});
|
||||
|
||||
describe("warm directory ownership", () => {
|
||||
async function nextRun() {
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", responsibleUserId: userId });
|
||||
return runId;
|
||||
}
|
||||
it("adopts one physical root and fences stale cleanup through final save", async () => {
|
||||
const first = await run();
|
||||
expect(await copies.hasChanges({ companyId, runId: first.runId })).toBe(false);
|
||||
const secondId = await nextRun();
|
||||
const second = await copies.adopt({ companyId, agentId, runId: secondId, previousRunId: first.runId, cwd: home });
|
||||
expect(second).toMatchObject({ runId: secondId, localRoot: first.localRoot, executionRoot: first.executionRoot,
|
||||
responsibleUserId: userId, receipt: { materializationRunId: first.runId } });
|
||||
expect((await copies.get(companyId, first.runId))?.receipt?.retainedByRunId).toBe(secondId);
|
||||
await copies.release(companyId, first.runId);
|
||||
expect((await copies.collectStopped({ companyId, runId: first.runId }))?.processStoppedAt).toBeNull();
|
||||
await copies.reportUnavailable(companyId, first.runId);
|
||||
expect(await fs.readFile(path.join(first.localRoot, entryFile), "utf8")).toBe(initial);
|
||||
await copies.release(companyId, secondId);
|
||||
expect(await copies.hasChanges({ companyId, runId: secondId })).toBe(false);
|
||||
const thirdId = await nextRun();
|
||||
const third = await copies.adopt({ companyId, agentId, runId: thirdId, previousRunId: secondId, cwd: home });
|
||||
expect(third?.executionRoot).toBe(first.executionRoot);
|
||||
await fs.writeFile(path.join(first.localRoot, "warm-memory.txt"), "third turn edit");
|
||||
expect(await copies.hasChanges({ companyId, runId: thirdId })).toBe(true);
|
||||
expect(await copies.collectStopped({ companyId, runId: thirdId })).toMatchObject({ state: "saved", receipt: { cleanupPending: false } });
|
||||
expect(await fs.readFile(path.join(root, "warm-memory.txt"), "utf8")).toBe("third turn edit");
|
||||
await expect(fs.stat(first.localRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await copies.release(companyId, first.runId);
|
||||
});
|
||||
it.each(["company", "agent", "workspace", "environment"])("rejects foreign %s adoption", async mismatch => {
|
||||
const first = await run();
|
||||
expect(await copies.hasChanges({ companyId, runId: first.runId })).toBe(false);
|
||||
const secondId = await nextRun();
|
||||
expect(await copies.adopt({ companyId: mismatch === "company" ? randomUUID() : companyId,
|
||||
agentId: mismatch === "agent" ? randomUUID() : agentId, runId: secondId, previousRunId: first.runId,
|
||||
cwd: mismatch === "workspace" ? `${home}-other` : home,
|
||||
...(mismatch === "environment" ? { target: { kind: "remote" as const, transport: "sandbox" as const, environmentId: randomUUID(), remoteCwd: "/workspace" } } : {}),
|
||||
})).toBeNull();
|
||||
expect((await copies.get(companyId, first.runId))?.receipt?.retainedByRunId).toBeUndefined();
|
||||
await copies.collectStopped({ companyId, runId: first.runId });
|
||||
});
|
||||
it("allows only one stale concurrent adoption with identical updatedAt timestamps", async () => {
|
||||
const fixed = new Date("2030-01-01T00:00:00.000Z");
|
||||
vi.useFakeTimers({ toFake: ["Date"] });
|
||||
vi.setSystemTime(fixed);
|
||||
try {
|
||||
const first = await run();
|
||||
expect(await copies.hasChanges({ companyId, runId: first.runId })).toBe(false);
|
||||
const next = await Promise.all([nextRun(), nextRun()]);
|
||||
let reads = 0;
|
||||
let releaseReads!: () => void;
|
||||
const bothRead = new Promise<void>(resolve => { releaseReads = resolve; });
|
||||
// Both claimants see the same unclaimed row before either can enter
|
||||
// the agent-row transaction. Date precision cannot distinguish them.
|
||||
const staleReader = async () => {
|
||||
const captured = await copies.get(companyId, first.runId);
|
||||
reads++;
|
||||
if (reads >= 3) { if (reads === 4) releaseReads(); await bothRead; }
|
||||
return captured;
|
||||
};
|
||||
const service = agentDirectoryWorkingCopyService(db, staleReader, async (row, values) => {
|
||||
const [updated] = await db.update(agentInstructionWorkingCopies).set({ ...values, updatedAt: fixed })
|
||||
.where(eq(agentInstructionWorkingCopies.runId, row.runId)).returning();
|
||||
return updated;
|
||||
});
|
||||
const settled = await Promise.allSettled(next.map(runId => service.adopt({ companyId, agentId, runId, previousRunId: first.runId, cwd: home })));
|
||||
const winners = settled.filter(result => result.status === "fulfilled" && result.value);
|
||||
expect(winners).toHaveLength(1);
|
||||
expect(settled.filter(result => result.status === "rejected")).toHaveLength(1);
|
||||
const prior = await copies.get(companyId, first.runId);
|
||||
expect(prior?.updatedAt).toEqual(fixed);
|
||||
const winnerId = prior?.receipt?.retainedByRunId as string;
|
||||
expect(next).toContain(winnerId);
|
||||
expect(await copies.get(companyId, next.find(id => id !== winnerId)!)).toBeNull();
|
||||
await copies.collectStopped({ companyId, runId: winnerId });
|
||||
} finally { vi.useRealTimers(); }
|
||||
});
|
||||
it("fences a stale collector when adoption commits immediately before its stop CAS", async () => {
|
||||
const first = await run();
|
||||
expect(await copies.hasChanges({ companyId, runId: first.runId })).toBe(false);
|
||||
const secondId = await nextRun();
|
||||
const update = db.update.bind(db);
|
||||
let injected = false;
|
||||
const spy = vi.spyOn(db, "update").mockImplementation(table => {
|
||||
const builder = update(table);
|
||||
if (table !== agentInstructionWorkingCopies) return builder;
|
||||
const set = builder.set.bind(builder);
|
||||
builder.set = (values => {
|
||||
const query = set(values);
|
||||
if (!("processStoppedAt" in values) || !values.processStoppedAt || injected) return query;
|
||||
const returning = query.returning.bind(query);
|
||||
query.returning = (async () => {
|
||||
injected = true;
|
||||
expect(await copies.adopt({ companyId, agentId, runId: secondId, previousRunId: first.runId, cwd: home })).not.toBeNull();
|
||||
return returning();
|
||||
}) as typeof query.returning;
|
||||
return query;
|
||||
}) as typeof builder.set;
|
||||
return builder;
|
||||
});
|
||||
try {
|
||||
expect(await copies.collectStopped({ companyId, runId: first.runId })).toMatchObject({
|
||||
receipt: { retainedByRunId: secondId }, processStoppedAt: null, attempts: 0,
|
||||
});
|
||||
expect(injected).toBe(true);
|
||||
expect(await fs.readFile(path.join(first.localRoot, entryFile), "utf8")).toBe(initial);
|
||||
expect((await copies.get(companyId, secondId))?.processStoppedAt).toBeNull();
|
||||
} finally { spy.mockRestore(); }
|
||||
await copies.collectStopped({ companyId, runId: secondId });
|
||||
});
|
||||
it("requires current-run authorization before transferring collection authority", async () => {
|
||||
const first = await run();
|
||||
expect(await copies.hasChanges({ companyId, runId: first.runId })).toBe(false);
|
||||
const secondId = await nextRun();
|
||||
await db.update(heartbeatRuns).set({ responsibleUserId: null }).where(eq(heartbeatRuns.id, secondId));
|
||||
await expect(copies.adopt({ companyId, agentId, runId: secondId, previousRunId: first.runId, cwd: home })).rejects.toMatchObject({ status: 403 });
|
||||
expect((await copies.get(companyId, first.runId))?.receipt?.retainedByRunId).toBeUndefined();
|
||||
await copies.collectStopped({ companyId, runId: first.runId });
|
||||
});
|
||||
it.each(["owner-close", "restart-proof"])("durably preserves an adopted directory after retirement failure and final cleanup: %s", async recovery => {
|
||||
const first = await run();
|
||||
expect(await copies.hasChanges({ companyId, runId: first.runId })).toBe(false);
|
||||
const secondId = await nextRun();
|
||||
await copies.adopt({ companyId, agentId, runId: secondId, previousRunId: first.runId, cwd: home });
|
||||
await db.update(heartbeatRuns).set({ status: "failed", runtimeMode: "native" }).where(eq(heartbeatRuns.id, secondId));
|
||||
expect(await copies.reportRetirementUnconfirmed(companyId, secondId)).toMatchObject({
|
||||
state: "pending_collection", errorCode: "INSTRUCTION_STOP_UNCONFIRMED", processStoppedAt: null,
|
||||
});
|
||||
await fs.writeFile(path.join(first.localRoot, "unsaved-memory.txt"), "Keep failed retirement edits");
|
||||
// Mirror heartbeat's final cleanup after preparation throws. Neither the
|
||||
// current pending owner nor the prior alias may become terminal loss.
|
||||
await copies.reportUnavailable(companyId, secondId);
|
||||
await copies.reportUnavailable(companyId, first.runId);
|
||||
expect(await copies.get(companyId, secondId)).toMatchObject({ state: "pending_collection", errorCode: "INSTRUCTION_STOP_UNCONFIRMED", processStoppedAt: null });
|
||||
await copies.reportRetirementUnconfirmed(companyId, first.runId);
|
||||
await copies.release(companyId, first.runId);
|
||||
await copies.release(companyId, secondId);
|
||||
await copies.recoverStopped(); // A terminal run with no stop receipt grants no collection authority.
|
||||
expect((await copies.get(companyId, secondId))?.processStoppedAt).toBeNull();
|
||||
expect(await fs.readFile(path.join(first.localRoot, entryFile), "utf8")).toBe(initial);
|
||||
if (recovery === "owner-close") await copies.collectStopped({ companyId, runId: secondId }); // Explicit test owner retirement.
|
||||
else {
|
||||
const { appendHeartbeatRunEvent } = await import("../services/heartbeat-run-events.js");
|
||||
const stop = (runId: string) => appendHeartbeatRunEvent(db, { companyId, runId, agentId,
|
||||
eventType: "native.local_process_stopped", stream: "system", level: "info", message: "Test-owned process retirement proof", payload: {} });
|
||||
await stop(first.runId);
|
||||
copies = agentInstructionWorkingCopyService(db); // No in-memory owner or transport survives.
|
||||
await copies.recoverStopped();
|
||||
expect((await copies.get(companyId, secondId))?.processStoppedAt).toBeNull();
|
||||
await stop(secondId);
|
||||
// Make the deferred maintenance retry due after the new owner stop proof.
|
||||
await db.update(agentInstructionWorkingCopies).set({ nextAttemptAt: null }).where(eq(agentInstructionWorkingCopies.runId, secondId));
|
||||
await copies.recoverStopped();
|
||||
}
|
||||
expect(await copies.get(companyId, secondId)).toMatchObject({ state: "saved", receipt: { cleanupPending: false } });
|
||||
expect(await fs.readFile(path.join(root, "unsaved-memory.txt"), "utf8")).toBe("Keep failed retirement edits");
|
||||
await expect(fs.stat(first.localRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
});
|
||||
it("declines reuse after canonical instruction changes", async () => {
|
||||
const first = await run();
|
||||
expect(await copies.hasChanges({ companyId, runId: first.runId })).toBe(false);
|
||||
await fs.writeFile(path.join(root, entryFile), "Changed canonical instructions");
|
||||
const secondId = await nextRun();
|
||||
expect(await copies.adopt({ companyId, agentId, runId: secondId, previousRunId: first.runId, cwd: home })).toBeNull();
|
||||
await copies.collectStopped({ companyId, runId: first.runId });
|
||||
});
|
||||
it("requires stop for matching bytes at a replaced materialized root", async () => {
|
||||
const first = await run();
|
||||
const replaced = `${first.localRoot}-old`;
|
||||
await fs.rename(first.localRoot, replaced);
|
||||
await fs.cp(replaced, first.localRoot, { recursive: true });
|
||||
expect(await copies.hasChanges({ companyId, runId: first.runId })).toBe(true);
|
||||
await copies.collectStopped({ companyId, runId: first.runId });
|
||||
});
|
||||
});
|
||||
|
||||
it.each([".paperclip-runtime/state", "notes/.paperclip-runtime/state", "promptTemplate.legacy.md"])("rejects reserved board path %s before mutation", async (reserved) => {
|
||||
await expect(agentFileStore(db).write({ ...target(), path: reserved, bytes: Buffer.from("reserved"), baseHash: null }, board())).rejects.toMatchObject({ status: 422 });
|
||||
await expect(fs.stat(path.join(root, reserved))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
@@ -381,7 +558,8 @@ describe("persistent agent directories", () => {
|
||||
}
|
||||
expect(execute).toHaveBeenCalledTimes(2);
|
||||
expect(execute).toHaveBeenLastCalledWith(expect.objectContaining({ lease: expect.objectContaining({ id: leaseId, providerLeaseId: "original-sandbox" }),
|
||||
command: "rm", args: ["-rf", "--", executionRoot], bypassSession: true }));
|
||||
command: "rm", args: ["-rf", "--", executionRoot,
|
||||
path.posix.join(remoteCwd, ".paperclip-runtime", "paperclip-runner", "agent-file-transfers", agentId, copy.runId)], bypassSession: true }));
|
||||
await copies.recoverCaptured();
|
||||
expect(execute).toHaveBeenCalledTimes(2);
|
||||
await copies.release(companyId, copy.runId);
|
||||
@@ -389,6 +567,60 @@ describe("persistent agent directories", () => {
|
||||
expect(execute).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it.each(["missing", "stopped", "destroyed"])("recovers a retained remote unchanged turn only with exact stop proof: %s", async stopped => {
|
||||
const copy = await run();
|
||||
const environmentId = randomUUID(), leaseId = randomUUID(), remoteCwd = "/fixture/task";
|
||||
const lease = { id: leaseId, companyId, environmentId, heartbeatRunId: copy.runId, provider: "daytona", providerLeaseId: "retained-sandbox" };
|
||||
await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "sandbox" });
|
||||
await db.insert(environmentLeases).values({ ...lease, status: "released", releasedAt: new Date(), cleanupStatus: "success",
|
||||
metadata: stopped !== "missing" ? { remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: stopped }) } : {} });
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded", runtimeMode: "native" }).where(eq(heartbeatRuns.id, copy.runId));
|
||||
await db.update(agentInstructionWorkingCopies).set({ state: "unchanged_turn", location: `remote:${environmentId}`,
|
||||
executionRoot: path.posix.join(remoteCwd, ".paperclip-runtime", "agent-files", agentId, copy.runId),
|
||||
receipt: { ...copy.receipt, cleanup: { leaseId, remoteCwd } } }).where(eq(agentInstructionWorkingCopies.runId, copy.runId));
|
||||
const execute = vi.fn();
|
||||
copies = agentInstructionWorkingCopyService(db, { environmentRuntime: { execute } as unknown as EnvironmentRuntimeService });
|
||||
expect((await copies.reportUnavailable(companyId, copy.runId))?.state).toBe("unchanged_turn");
|
||||
await copies.recoverStopped();
|
||||
const recovered = (await copies.get(companyId, copy.runId))!;
|
||||
if (stopped === "destroyed") {
|
||||
expect(recovered).toMatchObject({ state: "unavailable", errorCode: "INSTRUCTION_COLLECTION_UNAVAILABLE", receipt: { cleanupPending: false } });
|
||||
expect(recovered.processStoppedAt).toBeInstanceOf(Date);
|
||||
await expect(fs.stat(copy.localRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await copies.recoverStopped();
|
||||
expect((await copies.get(companyId, copy.runId))?.updatedAt).toEqual(recovered.updatedAt);
|
||||
} else {
|
||||
expect(recovered).toMatchObject({ state: stopped === "stopped" ? "pending_collection" : "unchanged_turn", processStoppedAt: null });
|
||||
if (stopped === "stopped") {
|
||||
expect(recovered.errorCode).toBe("INSTRUCTION_STOPPED_REMOTE_COLLECTION_PENDING");
|
||||
expect(recovered.receipt?.baseline).toBeDefined();
|
||||
await copies.release(companyId, copy.runId);
|
||||
await copies.recoverCaptured();
|
||||
}
|
||||
expect(await fs.readFile(path.join(copy.localRoot, entryFile), "utf8")).toBe(initial);
|
||||
}
|
||||
expect(execute).not.toHaveBeenCalled(); // Never restart a remote provider to recover bytes.
|
||||
});
|
||||
|
||||
it.each([0, 2])("preserves a legacy no-ID remote copy with %s matching leases", async count => {
|
||||
const copy = await run();
|
||||
const environmentId = randomUUID(), remoteCwd = "/fixture/task";
|
||||
await db.insert(environments).values({ id: environmentId, name: `Legacy remote ${environmentId}`, driver: "sandbox" });
|
||||
for (let index = 0; index < count; index++) await db.insert(environmentLeases).values({
|
||||
companyId, environmentId, heartbeatRunId: copy.runId, provider: "daytona", providerLeaseId: `allocation-${index}` });
|
||||
await db.update(agentInstructionWorkingCopies).set({ location: `remote:${environmentId}`,
|
||||
executionRoot: path.posix.join(remoteCwd, ".paperclip-runtime", "agent-files", agentId, copy.runId),
|
||||
receipt: { ...copy.receipt, cleanup: { remoteCwd } } }).where(eq(agentInstructionWorkingCopies.runId, copy.runId));
|
||||
const execute = vi.fn();
|
||||
copies = agentInstructionWorkingCopyService(db, { environmentRuntime: { execute } as unknown as EnvironmentRuntimeService });
|
||||
const pending = await copies.collectStopped({ companyId, runId: copy.runId });
|
||||
expect(pending).toMatchObject({ state: "pending_collection", attempts: 0, processStoppedAt: null, errorCode: "INSTRUCTION_REMOTE_LEASE_UNVERIFIED" });
|
||||
expect(pending!.receipt?.baseline).toBeDefined();
|
||||
await copies.release(companyId, copy.runId);
|
||||
expect(await fs.readFile(path.join(copy.localRoot, entryFile), "utf8")).toBe(initial);
|
||||
expect(execute).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("finishes remote cleanup from a destruction receipt after the environment is deleted", async () => {
|
||||
const copy = await run();
|
||||
const environmentId = randomUUID(), leaseId = randomUUID(), remoteCwd = "/fixture/task";
|
||||
@@ -614,15 +846,22 @@ describe("persistent agent directories", () => {
|
||||
}
|
||||
},
|
||||
};
|
||||
const executionTarget = { kind: "remote" as const, transport: "sandbox" as const, environmentId: randomUUID(), remoteCwd, runner };
|
||||
const executionTarget = { kind: "remote" as const, transport: "sandbox" as const, environmentId: randomUUID(), leaseId: "", remoteCwd, runner };
|
||||
await db.insert(environments).values({ id: executionTarget.environmentId, name: "Fixture sandbox", driver: "sandbox" });
|
||||
const prepare = async () => {
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", responsibleUserId: userId });
|
||||
executionTarget.leaseId = randomUUID();
|
||||
await db.insert(environmentLeases).values({ id: executionTarget.leaseId, companyId, environmentId: executionTarget.environmentId,
|
||||
heartbeatRunId: runId, provider: "daytona", providerLeaseId: "fixture-allocation" });
|
||||
return (await copies.prepare({ ...target(), runId, cwd: home, target: executionTarget }))!;
|
||||
};
|
||||
await fs.mkdir(path.join(root, "build"));
|
||||
await fs.writeFile(path.join(root, "build", "personal.txt"), "cache-like names are still agent files");
|
||||
const first = await prepare();
|
||||
const firstScratch = path.join(remoteCwd, ".paperclip-runtime", "paperclip-runner", "agent-file-transfers", agentId, first.runId);
|
||||
expect(await fs.readdir(firstScratch)).toEqual([]);
|
||||
await expect(fs.stat(path.join(first.executionRoot, ".paperclip-runtime"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect(await fs.readFile(path.join(first.executionRoot, "build", "personal.txt"), "utf8")).toBe("cache-like names are still agent files");
|
||||
await fs.mkdir(path.join(first.executionRoot, "notes"));
|
||||
await fs.writeFile(path.join(first.executionRoot, "notes", "bytes.bin"), Buffer.from([0, 128, 255]));
|
||||
@@ -632,6 +871,7 @@ describe("persistent agent directories", () => {
|
||||
expect((await execFile("git", ["-C", remoteCwd, "status", "--porcelain", "--untracked-files=all"])).stdout).toBe("?? task-only.txt\n");
|
||||
expect((await copies.collectStopped({ companyId, runId: first.runId, target: executionTarget }))?.state).toBe("saved");
|
||||
await copies.release(companyId, first.runId);
|
||||
await expect(fs.stat(firstScratch)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect((await copies.get(companyId, first.runId))?.receipt?.baseline).toBeUndefined();
|
||||
await expect(fs.stat(first.localRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await fs.rm(remoteCwd, { recursive: true });
|
||||
@@ -666,10 +906,14 @@ describe("persistent agent directories", () => {
|
||||
}
|
||||
},
|
||||
};
|
||||
const executionTarget = { kind: "remote" as const, transport: "sandbox" as const, environmentId: randomUUID(), remoteCwd, runner };
|
||||
const executionTarget = { kind: "remote" as const, transport: "sandbox" as const, environmentId: randomUUID(), leaseId: "", remoteCwd, runner };
|
||||
await db.insert(environments).values({ id: executionTarget.environmentId, name: "Warm fixture sandbox", driver: "sandbox" });
|
||||
const prepare = async (reuseRunId?: string) => {
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", responsibleUserId: userId });
|
||||
executionTarget.leaseId = randomUUID();
|
||||
await db.insert(environmentLeases).values({ id: executionTarget.leaseId, companyId, environmentId: executionTarget.environmentId,
|
||||
heartbeatRunId: runId, provider: "daytona", providerLeaseId: "warm-fixture-allocation" });
|
||||
return (await copies.prepare({ ...target(), runId, cwd: home, target: executionTarget, warm: true, reuseRunId }))!;
|
||||
};
|
||||
let copy = await prepare();
|
||||
@@ -687,7 +931,7 @@ describe("persistent agent directories", () => {
|
||||
}
|
||||
await copies.collectStopped({ companyId, runId: original.runId, target: executionTarget });
|
||||
expect(await fs.stat(copy.executionRoot)).toBeTruthy();
|
||||
await copies.collectStopped({ companyId, runId: copy.runId, target: executionTarget });
|
||||
expect((await copies.collectStopped({ companyId, runId: copy.runId, target: executionTarget }))?.receipt).toMatchObject({ cleanupPending: false });
|
||||
await expect(fs.stat(copy.executionRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
});
|
||||
|
||||
@@ -706,6 +950,9 @@ describe("persistent agent directories", () => {
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", responsibleUserId: userId });
|
||||
const target = { kind: "remote" as const, transport: "ssh" as const, environmentId: randomUUID(), remoteCwd,
|
||||
spec: { host: "unused.invalid", port: 22, username: "test", remoteCwd } };
|
||||
await db.insert(environments).values({ id: target.environmentId, name: "Fixture SSH", driver: "ssh" });
|
||||
// Legacy no-ID receipt: only the unique run/environment lease authorizes retrieval.
|
||||
await db.insert(environmentLeases).values({ companyId, environmentId: target.environmentId, heartbeatRunId: runId, provider: "ssh", providerLeaseId: "fixture-host" });
|
||||
const copy = (await copies.prepare({ companyId, agentId, runId, cwd: home, target }))!;
|
||||
expect(stage).toHaveBeenCalledWith(expect.objectContaining({ remoteDir: copy.executionRoot }));
|
||||
expect(await fs.readFile(path.join(copy.executionRoot, entryFile), "utf8")).toBe(initial);
|
||||
@@ -948,8 +1195,8 @@ describe("persistent agent directories", () => {
|
||||
const execute = vi.fn(), restoreWorkspace = vi.fn(), cleanupWorkspaceSnapshot = vi.fn().mockResolvedValue(undefined);
|
||||
const executionTarget = { kind: "remote" as const, transport: "sandbox" as const, environmentId, leaseId, remoteCwd, runner: { execute } };
|
||||
const shell = vi.spyOn(executionTargetTools, "runAdapterExecutionTargetShellCommand").mockResolvedValue({ exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" });
|
||||
const stage = vi.spyOn(executionTargetTools, "prepareAdapterExecutionTargetRuntime").mockImplementation(async () => ({ target: executionTarget,
|
||||
workspaceRemoteDir: path.posix.join(remoteCwd, ".paperclip-runtime", "agent-files", agentId, runId), runtimeRootDir: null,
|
||||
const stage = vi.spyOn(executionTargetTools, "prepareAdapterExecutionTargetRuntime").mockImplementation(async options => ({ target: executionTarget,
|
||||
workspaceRemoteDir: path.posix.join(remoteCwd, ".paperclip-runtime", "agent-files", agentId, runId), runtimeRootDir: options.runtimeRootDir ?? null,
|
||||
assetDirs: {}, additionalSourceDirs: {}, additionalSourceFailures: [], workspaceSyncSnapshot: null, restoreWorkspace, cleanupWorkspaceSnapshot }));
|
||||
try {
|
||||
const copy = (await copies.prepare({ ...target(), runId, cwd: home, target: executionTarget }))!;
|
||||
|
||||
@@ -300,6 +300,10 @@ describe("registered run instruction copies", () => {
|
||||
expect((await revisions.readCurrent(target(), board()))?.content).toBe(initial);
|
||||
expect((await copies.list(companyId, agentId, board()))[0]).toMatchObject({ state: "pending_collection", content: null });
|
||||
await appendHeartbeatRunEvent(db, { ...target(), runId: copy.runId, eventType: "native.local_process_stopped", stream: "system" });
|
||||
// Recovery scheduled a retry while stop authority was missing. Make that
|
||||
// retry due before asking the restarted controller to collect the copy.
|
||||
await db.update(agentInstructionWorkingCopies).set({ nextAttemptAt: new Date(0) })
|
||||
.where(eq(agentInstructionWorkingCopies.runId, copy.runId));
|
||||
await copies.recoverStopped();
|
||||
expect((await revisions.readCurrent(target(), board()))?.content).toBe("edit before controller restart");
|
||||
});
|
||||
|
||||
@@ -1084,17 +1084,30 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
.where(and(inArray(chatConversations.companyId, companyIds), inArray(chatConversations.state, ["active", "waiting"])));
|
||||
}
|
||||
|
||||
async function seedCompany() {
|
||||
const companyPrefixAttempts = 8;
|
||||
const freshCompanyPrefix = () => `C${randomUUID().replaceAll("-", "").slice(0, 7).toUpperCase()}`;
|
||||
|
||||
async function seedCompany(nextIssuePrefix = freshCompanyPrefix) {
|
||||
const companyId = randomUUID();
|
||||
fixtureCompanies.add(companyId);
|
||||
const assignedAgentId = randomUUID();
|
||||
const replacementAgentId = randomUUID();
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: `Chat Test ${companyId.slice(0, 8)}`,
|
||||
issuePrefix: `C${companyId.replace(/-/g, "").toUpperCase()}`,
|
||||
requireBoardApprovalForNewAgents: false,
|
||||
});
|
||||
let inserted = false;
|
||||
// Retired fixtures retain company rows. Retry only the short-prefix unique
|
||||
// conflict; every other database error must still fail the test immediately.
|
||||
for (let attempt = 0; attempt < companyPrefixAttempts; attempt += 1) {
|
||||
const [company] = await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: `Chat Test ${companyId.slice(0, 8)}`,
|
||||
issuePrefix: nextIssuePrefix(),
|
||||
requireBoardApprovalForNewAgents: false,
|
||||
}).onConflictDoNothing({ target: companies.issuePrefix }).returning({ id: companies.id });
|
||||
if (company) {
|
||||
inserted = true;
|
||||
fixtureCompanies.add(companyId);
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (!inserted) throw new Error(`Could not allocate a chat fixture company prefix after ${companyPrefixAttempts} attempts`);
|
||||
const now = new Date();
|
||||
await db
|
||||
.insert(authUsers)
|
||||
@@ -1149,6 +1162,46 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
return { companyId, assignedAgentId, replacementAgentId };
|
||||
}
|
||||
|
||||
it("retries a colliding company fixture prefix and creates a distinct complete fixture", async () => {
|
||||
const existing = await seedCompany();
|
||||
const [original] = await db.select().from(companies).where(eq(companies.id, existing.companyId));
|
||||
const candidates: string[] = [];
|
||||
const nextPrefix = vi.fn(() => {
|
||||
const prefix = candidates.length === 0 ? original!.issuePrefix : freshCompanyPrefix();
|
||||
candidates.push(prefix);
|
||||
return prefix;
|
||||
});
|
||||
const fixture = await seedCompany(nextPrefix);
|
||||
const [allocated] = await db.select().from(companies).where(eq(companies.id, fixture.companyId));
|
||||
expect(nextPrefix.mock.calls.length).toBeGreaterThanOrEqual(2);
|
||||
expect(allocated!.issuePrefix).toBe(candidates.at(-1));
|
||||
expect(allocated!.issuePrefix).toMatch(/^C[A-F0-9]{7}$/);
|
||||
expect(allocated!.issuePrefix).not.toBe(original!.issuePrefix);
|
||||
expect(fixture.companyId).not.toBe(existing.companyId);
|
||||
expect(await db.select().from(companies).where(eq(companies.id, existing.companyId))).toEqual([original]);
|
||||
expect((await db.select().from(agents).where(eq(agents.companyId, fixture.companyId))).map(agent => agent.id).sort())
|
||||
.toEqual([fixture.assignedAgentId, fixture.replacementAgentId].sort());
|
||||
expect(await db.select().from(companyMemberships).where(eq(companyMemberships.companyId, fixture.companyId)))
|
||||
.toEqual([expect.objectContaining({ principalId: "owner-user", membershipRole: "operator" })]);
|
||||
});
|
||||
|
||||
it("bounds company fixture prefix collision retries without creating a partial fixture", async () => {
|
||||
const fixture = await seedCompany();
|
||||
const [company] = await db.select().from(companies).where(eq(companies.id, fixture.companyId));
|
||||
const before = await db.select({ id: companies.id }).from(companies);
|
||||
const nextPrefix = vi.fn(() => company!.issuePrefix);
|
||||
await expect(seedCompany(nextPrefix)).rejects.toThrow(`after ${companyPrefixAttempts} attempts`);
|
||||
expect(nextPrefix).toHaveBeenCalledTimes(companyPrefixAttempts);
|
||||
expect((await db.select({ id: companies.id }).from(companies)).map(row => row.id).sort())
|
||||
.toEqual(before.map(row => row.id).sort());
|
||||
});
|
||||
|
||||
it("does not retry unrelated company fixture database errors", async () => {
|
||||
const nextPrefix = vi.fn(() => "C\0INVALID");
|
||||
await expect(seedCompany(nextPrefix)).rejects.toMatchObject({ cause: { code: "22021" } });
|
||||
expect(nextPrefix).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
// A truthy return is not a durable scheduler receipt. These transport tests
|
||||
// record the same exact receipt identity; real scheduling/coalescing is
|
||||
// separately exercised by durable-chat-wakeup.test.ts against heartbeat.
|
||||
@@ -1572,6 +1625,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
await service.processPendingPublications();
|
||||
const providerRuntime = fakeRuntime.endpoints.get(endpointId);
|
||||
if (providerRuntime) providerRuntime.posts.length = 0;
|
||||
return setupFollowUpMessageId;
|
||||
}
|
||||
|
||||
async function configuredSlackEndpoint(
|
||||
@@ -28187,8 +28241,18 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
}),
|
||||
trigger: "direct_message",
|
||||
});
|
||||
await qualifySetupRoundTrip(service, endpoint.id, userId);
|
||||
const setupMessageId = await qualifySetupRoundTrip(service, endpoint.id, userId);
|
||||
await service.test(endpoint.id, "owner-user");
|
||||
if (provider === "telegram") {
|
||||
// Setup dispatches receipt cleanup asynchronously. Finish it before
|
||||
// measuring reaction removals owned by this fixture's working run.
|
||||
await service.processPendingReceiptReactions();
|
||||
await waitForProcessedReceiptRemoval(endpoint.id, {
|
||||
threadId: thread.thread.id,
|
||||
messageId: setupMessageId,
|
||||
emoji: "eyes",
|
||||
});
|
||||
}
|
||||
const [conversation] = await service.listConversations(endpoint.id);
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({
|
||||
@@ -31689,6 +31753,28 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
});
|
||||
|
||||
it("quarantines Telegram maintenance success when credential-lease ownership is reclaimed", async () => {
|
||||
// Global sweeps may also reconcile a retired fixture's receipt. Keep that
|
||||
// work eligible so the fault hook proves it cannot steal another owner.
|
||||
const retiredFixture = await seedCompany();
|
||||
const retired = await configuredSlackEndpoint(retiredFixture);
|
||||
const retiredThread = makeThread({ channelId: "C-LEASE-SCOPE", id: "slack:C-LEASE-SCOPE:7002.1" });
|
||||
await deliverMessage({
|
||||
callbacks: retired.callbacks,
|
||||
endpointId: retired.endpoint.id,
|
||||
thread: retiredThread.thread,
|
||||
message: makeMessage({ id: "7002.1", text: "@maya check lease isolation", mentioned: true }),
|
||||
trigger: "mention",
|
||||
});
|
||||
await retired.service.shutdown();
|
||||
const [retiredReceipt] = await db.select().from(chatActions).where(and(
|
||||
eq(chatActions.endpointId, retired.endpoint.id),
|
||||
eq(chatActions.kind, "receipt_reaction"),
|
||||
));
|
||||
expect(retiredReceipt).toBeDefined();
|
||||
await db.update(chatActions).set({ status: "received" })
|
||||
.where(eq(chatActions.id, retiredReceipt!.id));
|
||||
await retireFixtureState([retiredFixture.companyId]);
|
||||
|
||||
const fixture = await seedCompany();
|
||||
const botToken = "123456:telegram-lease-reclaim";
|
||||
let reclaimLease = false;
|
||||
@@ -31741,7 +31827,12 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
const { service } = createService(new FakeChatSdkRuntime(), providerFetch, {
|
||||
credentialMutationLeaseRenewalIntervalMs: 5,
|
||||
renewCredentialMutationLease: async (input) => {
|
||||
if (!reclaimLease) return true;
|
||||
// Force the unrelated receipt to reach its ownership check only
|
||||
// after the Telegram fault is armed, independent of query timing.
|
||||
if (blockCommands && input.companyId === retiredFixture.companyId) {
|
||||
await leaseReclaimed;
|
||||
}
|
||||
if (input.companyId !== fixture.companyId || !reclaimLease) return true;
|
||||
const reclaimed = await db
|
||||
.update(chatEndpointLeases)
|
||||
.set({
|
||||
@@ -31799,6 +31890,13 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
|
||||
await service.processPendingDeliveries();
|
||||
|
||||
await expect(db.select({ status: chatActions.status }).from(chatActions)
|
||||
.where(eq(chatActions.id, retiredReceipt!.id)))
|
||||
.resolves.toEqual([{ status: "cancelled" }]);
|
||||
await expect(db.select().from(chatEndpointLeases)
|
||||
.where(eq(chatEndpointLeases.endpointId, retired.endpoint.id)))
|
||||
.resolves.toEqual([]);
|
||||
|
||||
await expect(
|
||||
db
|
||||
.select({ status: chatActions.status, result: chatActions.result })
|
||||
@@ -62486,7 +62584,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
},
|
||||
);
|
||||
|
||||
it("orders reversed GitHub edit and delete callbacks behind their durable root", async () => {
|
||||
it.each([false, true])("orders reversed GitHub edit and delete callbacks behind their durable root (root processed before replay: %s)", async (rootProcessedFirst) => {
|
||||
const fixture = await seedCompany();
|
||||
const deferred: Array<() => void | Promise<void>> = [];
|
||||
const { callbacks, endpoint, service, wakeup, webhookSecret } =
|
||||
@@ -62589,8 +62687,28 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
// One root-conversation drain plus one durable-ingress callback per HTTP
|
||||
// request is queued. The duplicate delivery callback becomes a no-op.
|
||||
expect(deferred).toHaveLength(4);
|
||||
if (rootProcessedFirst) {
|
||||
await deferred.shift()?.();
|
||||
await vi.waitFor(async () => {
|
||||
const [root] = await db
|
||||
.select({ state: chatDeliveries.state })
|
||||
.from(chatDeliveries)
|
||||
.where(eq(chatDeliveries.endpointId, endpoint.id));
|
||||
expect(root.state).toBe("processed");
|
||||
});
|
||||
}
|
||||
await drainDeferred();
|
||||
// The scheduler callbacks start background promises. Wait for durable
|
||||
// replay admission before draining the conversation work it schedules.
|
||||
await vi.waitFor(async () => {
|
||||
const deliveries = await db
|
||||
.select({ id: chatDeliveries.id })
|
||||
.from(chatDeliveries)
|
||||
.where(eq(chatDeliveries.endpointId, endpoint.id));
|
||||
expect(deliveries).toHaveLength(3);
|
||||
});
|
||||
await vi.waitFor(async () => {
|
||||
await drainDeferred();
|
||||
const deliveries = await db
|
||||
.select({
|
||||
eventKind: chatDeliveries.eventKind,
|
||||
|
||||
@@ -12,6 +12,7 @@ const mockUpdateExecutionWorkspace = vi.hoisted(() => vi.fn());
|
||||
const mockLogActivity = vi.hoisted(() => vi.fn());
|
||||
const mockLoggerInfo = vi.hoisted(() => vi.fn());
|
||||
const mockGetEnvironment = vi.hoisted(() => vi.fn());
|
||||
const mockGetLease = vi.hoisted(() => vi.fn());
|
||||
|
||||
vi.mock("../services/environment-execution-target.js", () => ({
|
||||
resolveEnvironmentExecutionTarget: mockResolveEnvironmentExecutionTarget,
|
||||
@@ -30,6 +31,7 @@ vi.mock("../services/environments.js", () => ({
|
||||
environmentService: vi.fn(() => ({
|
||||
ensureLocalEnvironment: vi.fn(),
|
||||
getById: mockGetEnvironment,
|
||||
getLeaseById: mockGetLease,
|
||||
acquireLease: vi.fn(),
|
||||
releaseLease: vi.fn(),
|
||||
updateLeaseMetadata: mockUpdateLeaseMetadata,
|
||||
@@ -815,3 +817,56 @@ describe("admitted native lifecycle recovery", () => {
|
||||
expect(environment.config.reuseLease).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("live remote runner lease reattachment", () => {
|
||||
const input = {
|
||||
companyId: "company-1", selectedEnvironmentId: "env-1", localEnvironmentId: "local",
|
||||
adapterType: "paperclip_runner", admittedLifecycleMode: "per_turn" as const,
|
||||
issueId: "task-1", heartbeatRunId: "run-1", agentId: "agent-1",
|
||||
persistedExecutionWorkspace: { id: "ew-1", mode: "shared_workspace" as const },
|
||||
executionWorkspaceSettings: null,
|
||||
reattachRemoteLease: { leaseId: "lease-1", providerLeaseId: "original-sandbox", remoteCwd: "/remote/workspace" },
|
||||
};
|
||||
const originalLease = () => makeLease({
|
||||
provider: "daytona", providerLeaseId: "original-sandbox", issueId: "task-1", executionWorkspaceId: "ew-1",
|
||||
metadata: { agentId: "agent-1", remoteCwd: "/remote/workspace",
|
||||
sandboxLeaseAcquisition: { outcome: "created", providerLeaseId: "original-sandbox" } },
|
||||
});
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
mockGetEnvironment.mockResolvedValue({ ...makeEnvironment("sandbox"), config: { provider: "daytona", reuseLease: false } });
|
||||
mockGetLease.mockResolvedValue(originalLease());
|
||||
});
|
||||
it("reattaches the original active ephemeral lease without acquiring or resuming a sandbox", async () => {
|
||||
const savedLease = originalLease();
|
||||
mockGetLease.mockResolvedValue(savedLease);
|
||||
const runtime = makeMockRuntime();
|
||||
const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input);
|
||||
expect(mockGetLease).toHaveBeenCalledWith("lease-1");
|
||||
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
|
||||
expect(result.lease).toEqual(savedLease);
|
||||
expect(result.leaseContext.executionWorkspaceId).toBe("ew-1");
|
||||
expect(result.environment.config.reuseLease).toBe(false);
|
||||
});
|
||||
it.each([
|
||||
{ companyId: "foreign" }, { environmentId: "foreign" }, { heartbeatRunId: "another-run" },
|
||||
{ issueId: "another-task" }, { executionWorkspaceId: "another-workspace" },
|
||||
{ providerLeaseId: "replacement" }, { provider: "another-provider" },
|
||||
{ status: "released" }, { expiresAt: new Date(0) }, { releasedAt: new Date() }, { cleanupStatus: "pending" },
|
||||
{ metadata: { agentId: "another-agent", remoteCwd: "/remote/workspace" } },
|
||||
{ metadata: { agentId: "agent-1", remoteCwd: "/other/workspace" } },
|
||||
] as Partial<EnvironmentLease>[])("fails closed before provider acquisition for a mismatched lease %j", async (override) => {
|
||||
mockGetLease.mockResolvedValue({ ...originalLease(), ...override });
|
||||
const runtime = makeMockRuntime();
|
||||
await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input))
|
||||
.rejects.toThrow("native_remote_recovery_lease_mismatch");
|
||||
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
|
||||
});
|
||||
it("never replaces a missing original lease", async () => {
|
||||
mockGetLease.mockResolvedValue(null);
|
||||
const runtime = makeMockRuntime();
|
||||
await expect(environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input))
|
||||
.rejects.toThrow("native_remote_recovery_lease_mismatch");
|
||||
expect(runtime.acquireRunLease).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
@@ -6679,13 +6679,24 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("destroys scoped reusable plugin-backed sandbox leases", async () => {
|
||||
it.each([
|
||||
{ status: "active", scope: "workspace" },
|
||||
{ status: "failed", scope: "workspace" },
|
||||
{ status: "failed", scope: "issue" },
|
||||
] as const)("destroys $status reusable plugin-backed sandbox leases scoped to an $scope", async ({ status, scope }) => {
|
||||
const { pluginId, companyId, runId, executionWorkspaceId, reusableLease } =
|
||||
await seedReusablePluginSandboxLease();
|
||||
await db
|
||||
.update(heartbeatRuns)
|
||||
.set({ status: "succeeded" })
|
||||
.set({ status: status === "failed" ? "failed" : "succeeded" })
|
||||
.where(eq(heartbeatRuns.id, runId));
|
||||
if (status === "failed") await environmentService(db).releaseLease(reusableLease.id, "failed");
|
||||
const issueId = randomUUID();
|
||||
if (scope === "issue") {
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Failed reusable lease cleanup", status: "done", priority: "medium" });
|
||||
await db.update(environmentLeases).set({ issueId }).where(eq(environmentLeases.id, reusableLease.id));
|
||||
}
|
||||
const selection = scope === "issue" ? { issueId } : { executionWorkspaceId };
|
||||
|
||||
const workerManager = {
|
||||
isRunning: vi.fn((id: string) => id === pluginId),
|
||||
@@ -6699,9 +6710,12 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
||||
} as unknown as PluginWorkerManager;
|
||||
const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager });
|
||||
|
||||
// A known issue/workspace id does not authorize another company's teardown.
|
||||
expect(await runtimeWithPlugin.destroyReusableSandboxLeases({ companyId: randomUUID(), ...selection })).toEqual([]);
|
||||
expect(workerManager.call).not.toHaveBeenCalled();
|
||||
const destroyed = await runtimeWithPlugin.destroyReusableSandboxLeases({
|
||||
companyId,
|
||||
executionWorkspaceId,
|
||||
...selection,
|
||||
failureReason: "execution_workspace_closed",
|
||||
});
|
||||
|
||||
@@ -6724,9 +6738,10 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("does not destroy a scoped reusable lease while its run is active", async () => {
|
||||
it.each(["active", "failed"] as const)("does not destroy a %s scoped reusable lease while its run is active", async (status) => {
|
||||
const { pluginId, companyId, executionWorkspaceId, reusableLease } =
|
||||
await seedReusablePluginSandboxLease();
|
||||
if (status === "failed") await environmentService(db).releaseLease(reusableLease.id, "failed");
|
||||
const workerManager = {
|
||||
isRunning: vi.fn((id: string) => id === pluginId),
|
||||
call: vi.fn(async () => undefined),
|
||||
@@ -6752,9 +6767,106 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
||||
expect(workerManager.call).not.toHaveBeenCalled();
|
||||
await expect(
|
||||
environmentService(db).getLeaseById(reusableLease.id),
|
||||
).resolves.toMatchObject({ status: "active" });
|
||||
).resolves.toMatchObject({ status });
|
||||
});
|
||||
|
||||
it.each(["issue", "workspace", "environment"] as const)(
|
||||
"durably claims failed reusable cleanup before %s teardown and recovers a provider failure",
|
||||
async (scope) => {
|
||||
const seeded = await seedReusablePluginSandboxLease();
|
||||
await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, seeded.runId));
|
||||
await environmentService(db).releaseLease(seeded.reusableLease.id, "failed");
|
||||
const issueId = randomUUID();
|
||||
if (scope === "issue") {
|
||||
await db.insert(issues).values({ id: issueId, companyId: seeded.companyId, title: "Cleanup crash window", status: "done" });
|
||||
await db.update(environmentLeases).set({ issueId }).where(eq(environmentLeases.id, seeded.reusableLease.id));
|
||||
}
|
||||
let enterProvider!: () => void;
|
||||
let finishProvider!: () => void;
|
||||
const entered = new Promise<void>((resolve) => { enterProvider = resolve; });
|
||||
const finish = new Promise<void>((resolve) => { finishProvider = resolve; });
|
||||
let unavailable = true;
|
||||
const call = vi.fn(async (_id: string, method: string) => {
|
||||
if (method !== "environmentDestroyLease") throw new Error(`Unexpected method ${method}`);
|
||||
if (unavailable) {
|
||||
enterProvider();
|
||||
await finish;
|
||||
throw new Error("Provider response lost");
|
||||
}
|
||||
return { providerLeaseId: seeded.reusableLease.providerLeaseId, state: "destroyed" };
|
||||
});
|
||||
const makeRuntime = () => environmentRuntimeService(db, { pluginWorkerManager: {
|
||||
isRunning: () => true, call,
|
||||
getWorker: () => ({ supportedMethods: ["environmentDestroyLease"] }),
|
||||
} as unknown as PluginWorkerManager });
|
||||
const runtime = makeRuntime();
|
||||
const close = (service: ReturnType<typeof makeRuntime>) => scope === "environment"
|
||||
? service.destroyReusableSandboxLeasesForEnvironment({ environmentId: seeded.environment.id })
|
||||
: service.destroyReusableSandboxLeases({ companyId: seeded.companyId,
|
||||
...(scope === "issue" ? { issueId } : { executionWorkspaceId: seeded.executionWorkspaceId }) });
|
||||
const closing = close(runtime);
|
||||
try {
|
||||
await entered;
|
||||
// This independently persisted row is what survives controller loss
|
||||
// during the provider call. No provider receipt has arrived yet.
|
||||
expect(await environmentService(db).getLeaseById(seeded.reusableLease.id)).toMatchObject({
|
||||
status: "pending_cleanup", cleanupStatus: "failed", companyId: seeded.companyId,
|
||||
providerLeaseId: seeded.reusableLease.providerLeaseId,
|
||||
metadata: { pendingCleanupAttemptId: expect.any(String), pendingCleanupInFlight: true,
|
||||
pendingCleanupLeaseExpiresAtMs: expect.any(Number) },
|
||||
});
|
||||
await close(makeRuntime());
|
||||
expect(await heartbeatService(db, { environmentRuntime: makeRuntime() }).sweepPendingCleanupLeases({ backoffMs: 0 }))
|
||||
.toEqual({ swept: 0, destroyed: 0, capped: 0 });
|
||||
expect(call).toHaveBeenCalledOnce();
|
||||
} finally {
|
||||
finishProvider();
|
||||
await closing;
|
||||
}
|
||||
expect(await environmentService(db).getLeaseById(seeded.reusableLease.id)).toMatchObject({
|
||||
status: "pending_cleanup", metadata: { pendingCleanupInFlight: false },
|
||||
});
|
||||
unavailable = false;
|
||||
expect(await heartbeatService(db, { environmentRuntime: makeRuntime() }).sweepPendingCleanupLeases({ backoffMs: 0 }))
|
||||
.toEqual({ swept: 1, destroyed: 1, capped: 0 });
|
||||
expect(call).toHaveBeenCalledTimes(2);
|
||||
expect(await environmentService(db).getLeaseById(seeded.reusableLease.id)).toMatchObject({
|
||||
status: "expired", cleanupStatus: "success", metadata: { remoteExecutionTermination: { state: "destroyed" } },
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["queued", "scheduled_retry", "running"] as const)(
|
||||
"fences scoped cleanup when its holding run becomes %s after selection", async (status) => {
|
||||
const seeded = await seedReusablePluginSandboxLease();
|
||||
await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, seeded.runId));
|
||||
await environmentService(db).releaseLease(seeded.reusableLease.id, "failed");
|
||||
const originalService = environmentsModule.environmentService;
|
||||
const serviceSpy = vi.spyOn(environmentsModule, "environmentService").mockImplementation((client) => {
|
||||
const service = originalService(client);
|
||||
return { ...service, getById: async (id: string) => {
|
||||
const environment = await service.getById(id);
|
||||
if (id === seeded.environment.id) {
|
||||
await db.update(heartbeatRuns).set({ status }).where(eq(heartbeatRuns.id, seeded.runId));
|
||||
}
|
||||
return environment;
|
||||
} };
|
||||
});
|
||||
const call = vi.fn(async () => undefined);
|
||||
const runtime = environmentRuntimeService(db, { pluginWorkerManager: {
|
||||
isRunning: () => true, call, getWorker: () => ({ supportedMethods: ["environmentDestroyLease"] }),
|
||||
} as unknown as PluginWorkerManager });
|
||||
try {
|
||||
expect(await runtime.destroyReusableSandboxLeases({ companyId: seeded.companyId,
|
||||
executionWorkspaceId: seeded.executionWorkspaceId })).toEqual([]);
|
||||
expect(call).not.toHaveBeenCalled();
|
||||
expect(await originalService(db).getLeaseById(seeded.reusableLease.id)).toMatchObject({ status: "failed" });
|
||||
} finally {
|
||||
serviceSpy.mockRestore();
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it("destroys reusable plugin-backed sandbox leases scoped to an environment", async () => {
|
||||
const { pluginId, runId, reusableLease } = await seedReusablePluginSandboxLease();
|
||||
// The holding run is finished, so the reservation is stale and destroyable.
|
||||
@@ -6796,9 +6908,51 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps a reusable lease held by an in-flight run out of the environment-scoped destroy", async () => {
|
||||
it("destroys a failed reusable sandbox after successful release before deleting its environment", async () => {
|
||||
const { pluginId, companyId, runId, environment, reusableLease } = await seedReusablePluginSandboxLease();
|
||||
const workerManager = {
|
||||
isRunning: vi.fn((id: string) => id === pluginId),
|
||||
call: vi.fn(async (_pluginId: string, method: string) => {
|
||||
if (method === "environmentReleaseLease") {
|
||||
return { providerLeaseId: reusableLease.providerLeaseId, state: "stopped" };
|
||||
}
|
||||
if (method === "environmentDestroyLease") {
|
||||
// The provider call must already have a durable cleanup reference.
|
||||
expect(await environmentService(db).getLeaseById(reusableLease.id)).toMatchObject({
|
||||
status: "pending_cleanup", environmentId: environment.id,
|
||||
});
|
||||
return { providerLeaseId: reusableLease.providerLeaseId, state: "destroyed" };
|
||||
}
|
||||
throw new Error(`Unexpected plugin method: ${method}`);
|
||||
}),
|
||||
getWorker: vi.fn(() => ({ supportedMethods: ["environmentReleaseLease", "environmentDestroyLease"] })),
|
||||
} as unknown as PluginWorkerManager;
|
||||
const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager });
|
||||
await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, runId));
|
||||
|
||||
const released = await runtimeWithPlugin.releaseRunLeases(runId, "failed", undefined, undefined, true);
|
||||
expect(released[0]?.lease).toMatchObject({
|
||||
id: reusableLease.id, status: "failed", cleanupStatus: "success",
|
||||
metadata: { remoteExecutionTermination: { state: "stopped" } },
|
||||
});
|
||||
const result = await runtimeWithPlugin.destroyReusableSandboxLeasesForEnvironment({
|
||||
environmentId: environment.id, failureReason: "environment_deleted",
|
||||
});
|
||||
expect(result).toEqual({ destroyed: 1, failed: 0, skippedLiveRun: 0 });
|
||||
expect(workerManager.call).toHaveBeenCalledWith(pluginId, "environmentDestroyLease",
|
||||
expect.objectContaining({ companyId, environmentId: environment.id, providerLeaseId: reusableLease.providerLeaseId }),
|
||||
31234);
|
||||
await expect(environmentService(db).getLeaseById(reusableLease.id)).resolves.toMatchObject({
|
||||
status: "expired", cleanupStatus: "success",
|
||||
metadata: { remoteExecutionTermination: { state: "destroyed" } },
|
||||
});
|
||||
expect((await environmentService(db).removeIfDeletable(environment.id))?.id).toBe(environment.id);
|
||||
});
|
||||
|
||||
it.each(["active", "failed"] as const)("keeps a %s reusable lease held by an in-flight run out of the environment-scoped destroy", async (status) => {
|
||||
const { pluginId, reusableLease } = await seedReusablePluginSandboxLease();
|
||||
// seedEnvironment leaves the holding run in `running` status.
|
||||
if (status === "failed") await environmentService(db).releaseLease(reusableLease.id, "failed");
|
||||
|
||||
const workerManager = {
|
||||
isRunning: vi.fn((id: string) => id === pluginId),
|
||||
@@ -6816,7 +6970,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
||||
expect(workerManager.call).not.toHaveBeenCalled();
|
||||
// The lease keeps its reusable status, so the delete guard still blocks.
|
||||
await expect(environmentService(db).getLeaseById(reusableLease.id)).resolves.toMatchObject({
|
||||
status: "active",
|
||||
status,
|
||||
leasePolicy: "reuse_by_environment",
|
||||
});
|
||||
});
|
||||
|
||||
@@ -636,6 +636,18 @@ describeEmbeddedPostgres("environmentService leases", () => {
|
||||
expect(releasedImpact?.reusableSandboxLeaseCount).toBe(1);
|
||||
expect(await svc.removeIfDeletable(environmentId)).toBeNull();
|
||||
|
||||
// A failed run can release its reusable sandbox successfully without
|
||||
// destroying it. The provider handle still needs environment-scoped cleanup.
|
||||
await svc.releaseLease(lease.id, "failed", {
|
||||
failureReason: "adapter_or_run_failure",
|
||||
cleanupStatus: "success",
|
||||
});
|
||||
const failedImpact = await svc.getDeleteBlastRadius(environmentId);
|
||||
expect.soft(failedImpact?.canDelete).toBe(false);
|
||||
expect.soft(failedImpact?.deleteBlockedReasons).toContain("reusable_sandbox_lease");
|
||||
expect.soft(failedImpact?.reusableSandboxLeaseCount).toBe(1);
|
||||
expect(await svc.removeIfDeletable(environmentId)).toBeNull();
|
||||
|
||||
const rows = await db.select().from(environments).where(eq(environments.id, environmentId));
|
||||
expect(rows).toHaveLength(1);
|
||||
const storedLease = await svc.getLeaseById(lease.id);
|
||||
|
||||
@@ -134,50 +134,65 @@ describe("workspace Git scan route load regression", () => {
|
||||
...unavailableMethods(),
|
||||
};
|
||||
const app = createLoadApp(db, companyId, service);
|
||||
const pendingResponses = Array.from({ length: 500 }, (_, index) => request(app)
|
||||
// Model one busy API server, not 500 independent ephemeral listeners whose
|
||||
// accept/close races can reset clients before they reach the scheduler.
|
||||
const server = app.listen(0, "127.0.0.1");
|
||||
await new Promise<void>(resolve => server.once("listening", resolve));
|
||||
const pendingResponses = Array.from({ length: 500 }, (_, index) => request(server)
|
||||
.get(`/api/issues/issue-${index}/file-resources/list`)
|
||||
.set("x-test-actor", `actor-${index % 73}`)
|
||||
.query({ mode: "changed" })
|
||||
.then((response) => response));
|
||||
// Observe every client rejection immediately, including on assertion failure.
|
||||
const settledResponses = Promise.allSettled(pendingResponses);
|
||||
try {
|
||||
await vi.waitFor(
|
||||
() => expect(scheduler.snapshot().totals.singleFlightJoins).toBe(498),
|
||||
{ timeout: 15_000, interval: 20 },
|
||||
);
|
||||
const loadedSnapshot = scheduler.snapshot();
|
||||
expect(loadedSnapshot).toMatchObject({ activeCount: 2, queuedCount: 0, inFlightCount: 2 });
|
||||
|
||||
await vi.waitFor(
|
||||
() => expect(scheduler.snapshot().totals.singleFlightJoins).toBe(498),
|
||||
{ timeout: 15_000, interval: 20 },
|
||||
);
|
||||
const loadedSnapshot = scheduler.snapshot();
|
||||
expect(loadedSnapshot).toMatchObject({ activeCount: 2, queuedCount: 0, inFlightCount: 2 });
|
||||
const healthLatencies: number[] = [];
|
||||
for (let index = 0; index < 25; index += 1) {
|
||||
const startedAt = performance.now();
|
||||
const response = await request(server).get("/api/health");
|
||||
healthLatencies.push(performance.now() - startedAt);
|
||||
expect(response.status).toBe(200);
|
||||
}
|
||||
healthLatencies.sort((left, right) => left - right);
|
||||
const healthP99Ms = healthLatencies[Math.ceil(healthLatencies.length * 0.99) - 1]!;
|
||||
expect(healthP99Ms).toBeLessThan(250);
|
||||
|
||||
const healthLatencies: number[] = [];
|
||||
for (let index = 0; index < 25; index += 1) {
|
||||
const startedAt = performance.now();
|
||||
const response = await request(app).get("/api/health");
|
||||
healthLatencies.push(performance.now() - startedAt);
|
||||
expect(response.status).toBe(200);
|
||||
releaseScans();
|
||||
const responses = (await settledResponses).map(result => {
|
||||
if (result.status === "rejected") throw result.reason;
|
||||
return result.value;
|
||||
});
|
||||
const outcomeCounts = responses.reduce<Record<number, number>>((counts, response) => {
|
||||
counts[response.status] = (counts[response.status] ?? 0) + 1;
|
||||
return counts;
|
||||
}, {});
|
||||
expect(outcomeCounts).toEqual({ 200: 500 });
|
||||
expect({ runnerCalls, peakActive }).toEqual({ runnerCalls: 2, peakActive: 2 });
|
||||
expect(scheduler.snapshot()).toMatchObject({ activeCount: 0, queuedCount: 0, inFlightCount: 0 });
|
||||
|
||||
console.info("workspace Git scan regression metrics", {
|
||||
requests: responses.length,
|
||||
repositories: roots.length,
|
||||
underlyingScans: runnerCalls,
|
||||
peakActive,
|
||||
peakQueued: loadedSnapshot.queuedCount,
|
||||
outcomes: outcomeCounts,
|
||||
healthP99Ms: Math.round(healthP99Ms * 100) / 100,
|
||||
unreapedChildCount: 0,
|
||||
});
|
||||
} finally {
|
||||
releaseScans();
|
||||
server.closeAllConnections();
|
||||
await settledResponses;
|
||||
await new Promise<void>((resolve, reject) => server.close(error => error ? reject(error) : resolve()));
|
||||
}
|
||||
healthLatencies.sort((left, right) => left - right);
|
||||
const healthP99Ms = healthLatencies[Math.ceil(healthLatencies.length * 0.99) - 1]!;
|
||||
expect(healthP99Ms).toBeLessThan(250);
|
||||
|
||||
releaseScans();
|
||||
const responses = await Promise.all(pendingResponses);
|
||||
const outcomeCounts = responses.reduce<Record<number, number>>((counts, response) => {
|
||||
counts[response.status] = (counts[response.status] ?? 0) + 1;
|
||||
return counts;
|
||||
}, {});
|
||||
expect(outcomeCounts).toEqual({ 200: 500 });
|
||||
expect({ runnerCalls, peakActive }).toEqual({ runnerCalls: 2, peakActive: 2 });
|
||||
expect(scheduler.snapshot()).toMatchObject({ activeCount: 0, queuedCount: 0, inFlightCount: 0 });
|
||||
|
||||
console.info("workspace Git scan regression metrics", {
|
||||
requests: responses.length,
|
||||
repositories: roots.length,
|
||||
underlyingScans: runnerCalls,
|
||||
peakActive,
|
||||
peakQueued: loadedSnapshot.queuedCount,
|
||||
outcomes: outcomeCounts,
|
||||
healthP99Ms: Math.round(healthP99Ms * 100) / 100,
|
||||
unreapedChildCount: 0,
|
||||
});
|
||||
}, 60_000);
|
||||
|
||||
it.each([
|
||||
|
||||
@@ -41,6 +41,13 @@ describe("native heartbeat cancellation authority", () => {
|
||||
);
|
||||
});
|
||||
|
||||
it("passes the caller identity into the audited native boundary", async () => {
|
||||
const cancel = vi.fn(async () => ({ decision: {}, auditId: "audit" }));
|
||||
const db = {} as Db, cancellationRequestId = "11111111-1111-4111-8111-111111111111";
|
||||
await cancelHeartbeatNativeRun({ db, runId: "run", reason: "Stop", runtimeMode: "native", cancellationRequestId, cancel });
|
||||
expect(cancel).toHaveBeenCalledWith("run", "Stop", { db, scope: "run", cancellationRequestId });
|
||||
});
|
||||
|
||||
it("fails closed when a native cancellation lacks its decision audit", async () => {
|
||||
const cancel = vi.fn(async () => ({
|
||||
decision: null,
|
||||
|
||||
@@ -1,3 +1,7 @@
|
||||
import { ensureNativeCompletionContract } from "../services/native-runtime/completion-contracts.js";
|
||||
import { readNativePlanWait, hasCommittedNativePlanWait } from "../services/native-runtime/native-plan-wait.js";
|
||||
import { nativeSha256 } from "../services/native-runtime/canonical.js";
|
||||
import { buildQuestionResponseDeliveryEnvelope } from "../services/question-response-delivery.js";
|
||||
import * as aiConnectionRuntime from "../services/ai-connection-runtime.js";
|
||||
import { unprocessable } from "../errors.js";
|
||||
import * as executionContinuation from "../services/execution-continuation.js";
|
||||
@@ -23,6 +27,7 @@ import {
|
||||
afterAll,
|
||||
afterEach,
|
||||
beforeAll,
|
||||
beforeEach,
|
||||
describe,
|
||||
expect,
|
||||
it,
|
||||
@@ -66,6 +71,7 @@ import {
|
||||
heartbeatRunEvents,
|
||||
heartbeatRuns,
|
||||
issueComments,
|
||||
issueQuestionResponseDeliveries,
|
||||
issueApprovals,
|
||||
issueDocuments,
|
||||
issuePlanDecompositions,
|
||||
@@ -16451,4 +16457,237 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
|
||||
.where(eq(heartbeatRuns.id, runId));
|
||||
expect(runs).toHaveLength(1);
|
||||
});
|
||||
describe("accepted Cursor planning boundaries", () => {
|
||||
const nativeImplementation = mockExecutePaperclipNativeSession.getMockImplementation();
|
||||
beforeEach(() => {
|
||||
mockExecutePaperclipNativeSession.mockImplementation(async () => { throw new Error("Accepted-plan tests must never execute a provider"); });
|
||||
});
|
||||
afterEach(() => { mockExecutePaperclipNativeSession.mockImplementation(nativeImplementation!); });
|
||||
|
||||
async function seedAcceptedCursorPlanWait(semanticFinish = false, sourceSequenceOffset = 0) {
|
||||
const f = await seedStrandedIssueFixture({ status: "in_progress", runStatus: "succeeded", livenessState: "advanced" });
|
||||
const instance = randomUUID(), interactionId = randomUUID();
|
||||
const contractInput = { db, companyId: f.companyId,
|
||||
issue: { id: f.issueId, title: "Review the plan before further work", description: "Explicit completion required" }, actorId: "test" };
|
||||
const { row: persistedContract, contract } = await ensureNativeCompletionContract(contractInput);
|
||||
const reused = await ensureNativeCompletionContract(contractInput);
|
||||
expect(reused.row.id).toBe(persistedContract.id);
|
||||
expect(reused.contract).toEqual(contract);
|
||||
const contractId = persistedContract.id, contractSha = persistedContract.canonicalSha256;
|
||||
// Production binds policy/schema as well as the body; a body-only hash is not a valid contract receipt.
|
||||
expect(contractSha).not.toBe(nativeSha256(contract));
|
||||
const model = "gpt-5.6-luna[context=272k,reasoning=medium,fast=false]";
|
||||
await db.update(agents).set({ adapterType: "paperclip_runner", adapterConfig: { provider: "acpx", acpxAgent: "cursor", model, acpxSessionMode: "plan", acpxPermissionMode: "approve-all" } }).where(eq(agents.id, f.agentId));
|
||||
await db.update(agentWakeupRequests).set({ status: "completed" }).where(eq(agentWakeupRequests.id, f.wakeupRequestId));
|
||||
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: f.issueId, nativeSessionId: f.runId,
|
||||
completionContractId: contractId, completionContractSha256: contractSha, runnerInstanceId: instance,
|
||||
runnerProfileJson: { nativeExecutionInput: { binding: { companyId: f.companyId, issueId: f.issueId, runId: f.runId, agentId: f.agentId }, provider: { kind: "acpx", agent: "cursor", model, mode: "plan", permissionMode: "approve-all",
|
||||
profile: paperclipRunner.resolveQualifiedAcpxProfile("cursor", model) }, session: { normalizedSessionId: f.runId },
|
||||
completionContract: { id: contractId, sha256: contractSha, contract } } } }).where(eq(heartbeatRuns.id, f.runId));
|
||||
const planId = `plan-${"a".repeat(64)}`;
|
||||
const questionSet = { schema: "paperclip.question_set.v1", title: "Native plan", description: "Exact accepted revision", questions: [{ id: planId, prompt: "Proceed?", required: true,
|
||||
answerMode: "single_select", options: [{ id: "accept", label: "Accept" }, { id: "reject", label: "Reject" }, { id: "cancel", label: "Cancel" }] }] };
|
||||
const [interaction] = await db.insert(issueThreadInteractions).values({ id: interactionId, companyId: f.companyId, issueId: f.issueId, sourceRunId: f.runId,
|
||||
createdByAgentId: f.agentId, kind: "ask_user_questions", status: "answered", continuationPolicy: "none",
|
||||
idempotencyKey: `paperclip-runner-question:${f.runId}:request`, resolvedByUserId: "responsible-user", resolvedAt: new Date(Date.now() - 1000),
|
||||
payload: { version: 1, questions: [{ id: planId, prompt: "Proceed?", selectionMode: "single", required: true, allowOther: false, options: [{ id: "accept", label: "Accept" }, { id: "reject", label: "Reject" }, { id: "cancel", label: "Cancel" }] }], runtimeRequestId: "request", questionSet: questionSet as never }, result: { version: 1, answers: [{ questionId: planId, optionIds: ["accept"] }] } }).returning();
|
||||
const answer = buildQuestionResponseDeliveryEnvelope(interaction as never);
|
||||
const [delivery] = await db.insert(issueQuestionResponseDeliveries).values({ companyId: f.companyId, issueId: f.issueId, interactionId,
|
||||
sourceRunId: f.runId, targetRunId: f.runId, correlationId: randomUUID(), status: "delivered", deliveryMode: "steered", acknowledgedAt: new Date(), payloadSha256: nativeSha256(answer) }).returning();
|
||||
const port = new PaperclipControlPlanePort(db, { companyId: f.companyId, issueId: f.issueId, runId: f.runId, agentId: f.agentId,
|
||||
sessionId: f.runId, completionContractId: contractId, completionContractSha256: contractSha, sourceInstanceId: instance, controlPlaneSourceInstanceId: `control-${f.runId}` });
|
||||
const events = [
|
||||
{ eventType: "runtime_request.created", payload: { request: { schema: "paperclip.runtime_request.v2", requestKind: "runtime", requestId: "request", type: "input", status: "pending", turnId: "turn", itemId: "native-plan-tool", prompt: "Review plan",
|
||||
origin: { adapter: "acpx-runtime-sidecar", provider: "cursor", method: "cursor/create_plan" }, input: questionSet } } },
|
||||
{ eventType: "tool.execution.started", payload: { schema: "paperclip.tool.execution.v1", executionId: "native-plan-tool", transport: "builtin", operation: "execute", status: "running", readOnly: false, namespace: null, name: "Create Plan", target: null, inputUpdated: true, output: null, outputBytes: 0, outputTruncated: false, outputDigest: null, progress: null, exitCode: null, durationMs: null } },
|
||||
{ eventType: "runtime_request.resolved", payload: { requestId: "request", turnId: "turn", action: "submit", response: answer.response } },
|
||||
{ eventType: "tool.execution.completed", payload: { schema: "paperclip.tool.execution.v1", executionId: "native-plan-tool", transport: "builtin", operation: "execute", status: "completed", readOnly: false, namespace: null, name: "Create Plan", target: null, inputUpdated: false, output: null, outputBytes: 0, outputTruncated: false, outputDigest: null, progress: null, exitCode: null, durationMs: null } },
|
||||
{ eventType: "turn.completed", payload: { status: "completed", error: null } },
|
||||
];
|
||||
for (const [index, event] of events.entries()) await port.appendEvent({ schema: "paperclip.prp.event.v1", sourceEventId: `${instance}:${sourceSequenceOffset + index + 1}`, sourceSeq: sourceSequenceOffset + index + 1,
|
||||
sourceInstanceId: instance, sourceKind: "runner", runId: f.runId, normalizedSessionId: f.runId, turnId: "turn", schemaVersion: 1, priority: 0,
|
||||
emittedAt: new Date().toISOString(), ...event } as paperclipRunner.PrpEvent);
|
||||
const proof = await readNativePlanWait(db, f);
|
||||
expect(proof).not.toBeNull();
|
||||
const result = semanticFinish ? { ...proof!.result, reportedWorkDisposition: "done" as const, summary: "Explicit semantic finish wins", continuation: undefined,
|
||||
completionClaim: { contractRevision: "1", objectiveSatisfied: true, criteria: [{ criterionId: "objective", status: "satisfied" as const, evidenceRefs: [] }], remainingWork: [] } } : proof!.result;
|
||||
await port.completeRun({ result, turnId: "turn", terminal: { schema: "paperclip.prp.terminal.v1", runTerminalState: "succeeded", turnTerminalState: "completed", reportedWorkDisposition: result.reportedWorkDisposition } });
|
||||
return { ...f, interactionId, deliveryId: delivery!.id, proof: proof! };
|
||||
}
|
||||
|
||||
it("keeps an accepted Cursor plan passive across restart, future configuration changes, and paused recovery", async () => {
|
||||
const f = await seedAcceptedCursorPlanWait();
|
||||
await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
|
||||
await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
|
||||
expect(await hasCommittedNativePlanWait(db, f)).toBe(true);
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.runId, f.runId))).toEqual([expect.objectContaining({ reasonCode: "native_plan_accepted_waiting_for_continuation", toStatus: "in_progress" })]);
|
||||
expect(await db.select().from(statusDecisionEffects).where(eq(statusDecisionEffects.issueId, f.issueId))).toEqual([expect.objectContaining({ effectKind: "issue_status_projection", targetType: "issue", deliveryState: "delivered" })]);
|
||||
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, f.runId))).toEqual([expect.objectContaining({ body: expect.stringContaining("next message") })]);
|
||||
const current = paperclipRunner.resolveQualifiedAcpxProfile("cursor", "gpt-5.6-luna[context=272k,reasoning=medium,fast=false]");
|
||||
const resolver = vi.spyOn(paperclipRunner, "resolveQualifiedAcpxProfile").mockReturnValue({ ...current,
|
||||
agentProfileVersion: current.agentProfileVersion + 1, commandDigest: `sha256:${"b".repeat(64)}` });
|
||||
try {
|
||||
expect(await readNativePlanWait(db, f)).toBeNull(); // Old profile cannot create a new wait.
|
||||
for (const status of ["idle", "paused"] as const) {
|
||||
await db.update(agents).set({ status, adapterConfig: { provider: "acpx", acpxAgent: "cursor",
|
||||
model: "future-model", acpxSessionMode: "agent", acpxPermissionMode: "approve-reads" } }).where(eq(agents.id, f.agentId));
|
||||
expect(await hasCommittedNativePlanWait(db, f)).toBe(true);
|
||||
const recovered = await heartbeatService(db).reconcileStrandedAssignedIssues();
|
||||
expect(recovered.continuationRequeued).toBe(0); expect(recovered.escalated).toBe(0);
|
||||
}
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
||||
const changed = structuredClone(run!.runnerProfileJson!);
|
||||
(changed.nativeExecutionInput as { provider: { profile: { commandDigest: string } } }).provider.profile.commandDigest = "tampered-original-profile";
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: changed }).where(eq(heartbeatRuns.id, f.runId));
|
||||
expect(await hasCommittedNativePlanWait(db, f)).toBe(false);
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: run!.runnerProfileJson }).where(eq(heartbeatRuns.id, f.runId));
|
||||
expect(await hasCommittedNativePlanWait(db, f)).toBe(true);
|
||||
const [tool] = await db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, f.runId), eq(heartbeatRunEvents.eventType, "tool.execution.completed")));
|
||||
const changedTool = structuredClone(tool!.payload!) as { prpEvent: { payload: { name: string } } };
|
||||
changedTool.prpEvent.payload.name = "mutated committed tool evidence";
|
||||
await db.update(heartbeatRunEvents).set({ payload: changedTool, sourcePayloadSha256: nativeSha256(changedTool.prpEvent) }).where(eq(heartbeatRunEvents.id, tool!.id));
|
||||
expect(await hasCommittedNativePlanWait(db, f)).toBe(false);
|
||||
await db.update(heartbeatRunEvents).set({ payload: tool!.payload, sourcePayloadSha256: tool!.sourcePayloadSha256 }).where(eq(heartbeatRunEvents.id, tool!.id));
|
||||
expect(await hasCommittedNativePlanWait(db, f)).toBe(true);
|
||||
} finally { resolver.mockRestore(); }
|
||||
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(1);
|
||||
expect(mockAdapterExecute).not.toHaveBeenCalled(); expect(mockExecutePaperclipNativeSession).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("retains an exact historical Cursor6 committed wait across more than 1000 ignored progress rows without allowing new unbound admission", async () => {
|
||||
const progressCount = 1002;
|
||||
const f = await seedAcceptedCursorPlanWait(false, progressCount);
|
||||
await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
||||
const profile = structuredClone(run!.runnerProfileJson!) as any;
|
||||
Object.assign(profile.nativeExecutionInput.provider.profile, { agentProfileVersion: 6, commandDigest: "sha256:377dcea64a727ce799cc112458d4b40ba4bc6574cd6c6f7233b6efd5917a6c4b" });
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: profile }).where(eq(heartbeatRuns.id, f.runId));
|
||||
await db.delete(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, f.runId), inArray(heartbeatRunEvents.eventType, ["tool.execution.started", "tool.execution.completed"])));
|
||||
// Reconstruct the persisted Cursor6 receipt format using its unchanged
|
||||
// source facts. This historical format had no tool lifecycle binding.
|
||||
const rows = await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.runId));
|
||||
const event = (kind: string) => (rows.find(r => r.eventType === kind)!.payload as any).prpEvent;
|
||||
const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.id, run!.completionContractId!));
|
||||
const [interaction] = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, f.interactionId));
|
||||
const [delivery] = await db.select().from(issueQuestionResponseDeliveries).where(eq(issueQuestionResponseDeliveries.id, f.deliveryId));
|
||||
profile.nativeExecutionInput.provider.cursorMode = profile.nativeExecutionInput.provider.mode;
|
||||
delete profile.nativeExecutionInput.provider.mode;
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: profile }).where(eq(heartbeatRuns.id, f.runId));
|
||||
const legacy = { ...f.proof.source, schema: "paperclip.native_cursor_plan_wait.v1" }; delete legacy.toolExecutionId; delete legacy.toolLifecycleSha256;
|
||||
legacy.authoritySha256 = nativeSha256({ admission: profile.nativeExecutionInput, contract, created: event("runtime_request.created"), resolved: event("runtime_request.resolved"), terminal: event("turn.completed"), interaction, delivery, resolvedAt: interaction!.resolvedAt!.toISOString(), acknowledgedAt: delivery!.acknowledgedAt!.toISOString() });
|
||||
const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, f.runId));
|
||||
const [savedResult] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId));
|
||||
const resultJson = structuredClone(savedResult!.resultJson!) as any;
|
||||
resultJson.result.continuation.idempotencyKey = resultJson.result.continuation.idempotencyKey.replace("native-plan-wait:", "cursor-plan-wait:");
|
||||
const canonicalSha256 = nativeSha256(resultJson);
|
||||
await db.update(nativeRunResults).set({ resultJson, canonicalSha256 }).where(eq(nativeRunResults.id, savedResult!.id));
|
||||
const decisionJson = { ...decision!.decisionJson } as any;
|
||||
delete decisionJson.planWait; delete decisionJson.planWaitResult;
|
||||
decisionJson.cursorPlanWait = legacy;
|
||||
decisionJson.cursorPlanWaitResult = { resultId: savedResult!.id, resultSha256: canonicalSha256 };
|
||||
await db.update(statusDecisions).set({ decisionJson }).where(eq(statusDecisions.id, decision!.id));
|
||||
expect(await readNativePlanWait(db, f)).toBeNull();
|
||||
expect(await hasCommittedNativePlanWait(db, f)).toBe(true);
|
||||
// Fill the reserved source prefix with genuine durable progress rows. Shift
|
||||
// only DB ordering keys; the original PRP facts and committed receipt stay
|
||||
// byte-identical. Cursor6 never queried these rows, even above its budget.
|
||||
await db.update(heartbeatRunEvents).set({ seq: sql`${heartbeatRunEvents.seq} + ${progressCount}` }).where(eq(heartbeatRunEvents.runId, f.runId));
|
||||
const progress = Array.from({ length: progressCount }, (_, index) => {
|
||||
const seq = index + 1;
|
||||
const prpEvent = { ...event("runtime_request.created"), sourceEventId: `${run!.runnerInstanceId}:${seq}`, sourceSeq: seq, eventType: "tool.execution.progressed",
|
||||
payload: { schema: "paperclip.tool.execution.v1", executionId: "earlier-tool", transport: "builtin", operation: "read", status: "running" } };
|
||||
return { companyId: f.companyId, runId: f.runId, agentId: f.agentId, seq, eventType: prpEvent.eventType, payload: { prpEvent }, sourceInstanceId: run!.runnerInstanceId,
|
||||
sourceEventId: prpEvent.sourceEventId, sourceSeq: seq, sourcePayloadSha256: nativeSha256(prpEvent), protocolSchemaVersion: 1 };
|
||||
});
|
||||
await db.insert(heartbeatRunEvents).values(progress);
|
||||
expect(await hasCommittedNativePlanWait(db, f)).toBe(true);
|
||||
const [unchangedDecision] = await db.select().from(statusDecisions).where(eq(statusDecisions.id, decision!.id));
|
||||
expect(unchangedDecision!.decisionJson!.cursorPlanWait).toEqual(legacy);
|
||||
// Completed rows belonged to the old query. A later completion must still
|
||||
// invalidate the normal-terminal boundary rather than being filtered away.
|
||||
const completion = { ...event("turn.completed"), sourceEventId: `${run!.runnerInstanceId}:99999`, sourceSeq: 99999, eventType: "tool.execution.completed", payload: { ...progress[0]!.payload.prpEvent.payload, status: "completed" } };
|
||||
const [extra] = await db.insert(heartbeatRunEvents).values({ ...progress[0]!, seq: 99999, eventType: completion.eventType, payload: { prpEvent: completion }, sourceEventId: completion.sourceEventId, sourceSeq: 99999, sourcePayloadSha256: nativeSha256(completion) }).returning();
|
||||
expect(await hasCommittedNativePlanWait(db, f)).toBe(false);
|
||||
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.id, extra!.id));
|
||||
expect(await hasCommittedNativePlanWait(db, f)).toBe(true);
|
||||
profile.nativeExecutionInput.provider.profile.commandDigest = "tampered-history";
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: profile }).where(eq(heartbeatRuns.id, f.runId));
|
||||
expect(await hasCommittedNativePlanWait(db, f)).toBe(false);
|
||||
});
|
||||
|
||||
it("admits a later ordinary user message after an accepted Cursor plan without changing Plan mode or replaying its run", async () => {
|
||||
const f = await seedAcceptedCursorPlanWait();
|
||||
await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
|
||||
// Occupy the single dispatch slot: this checks actual user-wake admission
|
||||
// without executing any provider or fabricating a second semantic result.
|
||||
const occupied = randomUUID();
|
||||
await db.update(agents).set({ runtimeConfig: { heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 1 } } }).where(eq(agents.id, f.agentId));
|
||||
await db.insert(heartbeatRuns).values({ id: occupied, companyId: f.companyId, agentId: f.agentId, status: "running", startedAt: new Date(), contextSnapshot: {} });
|
||||
const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId, authorType: "user", authorUserId: "responsible-user", body: "Continue reviewing the accepted plan in Plan mode." }).returning();
|
||||
try {
|
||||
const heartbeat = heartbeatService(db);
|
||||
const next = await heartbeat.wakeup(f.agentId, { source: "automation", triggerDetail: "system", reason: "issue_commented", requestedByActorType: "user", requestedByActorId: "responsible-user",
|
||||
payload: { issueId: f.issueId, commentId: comment!.id }, contextSnapshot: { issueId: f.issueId, taskId: f.issueId, wakeCommentId: comment!.id, wakeCommentIds: [comment!.id] } });
|
||||
expect(next).not.toBeNull(); expect(next!.id).not.toBe(f.runId);
|
||||
const [admitted] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, next!.id));
|
||||
expect(admitted).toMatchObject({ status: "queued", retryOfRunId: null });
|
||||
expect(admitted!.contextSnapshot?.wakeCommentIds).toContain(comment!.id);
|
||||
const [agent] = await db.select().from(agents).where(eq(agents.id, f.agentId));
|
||||
expect(agent!.adapterConfig.acpxSessionMode).toBe("plan");
|
||||
expect(await hasCommittedNativePlanWait(db, f)).toBe(false);
|
||||
expect(mockExecutePaperclipNativeSession).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(and(eq(heartbeatRuns.companyId, f.companyId), inArray(heartbeatRuns.status, ["running", "queued"])));
|
||||
await db.update(agentWakeupRequests).set({ status: "cancelled", finishedAt: new Date() }).where(and(eq(agentWakeupRequests.companyId, f.companyId), inArray(agentWakeupRequests.status, ["queued", "claimed"])));
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["delivery", "assignment", "admission", "user_request", "result_or_decision"] as const)("revokes an accepted Cursor plan passive wait after %s changes", async change => {
|
||||
const f = await seedAcceptedCursorPlanWait();
|
||||
await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
|
||||
if (change === "delivery") await db.update(issueQuestionResponseDeliveries).set({ acknowledgedAt: null }).where(eq(issueQuestionResponseDeliveries.id, f.deliveryId));
|
||||
if (change === "assignment") await db.update(issues).set({ assigneeAgentId: null }).where(eq(issues.id, f.issueId));
|
||||
if (change === "admission") {
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
||||
const profile = structuredClone(run!.runnerProfileJson!);
|
||||
(profile.nativeExecutionInput as { provider: { cursorMode: string } }).provider.cursorMode = "agent";
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: profile }).where(eq(heartbeatRuns.id, f.runId));
|
||||
}
|
||||
if (change === "user_request") await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId, authorType: "user", authorUserId: "responsible-user", body: "Continue reviewing this plan in Plan mode." });
|
||||
if (change === "result_or_decision") {
|
||||
const [accepted] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId));
|
||||
await db.update(nativeRunResults).set({ canonicalSha256: "changed" }).where(eq(nativeRunResults.id, accepted!.id));
|
||||
expect(await hasCommittedNativePlanWait(db, f)).toBe(false);
|
||||
await db.update(nativeRunResults).set({ canonicalSha256: accepted!.canonicalSha256 }).where(eq(nativeRunResults.id, accepted!.id));
|
||||
expect(await hasCommittedNativePlanWait(db, f)).toBe(true);
|
||||
await db.update(issues).set({ lastStatusDecisionId: null }).where(eq(issues.id, f.issueId));
|
||||
}
|
||||
expect(await hasCommittedNativePlanWait(db, f)).toBe(false);
|
||||
});
|
||||
|
||||
it("rechecks accepted Cursor plan delivery under the status transaction before presentation or effects", async () => {
|
||||
const f = await seedAcceptedCursorPlanWait();
|
||||
await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", failpoint: "status_projection" });
|
||||
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
||||
const [issue] = await db.select().from(issues).where(eq(issues.id, f.issueId));
|
||||
await db.update(issueQuestionResponseDeliveries).set({ payloadSha256: "changed" }).where(eq(issueQuestionResponseDeliveries.id, f.deliveryId));
|
||||
await expect(commitNativeStatusDecision({ db, companyId: f.companyId, issueId: f.issueId, runId: f.runId, assessmentId: coordinator!.assessmentId!,
|
||||
priorStatus: issue!.status, priorStatusVersion: issue!.statusVersion, priorDecisionId: issue!.lastStatusDecisionId,
|
||||
decision: { policyVersion: "phase6-v7", statusAction: "in_progress", toStatus: "in_progress", reasonCode: "native_plan_accepted_waiting_for_continuation", unblockDescriptor: null, effects: [] },
|
||||
requirePlanWaitSource: f.proof.source })).rejects.toBeInstanceOf(NativeStatusRaceError);
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.runId, f.runId))).toHaveLength(0);
|
||||
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, f.runId))).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("preserves explicit semantic finish priority over accepted Cursor plan evidence", async () => {
|
||||
const f = await seedAcceptedCursorPlanWait(true);
|
||||
await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
|
||||
expect(await hasCommittedNativePlanWait(db, f)).toBe(false);
|
||||
const decisions = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, f.runId));
|
||||
expect(decisions).toHaveLength(1); expect(decisions[0]!.reasonCode).not.toBe("native_plan_accepted_waiting_for_continuation");
|
||||
});
|
||||
|
||||
});
|
||||
|
||||
|
||||
});
|
||||
@@ -1,6 +1,6 @@
|
||||
import express from "express";
|
||||
import request from "supertest";
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
function createSelectChain(rows: unknown[]) {
|
||||
const query = {
|
||||
@@ -44,6 +44,13 @@ function createInvite(overrides: Record<string, unknown> = {}) {
|
||||
};
|
||||
}
|
||||
|
||||
function loadAppModules() {
|
||||
return Promise.all([
|
||||
import("../routes/access.js"),
|
||||
import("../middleware/index.js"),
|
||||
]);
|
||||
}
|
||||
|
||||
async function createApp(
|
||||
db: Record<string, unknown>,
|
||||
network: {
|
||||
@@ -51,10 +58,7 @@ async function createApp(
|
||||
requestHead: ReturnType<typeof vi.fn>;
|
||||
},
|
||||
) {
|
||||
const [access, middleware] = await Promise.all([
|
||||
import("../routes/access.js"),
|
||||
import("../middleware/index.js"),
|
||||
]);
|
||||
const [access, middleware] = await loadAppModules();
|
||||
const app = express();
|
||||
app.use((req, _res, next) => {
|
||||
(req as any).actor = { type: "anon" };
|
||||
@@ -74,7 +78,12 @@ async function createApp(
|
||||
return app;
|
||||
}
|
||||
|
||||
describe("GET /invites/:token/test-resolution", () => {
|
||||
describe("GET /invites/:token/test-resolution", { sequential: true }, () => {
|
||||
beforeAll(async () => {
|
||||
// Route transformation is fixture setup, not part of the network assertions.
|
||||
await loadAppModules();
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
});
|
||||
|
||||
@@ -173,11 +173,15 @@ type TestActor =
|
||||
runId: string | null;
|
||||
};
|
||||
|
||||
async function createApp(actor?: TestActor) {
|
||||
const [{ errorHandler }, { issueRoutes }] = await Promise.all([
|
||||
function loadAppModules() {
|
||||
return Promise.all([
|
||||
import("../middleware/index.js"),
|
||||
import("../routes/issues.js"),
|
||||
]);
|
||||
}
|
||||
|
||||
async function createApp(actor?: TestActor) {
|
||||
const [{ errorHandler }, { issueRoutes }] = await loadAppModules();
|
||||
const app = express();
|
||||
app.use(express.json());
|
||||
app.use((req, _res, next) => {
|
||||
@@ -196,7 +200,7 @@ async function createApp(actor?: TestActor) {
|
||||
}
|
||||
|
||||
describe("issue execution policy routes", () => {
|
||||
beforeEach(() => {
|
||||
beforeEach(async () => {
|
||||
vi.resetModules();
|
||||
vi.doUnmock("../services/index.js");
|
||||
vi.doUnmock("../routes/issues.js");
|
||||
@@ -263,6 +267,9 @@ describe("issue execution policy routes", () => {
|
||||
};
|
||||
});
|
||||
mockAccessService.hasPermission.mockResolvedValue(false);
|
||||
// Finish cold imports before a test configures its request-specific mocks.
|
||||
// A timed-out import must not resume a request against the next test's mocks.
|
||||
await loadAppModules();
|
||||
});
|
||||
|
||||
it("reauthorizes a terminal verdict against the review policy held under the update lock", async () => {
|
||||
|
||||
@@ -23,21 +23,66 @@ import {
|
||||
sanitizeRecord,
|
||||
} from "../redaction.js";
|
||||
|
||||
import { createCopilotToolEvidence } from "../../../packages/paperclip-runner/src/drivers/acpx/copilot-tool-evidence.js";
|
||||
import { appendSemanticToolReceipt } from "../../../packages/paperclip-runner/src/drivers/semantic-tool-receipt.js";
|
||||
|
||||
function receiptNotice(version: 1 | 2 = 1): Record<string, any> {
|
||||
return {
|
||||
schema: "paperclip.provider.notice.v1", noticeId: `copilot-evidence-${"a".repeat(24)}-1`,
|
||||
severity: "info", category: `paperclip_semantic_tool_receipt_v${version}`, scope: "turn",
|
||||
recoverable: true, userActionable: false, summary: "Paperclip returned a semantic tool result.",
|
||||
provenance: { method: "paperclip/semantic_tool_result", eventType: "semantic_result", sessionId: "session", turnId: "turn" },
|
||||
details: Object.entries({ stage: "semantic_result", schema: `paperclip.semantic_tool_receipt.v${version}`,
|
||||
operationId: "finish_task", callIdentitySha256: "a".repeat(64), inputSha256: "b".repeat(64),
|
||||
resultSha256: "c".repeat(64), outcome: "returned", ...(version === 2 ? { normalizedInputSha256: "d".repeat(64) } : {}),
|
||||
}).map(([name, value]) => ({ name, value })),
|
||||
};
|
||||
}
|
||||
const receiptSchemaValue = (notice: Record<string, any>) => notice.details.find((detail: any) => detail.name === "schema").value;
|
||||
|
||||
describe("redaction", () => {
|
||||
it("preserves credential-related prose, metadata, and dotted filenames", () => {
|
||||
const input = {
|
||||
body: "Keep the private key in a secret manager. Document credential handling and token permissions. Use bearer tokens for authentication. Prefer bearer authentication.",
|
||||
tokenPolicy: "least privilege",
|
||||
secretStorage: "vault",
|
||||
credentialHandling: "harness",
|
||||
authorizationRequired: true,
|
||||
path: "deployment.credentials.example.md",
|
||||
};
|
||||
expect(sanitizeRecord(input)).toEqual(input);
|
||||
expect(redactSensitiveText(input.body)).toBe(input.body);
|
||||
expect(sanitizeRecord({ credentials: { provider: "opaque-value" }, passwordValue: "opaque-value", authorization_code: "opaque-value" }))
|
||||
.toEqual({ credentials: REDACTED_EVENT_VALUE, passwordValue: REDACTED_EVENT_VALUE, authorization_code: REDACTED_EVENT_VALUE });
|
||||
expect(redactSensitiveText("--token-budget 4000 SECRET_STORAGE=vault")).toBe("--token-budget 4000 SECRET_STORAGE=vault");
|
||||
it("preserves the actual Copilot receipt producer discriminator through nested durable redaction", () => {
|
||||
const events: Array<Record<string, any>> = [];
|
||||
const projector = createCopilotToolEvidence({ sessionId: "session", turnId: "turn", workingDirectory: "/workspace",
|
||||
active: () => true, emit: event => events.push(event) });
|
||||
const receipt = appendSemanticToolReceipt({ tool: "finish_task", callId: "call", arguments: {} },
|
||||
{ content: [{ type: "text", text: "accepted" }] }).receipt;
|
||||
expect(receipt.schema).toBe("paperclip.semantic_tool_receipt.v2");
|
||||
projector.captureSemanticReceipt()!(receipt);
|
||||
expect(events).toHaveLength(1);
|
||||
expect(events[0]!.payload.category).toBe("paperclip_semantic_tool_receipt_v2");
|
||||
expect(events[0]!.payload.details).toHaveLength(8);
|
||||
const input = { prpEvent: { schema: "paperclip.prp.event.v1", schemaVersion: 1,
|
||||
eventType: "provider.notice.recorded", payload: events[0]!.payload } };
|
||||
expect(redactEventPayload(input)).toEqual(input);
|
||||
expect(redactEventPayload(redactEventPayload(input))).toEqual(input);
|
||||
});
|
||||
|
||||
it.each([1, 2] as const)("preserves public v%s receipt identifiers as text", version => {
|
||||
const notice = receiptNotice(version);
|
||||
expect(redactEventPayload(notice)).toEqual(notice);
|
||||
// Master's JWT detector recognizes encoded headers, so a dotted public
|
||||
// identifier needs no receipt-shaped exemption to survive redaction.
|
||||
expect(redactEventPayload({ value: `paperclip.semantic_tool_receipt.v${version}` }))
|
||||
.toEqual({ value: `paperclip.semantic_tool_receipt.v${version}` });
|
||||
});
|
||||
|
||||
it("redacts credentials in receipt-shaped data and every adjacent field", () => {
|
||||
const jwt = "eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0.signature12345678";
|
||||
const notice = receiptNotice();
|
||||
notice.details.find((d: any) => d.name === "operationId").value = jwt;
|
||||
notice.provenance.sessionId = jwt;
|
||||
const redacted = redactEventPayload(notice)! as Record<string, any>;
|
||||
expect(receiptSchemaValue(redacted)).toBe("paperclip.semantic_tool_receipt.v1");
|
||||
expect(redacted.details.find((d: any) => d.name === "operationId").value).toBe(REDACTED_EVENT_VALUE);
|
||||
expect(redacted.provenance.sessionId).toBe(REDACTED_EVENT_VALUE);
|
||||
expect(redactEventPayload({ notice, password: "canary", arbitrary: jwt }))
|
||||
.toMatchObject({ password: REDACTED_EVENT_VALUE, arbitrary: REDACTED_EVENT_VALUE });
|
||||
notice.details.find((d: any) => d.name === "schema").value = jwt;
|
||||
notice.summary = "Authorization: Bearer canary-token";
|
||||
const hostile = redactEventPayload(notice)!;
|
||||
expect(receiptSchemaValue(hostile)).toBe(REDACTED_EVENT_VALUE);
|
||||
expect(JSON.stringify(hostile)).not.toContain("canary-token");
|
||||
});
|
||||
|
||||
it("keeps the discriminator allowlist in exact PRP v1 schema parity", () => {
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { chmodSync, linkSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, realpathSync, renameSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, expect, it, vi } from "vitest";
|
||||
const hooks = vi.hoisted(() => ({ mkdir: undefined as undefined | ((path: string) => void), rename: undefined as undefined | ((source: string, destination: string) => void), open: undefined as undefined | ((path: string) => void), read: undefined as undefined | ((path: string, bytes: number) => void) }));
|
||||
vi.mock("node:fs/promises", async (original) => {
|
||||
const fs = await original<typeof import("node:fs/promises")>();
|
||||
return { ...fs,
|
||||
mkdir: async (...args: Parameters<typeof fs.mkdir>) => { hooks.mkdir?.(String(args[0])); return fs.mkdir(...args); },
|
||||
rename: async (...args: Parameters<typeof fs.rename>) => { hooks.rename?.(String(args[0]), String(args[1])); return fs.rename(...args); },
|
||||
open: async (...args: Parameters<typeof fs.open>) => { hooks.open?.(String(args[0])); const file = await fs.open(...args); const read = file.read.bind(file); file.read = (async (...readArgs: any[]) => { const result = await (read as any)(...readArgs); hooks.read?.(String(args[0]), result.bytesRead); return result; }) as typeof file.read; return file; },
|
||||
};
|
||||
});
|
||||
vi.mock("../vendor/paperclip-runner/index.js", async () => await import("../../../packages/paperclip-runner/src/drivers/acpx/qualified-profiles.js"));
|
||||
import { QUALIFIED_ACPX_PROFILES } from "../../../packages/paperclip-runner/src/drivers/acpx/qualified-profiles.js";
|
||||
import { createRemotePiCompanionManifest, importRemotePiCompanion, inventoryRemoteCompanion, resolveRemotePiCompanion, selectRemotePiCompanion } from "../services/native-runtime/remote-pi-companion.js";
|
||||
const roots: string[] = [];
|
||||
afterEach(() => { hooks.mkdir = undefined; hooks.rename = undefined; hooks.open = undefined; hooks.read = undefined; for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); vi.restoreAllMocks(); });
|
||||
const hash = (bytes: string | Buffer) => createHash("sha256").update(bytes).digest("hex");
|
||||
const canonical = (v: any): string => Array.isArray(v) ? `[${v.map(canonical).join(",")}]` : v && typeof v === "object" ? `{${Object.keys(v).sort().map(k => `${JSON.stringify(k)}:${canonical(v[k])}`).join(",")}}` : JSON.stringify(v);
|
||||
async function fixture() {
|
||||
const root = realpathSync(mkdtempSync(join(tmpdir(), "paperclip-companion-"))); roots.push(root);
|
||||
const source = "a".repeat(40); const serverRoot = join(root, "server"); const directory = join(root, "release");
|
||||
mkdirSync(join(serverRoot, "dist"), { recursive: true }); writeFileSync(join(serverRoot, "package.json"), '{"name":"@paperclipai/server"}'); writeFileSync(join(serverRoot, "dist/build-info.json"), JSON.stringify({ commit: source }));
|
||||
mkdirSync(join(directory, "bin"), { recursive: true }); mkdirSync(join(directory, "provider-pack"));
|
||||
// Synthetic ELF header; these tests never launch it or claim native qualification.
|
||||
const elf = Buffer.alloc(128); Buffer.from([127, 69, 76, 70, 2, 1]).copy(elf); elf.writeUInt16LE(62, 18); writeFileSync(join(directory, "bin/paperclip-runnerd"), elf, { mode: 0o755 });
|
||||
const payload = { runnerSourceRevision: source, target: { platform: "linux", architecture: "x64" }, candidateProviders: { pi: { qualification: "qualified", profileDigest: QUALIFIED_ACPX_PROFILES.pi.commandDigest, path: "provider-assets/pi/linux-x64" } } };
|
||||
writeFileSync(join(directory, "provider-pack/provider-pack.json"), JSON.stringify({ schema: "paperclip-runner/remote-provider-pack/v1", digest: `sha256:${hash(canonical(payload))}`, payload }));
|
||||
const manifest = await createRemotePiCompanionManifest(directory, source); const bytes = JSON.stringify(manifest); writeFileSync(join(directory, "companion.json"), bytes); return { root, directory, serverRoot, sha256: hash(bytes), manifest, source };
|
||||
}
|
||||
it("imports the release-generated closure and resolves target bytes without environment overrides", async () => {
|
||||
const f = await fixture(); const result = await importRemotePiCompanion(f); expect(result.status).toBe("imported_verified");
|
||||
expect(result.runnerBinary).toBe(join(f.serverRoot, "remote-companions/linux-x64/bin/paperclip-runnerd"));
|
||||
expect(readFileSync(result.runnerBinary!)).toEqual(readFileSync(join(f.directory, "bin/paperclip-runnerd")));
|
||||
expect((await importRemotePiCompanion(f)).status).toBe("verified_existing"); expect((await resolveRemotePiCompanion({ serverRoot: f.serverRoot }))?.manifestSha256).toBe(f.sha256);
|
||||
expect(readdirSync(join(f.serverRoot, "remote-companions"))).toEqual(["linux-x64"]);
|
||||
});
|
||||
it.each(["sha", "source", "profile", "target", "daemon", "extra", "mode", "outside-link", "outside-hardlink", "pack"])("rejects %s before publishing", async kind => {
|
||||
const f = await fixture();
|
||||
if (kind === "sha") f.sha256 = "b".repeat(64);
|
||||
if (kind === "source") writeFileSync(join(f.serverRoot, "dist/build-info.json"), JSON.stringify({ commit: "b".repeat(40) }));
|
||||
if (kind === "profile" || kind === "target") { Object.assign(f.manifest, kind === "profile" ? { profileDigest: "sha256:" + "b".repeat(64) } : { target: "darwin-arm64" }); const bytes = JSON.stringify(f.manifest); writeFileSync(join(f.directory, "companion.json"), bytes); f.sha256 = hash(bytes); }
|
||||
if (kind === "daemon") writeFileSync(join(f.directory, "bin/paperclip-runnerd"), "changed");
|
||||
if (kind === "extra") writeFileSync(join(f.directory, "extra"), "unlisted");
|
||||
if (kind === "mode") chmodSync(join(f.directory, "bin/paperclip-runnerd"), 0o777);
|
||||
if (kind === "outside-link") symlinkSync("../../server/package.json", join(f.directory, "provider-pack/escape"));
|
||||
if (kind === "outside-hardlink") linkSync(join(f.directory, "bin/paperclip-runnerd"), join(f.root, "alias"));
|
||||
if (kind === "pack") writeFileSync(join(f.directory, "provider-pack/provider-pack.json"), "{}");
|
||||
await expect(importRemotePiCompanion(f)).rejects.toThrow(); expect(() => readdirSync(join(f.serverRoot, "remote-companions/linux-x64"))).toThrow();
|
||||
});
|
||||
it("copies contained symlinks and breaks only fully owned internal hardlinks", async () => {
|
||||
const f = await fixture(); linkSync(join(f.directory, "bin/paperclip-runnerd"), join(f.directory, "bin/alias")); symlinkSync("paperclip-runnerd", join(f.directory, "bin/link"));
|
||||
f.manifest = await createRemotePiCompanionManifest(f.directory, f.source); const bytes = JSON.stringify(f.manifest); writeFileSync(join(f.directory, "companion.json"), bytes); f.sha256 = hash(bytes);
|
||||
expect((await importRemotePiCompanion(f)).status).toBe("imported_verified");
|
||||
});
|
||||
it("fails closed on cache corruption, foreign authority and workspace-contained cache", async () => {
|
||||
const f = await fixture(); const result = await importRemotePiCompanion(f);
|
||||
await expect(resolveRemotePiCompanion({ serverRoot: f.serverRoot, workspaceRoot: f.root })).rejects.toThrow("workspace");
|
||||
await expect(resolveRemotePiCompanion({ serverRoot: f.serverRoot, workspaceRoot: join(result.root!, "provider-pack") })).rejects.toThrow("workspace");
|
||||
writeFileSync(result.runnerBinary!, "corrupted"); await expect(resolveRemotePiCompanion({ serverRoot: f.serverRoot })).rejects.toThrow("inventory");
|
||||
await expect(importRemotePiCompanion(f)).rejects.toThrow();
|
||||
});
|
||||
it("preserves an existing empty destination and releases only its import lock", async () => {
|
||||
const f = await fixture(); mkdirSync(join(f.serverRoot, "remote-companions/linux-x64"), { recursive: true, mode: 0o700 }); chmodSync(join(f.serverRoot, "remote-companions"), 0o700);
|
||||
await expect(importRemotePiCompanion(f)).rejects.toThrow(); expect(readdirSync(join(f.serverRoot, "remote-companions/linux-x64"))).toEqual([]); expect(readdirSync(join(f.serverRoot, "remote-companions"))).toEqual(["linux-x64"]);
|
||||
});
|
||||
it("rejects an unsafe cache parent and leaves it intact", async () => {
|
||||
const f = await fixture(); const outside = join(f.root, "outside"); mkdirSync(outside); symlinkSync(outside, join(f.serverRoot, "remote-companions"));
|
||||
await expect(importRemotePiCompanion(f)).rejects.toThrow(); expect(readdirSync(outside)).toEqual([]);
|
||||
});
|
||||
it("reports missing installation without manufacturing a companion", async () => { const f = await fixture(); expect(await resolveRemotePiCompanion({ serverRoot: f.serverRoot })).toBeNull(); });
|
||||
|
||||
it("selects only normal remote Pi, preserving explicit overrides and C/C admission", async () => {
|
||||
const f = await fixture(); await importRemotePiCompanion(f); const workspaceRoot = join(f.root, "workspace"); mkdirSync(workspaceRoot);
|
||||
const input = { serverRoot: f.serverRoot, workspaceRoot, remote: true, provider: { kind: "acpx", agent: "pi" } };
|
||||
expect((await selectRemotePiCompanion(input))?.target).toBe("linux-x64");
|
||||
for (const patch of [{ remote: false }, { provider: { kind: "acpx", agent: "cursor" } }, { provider: { kind: "acpx", agent: "copilot" } }, { provider: { kind: "codex" } }, { binaryOverride: "/operator/runnerd" }, { packOverride: "/operator/pack" }]) expect(await selectRemotePiCompanion({ ...input, ...patch })).toBeNull();
|
||||
});
|
||||
|
||||
it("rejects an appeared empty destination without replacing it", async () => {
|
||||
const f = await fixture(); const output = join(f.serverRoot, "remote-companions/linux-x64");
|
||||
hooks.mkdir = path => { if (path !== output) return; hooks.mkdir = undefined; mkdirSync(output, { mode: 0o700 }); };
|
||||
await expect(importRemotePiCompanion(f)).rejects.toThrow(); expect(readdirSync(output)).toEqual([]);
|
||||
expect(readdirSync(join(f.serverRoot, "remote-companions"))).toEqual(["linux-x64"]);
|
||||
});
|
||||
it("drains owned output, staging and lock cleanup after publication failure", async () => {
|
||||
const f = await fixture(); hooks.rename = () => { hooks.rename = undefined; throw new Error("injected publication failure"); };
|
||||
await expect(importRemotePiCompanion(f)).rejects.toThrow("injected publication failure");
|
||||
expect(readdirSync(join(f.serverRoot, "remote-companions"))).toEqual([]);
|
||||
});
|
||||
it("preserves a replaced staging root while draining the other owned cleanup", async () => {
|
||||
const f = await fixture(); let replaced = "";
|
||||
hooks.rename = source => {
|
||||
hooks.rename = undefined; replaced = source.slice(0, source.lastIndexOf("/"));
|
||||
renameSync(replaced, replaced + "-original"); mkdirSync(replaced, { mode: 0o700 }); writeFileSync(join(replaced, "foreign"), "preserve");
|
||||
throw new Error("replaced staging");
|
||||
};
|
||||
await expect(importRemotePiCompanion(f)).rejects.toThrow("cleanup incomplete"); expect(readFileSync(join(replaced, "foreign"), "utf8")).toBe("preserve");
|
||||
expect(readdirSync(join(f.serverRoot, "remote-companions"))).not.toContain(".import-linux-x64.lock");
|
||||
expect(readdirSync(join(f.serverRoot, "remote-companions"))).not.toContain("linux-x64");
|
||||
});
|
||||
it.each(["new", "existing"])("defers actual SIGTERM on the %s import path and drains owned cleanup", async (mode) => {
|
||||
const f = await fixture(); if (mode === "existing") await importRemotePiCompanion(f);
|
||||
const { build } = await import("esbuild");
|
||||
const { execFile } = await import("node:child_process");
|
||||
const { promisify } = await import("node:util");
|
||||
const bundle = join(f.root, "companion.mjs");
|
||||
await build({ entryPoints: [new URL("../services/native-runtime/remote-pi-companion.ts", import.meta.url).pathname], outfile: bundle,
|
||||
platform: "node", format: "esm", target: "node24", bundle: true, logLevel: "silent",
|
||||
plugins: [{ name: "source-owned-profile-only", setup(builder) {
|
||||
builder.onResolve({ filter: /vendor\/paperclip-runner\/index\.js$/ }, () => ({ path: new URL("../../../packages/paperclip-runner/src/drivers/acpx/qualified-profiles.ts", import.meta.url).pathname }));
|
||||
} }],
|
||||
});
|
||||
const script = join(f.root, "cancel.mjs");
|
||||
writeFileSync(script, `import {importRemotePiCompanion} from './companion.mjs';
|
||||
import {readdirSync,existsSync} from 'node:fs';
|
||||
let cancelled=false, checkpoints=0,sent=false; const cancel=()=>{cancelled=true}; process.on('SIGTERM',cancel);
|
||||
try { await importRemotePiCompanion({...JSON.parse(process.argv[2]),checkCancelled(){ checkpoints++; if(!sent&&existsSync(process.argv[3]+'/.import-linux-x64.lock')){sent=true;process.kill(process.pid,'SIGTERM');} if(cancelled)throw Error('owned cancellation'); }}); throw Error('unexpected success'); }
|
||||
catch(error){ if(error.message!=='owned cancellation')throw error; console.log(JSON.stringify({cancelled,checkpoints,remaining:readdirSync(process.argv[3])})); }
|
||||
finally {process.off('SIGTERM',cancel);}
|
||||
`);
|
||||
const { stdout } = await promisify(execFile)(process.execPath, [script, JSON.stringify({ directory: f.directory, sha256: f.sha256, serverRoot: f.serverRoot }), join(f.serverRoot, "remote-companions")], { env: { PATH: "/usr/bin:/bin", LANG: "C.UTF-8" }, timeout: 10_000, maxBuffer: 65536 });
|
||||
expect(JSON.parse(stdout)).toMatchObject({ cancelled: true, remaining: mode === "existing" ? ["linux-x64"] : [] });
|
||||
});
|
||||
|
||||
it("honors deadline expiry after re-verifying an existing cache without deleting it", async () => {
|
||||
const f = await fixture(); await importRemotePiCompanion(f); let expired = false;
|
||||
hooks.open = path => { if(path === join(f.serverRoot, "remote-companions/linux-x64/companion.json")) expired = true; };
|
||||
vi.spyOn(Date, "now").mockImplementation(() => expired ? 600_001 : 0);
|
||||
await expect(importRemotePiCompanion(f)).rejects.toThrow("deadline exceeded");
|
||||
expect(readdirSync(join(f.serverRoot, "remote-companions"))).toEqual(["linux-x64"]);
|
||||
});
|
||||
|
||||
async function largeFixture() {
|
||||
const f = await fixture(); writeFileSync(join(f.directory, "provider-pack/large"), Buffer.alloc(8 * 1024 ** 2, 0x61));
|
||||
f.manifest = await createRemotePiCompanionManifest(f.directory, f.source);
|
||||
const bytes = JSON.stringify(f.manifest); writeFileSync(join(f.directory, "companion.json"), bytes); f.sha256 = hash(bytes); return f;
|
||||
}
|
||||
it("yields to unrelated event-loop work between chunks of a real cache file", async () => {
|
||||
const f = await largeFixture(); await importRemotePiCompanion(f);
|
||||
const file = join(f.serverRoot, "remote-companions/linux-x64/provider-pack/large");
|
||||
let reads = 0; let serviced = false; let servicedBeforeNextRead = false;
|
||||
hooks.read = (path, bytes) => { if(path !== file || !bytes) return; reads++; if(reads === 1) setImmediate(() => { serviced = true; }); else servicedBeforeNextRead ||= serviced; };
|
||||
expect((await resolveRemotePiCompanion({serverRoot:f.serverRoot}))?.manifestSha256).toBe(f.sha256);
|
||||
expect(reads).toBeGreaterThanOrEqual(8); expect(servicedBeforeNextRead).toBe(true);
|
||||
});
|
||||
it("cancels during a large existing-cache file before reading the whole file and preserves it", async () => {
|
||||
const f = await largeFixture(); await importRemotePiCompanion(f);
|
||||
const file = join(f.serverRoot, "remote-companions/linux-x64/provider-pack/large"); let readBytes = 0;
|
||||
hooks.read = (path, bytes) => { if(path === file) readBytes += bytes; };
|
||||
await expect(importRemotePiCompanion({...f, checkCancelled(){ if(readBytes) throw Error("cancel during cache bytes"); }})).rejects.toThrow("cancel during cache bytes");
|
||||
expect(readBytes).toBeGreaterThan(0); expect(readBytes).toBeLessThan(8 * 1024 ** 2);
|
||||
expect(readdirSync(join(f.serverRoot, "remote-companions"))).toEqual(["linux-x64"]);
|
||||
});
|
||||
it("rejects read-only directory modes before claiming a staging or output path", async () => {
|
||||
const f = await fixture(); const dir = join(f.directory, "provider-pack"); chmodSync(dir, 0o500);
|
||||
try {
|
||||
await expect(importRemotePiCompanion({...f, checkCancelled(){}})).rejects.toThrow("owner read/write/search");
|
||||
expect(() => readdirSync(join(f.serverRoot, "remote-companions"))).toThrow();
|
||||
} finally { chmodSync(dir, 0o755); }
|
||||
});
|
||||
@@ -681,6 +681,21 @@ describe("routine routes", () => {
|
||||
userId: "board-user",
|
||||
runId: null,
|
||||
});
|
||||
expect(mockTrackRoutineCreated).toHaveBeenCalledWith(expect.anything());
|
||||
try {
|
||||
expect(mockTrackRoutineCreated).toHaveBeenCalledWith(expect.anything());
|
||||
} catch (error) {
|
||||
// Report mock wiring without importing modules again or changing timing.
|
||||
console.error("Routine creation telemetry mock diagnostic", {
|
||||
clientGetterCalls: mockGetTelemetryClient.mock.calls.length,
|
||||
clientGetterResults: mockGetTelemetryClient.mock.results.slice(-4).map(result => ({
|
||||
type: result.type,
|
||||
truthy: Boolean(result.value),
|
||||
})),
|
||||
trackingCalls: mockTrackRoutineCreated.mock.calls.length,
|
||||
createCalls: mockRoutineService.create.mock.calls.length,
|
||||
activityCalls: mockLogActivity.mock.calls.length,
|
||||
});
|
||||
throw error;
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,25 @@
|
||||
import Ajv2020 from "ajv/dist/2020.js";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
capabilitySemanticToolDescriptor,
|
||||
paperclipSemanticAction,
|
||||
} from "@paperclipai/paperclip-runner";
|
||||
import { PROJECT_ICON_NAMES } from "@paperclipai/shared";
|
||||
|
||||
// Cross-package parity belongs at the App boundary: the standalone runner
|
||||
// must not import the App's shared package, even from its own tests.
|
||||
describe("runner project icon contract", () => {
|
||||
it("advertises only icons accepted by the project API on both tool surfaces", () => {
|
||||
const ajv = new Ajv2020({ allErrors: true, allowUnionTypes: true, strict: true });
|
||||
for (const schema of [
|
||||
capabilitySemanticToolDescriptor("create_project")!.inputSchema,
|
||||
paperclipSemanticAction("create_project")!.inputSchema,
|
||||
]) {
|
||||
const validate = ajv.compile(schema);
|
||||
const input = { name: "Onboarding", idempotencyKey: "onboarding" };
|
||||
expect(schema).toMatchObject({ properties: { icon: { enum: [...PROJECT_ICON_NAMES, null] } } });
|
||||
for (const icon of [...PROJECT_ICON_NAMES, null]) expect(validate({ ...input, icon }), String(icon)).toBe(true);
|
||||
expect(validate({ ...input, icon: "users" })).toBe(false);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -6678,7 +6678,7 @@ describeEmbeddedPostgres("workspace runtime service control persistence", () =>
|
||||
}
|
||||
throw new Error("Timed out waiting for runtime service process marker");
|
||||
};
|
||||
const waitForPersistedStatus = async (status: string) => {
|
||||
const waitForPersistedStatus = async (status: string, requireProviderRef = false) => {
|
||||
const deadline = Date.now() + 5_000;
|
||||
while (Date.now() < deadline) {
|
||||
const row = await db
|
||||
@@ -6686,7 +6686,9 @@ describeEmbeddedPostgres("workspace runtime service control persistence", () =>
|
||||
.from(workspaceRuntimeServices)
|
||||
.where(eq(workspaceRuntimeServices.executionWorkspaceId, executionWorkspaceId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (row?.status === status) return row;
|
||||
// The provisional starting row is durable before spawn; the child can
|
||||
// write its marker before the subsequent PID update reaches the DB.
|
||||
if (row?.status === status && (!requireProviderRef || /^\d+$/.test(row.providerRef ?? ""))) return row;
|
||||
await new Promise((resolve) => setTimeout(resolve, 25));
|
||||
}
|
||||
throw new Error(`Timed out waiting for persisted runtime service status ${status}`);
|
||||
|
||||
@@ -4,13 +4,15 @@ import { listServerAdapters, requireServerAdapter } from "./registry.js";
|
||||
import * as executionTarget from "@paperclipai/adapter-utils/execution-target";
|
||||
import { BUILTIN_ADAPTER_TYPES } from "./builtin-adapter-types.js";
|
||||
|
||||
const { probeInstallation, probeGrokInstallation } = vi.hoisted(() => ({
|
||||
const { probeInstallation, probeGrokInstallation, probePiInstallation } = vi.hoisted(() => ({
|
||||
probeInstallation: vi.fn(),
|
||||
probeGrokInstallation: vi.fn(),
|
||||
probePiInstallation: vi.fn(),
|
||||
}));
|
||||
vi.mock("@paperclipai/paperclip-runner/live", () => ({
|
||||
vi.mock("../vendor/paperclip-runner/live/index.js", () => ({
|
||||
probeAcpxClaudeInstallation: probeInstallation,
|
||||
probeAcpxGrokInstallation: probeGrokInstallation,
|
||||
probeAcpxPiInstallation: probePiInstallation,
|
||||
probeAcpxCursorInstallation: vi.fn(async () => undefined),
|
||||
}));
|
||||
|
||||
@@ -96,6 +98,7 @@ describe("native ACPX environment checks", () => {
|
||||
beforeEach(() => {
|
||||
probeInstallation.mockReset().mockResolvedValue(undefined);
|
||||
probeGrokInstallation.mockReset().mockResolvedValue(undefined);
|
||||
probePiInstallation.mockReset().mockResolvedValue(undefined);
|
||||
});
|
||||
afterEach(() => vi.restoreAllMocks());
|
||||
|
||||
@@ -132,6 +135,19 @@ describe("native ACPX environment checks", () => {
|
||||
expect(probeInstallation).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([true, false])("checks qualified Pi's own installation readiness (%s)", async (ready) => {
|
||||
if (!ready) probePiInstallation.mockRejectedValueOnce(new Error("Pi installation integrity mismatch"));
|
||||
const model = "openrouter/deepseek/deepseek-v4-flash-0731";
|
||||
const result = await requireServerAdapter("paperclip_runner").testEnvironment!({
|
||||
...context,
|
||||
config: { provider: "acpx", acpxAgent: "pi", model },
|
||||
});
|
||||
expect(result.status).toBe(ready ? "pass" : "fail");
|
||||
expect(probePiInstallation).toHaveBeenCalledWith(model);
|
||||
expect(probeInstallation).not.toHaveBeenCalled();
|
||||
expect(probeGrokInstallation).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("does not use the host platform to reject a remote environment", async () => {
|
||||
vi.spyOn(process, "platform", "get").mockReturnValue("darwin");
|
||||
const result = await requireServerAdapter("paperclip_runner").testEnvironment!({
|
||||
|
||||
@@ -410,7 +410,7 @@ const paperclipRunnerAdapter: ServerAdapterModule = {
|
||||
checks: [{ code: "dot_event_test_required", level: "warn" as const, message: "Dot manages its model and billing. Validate the dedicated agent binding and event round trip in Paperclip; this read-only check does not wake the Dot." }] };
|
||||
}
|
||||
if (profile.provider === "acpx") {
|
||||
if (["copilot", "pi"].includes(profile.acpxAgent)) {
|
||||
if (["copilot"].includes(profile.acpxAgent)) {
|
||||
// The profile resolver already validated the isolated host's exact
|
||||
// qualification pair. Do not report a production readiness pass.
|
||||
return {
|
||||
@@ -420,7 +420,7 @@ const paperclipRunnerAdapter: ServerAdapterModule = {
|
||||
};
|
||||
}
|
||||
try {
|
||||
if (profile.acpxAgent !== "claude" && profile.acpxAgent !== "grok" && profile.acpxAgent !== "cursor") throw new Error("Select Codex to use the native Codex runner.");
|
||||
if (profile.acpxAgent !== "claude" && profile.acpxAgent !== "grok" && profile.acpxAgent !== "cursor" && profile.acpxAgent !== "pi") throw new Error("Select Codex to use the native Codex runner.");
|
||||
const target = context.executionTarget;
|
||||
if (target?.kind === "remote") {
|
||||
const probe = await runAdapterExecutionTargetShellCommand(
|
||||
@@ -438,8 +438,8 @@ const paperclipRunnerAdapter: ServerAdapterModule = {
|
||||
message: "The remote platform is supported. Runtime package integrity and readiness must still be verified by the remote runner before launch." }],
|
||||
};
|
||||
}
|
||||
const { probeAcpxClaudeInstallation, probeAcpxGrokInstallation, probeAcpxCursorInstallation } = await import("../vendor/paperclip-runner/live/index.js");
|
||||
await (profile.acpxAgent === "grok" ? probeAcpxGrokInstallation : profile.acpxAgent === "cursor" ? probeAcpxCursorInstallation : probeAcpxClaudeInstallation)(profile.model);
|
||||
const { probeAcpxClaudeInstallation, probeAcpxGrokInstallation, probeAcpxCursorInstallation, probeAcpxPiInstallation } = await import("../vendor/paperclip-runner/live/index.js");
|
||||
await (profile.acpxAgent === "pi" ? probeAcpxPiInstallation : profile.acpxAgent === "grok" ? probeAcpxGrokInstallation : profile.acpxAgent === "cursor" ? probeAcpxCursorInstallation : probeAcpxClaudeInstallation)(profile.model);
|
||||
return {
|
||||
adapterType: "paperclip_runner", status: "pass" as const, testedAt: new Date().toISOString(),
|
||||
checks: [{ code: "acpx_runtime_ready", level: "info" as const, message: `ACPX ${profile.acpxAgent} runtime is installed and verified. Model access is checked when it runs.` }],
|
||||
@@ -525,7 +525,7 @@ const paperclipRunnerAdapter: ServerAdapterModule = {
|
||||
)
|
||||
: buildNpmRuntimeCommandSpec(config, "codex", "@openai/codex@0.160.0"),
|
||||
agentConfigurationDoc:
|
||||
"# Paperclip Runner\n\nAdapter: paperclip_runner\n\nRuns Codex, OpenCode, Claude Managed, AWS AgentCore, or ACPX Claude/Grok Build/Cursor through the Rust Paperclip runner and authenticated PRP transport. GitHub Copilot and Pi are awaiting local and Daytona qualification and are not enabled for production runs. Managed providers use company-scoped qualified profiles, explicit retention acknowledgement, and spend limits.\n",
|
||||
"# Paperclip Runner\n\nAdapter: paperclip_runner\n\nRuns Codex, OpenCode, Claude Managed, AWS AgentCore, or ACPX Claude/Grok Build/Cursor/Pi through the Rust Paperclip runner and authenticated PRP transport. Pi accepts an explicit provider/model ID and uses the company credentials bound to the agent environment. GitHub Copilot awaits local and Daytona qualification. Managed providers use company-scoped qualified profiles, explicit retention acknowledgement, and spend limits.\n",
|
||||
getConfigSchema: () => ({
|
||||
fields: [
|
||||
{
|
||||
@@ -569,6 +569,19 @@ const paperclipRunnerAdapter: ServerAdapterModule = {
|
||||
hint: PAPERCLIP_RUNNER_PERMISSION_CAPABILITIES.opencode.description,
|
||||
meta: { visibleWhen: { key: "provider", value: "opencode" } },
|
||||
},
|
||||
{
|
||||
key: "acpxSessionMode",
|
||||
label: "Cursor mode",
|
||||
type: "select" as const,
|
||||
default: "agent",
|
||||
options: [
|
||||
{ value: "agent", label: "Agent" },
|
||||
{ value: "plan", label: "Plan" },
|
||||
{ value: "ask", label: "Ask" },
|
||||
],
|
||||
hint: "Select Cursor's session mode. Permissions and company approval rules still apply.",
|
||||
meta: { visibleWhen: { key: "acpxAgent", value: "cursor" } },
|
||||
},
|
||||
{
|
||||
key: "acpxPermissionMode",
|
||||
label: "ACPX permission mode",
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { cancellationRequestId } from "../services/native-runtime/native-cancellation-request.js";
|
||||
import { aiRoutingHarness } from "@paperclipai/shared";
|
||||
import { agentIdentityService } from "../services/agent-identity.js";
|
||||
import { aiConnectionRouterService, poolMemberRuntimeConfig } from "../services/ai-connection-router.js";
|
||||
@@ -6,7 +7,6 @@ import { dotRunnerBroker } from "../services/dot-runner-broker.js";
|
||||
import { publicMcpConfig } from "../services/public-mcp/oauth.js";
|
||||
import { connectionIntentDeliveryService } from "../services/connection-intent-delivery.js";
|
||||
import { completeConnectionIntentSchema } from "@paperclipai/shared";
|
||||
import { cancellationRequestId } from "../services/native-runtime/native-cancellation-request.js";
|
||||
import { agentFileStore, agentFileTokenFromHash } from "../services/agent-file-store.js";
|
||||
import { pipeline } from "node:stream/promises";
|
||||
import { resolveAgentAppearance, agentAvatarUrl } from "@paperclipai/shared";
|
||||
|
||||
@@ -0,0 +1,169 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import childProcess from "node:child_process";
|
||||
import type { DirectorySnapshot } from "@paperclipai/adapter-utils/workspace-restore-merge";
|
||||
|
||||
/** Only for a live unchanged-turn observation. Never a save or stop receipt. */
|
||||
export function agentDirectoryBaselineDigest(snapshot: DirectorySnapshot): string {
|
||||
if (snapshot.exclude.length || snapshot.ignoredPaths) throw new Error("Incomplete agent directory baseline");
|
||||
const entries = [...snapshot.entries].sort(([a], [b]) => a < b ? -1 : a > b ? 1 : 0);
|
||||
return createHash("sha256").update(JSON.stringify(entries)).digest("hex");
|
||||
}
|
||||
|
||||
// This self-contained function also runs inside the original remote target.
|
||||
// Keep all dependencies explicit so remote observation never reads a local mirror.
|
||||
export function probeAgentDirectory(
|
||||
root: string,
|
||||
modules = { fs, path, createHash, platform: process.platform },
|
||||
): { digest: string; identity: string } {
|
||||
const { fs, path, createHash } = modules;
|
||||
const deadline = Date.now() + 5_000;
|
||||
const checkTime = () => { if (Date.now() > deadline) throw new Error("Agent directory probe deadline"); };
|
||||
const stamp = (s: fs.BigIntStats) => [s.dev, s.ino, s.mode, s.nlink, s.size, s.mtimeNs, s.ctimeNs].join(":");
|
||||
const identity = (s: fs.BigIntStats) => [s.dev, s.ino, s.birthtimeNs].join(":");
|
||||
if (!path.isAbsolute(root) || path.resolve(root) !== root || root.includes("\0")) throw new Error("Invalid agent directory root");
|
||||
// Resolve only the observing host's fixed Darwin aliases. The exact full
|
||||
// counterpart excludes nested/user symlinks; canonical ancestors remain strict.
|
||||
const requestedRoot = root;
|
||||
if (modules.platform === "darwin" && /^\/(tmp|var|etc)\//.test(root)) {
|
||||
const resolved = fs.realpathSync(root);
|
||||
if (resolved !== `/private${root}`) throw new Error("Unsafe agent directory alias");
|
||||
root = resolved;
|
||||
}
|
||||
const ancestors = new Map<string, string>();
|
||||
let ancestor = path.parse(root).root;
|
||||
for (const segment of root.slice(ancestor.length).split(path.sep)) {
|
||||
ancestor = path.join(ancestor, segment);
|
||||
const s = fs.lstatSync(ancestor, { bigint: true });
|
||||
if (!s.isDirectory() || s.isSymbolicLink()) throw new Error("Unsafe agent directory ancestor");
|
||||
ancestors.set(ancestor, identity(s));
|
||||
}
|
||||
const metadata = new Map<string, string>();
|
||||
const entries: Array<[string, { kind: "dir" } | { kind: "file"; mode: number; hash: string }]> = [];
|
||||
let total = 0;
|
||||
function walk(relative: string, verify: boolean) {
|
||||
checkTime();
|
||||
const absolute = path.join(root, relative);
|
||||
const before = fs.lstatSync(absolute, { bigint: true });
|
||||
if (!before.isDirectory() || before.isSymbolicLink()) throw new Error("Unsafe agent directory");
|
||||
const names = fs.readdirSync(absolute).sort();
|
||||
if (verify) {
|
||||
if (metadata.get(relative) !== stamp(before)) throw new Error("Agent directory changed during probe");
|
||||
} else metadata.set(relative, stamp(before));
|
||||
for (const name of names) {
|
||||
checkTime();
|
||||
const next = relative ? `${relative}/${name}` : name;
|
||||
if (next.length > 512 || name === "." || name === ".." || name.includes("\\") || name.includes("\0")
|
||||
|| name === ".paperclip-runtime" || next === "promptTemplate.legacy.md") throw new Error("Unsafe agent file path");
|
||||
const filename = path.join(root, next);
|
||||
const s = fs.lstatSync(filename, { bigint: true });
|
||||
if (s.isDirectory()) {
|
||||
if (!verify) entries.push([next, { kind: "dir" }]);
|
||||
walk(next, verify);
|
||||
} else {
|
||||
if (!s.isFile() || s.nlink !== 1n || s.size > 256n * 1024n * 1024n) throw new Error("Unsafe or oversized agent file");
|
||||
if (verify) {
|
||||
if (metadata.get(next) !== stamp(s)) throw new Error("Agent file changed during probe");
|
||||
} else {
|
||||
total += Number(s.size);
|
||||
if (total > 2 * 1024 * 1024 * 1024) throw new Error("Agent directory probe byte limit");
|
||||
const fd = fs.openSync(filename, fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW);
|
||||
try {
|
||||
if (stamp(fs.fstatSync(fd, { bigint: true })) !== stamp(s)) throw new Error("Agent file replaced during open");
|
||||
const hash = createHash("sha256"), buffer = Buffer.alloc(64 * 1024);
|
||||
let size = 0;
|
||||
for (;;) {
|
||||
checkTime();
|
||||
const count = fs.readSync(fd, buffer, 0, buffer.length, null);
|
||||
if (!count) break;
|
||||
size += count;
|
||||
if (size > Number(s.size)) throw new Error("Agent file grew during probe");
|
||||
hash.update(buffer.subarray(0, count));
|
||||
}
|
||||
if (size !== Number(s.size) || stamp(fs.fstatSync(fd, { bigint: true })) !== stamp(s)
|
||||
|| stamp(fs.lstatSync(filename, { bigint: true })) !== stamp(s)) throw new Error("Agent file changed during read");
|
||||
metadata.set(next, stamp(s));
|
||||
entries.push([next, { kind: "file", mode: Number(s.mode), hash: hash.digest("hex") }]);
|
||||
} finally { fs.closeSync(fd); }
|
||||
}
|
||||
}
|
||||
if (entries.length > 100_000) throw new Error("Agent directory probe entry limit");
|
||||
}
|
||||
if (stamp(fs.lstatSync(absolute, { bigint: true })) !== stamp(before)
|
||||
|| JSON.stringify(fs.readdirSync(absolute).sort()) !== JSON.stringify(names)) throw new Error("Agent directory membership changed");
|
||||
}
|
||||
const initial = fs.lstatSync(root, { bigint: true });
|
||||
walk("", false);
|
||||
walk("", true);
|
||||
if (stamp(fs.lstatSync(root, { bigint: true })) !== stamp(initial)) throw new Error("Agent directory root changed");
|
||||
for (const [ancestor, expected] of ancestors) {
|
||||
const current = fs.lstatSync(ancestor, { bigint: true });
|
||||
if (!current.isDirectory() || current.isSymbolicLink() || identity(current) !== expected) throw new Error("Agent directory ancestor changed");
|
||||
}
|
||||
if (requestedRoot !== root && fs.realpathSync(requestedRoot) !== root) throw new Error("Agent directory alias changed");
|
||||
entries.sort(([a], [b]) => a < b ? -1 : a > b ? 1 : 0);
|
||||
return { digest: createHash("sha256").update(JSON.stringify(entries)).digest("hex"), identity: identity(initial) };
|
||||
}
|
||||
|
||||
// tsx/esbuild keepNames injects this lexical helper into nested functions.
|
||||
// Include its exact name-property behavior in each plain-Node child program;
|
||||
// function.toString() alone does not carry the controller module's bindings.
|
||||
const childNameHelper = `const __name=(target,value)=>Object.defineProperty(target,"name",{value,configurable:true});`;
|
||||
|
||||
export const agentDirectoryProbeProgram = `${childNameHelper} const probe=${probeAgentDirectory.toString()}; process.stdout.write(JSON.stringify(probe(process.argv[1], {fs:require('node:fs'),path:require('node:path'),createHash:require('node:crypto').createHash,platform:process.platform})))`;
|
||||
|
||||
/** Retire only the two empty directories created by initial sandbox transfer.
|
||||
* Run before provider admission; this is never an exclusion from live probing. */
|
||||
export function retireAgentDirectoryTransferScratch(root: string, modules = { fs, path, platform: process.platform }): void {
|
||||
const { fs, path } = modules;
|
||||
if (!path.isAbsolute(root) || path.resolve(root) !== root || root.includes("\0")) throw new Error("Invalid agent directory root");
|
||||
const requestedRoot = root;
|
||||
if (modules.platform === "darwin" && /^\/(tmp|var|etc)\//.test(root)) {
|
||||
const resolved = fs.realpathSync(root);
|
||||
if (resolved !== `/private${root}`) throw new Error("Unsafe agent directory alias");
|
||||
root = resolved;
|
||||
}
|
||||
const identities = new Map<string, string>();
|
||||
const identify = (directory: string) => {
|
||||
const stat = fs.lstatSync(directory, { bigint: true });
|
||||
if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error("Unsafe transfer scratch directory");
|
||||
return [stat.dev, stat.ino, stat.birthtimeNs].join(":");
|
||||
};
|
||||
let ancestor = path.parse(root).root;
|
||||
for (const segment of root.slice(ancestor.length).split(path.sep)) {
|
||||
ancestor = path.join(ancestor, segment); identities.set(ancestor, identify(ancestor));
|
||||
}
|
||||
const parent = path.join(root, ".paperclip-runtime"), scratch = path.join(parent, "agent-files");
|
||||
identities.set(parent, identify(parent)); identities.set(scratch, identify(scratch));
|
||||
const verify = () => {
|
||||
for (const [directory, expected] of identities) if (identify(directory) !== expected) throw new Error("Transfer scratch identity changed");
|
||||
if (requestedRoot !== root && fs.realpathSync(requestedRoot) !== root) throw new Error("Agent directory alias changed");
|
||||
};
|
||||
const entries = fs.readdirSync(parent);
|
||||
if (entries.length !== 1 || entries[0] !== "agent-files" || fs.readdirSync(scratch).length) throw new Error("Unexpected transfer scratch contents");
|
||||
verify();
|
||||
// rmdir atomically refuses nonempty directories. Never recursively delete or
|
||||
// follow a link; any change stops preparation before the provider can start.
|
||||
fs.rmdirSync(scratch); identities.delete(scratch);
|
||||
verify();
|
||||
if (fs.readdirSync(parent).length) throw new Error("Transfer scratch changed during retirement");
|
||||
fs.rmdirSync(parent); identities.delete(parent);
|
||||
verify();
|
||||
}
|
||||
|
||||
export const agentDirectoryTransferCleanupProgram = `${childNameHelper} const retire=${retireAgentDirectoryTransferScratch.toString()}; retire(process.argv[1], {fs:require('node:fs'),path:require('node:path'),platform:process.platform})`;
|
||||
|
||||
/** Bound child: no shell, inherited credentials, or event-loop hashing. */
|
||||
export async function observeLocalAgentDirectory(root: string) {
|
||||
const result = await new Promise<{ stdout: string; stderr: string }>((resolve, reject) => {
|
||||
childProcess.execFile(process.execPath, ["-e", agentDirectoryProbeProgram, root], {
|
||||
cwd: root, env: {}, timeout: 10_000, maxBuffer: 1024, killSignal: "SIGKILL", encoding: "utf8",
|
||||
}, (error, stdout, stderr) => { if (error) reject(error); else resolve({ stdout, stderr }); });
|
||||
});
|
||||
if (result.stdout.length > 1024) throw new Error("Agent directory observation exceeds bound");
|
||||
const value = JSON.parse(result.stdout);
|
||||
if (!value || typeof value.digest !== "string" || !/^[a-f0-9]{64}$/.test(value.digest)
|
||||
|| typeof value.identity !== "string" || !/^[0-9]+:[0-9]+:[0-9]+$/.test(value.identity)) throw new Error("Invalid agent directory observation");
|
||||
return value as { digest: string; identity: string };
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { and, eq, inArray, isNull, sql } from "drizzle-orm";
|
||||
import { agents, environmentLeases, environments, agentInstructionWorkingCopies as copies, type Db } from "@paperclipai/db";
|
||||
import { syncDirectoryToSsh, restoreWorkspaceFromSshExecution } from "@paperclipai/adapter-utils/ssh";
|
||||
import { prepareAdapterExecutionTargetRuntime, runAdapterExecutionTargetShellCommand, type AdapterExecutionTarget, type PreparedAdapterExecutionTargetRuntime } from "@paperclipai/adapter-utils/execution-target";
|
||||
@@ -13,6 +13,7 @@ import { HttpError, conflict, notFound } from "../errors.js";
|
||||
import type { AuthorizationActor } from "./authorization.js";
|
||||
import type { EnvironmentRuntimeService } from "./environment-runtime.js";
|
||||
import type { Environment, EnvironmentLease } from "@paperclipai/shared";
|
||||
import { agentDirectoryBaselineDigest, agentDirectoryProbeProgram, observeLocalAgentDirectory } from "./agent-directory-probe.js";
|
||||
import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
|
||||
import { cachedAgentFileManifest, captureAgentFileCheckpoint, checkpointBaseline, checkpointSnapshot, type AgentFileManifest } from "./agent-file-checkpoints.js";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
@@ -22,6 +23,10 @@ export class AgentDirectoryReuseInvalidatedError extends Error {}
|
||||
const completed = new Set(["saved", "unchanged", "resolved", "unavailable"]);
|
||||
const transports = new Map<string, PreparedAdapterExecutionTargetRuntime>();
|
||||
const key = (row: Pick<Copy, "companyId" | "runId">) => `${row.companyId}:${row.runId}`;
|
||||
function sandboxTransferRoot(row: Copy, remoteCwd: string): string {
|
||||
const origin = typeof row.receipt?.materializationRunId === "string" ? row.receipt.materializationRunId : String(row.receipt?.directoryRunId ?? row.runId);
|
||||
return path.posix.join(remoteCwd, ".paperclip-runtime", "paperclip-runner", "agent-file-transfers", row.agentId, origin);
|
||||
}
|
||||
export function isAgentDirectoryCopy(row: Pick<Copy, "receipt"> | null): boolean {
|
||||
return row?.receipt?.schema === AGENT_FILES_CONTRACT;
|
||||
}
|
||||
@@ -50,11 +55,10 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
|
||||
}
|
||||
async function transport(row: Copy, target: AdapterExecutionTarget, recovering: boolean) {
|
||||
if (target.kind !== "remote" || row.location !== `remote:${target.environmentId ?? ""}`) throw new Error("Agent directory environment changed");
|
||||
if (recovering && target.transport === "ssh") throw new Error("The original SSH collector is unavailable");
|
||||
if (target.transport === "ssh") {
|
||||
// Reuse SSH's plain directory transfer without its task-workspace suffix
|
||||
// or Git-history discovery. The registered root is the exact writable root.
|
||||
await syncDirectoryToSsh({ spec: target.spec, localDir: row.localRoot, remoteDir: row.executionRoot, exclude: [".paperclip-runtime"] });
|
||||
if (!recovering) await syncDirectoryToSsh({ spec: target.spec, localDir: row.localRoot, remoteDir: row.executionRoot, exclude: [".paperclip-runtime"] });
|
||||
return { target, workspaceRemoteDir: row.executionRoot, runtimeRootDir: null,
|
||||
assetDirs: {}, additionalSourceDirs: {}, additionalSourceFailures: [], workspaceSyncSnapshot: null,
|
||||
restoreWorkspace: () => restoreWorkspaceFromSshExecution({ spec: target.spec, localDir: row.localRoot,
|
||||
@@ -62,10 +66,30 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
|
||||
}
|
||||
return prepareAdapterExecutionTargetRuntime({ target, runId: row.runId, adapterKey: "agent-files",
|
||||
workspaceLocalDir: row.localRoot, workspaceRemoteDir: row.executionRoot,
|
||||
// Agent files are an independently observed native directory. Transfer
|
||||
// scratch belongs to the host runtime, including the fallback selected
|
||||
// while plugin capability discovery is cold after controller restart.
|
||||
runtimeRootDir: sandboxTransferRoot(row, target.remoteCwd),
|
||||
syncWorkspace: true, workspaceInboundMode: recovering ? "adopt_remote" : undefined,
|
||||
workspaceBaseline: baseline(row), workspaceGitSnapshot: null, workspaceFileMode: "all",
|
||||
workspaceExclude: [".paperclip-runtime", ".paperclip-runtime/**"] });
|
||||
}
|
||||
async function observe(row: Copy, target?: AdapterExecutionTarget | null) {
|
||||
if (row.location === "local") {
|
||||
// The same bounded program validates aliases on its actual host.
|
||||
return observeLocalAgentDirectory(row.localRoot);
|
||||
}
|
||||
if (!target || target.kind !== "remote" || row.location !== `remote:${target.environmentId ?? ""}`) throw new Error("Agent directory environment changed");
|
||||
const origin = typeof row.receipt?.materializationRunId === "string" ? row.receipt.materializationRunId : row.runId;
|
||||
if (row.executionRoot !== path.posix.join(target.remoteCwd, ".paperclip-runtime", "agent-files", row.agentId, origin)) throw new Error("Agent directory root changed");
|
||||
const quote = (value: string) => `'${value.replaceAll("'", `'"'"'`)}'`;
|
||||
const result = await runAdapterExecutionTargetShellCommand(row.runId, target,
|
||||
`node -e ${quote(agentDirectoryProbeProgram)} ${quote(row.executionRoot)}`, { cwd: target.remoteCwd, env: {}, timeoutSec: 10 });
|
||||
if (result.exitCode !== 0 || result.timedOut || result.stdout.length > 1024) throw new Error("Agent directory observation unavailable");
|
||||
const value = JSON.parse(result.stdout);
|
||||
if (!value || typeof value.digest !== "string" || !/^[a-f0-9]{64}$/.test(value.digest) || typeof value.identity !== "string") throw new Error("Invalid agent directory observation");
|
||||
return value as { digest: string; identity: string };
|
||||
}
|
||||
async function prepareCopy(input: { companyId: string; agentId: string; runId: string; target?: AdapterExecutionTarget | null; cwd: string; warm?: boolean; reuseRunId?: string; onWarmHandoff?: (copy: Copy) => void }, serialHeld = false): Promise<Copy | null> {
|
||||
const [agent] = await db.select().from(agents).where(and(eq(agents.id, input.agentId), eq(agents.companyId, input.companyId)));
|
||||
if (!agent) throw notFound("Agent not found");
|
||||
@@ -110,15 +134,18 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
|
||||
}
|
||||
}, "agent_directory_handoff");
|
||||
}
|
||||
const localRoot = path.join(path.dirname(root), "file-sync", "runs", input.runId, "live");
|
||||
let row = await get(input.companyId, input.runId);
|
||||
if (row?.receipt?.retainedByRunId) throw conflict("Agent directory ownership passed to another run");
|
||||
const origin = typeof row?.receipt?.materializationRunId === "string" ? row.receipt.materializationRunId : input.runId;
|
||||
if (!/^[a-zA-Z0-9_-]{1,128}$/.test(origin)) throw conflict("Invalid agent directory owner");
|
||||
const localRoot = path.join(path.dirname(root), "file-sync", "runs", origin, "live");
|
||||
// Local copies live outside the task cwd. Remote copies use the reserved,
|
||||
// excluded runtime area inside the provider's confined workspace. They are
|
||||
// synchronized independently; provider HOME is never reinterpreted.
|
||||
const executionRoot = input.target?.kind === "remote"
|
||||
? path.posix.join(input.target.remoteCwd, ".paperclip-runtime", "agent-files", input.agentId, input.runId)
|
||||
? path.posix.join(input.target.remoteCwd, ".paperclip-runtime", "agent-files", input.agentId, origin)
|
||||
: localRoot;
|
||||
const location = input.target?.kind === "remote" ? `remote:${input.target.environmentId ?? ""}` : "local";
|
||||
let row = await get(input.companyId, input.runId);
|
||||
if (row && (row.localRoot !== localRoot || row.executionRoot !== executionRoot || row.agentId !== input.agentId || row.location !== location)) throw conflict("Agent directory belongs to a different execution environment");
|
||||
if (row && !serialHeld) {
|
||||
// Restart preparation can replace a completed copy under the same run ID.
|
||||
@@ -135,7 +162,7 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
|
||||
// leaves a recoverable preparing receipt instead of an orphan directory.
|
||||
const preparing = { entryFile: deriveBundleState(agent).entryFile, baseRevisionId: null, baseHash: "preparing",
|
||||
localRoot, executionRoot, location, state: "preparing", candidateBase64: null, candidateHash: null,
|
||||
receipt: { schema: AGENT_FILES_CONTRACT, ...(input.warm ? { warm: true, directoryRunId: input.runId } : {}), ...(input.target?.kind === "remote"
|
||||
receipt: { schema: AGENT_FILES_CONTRACT, workspaceCwd: input.cwd, materializationRunId: origin, ...(input.warm ? { warm: true, directoryRunId: input.runId } : {}), ...(input.target?.kind === "remote"
|
||||
? { cleanup: { leaseId: input.target.leaseId ?? null, remoteCwd: input.target.remoteCwd } } : {}) }, processStoppedAt: null, attempts: 0,
|
||||
nextAttemptAt: null, errorCode: null, errorMessage: null };
|
||||
if (row) row = await patch(row, preparing);
|
||||
@@ -183,9 +210,15 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
|
||||
`node -e ${quote(instructionGitExcludeProgram)} ${quote(input.target.remoteCwd)}`,
|
||||
{ cwd: input.target.remoteCwd, env: {}, timeoutSec: 15 });
|
||||
if (excluded.exitCode !== 0 || excluded.timedOut) throw new Error("Could not exclude agent files from task Git staging");
|
||||
transports.set(key(row), await transport(row, input.target, false));
|
||||
const runtime = await transport(row, input.target, false);
|
||||
transports.set(key(row), runtime);
|
||||
if (input.target.transport === "sandbox") {
|
||||
if (runtime.runtimeRootDir !== sandboxTransferRoot(row, input.target.remoteCwd)) throw new Error("Agent directory transfer scratch path changed");
|
||||
}
|
||||
}
|
||||
return await patch(row, { state: "prepared" });
|
||||
const observed = await observe(row, input.target).catch(() => null);
|
||||
return await patch(row, { state: "prepared", receipt: { ...row.receipt,
|
||||
...(observed?.digest === agentDirectoryBaselineDigest(snapshot) ? { materializationIdentity: observed.identity } : {}) } });
|
||||
} catch (error) {
|
||||
// Preparation failed before a provider could start using this copy.
|
||||
row = await patch(row, { state: "unavailable", processStoppedAt: new Date(), errorCode: "AGENT_FILES_PREPARE_FAILED",
|
||||
@@ -207,12 +240,81 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
|
||||
}
|
||||
return inspectAgentDirectory(row.localRoot);
|
||||
}
|
||||
async function hasChanges(_row: Copy, _target?: AdapterExecutionTarget | null) {
|
||||
// A whole-directory collector needs a quiescent provider, including its
|
||||
// child processes. Checkpoint and close at the turn boundary before reading
|
||||
// either local or remote files. The provider conversation remains resumable;
|
||||
// ordinary file edits do not change the session configuration digest.
|
||||
return true;
|
||||
async function hasChanges(row: Copy, target?: AdapterExecutionTarget | null) {
|
||||
try {
|
||||
if (row.receipt?.retainedByRunId || row.receipt?.retirementRequired || typeof row.receipt?.materializationIdentity !== "string") return true;
|
||||
const observed = await observe(row, target);
|
||||
if (observed.identity !== row.receipt.materializationIdentity || observed.digest !== agentDirectoryBaselineDigest(baseline(row))) return true;
|
||||
const unchanged = await patch(row, { state: "unchanged_turn", nextAttemptAt: null });
|
||||
return unchanged.state !== "unchanged_turn" || Boolean(unchanged.receipt?.retainedByRunId) || Boolean(unchanged.processStoppedAt);
|
||||
} catch { return true; }
|
||||
}
|
||||
/** Rebind one physical materialization; earlier runs lose collection authority. */
|
||||
async function adopt(input: { companyId: string; agentId: string; runId: string; previousRunId: string; target?: AdapterExecutionTarget | null; cwd: string; allowRetirementHandoff?: boolean }) {
|
||||
let prior = await get(input.companyId, input.previousRunId);
|
||||
if (!prior || prior.agentId !== input.agentId || prior.state !== "unchanged_turn" || prior.receipt?.retainedByRunId || prior.processStoppedAt
|
||||
|| prior.receipt?.workspaceCwd !== input.cwd || prior.location !== (input.target?.kind === "remote" ? `remote:${input.target.environmentId ?? ""}` : "local")) return null;
|
||||
const oldRuntime = transports.get(key(prior));
|
||||
const remote = input.target?.kind === "remote" ? input.target : null;
|
||||
const priorCleanup = prior.receipt?.cleanup as { leaseId?: string; remoteCwd?: string } | undefined;
|
||||
const validLeases = (leases: Array<typeof environmentLeases.$inferSelect>) => {
|
||||
if (!remote) return true;
|
||||
const previous = leases.find(lease => lease.id === priorCleanup?.leaseId);
|
||||
const successor = leases.find(lease => lease.id === remote.leaseId);
|
||||
return Boolean(previous && successor && previous.id !== successor.id
|
||||
&& previous.companyId === input.companyId && successor.companyId === input.companyId
|
||||
&& previous.heartbeatRunId === input.previousRunId && successor.heartbeatRunId === input.runId
|
||||
&& previous.environmentId === remote.environmentId && successor.environmentId === remote.environmentId
|
||||
&& previous.provider && previous.provider === successor.provider
|
||||
&& previous.providerLeaseId && previous.providerLeaseId === successor.providerLeaseId
|
||||
&& successor.status === "active" && !successor.releasedAt
|
||||
&& !hasRemoteTerminationReceipt(previous) && !hasRemoteTerminationReceipt(successor)
|
||||
&& (remote.transport !== "sandbox" || remote.providerKey === successor.provider)
|
||||
&& (successor.metadata?.remoteCwd === undefined || successor.metadata.remoteCwd === remote.remoteCwd));
|
||||
};
|
||||
const leaseIds = remote && priorCleanup?.leaseId && remote.leaseId ? [priorCleanup.leaseId, remote.leaseId] : [];
|
||||
if (remote && (!oldRuntime || oldRuntime.target.kind !== "remote" || oldRuntime.target.transport !== remote.transport
|
||||
|| oldRuntime.target.leaseId !== priorCleanup?.leaseId || oldRuntime.target.remoteCwd !== remote.remoteCwd
|
||||
|| priorCleanup?.remoteCwd !== remote.remoteCwd || leaseIds.length !== 2
|
||||
|| !validLeases(await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId), inArray(environmentLeases.id, leaseIds)))))) return null;
|
||||
const bound = await resolveInstructionActor(db, { type: "agent", companyId: input.companyId, agentId: input.agentId, runId: input.runId });
|
||||
const observedChange = await hasChanges(prior, input.target);
|
||||
if (observedChange && !input.allowRetirementHandoff) return null;
|
||||
prior = (await get(input.companyId, input.previousRunId))!;
|
||||
// adopt_remote constructs a collector on the successor target without
|
||||
// uploading the stale host mirror over the retained remote directory.
|
||||
const replacement = remote ? await transport({ ...prior, runId: input.runId }, remote, true) : undefined;
|
||||
let adopted: Copy | null;
|
||||
try { adopted = await store.locked(input.companyId, input.agentId, bound, false, async (tx, _agent, canonical) => {
|
||||
if (remote && !validLeases(await tx.select().from(environmentLeases).where(and(
|
||||
eq(environmentLeases.companyId, input.companyId), inArray(environmentLeases.id, leaseIds))).for("update"))) return null;
|
||||
const current = await inspectAgentDirectory(canonical);
|
||||
const canonicalChanged = agentDirectoryBaselineDigest(current.snapshot) !== agentDirectoryBaselineDigest(baseline(prior));
|
||||
if (canonicalChanged && !input.allowRetirementHandoff) return null;
|
||||
const [claimed] = await tx.update(copies).set({ receipt: { ...prior.receipt, retainedByRunId: input.runId }, updatedAt: new Date() })
|
||||
.where(and(eq(copies.companyId, input.companyId), eq(copies.runId, prior.runId), eq(copies.agentId, input.agentId), eq(copies.state, "unchanged_turn"),
|
||||
eq(copies.updatedAt, prior.updatedAt), eq(copies.baseHash, prior.baseHash), eq(copies.localRoot, prior.localRoot),
|
||||
isNull(copies.processStoppedAt), sql`${copies.receipt}->>'retainedByRunId' IS NULL`)).returning();
|
||||
if (!claimed) throw conflict("Agent directory ownership changed");
|
||||
const { createdAt: _createdAt, updatedAt: _updatedAt, ...values } = prior;
|
||||
const [next] = await tx.insert(copies).values({ ...values, runId: input.runId, responsibleUserId: bound.onBehalfOfUserId!, state: "prepared",
|
||||
receipt: { ...prior.receipt, materializationRunId: prior.receipt?.materializationRunId ?? prior.runId,
|
||||
...(remote ? { cleanup: { leaseId: remote.leaseId, remoteCwd: remote.remoteCwd } } : {}),
|
||||
...(observedChange || canonicalChanged ? { retirementRequired: true } : {}) }, updatedAt: new Date(),
|
||||
}).returning();
|
||||
return next;
|
||||
}); } catch (error) { await replacement?.cleanupWorkspaceSnapshot?.(); throw error; }
|
||||
// Do not move the live transport until the database ownership commit succeeds.
|
||||
if (adopted) {
|
||||
if (replacement && oldRuntime) {
|
||||
const cleanup = replacement.cleanupWorkspaceSnapshot;
|
||||
replacement.cleanupWorkspaceSnapshot = async () => {
|
||||
try { await cleanup?.(); } finally { await oldRuntime.cleanupWorkspaceSnapshot?.(); }
|
||||
};
|
||||
transports.delete(key(prior)); transports.set(key(adopted), replacement);
|
||||
}
|
||||
} else await replacement?.cleanupWorkspaceSnapshot?.();
|
||||
return adopted;
|
||||
}
|
||||
async function checkpoint(row: Copy, target?: AdapterExecutionTarget | null, stopped = false): Promise<Copy> {
|
||||
if (!await owns(row) || row.state === "superseded") return row;
|
||||
@@ -254,6 +356,7 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
|
||||
receipt: { ...row.receipt, storageWarning: storageLimit ? agentStorageWarning(failure instanceof AgentFileLimitError ? failure.message : "Agent folder exceeds a storage limit") : null } });
|
||||
}
|
||||
async function collectStopped(row: Copy, target?: AdapterExecutionTarget | null) {
|
||||
if (row.receipt?.retainedByRunId) return row;
|
||||
if (!await owns(row) || row.state === "superseded") return row;
|
||||
if (row.receipt?.warm === true && !completed.has(row.state)) {
|
||||
row = await checkpoint(row, target, true);
|
||||
@@ -261,7 +364,15 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
|
||||
return (await get(row.companyId, row.runId))!;
|
||||
}
|
||||
if (completed.has(row.state)) { await release(row); return (await get(row.companyId, row.runId))!; }
|
||||
const remoteLease = row.location !== "local" ? await ownedRemoteLease(row) : null;
|
||||
if (row.location !== "local" && (!remoteLease
|
||||
|| remoteLease.releasedAt || remoteLease.status !== "active" || hasRemoteTerminationReceipt(remoteLease))) {
|
||||
return recoverStoppedRemote(row);
|
||||
}
|
||||
row = await patch(row, { processStoppedAt: row.processStoppedAt ?? new Date() });
|
||||
// Adoption may win between the initial read and the stop CAS. The returned
|
||||
// current row is authoritative; a transferred alias cannot inspect or save.
|
||||
if (row.receipt?.retainedByRunId || !row.processStoppedAt) return row;
|
||||
// The stopped working copy is the only temporary tree. Retry transient I/O
|
||||
// at this boundary, then clean it up; there are no preserved run snapshots.
|
||||
let inspected: Awaited<ReturnType<typeof retrieve>> | undefined;
|
||||
@@ -303,22 +414,36 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
|
||||
async function release(row: Copy, target?: AdapterExecutionTarget | null) {
|
||||
if (releaseIsNoop(row) || !await owns(row)) return;
|
||||
const destroyedOnly = row.receipt?.cleanupDestroyedOnly === true;
|
||||
if (row.receipt?.retainedByRunId || row.state === "superseded" || !await owns(row) || !row.processStoppedAt) return;
|
||||
// Collection and environment teardown can both release the same copy.
|
||||
// A compact successful cleanup receipt is final, even after restart.
|
||||
if (completed.has(row.state) && row.processStoppedAt && row.receipt?.cleanupPending === false) return;
|
||||
const runtime = transports.get(key(row));
|
||||
transports.delete(key(row));
|
||||
let cleanupPending = false;
|
||||
await runtime?.cleanupWorkspaceSnapshot?.().catch(() => { cleanupPending = true; });
|
||||
const cleanupTarget = target ?? runtime?.target;
|
||||
if (!destroyedOnly && row.processStoppedAt && completed.has(row.state) && cleanupTarget?.kind === "remote") {
|
||||
const expected = path.posix.join(cleanupTarget.remoteCwd, ".paperclip-runtime", "agent-files", row.agentId, String(row.receipt?.directoryRunId ?? row.runId));
|
||||
const lease = row.location !== "local" ? await ownedRemoteLease(row) : null;
|
||||
const terminated = lease && hasRemoteTerminationReceipt(lease);
|
||||
const destroyed = destroyedOnly || terminated && (lease.metadata?.remoteExecutionTermination as { state?: string } | undefined)?.state === "destroyed";
|
||||
const cleanupLeaseId = (row.receipt?.cleanup as { leaseId?: string } | undefined)?.leaseId;
|
||||
// Maintenance may run while the collector is still cached. A cached target
|
||||
// cannot override the current lease's stopped/destroyed/uncertain state.
|
||||
const remoteUnavailable = row.location !== "local" && !destroyed && (!lease || terminated
|
||||
|| Boolean(lease && (lease.releasedAt || lease.status !== "active"))
|
||||
|| Boolean(cleanupLeaseId && (!lease || cleanupTarget?.kind === "remote" && cleanupTarget.leaseId !== lease.id)));
|
||||
if (remoteUnavailable) cleanupPending = true;
|
||||
else if (!destroyed && row.processStoppedAt && completed.has(row.state) && cleanupTarget?.kind === "remote") {
|
||||
const expected = path.posix.join(cleanupTarget.remoteCwd, ".paperclip-runtime", "agent-files", row.agentId, typeof row.receipt?.materializationRunId === "string" ? row.receipt.materializationRunId : String(row.receipt?.directoryRunId ?? row.runId));
|
||||
if (row.executionRoot !== expected) throw new Error("Agent directory cleanup path changed");
|
||||
const quoted = `'${expected.replaceAll("'", `'"'"'`)}'`;
|
||||
const paths = cleanupTarget.transport === "sandbox" ? [expected, sandboxTransferRoot(row, cleanupTarget.remoteCwd)] : [expected];
|
||||
const quoted = paths.map(p => `'${p.replaceAll("'", `'"'"'`)}'`).join(" ");
|
||||
const remoteCleanupFailed = await runAdapterExecutionTargetShellCommand(row.runId, cleanupTarget, `rm -rf -- ${quoted}`,
|
||||
{ cwd: cleanupTarget.remoteCwd, env: {}, timeoutSec: 15 }).then(result => result.exitCode !== 0 || result.timedOut, () => true);
|
||||
cleanupPending ||= remoteCleanupFailed;
|
||||
} else if (row.processStoppedAt && completed.has(row.state) && row.location !== "local") {
|
||||
// Rebind only the original host-owned lease. Never acquire a replacement
|
||||
// sandbox or persist transport credentials in a working-copy receipt.
|
||||
cleanupPending ||= !await cleanupRemoteAfterRestart(row, destroyedOnly).catch(() => false);
|
||||
} else if (!destroyed && row.processStoppedAt && completed.has(row.state) && row.location !== "local") {
|
||||
// Rebind only the current host-owned lease. Never acquire a replacement.
|
||||
cleanupPending ||= !await cleanupRemoteAfterRestart(row).catch(() => false);
|
||||
}
|
||||
if (completed.has(row.state) && row.processStoppedAt) {
|
||||
try { await fs.rm(path.dirname(row.localRoot), { recursive: true, force: true }); }
|
||||
@@ -328,6 +453,7 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
|
||||
return;
|
||||
}
|
||||
await patch(row, { receipt: { schema: AGENT_FILES_CONTRACT, state: row.state, appliedCandidateHash: row.candidateHash, storageWarning: row.receipt?.storageWarning ?? null, cleanupPending,
|
||||
materializationRunId: row.receipt?.materializationRunId ?? row.runId,
|
||||
...(row.receipt?.directoryRunId ? { directoryRunId: row.receipt.directoryRunId } : {}),
|
||||
...(cleanupPending ? { cleanup: row.receipt?.cleanup, ...(destroyedOnly ? { cleanupDestroyedOnly: true } : {}) } : {}) },
|
||||
nextAttemptAt: cleanupPending ? new Date(Date.now() + 30_000) : null });
|
||||
@@ -345,7 +471,7 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
|
||||
if (lease.environmentId !== null && row.location !== `remote:${lease.environmentId}`) return false;
|
||||
const remoteCwd = cleanup?.remoteCwd ?? lease.metadata?.remoteCwd;
|
||||
return typeof remoteCwd === "string" &&
|
||||
row.executionRoot === path.posix.join(remoteCwd, ".paperclip-runtime", "agent-files", row.agentId, String(row.receipt?.directoryRunId ?? row.runId)) &&
|
||||
row.executionRoot === path.posix.join(remoteCwd, ".paperclip-runtime", "agent-files", row.agentId, String(row.receipt?.materializationRunId ?? row.receipt?.directoryRunId ?? row.runId)) &&
|
||||
hasRemoteTerminationReceipt(lease) && (lease.metadata?.remoteExecutionTermination as { state?: string }).state === "destroyed";
|
||||
}
|
||||
async function recoverUnavailable(row: Copy) {
|
||||
@@ -363,28 +489,50 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
|
||||
if (confirmed.state === "unavailable" && confirmed.processStoppedAt) await release(confirmed);
|
||||
}, "agent_directory_release");
|
||||
}
|
||||
async function cleanupRemoteAfterRestart(row: Copy, destroyedOnly = false): Promise<boolean> {
|
||||
// Newly recovered loss receipts permit no command that could wake a stopped
|
||||
// provider. Recheck the exact destruction binding even after lock acquisition.
|
||||
if (destroyedOnly) return destroyedRemoteCopy(row);
|
||||
const cleanup = row.receipt?.cleanup as { leaseId?: string; remoteCwd?: string } | undefined;
|
||||
// Older preview receipts did not record the lease ID. Accept only a unique
|
||||
// lease still bound to this run and environment in that compatibility case.
|
||||
async function ownedRemoteLease(row: Copy) {
|
||||
const cleanup = row.receipt?.cleanup as { leaseId?: string } | undefined;
|
||||
const leases = await db.select().from(environmentLeases).where(and(
|
||||
eq(environmentLeases.companyId, row.companyId),
|
||||
cleanup?.leaseId ? eq(environmentLeases.id, cleanup.leaseId) : and(
|
||||
eq(environmentLeases.environmentId, row.location.slice("remote:".length)), eq(environmentLeases.heartbeatRunId, row.runId)),
|
||||
eq(environmentLeases.companyId, row.companyId), eq(environmentLeases.heartbeatRunId, row.runId),
|
||||
cleanup?.leaseId ? eq(environmentLeases.id, cleanup.leaseId) : eq(environmentLeases.environmentId, row.location.slice("remote:".length)),
|
||||
));
|
||||
if (leases.length !== 1) return false;
|
||||
if (leases.length !== 1) return null;
|
||||
const lease = leases[0]!;
|
||||
const destroyedWithoutEnvironment = lease.environmentId === null && hasRemoteTerminationReceipt(lease)
|
||||
&& (lease.metadata?.remoteExecutionTermination as { state?: string } | undefined)?.state === "destroyed";
|
||||
if (row.location !== `remote:${lease.environmentId}` && !destroyedWithoutEnvironment) return null;
|
||||
return lease;
|
||||
}
|
||||
async function recoverStoppedRemote(row: Copy) {
|
||||
const lease = await ownedRemoteLease(row);
|
||||
const destroyed = lease && hasRemoteTerminationReceipt(lease)
|
||||
&& (lease.metadata?.remoteExecutionTermination as { state?: string } | undefined)?.state === "destroyed";
|
||||
if (!destroyed) {
|
||||
// execute (including bypassSession) may restart a stopped sandbox. Keep
|
||||
// the only remaining bytes; a stop receipt is not a destruction receipt.
|
||||
return patch(row, { state: "pending_collection", errorCode: lease ? "INSTRUCTION_STOPPED_REMOTE_COLLECTION_PENDING" : "INSTRUCTION_REMOTE_LEASE_UNVERIFIED",
|
||||
errorMessage: lease ? "The remote provider stopped with its agent directory retained. Safe stopped-file retrieval is unavailable; no save or discard is claimed."
|
||||
: "The current remote lease could not be verified. Its agent directory is preserved; no retrieval, save or discard is claimed.", nextAttemptAt: new Date(Date.now() + 30_000) });
|
||||
}
|
||||
row = await patch(row, { state: "unavailable", processStoppedAt: row.processStoppedAt ?? new Date(),
|
||||
errorCode: "INSTRUCTION_COLLECTION_UNAVAILABLE", errorMessage: "The exact remote sandbox was destroyed before its agent directory was retrieved. No instruction save is claimed.", nextAttemptAt: null });
|
||||
await release(row);
|
||||
return (await get(row.companyId, row.runId))!;
|
||||
}
|
||||
async function cleanupRemoteAfterRestart(row: Copy): Promise<boolean> {
|
||||
const cleanup = row.receipt?.cleanup as { leaseId?: string; remoteCwd?: string } | undefined;
|
||||
const lease = await ownedRemoteLease(row);
|
||||
if (!lease) return false;
|
||||
const termination = lease.metadata?.remoteExecutionTermination as { state?: string } | undefined;
|
||||
if (hasRemoteTerminationReceipt(lease) && termination?.state === "destroyed") return true;
|
||||
if (hasRemoteTerminationReceipt(lease)) return termination?.state === "destroyed";
|
||||
// Never execute a cleanup command against a released/uncertain allocation.
|
||||
if (lease.releasedAt || lease.status !== "active") return false;
|
||||
if (!environmentRuntime || !lease.environmentId || row.location !== `remote:${lease.environmentId}`) return false;
|
||||
const [environment] = await db.select().from(environments).where(eq(environments.id, lease.environmentId));
|
||||
if (!environment) return false;
|
||||
const remoteCwd = cleanup?.remoteCwd ?? lease.metadata?.remoteCwd;
|
||||
if (typeof remoteCwd !== "string" || row.executionRoot !== path.posix.join(remoteCwd, ".paperclip-runtime", "agent-files", row.agentId, String(row.receipt?.directoryRunId ?? row.runId))) return false;
|
||||
const result = await environmentRuntime.execute({ environment: environment as Environment, lease: lease as EnvironmentLease, command: "rm", args: ["-rf", "--", row.executionRoot],
|
||||
if (typeof remoteCwd !== "string" || row.executionRoot !== path.posix.join(remoteCwd, ".paperclip-runtime", "agent-files", row.agentId, typeof row.receipt?.materializationRunId === "string" ? row.receipt.materializationRunId : String(row.receipt?.directoryRunId ?? row.runId))) return false;
|
||||
const paths = environment.driver === "sandbox" ? [row.executionRoot, sandboxTransferRoot(row, remoteCwd)] : [row.executionRoot];
|
||||
const result = await environmentRuntime.execute({ environment: environment as Environment, lease: lease as EnvironmentLease, command: "rm", args: ["-rf", "--", ...paths],
|
||||
cwd: remoteCwd, env: {}, timeoutMs: 15_000, bypassSession: true });
|
||||
return result.exitCode === 0 && !result.timedOut;
|
||||
}
|
||||
@@ -397,7 +545,8 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
|
||||
return fn(current);
|
||||
}, process.env, operation);
|
||||
}
|
||||
return { prepare: (input: Parameters<typeof prepareCopy>[0]) => prepareCopy(input), hasChanges, canReuse, recoverUnavailable,
|
||||
return { prepare: (input: Parameters<typeof prepareCopy>[0]) => prepareCopy(input), hasChanges, adopt, canReuse, recoverUnavailable,
|
||||
recoverStoppedRemote: (row: Copy) => serial(row, current => current.receipt?.retainedByRunId ? Promise.resolve(current) : recoverStoppedRemote(current), "agent_directory_collect"),
|
||||
checkpointWarm: (row: Copy, target?: AdapterExecutionTarget | null) => serial(row, current => current.receipt?.warm === true ? checkpoint(current, target) : Promise.resolve(current), "agent_directory_checkpoint"),
|
||||
collectStopped: (row: Copy, target?: AdapterExecutionTarget | null) => serial(row, current => collectStopped(current, target), "agent_directory_collect"),
|
||||
release: async (row: Copy) => {
|
||||
|
||||
@@ -33,6 +33,19 @@ const MAX_COLLECTION_ATTEMPTS = 3;
|
||||
const liveTargets = new Map<string, AdapterExecutionTarget>();
|
||||
const targetKey = (companyId: string, runId: string) => `${companyId}:${runId}`;
|
||||
|
||||
/** Finish only bounded immediate collection work; deferred ownership remains pending. */
|
||||
export async function collectStoppedInstructionCopyWithRetries<T extends Pick<Copy, "state" | "attempts" | "nextAttemptAt">>(collect: () => Promise<T | null>): Promise<T | null> {
|
||||
let saved: T | null = null;
|
||||
let previousAttempts = -1;
|
||||
for (let calls = 0; calls < MAX_COLLECTION_ATTEMPTS; calls++) {
|
||||
saved = await collect();
|
||||
if (!saved || saved.state !== "pending_collection" || saved.nextAttemptAt === null
|
||||
|| saved.attempts >= MAX_COLLECTION_ATTEMPTS || saved.attempts <= previousAttempts) break;
|
||||
previousAttempts = saved.attempts;
|
||||
}
|
||||
return saved;
|
||||
}
|
||||
|
||||
export function instructionWorkingCopyGuidance(copy: Pick<Copy, "executionRoot" | "entryFile" | "receipt">) {
|
||||
if (isAgentDirectoryCopy(copy)) return `Your persistent agent directory is ${copy.executionRoot} (AGENT_HOME). Your instruction entry is ${copy.executionRoot}/${copy.entryFile}. Read and write your own files and subfolders there. This directory belongs to this agent across tasks and sessions; task files belong in the task working directory. Paperclip restores this directory before execution and saves validated changes at turn boundaries. A warm native Codex session keeps the same writable directory between turns. Other sessions collect after the provider stops. Regular files, including binary files, persist; symlinks and special files are unsupported. Check the agent-files save receipt before claiming persistence; Only files you change or delete are synchronized. If another run changes the same file, the last completed synchronization wins. Temporary copies are removed when the owning session stops; there is no per-run file history. Storage allows 256 MiB per file, 2 GiB total, and 100,000 entries; the instruction entry must remain UTF-8 and at most 1 MiB. Reaching a storage limit never prevents this or future tasks from running. Remove or shrink files to free space; changes that exceed the limits will not be saved.${typeof copy.receipt?.storageWarning === "string" ? `\n\n${copy.receipt.storageWarning}` : ""}`;
|
||||
return `Your editable agent instruction file is ${copy.executionRoot}/${copy.entryFile}. Edit this registered private copy normally. After this run stops, Paperclip saves changed content as a persistent revision if your responsible user still has permission and the baseline has not changed. Check the run's instruction-save receipt before claiming persistence. Use read_agent_instructions, update_agent_instructions, get_agent_instruction_history, and restore_agent_instructions for immediate saves and history. Read first and pin the returned revision. Preserve conflicts; never silently retry against a newer head. Repository instructions, skills, and the loaded prompt are separate and are not collected.`;
|
||||
@@ -63,6 +76,7 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen
|
||||
async function patch(row: Copy, values: Partial<typeof copies.$inferInsert>) {
|
||||
const [updated] = await db.update(copies).set({ ...values, updatedAt: new Date() }).where(and(
|
||||
scope(row.companyId, row.runId), eq(copies.state, row.state), eq(copies.attempts, row.attempts), eq(copies.baseHash, row.baseHash),
|
||||
sql`${copies.receipt}->>'retainedByRunId' IS NOT DISTINCT FROM ${typeof row.receipt?.retainedByRunId === "string" ? row.receipt.retainedByRunId : null}`,
|
||||
)).returning();
|
||||
// A late cleanup must not overwrite an explicit resolution or a newer
|
||||
// baseline. Canonical content has its own independent head CAS.
|
||||
@@ -314,12 +328,13 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen
|
||||
async function recoverStopped() {
|
||||
const pending = await db.select({ copy: copies, runtimeMode: heartbeatRuns.runtimeMode }).from(copies)
|
||||
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, copies.companyId), eq(heartbeatRuns.id, copies.runId)))
|
||||
.where(and(or(and(or(inArray(copies.state, ["prepared", "pending_collection", "warm_saved"]),
|
||||
.where(and(or(and(or(inArray(copies.state, ["prepared", "pending_collection", "unchanged_turn", "warm_saved"]),
|
||||
and(eq(copies.state, "preparing"), sql`${copies.receipt}->>'schema' = 'paperclip.agent-files.v1'`)),
|
||||
lte(copies.attempts, MAX_COLLECTION_ATTEMPTS - 1)),
|
||||
and(eq(copies.state, "unavailable"), isNull(copies.processStoppedAt),
|
||||
sql`${copies.location} like 'remote:%'`,
|
||||
sql`${copies.receipt}->>'schema' = 'paperclip.agent-files.v1'`)),
|
||||
sql`NOT (coalesce(${copies.receipt}, '{}'::jsonb) ? 'retainedByRunId')`,
|
||||
inArray(heartbeatRuns.status, ["succeeded", "failed", "cancelled", "timed_out", "interrupted"]),
|
||||
or(isNull(copies.nextAttemptAt), lte(copies.nextAttemptAt, new Date())))).orderBy(asc(copies.updatedAt)).limit(20);
|
||||
for (const { copy: row, runtimeMode } of pending) {
|
||||
@@ -335,6 +350,7 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen
|
||||
const result = await collectStopped({ companyId: row.companyId, runId: row.runId });
|
||||
if (result && isAgentDirectoryCopy(result) && completed.has(result.state)) await directories.release(result);
|
||||
} else if (row.location !== "local" && await remoteExecutionHasStopped(db, row.companyId, row.runId)) {
|
||||
if (row.receipt?.warm === true) {
|
||||
const unavailable = row.receipt?.warm === true
|
||||
? await patch(row, { state: "unavailable", errorCode: "AGENT_FILES_FINAL_COLLECTION_UNAVAILABLE",
|
||||
errorMessage: "The last completed checkpoint remains saved. Files changed afterwards could not be collected after remote termination.", nextAttemptAt: null })
|
||||
@@ -342,10 +358,14 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen
|
||||
if (unavailable && isAgentDirectoryCopy(unavailable)) {
|
||||
await directories.release(await patch(unavailable, { processStoppedAt: unavailable.processStoppedAt ?? new Date() }));
|
||||
}
|
||||
} else {
|
||||
if (isAgentDirectoryCopy(row)) await directories.recoverStoppedRemote(row);
|
||||
else await reportUnavailable(row.companyId, row.runId);
|
||||
}
|
||||
} else if (row.state === "prepared") {
|
||||
await patch(row, { state: "pending_collection", errorCode: "INSTRUCTION_STOP_UNCONFIRMED",
|
||||
errorMessage: "The provider's stop has not been confirmed. Instruction collection is pending; no save is claimed.", nextAttemptAt: null });
|
||||
} else if (row.state === "warm_saved") {
|
||||
errorMessage: "The provider's stop has not been confirmed. Instruction collection is pending; no save is claimed.", nextAttemptAt: new Date(Date.now() + 30_000) });
|
||||
} else if (["warm_saved", "pending_collection", "unchanged_turn"].includes(row.state)) {
|
||||
// Live retained sessions must not occupy every batch and starve stopped
|
||||
// copies from other agents. This does not permit reads without stop proof.
|
||||
await patch(row, { nextAttemptAt: new Date(Date.now() + 30_000) });
|
||||
@@ -379,11 +399,28 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen
|
||||
async function reportUnavailable(companyId: string, runId: string) {
|
||||
const row = await get(companyId, runId);
|
||||
if (!row || completed.has(row.state) || ["unchanged_turn", "warm_saved", "superseded"].includes(row.state) || row.candidateBase64 !== null || (isAgentDirectoryCopy(row) && row.candidateHash !== null) || row.state === "conflict") return row;
|
||||
if (isAgentDirectoryCopy(row) && row.state === "unavailable") return row;
|
||||
// A failed close or deferred lease observation grants no stop proof. Keep
|
||||
// that row recoverable through generic heartbeat cleanup.
|
||||
if (isAgentDirectoryCopy(row) && (row.state === "unavailable"
|
||||
|| row.state === "pending_collection" && !row.processStoppedAt)) return row;
|
||||
return patch(row, { state: "unavailable", errorCode: "INSTRUCTION_COLLECTION_UNAVAILABLE",
|
||||
errorMessage: "The registered instruction copy could not be retrieved safely before environment release. No instruction save is claimed.", nextAttemptAt: null });
|
||||
}
|
||||
|
||||
async function release(companyId: string, runId: string) { liveTargets.delete(targetKey(companyId, runId)); const row = await get(companyId, runId); if (row && isAgentDirectoryCopy(row)) await directories.release(row); }
|
||||
return { prepare, get, hasChanges, checkpointWarm, canReuseWarm: directories.canReuse, acknowledgeExplicitSave, collectStopped, recoverCaptured, recoverStopped, list, resolve, reportUnavailable, release };
|
||||
async function reportRetirementUnconfirmed(companyId: string, runId: string) {
|
||||
const row = await get(companyId, runId);
|
||||
if (!row || !isAgentDirectoryCopy(row) || row.receipt?.retainedByRunId || row.processStoppedAt) return row;
|
||||
return patch(row, { state: "pending_collection", errorCode: "INSTRUCTION_STOP_UNCONFIRMED",
|
||||
errorMessage: "The retained provider did not confirm retirement. Its registered agent directory is preserved; no save is claimed.", nextAttemptAt: null });
|
||||
}
|
||||
async function adopt(input: Parameters<typeof directories.adopt>[0]) {
|
||||
const row = await directories.adopt(input);
|
||||
if (row) {
|
||||
liveTargets.delete(targetKey(input.companyId, input.previousRunId));
|
||||
if (input.target) liveTargets.set(targetKey(input.companyId, input.runId), input.target);
|
||||
}
|
||||
return row;
|
||||
}
|
||||
return { prepare, checkpointWarm, canReuseWarm: directories.canReuse, adopt, reportRetirementUnconfirmed, get, hasChanges, acknowledgeExplicitSave, collectStopped, recoverCaptured, recoverStopped, list, resolve, reportUnavailable, release };
|
||||
}
|
||||
@@ -273,6 +273,8 @@ export function environmentRunOrchestrator(
|
||||
agentId: string;
|
||||
persistedExecutionWorkspace: Pick<ExecutionWorkspace, "id" | "mode"> | null;
|
||||
executionWorkspaceSettings: IssueExecutionWorkspaceSettings | null;
|
||||
/** Existing live runner recovery must use its original active lease, including ephemeral leases. */
|
||||
reattachRemoteLease?: { leaseId: string; providerLeaseId: string; remoteCwd: string };
|
||||
}): Promise<EnvironmentAcquisitionResult> {
|
||||
// Step 1: Resolve environment
|
||||
const selectedEnvironment = await resolveEnvironment({
|
||||
@@ -286,7 +288,7 @@ export function environmentRunOrchestrator(
|
||||
);
|
||||
|
||||
// Step 2: Acquire lease
|
||||
const leaseRecord = await acquireLease({
|
||||
const acquisitionInput = {
|
||||
companyId: input.companyId,
|
||||
environment,
|
||||
issueId: input.issueId,
|
||||
@@ -295,7 +297,29 @@ export function environmentRunOrchestrator(
|
||||
persistedExecutionWorkspace: input.persistedExecutionWorkspace,
|
||||
executionWorkspaceSettings: input.executionWorkspaceSettings,
|
||||
adapterType: input.adapterType ?? null,
|
||||
});
|
||||
};
|
||||
let leaseRecord: EnvironmentRuntimeLeaseRecord;
|
||||
if (input.reattachRemoteLease) {
|
||||
// Reattachment is inspection of an already running sandbox, never a new
|
||||
// acquisition or a lifecycle resume. Those paths can create a replacement
|
||||
// when the admitted per-turn policy deliberately disables reusable leases.
|
||||
const expected = input.reattachRemoteLease;
|
||||
const lease = await environmentsSvc.getLeaseById(expected.leaseId);
|
||||
if (!lease || environment.driver !== "sandbox" ||
|
||||
lease.companyId !== input.companyId || lease.environmentId !== environment.id ||
|
||||
lease.heartbeatRunId !== input.heartbeatRunId || lease.issueId !== input.issueId ||
|
||||
lease.executionWorkspaceId !== (input.persistedExecutionWorkspace?.id ?? null) ||
|
||||
lease.metadata?.agentId !== input.agentId || lease.status !== "active" ||
|
||||
lease.releasedAt !== null || lease.cleanupStatus !== null ||
|
||||
(lease.expiresAt !== null && new Date(lease.expiresAt).getTime() <= Date.now()) ||
|
||||
lease.provider !== environment.config.provider || lease.providerLeaseId !== expected.providerLeaseId ||
|
||||
lease.metadata?.remoteCwd !== expected.remoteCwd) {
|
||||
throw new Error("native_remote_recovery_lease_mismatch");
|
||||
}
|
||||
leaseRecord = { environment, lease, leaseContext: buildEnvironmentLeaseContext(input) };
|
||||
} else {
|
||||
leaseRecord = await acquireLease(acquisitionInput);
|
||||
}
|
||||
|
||||
// Step 3: Log lease acquisition activity
|
||||
await logActivity(db, {
|
||||
|
||||
@@ -3875,6 +3875,86 @@ export function environmentRuntimeService(
|
||||
return driver;
|
||||
}
|
||||
|
||||
async function destroyScopedReusableSandboxLease(input: {
|
||||
environment: Environment;
|
||||
leaseRow: typeof environmentLeases.$inferSelect;
|
||||
scopeCondition?: ReturnType<typeof and>;
|
||||
failureReason: string;
|
||||
}): Promise<EnvironmentLease | null> {
|
||||
const now = new Date();
|
||||
const attemptId = randomUUID();
|
||||
// Persist recovery ownership before any provider work. Re-check scope and
|
||||
// run liveness in this write: the earlier selection cannot fence a resume.
|
||||
// The in-flight lease also excludes concurrent closures and cleanup sweeps.
|
||||
const [claimed] = await db.update(environmentLeases).set({
|
||||
status: "pending_cleanup",
|
||||
failureReason: input.failureReason,
|
||||
cleanupStatus: "failed",
|
||||
releasedAt: now,
|
||||
lastUsedAt: now,
|
||||
updatedAt: now,
|
||||
metadata: sql`(case when jsonb_typeof(${environmentLeases.metadata}) = 'object'
|
||||
then ${environmentLeases.metadata} else '{}'::jsonb end - 'remoteExecutionTermination')
|
||||
|| ${JSON.stringify({ pendingCleanupAttemptId: attemptId, pendingCleanupInFlight: true,
|
||||
pendingCleanupLeaseExpiresAtMs: Date.now() + 15 * 60_000 })}::jsonb`,
|
||||
}).where(and(
|
||||
eq(environmentLeases.id, input.leaseRow.id),
|
||||
eq(environmentLeases.companyId, input.leaseRow.companyId),
|
||||
eq(environmentLeases.environmentId, input.environment.id),
|
||||
eq(environmentLeases.leasePolicy, "reuse_by_environment"),
|
||||
eq(environmentLeases.status, input.leaseRow.status),
|
||||
input.scopeCondition,
|
||||
sql`coalesce(${environmentLeases.metadata}->>'pendingCleanupInFlight', 'false') != 'true'`,
|
||||
sql`NOT EXISTS (
|
||||
SELECT 1 FROM ${heartbeatRuns}
|
||||
WHERE ${heartbeatRuns.id} = ${environmentLeases.heartbeatRunId}
|
||||
AND ${heartbeatRuns.status} IN ('queued', 'scheduled_retry', 'running')
|
||||
)`,
|
||||
)).returning();
|
||||
if (!claimed) return null;
|
||||
const lease = toEnvironmentLeaseSnapshot(claimed);
|
||||
let renewalInFlight = false;
|
||||
const renewal = setInterval(() => {
|
||||
if (renewalInFlight) return;
|
||||
renewalInFlight = true;
|
||||
void db.update(environmentLeases).set({
|
||||
metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({
|
||||
pendingCleanupLeaseExpiresAtMs: Date.now() + 15 * 60_000,
|
||||
})}::jsonb`,
|
||||
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.status, "pending_cleanup"),
|
||||
sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${attemptId}`,
|
||||
sql`${environmentLeases.metadata}->>'pendingCleanupInFlight' = 'true'`,
|
||||
)).then(() => {}).catch(() => {
|
||||
logger.warn({ leaseId: lease.id }, "scoped cleanup ownership renewal failed");
|
||||
}).finally(() => { renewalInFlight = false; });
|
||||
}, 30_000);
|
||||
renewal.unref();
|
||||
try {
|
||||
const driver = getDriver(getLeaseDriverKey(lease, input.environment));
|
||||
if (!driver?.destroyRunLease) return lease;
|
||||
return await driver.destroyRunLease({
|
||||
environment: input.environment,
|
||||
lease,
|
||||
failureReason: input.failureReason,
|
||||
}) ?? lease;
|
||||
} catch {
|
||||
// Even a failed settlement write leaves the durable provider handle for
|
||||
// the sweep. Continue with later leases without logging provider errors.
|
||||
return lease;
|
||||
} finally {
|
||||
clearInterval(renewal);
|
||||
// A crash skips this write; the bounded ownership lease lets the sweep
|
||||
// recover. A completed failure becomes eligible for a later explicit retry.
|
||||
await db.update(environmentLeases).set({
|
||||
metadata: sql`${environmentLeases.metadata} || '{"pendingCleanupInFlight":false}'::jsonb`,
|
||||
}).where(and(eq(environmentLeases.id, lease.id),
|
||||
sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${attemptId}`,
|
||||
)).catch(() => {
|
||||
logger.warn({ leaseId: lease.id }, "scoped cleanup ownership settlement failed");
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
getDriver,
|
||||
|
||||
@@ -4180,7 +4260,7 @@ export function environmentRuntimeService(
|
||||
and(
|
||||
eq(environmentLeases.companyId, input.companyId),
|
||||
eq(environmentLeases.leasePolicy, "reuse_by_environment"),
|
||||
inArray(environmentLeases.status, ["active", "released", "retained", "pending_cleanup"]),
|
||||
inArray(environmentLeases.status, ["active", "released", "retained", "failed", "pending_cleanup"]),
|
||||
...scopeConditions,
|
||||
),
|
||||
);
|
||||
@@ -4222,18 +4302,12 @@ export function environmentRuntimeService(
|
||||
? await environmentsSvc.getById(leaseRow.environmentId)
|
||||
: null;
|
||||
if (!environment) continue;
|
||||
const leaseSnapshot = toEnvironmentLeaseSnapshot(leaseRow);
|
||||
const driver = getDriver(getLeaseDriverKey(leaseSnapshot, environment));
|
||||
const lease = driver?.destroyRunLease
|
||||
? await driver.destroyRunLease({
|
||||
environment,
|
||||
lease: leaseSnapshot,
|
||||
failureReason: input.failureReason ?? "reusable_lease_destroyed",
|
||||
})
|
||||
: await environmentsSvc.releaseLease(leaseSnapshot.id, "pending_cleanup", {
|
||||
failureReason: input.failureReason ?? "reusable_lease_destroyed",
|
||||
cleanupStatus: "failed",
|
||||
});
|
||||
const lease = await destroyScopedReusableSandboxLease({
|
||||
environment,
|
||||
leaseRow,
|
||||
scopeCondition: and(...scopeConditions),
|
||||
failureReason: input.failureReason ?? "reusable_lease_destroyed",
|
||||
});
|
||||
if (!lease) continue;
|
||||
destroyed.push({
|
||||
environment,
|
||||
@@ -4271,7 +4345,9 @@ export function environmentRuntimeService(
|
||||
and(
|
||||
eq(environmentLeases.environmentId, input.environmentId),
|
||||
eq(environmentLeases.leasePolicy, "reuse_by_environment"),
|
||||
inArray(environmentLeases.status, ["active", "released", "retained"]),
|
||||
// A failed run may have stopped its reusable sandbox without
|
||||
// destroying it; that provider handle still needs scoped teardown.
|
||||
inArray(environmentLeases.status, ["active", "released", "retained", "failed"]),
|
||||
),
|
||||
);
|
||||
|
||||
@@ -4302,73 +4378,20 @@ export function environmentRuntimeService(
|
||||
let failed = 0;
|
||||
let skippedLiveRun = 0;
|
||||
const failureReason = input.failureReason ?? "environment_delete_requested";
|
||||
const now = new Date();
|
||||
for (const leaseRow of leaseRows) {
|
||||
if (leaseRow.heartbeatRunId && liveRunIds.has(leaseRow.heartbeatRunId)) {
|
||||
skippedLiveRun += 1;
|
||||
continue;
|
||||
}
|
||||
// Claim the row BEFORE the provider call, mirroring the inline-teardown
|
||||
// invariant used elsewhere in this file: no provider destroy without a
|
||||
// durable `pending_cleanup` reference already on disk. The claim is one
|
||||
// conditional UPDATE, so it is the fence against a racing resume: a
|
||||
// resume that re-activates the lease first makes the status predicate
|
||||
// (or the run-liveness predicate) fail and the claim loses — the live
|
||||
// run keeps its sandbox. A claim that wins parks the lease where the
|
||||
// cleanup sweep owns it, so a crash or thrown destroy after this point
|
||||
// is recovered by the sweep's idempotent teardown, and a double write
|
||||
// failure cannot strand the lease in a reusable status.
|
||||
const claimedRow = await db
|
||||
.update(environmentLeases)
|
||||
.set({
|
||||
status: "pending_cleanup",
|
||||
failureReason,
|
||||
cleanupStatus: "failed",
|
||||
releasedAt: now,
|
||||
lastUsedAt: now,
|
||||
updatedAt: now,
|
||||
})
|
||||
.where(
|
||||
and(
|
||||
eq(environmentLeases.id, leaseRow.id),
|
||||
inArray(environmentLeases.status, ["active", "released", "retained"]),
|
||||
sql`NOT EXISTS (
|
||||
SELECT 1 FROM ${heartbeatRuns}
|
||||
WHERE ${heartbeatRuns.id} = ${environmentLeases.heartbeatRunId}
|
||||
AND ${heartbeatRuns.status} IN ('queued', 'scheduled_retry', 'running')
|
||||
)`,
|
||||
),
|
||||
)
|
||||
.returning()
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (!claimedRow) {
|
||||
const lease = await destroyScopedReusableSandboxLease({ environment, leaseRow, failureReason });
|
||||
if (!lease) {
|
||||
// Lost to a racing resume or a concurrent terminal transition — the
|
||||
// lease is no longer ours to destroy.
|
||||
skippedLiveRun += 1;
|
||||
continue;
|
||||
}
|
||||
const leaseSnapshot = toEnvironmentLeaseSnapshot(claimedRow);
|
||||
try {
|
||||
const driver = getDriver(getLeaseDriverKey(leaseSnapshot, environment));
|
||||
if (!driver?.destroyRunLease) {
|
||||
// No driver available: the claim already parked the lease for the
|
||||
// sweep, which retries once the driver's plugin is back.
|
||||
failed += 1;
|
||||
continue;
|
||||
}
|
||||
const lease = await driver.destroyRunLease({
|
||||
environment,
|
||||
lease: leaseSnapshot,
|
||||
failureReason,
|
||||
});
|
||||
if (lease && lease.status !== "pending_cleanup") destroyed += 1;
|
||||
else failed += 1;
|
||||
} catch {
|
||||
// The claim above already parked the lease in `pending_cleanup`, so
|
||||
// the sweep owns the retry; its teardown is idempotent, so a destroy
|
||||
// that reached the provider before the throw resolves as success.
|
||||
failed += 1;
|
||||
}
|
||||
if (lease.status !== "pending_cleanup") destroyed += 1;
|
||||
else failed += 1;
|
||||
}
|
||||
return { destroyed, failed, skippedLiveRun };
|
||||
},
|
||||
|
||||
@@ -1179,8 +1179,9 @@ export function environmentService(db: Db) {
|
||||
where ${environmentLeases.environmentId} = ${environments.id}
|
||||
and ${environmentLeases.status} = 'pending_cleanup'
|
||||
)`,
|
||||
// A reusable lease keeps a live provider sandbox after a run
|
||||
// releases it. Deleting the environment would set its reference to
|
||||
// A reusable lease keeps a provider sandbox after a run releases
|
||||
// it, including when the run failed but release succeeded.
|
||||
// Deleting the environment would set its reference to
|
||||
// null, and both the normal release path and scoped reusable cleanup
|
||||
// require that environment context. Refuse the delete atomically
|
||||
// until the owning issue/workspace destroys the reusable sandbox.
|
||||
@@ -1188,7 +1189,7 @@ export function environmentService(db: Db) {
|
||||
select 1 from ${environmentLeases}
|
||||
where ${environmentLeases.environmentId} = ${environments.id}
|
||||
and ${environmentLeases.leasePolicy} = 'reuse_by_environment'
|
||||
and ${environmentLeases.status} in ('active', 'released', 'retained')
|
||||
and ${environmentLeases.status} in ('active', 'released', 'retained', 'failed')
|
||||
)`,
|
||||
),
|
||||
)
|
||||
@@ -1303,7 +1304,7 @@ export function environmentService(db: Db) {
|
||||
and(
|
||||
eq(environmentLeases.environmentId, id),
|
||||
eq(environmentLeases.leasePolicy, "reuse_by_environment"),
|
||||
inArray(environmentLeases.status, ["active", "released", "retained"]),
|
||||
inArray(environmentLeases.status, ["active", "released", "retained", "failed"]),
|
||||
),
|
||||
),
|
||||
db
|
||||
|
||||
@@ -456,12 +456,12 @@ describe("Paperclip Runner native provider configuration", () => {
|
||||
).toThrow("provider changed after this run selected its native backend");
|
||||
});
|
||||
|
||||
it("rejects Pi before a native descriptor is persisted", () => {
|
||||
expect(() =>
|
||||
it("preserves a caller-selected Pi model in the native descriptor", () => {
|
||||
expect(
|
||||
resolvePaperclipRunnerNativeProviderInput({
|
||||
backend: "acpx_runtime",
|
||||
adapterConfig: { provider: "acpx", acpxAgent: "pi", model: "pi-model" },
|
||||
}),
|
||||
).toThrow("Pi is awaiting local and Daytona qualification");
|
||||
).toMatchObject({ provider: "acpx", acpxAgent: "pi", model: "pi-model" });
|
||||
});
|
||||
});
|
||||
@@ -230,7 +230,7 @@ import {
|
||||
configuredEnvironmentProjection,
|
||||
} from "../vendor/paperclip-runner/index.js";
|
||||
import { decisionModelService } from "./decision-models.js";
|
||||
import { activeIssueInteractionCondition } from "./issue-question-context.js";
|
||||
import { activeIssueInteractionCondition, TASK_QUESTION_GUIDANCE } from "./issue-question-context.js";
|
||||
import { createAgentIdentityRedactor } from "./agent-identity-redaction.js";
|
||||
import { agentIdentityService, supportsManagedAgentIdentity } from "./agent-identity.js";
|
||||
import { buildAgentIdentityEnv } from "@paperclipai/adapter-utils/server-utils";
|
||||
@@ -350,8 +350,7 @@ import {
|
||||
prepareGitHubExecutionEnvironment,
|
||||
startAdapterExecutionTargetPaperclipBridge,
|
||||
} from "@paperclipai/adapter-utils/execution-target";
|
||||
|
||||
import { agentInstructionWorkingCopyService, instructionWorkingCopyGuidance } from "./agent-instruction-working-copies.js";
|
||||
import { agentInstructionWorkingCopyService, collectStoppedInstructionCopyWithRetries, instructionWorkingCopyGuidance } from "./agent-instruction-working-copies.js";
|
||||
import {
|
||||
resolveManagedOpenAiBilling,
|
||||
} from "@paperclipai/adapter-utils";
|
||||
@@ -463,6 +462,8 @@ import {
|
||||
buildNativeExecutionWithCheckpoint,
|
||||
buildNativeRuntimeContext,
|
||||
cancelNativeSession,
|
||||
claimWarmNativeInstructionCopy,
|
||||
nativeSessionWorkspaceScope,
|
||||
closeWarmNativeSessionsForEnvironment,
|
||||
reserveWarmNativeInstructionDirectory,
|
||||
dispatchNativeSessionResumptions,
|
||||
@@ -484,6 +485,7 @@ import {
|
||||
nativeToolContractFingerprintForTarget,
|
||||
prepareNativeSessionBootstrapPersistence,
|
||||
prepareNativeWorkspaceSync,
|
||||
readNativeWorkspaceSyncReference,
|
||||
recordNativeFinalizationFailure,
|
||||
type NativeRestartRecoveryClaim,
|
||||
rebindNativeSessionCheckpoint,
|
||||
@@ -6656,6 +6658,7 @@ export function heartbeatService(
|
||||
Parameters<typeof cleanupGitHubOperationLaunchers>[0] | null = null;
|
||||
let nativeSessionResumeScheduled = false;
|
||||
let nativeOwnershipHeld = false;
|
||||
let releaseWarmInstructionPreparation: (() => Promise<void>) | null = null;
|
||||
let nativeInstructionReservation: Awaited<ReturnType<typeof reserveWarmNativeInstructionDirectory>> = null;
|
||||
let nativeDispatchStarted = false;
|
||||
let nativeWorkspaceFinalizeScheduled = false;
|
||||
@@ -8907,6 +8910,15 @@ export function heartbeatService(
|
||||
>;
|
||||
try {
|
||||
await controllerLease.assertOwned();
|
||||
const remoteRecovery = runOptions.nativeRestartRecovery?.kind === "reattach_remote_runner"
|
||||
? runOptions.nativeRestartRecovery : null;
|
||||
const recoveryWorkspace = remoteRecovery
|
||||
? readNativeWorkspaceSyncReference(parseObject(run.runnerProfileJson).nativeWorkspaceSync) : null;
|
||||
if (remoteRecovery && (!recoveryWorkspace || remoteRecovery.runId !== run.id ||
|
||||
recoveryWorkspace.providerLeaseId !== remoteRecovery.remote.providerLeaseId ||
|
||||
recoveryWorkspace.remoteCwd !== remoteRecovery.remote.remoteCwd)) {
|
||||
throw new Error("native_remote_recovery_lease_mismatch");
|
||||
}
|
||||
acquiredEnvironment = await envOrchestrator.acquireForRun({
|
||||
companyId: agent.companyId,
|
||||
selectedEnvironmentId,
|
||||
@@ -8919,6 +8931,11 @@ export function heartbeatService(
|
||||
agentId: agent.id,
|
||||
persistedExecutionWorkspace,
|
||||
executionWorkspaceSettings: environmentExecutionWorkspaceSettings,
|
||||
...(remoteRecovery && recoveryWorkspace ? { reattachRemoteLease: {
|
||||
leaseId: recoveryWorkspace.leaseId,
|
||||
providerLeaseId: remoteRecovery.remote.providerLeaseId,
|
||||
remoteCwd: remoteRecovery.remote.remoteCwd,
|
||||
} } : {}),
|
||||
});
|
||||
await controllerLease.assertOwned();
|
||||
nativeRunnerPreparationSpans.push({
|
||||
@@ -9041,6 +9058,13 @@ export function heartbeatService(
|
||||
const executionTarget = realizationResult.executionTarget;
|
||||
let instructionCopy: Awaited<ReturnType<typeof instructionCopies.prepare>> = null;
|
||||
let instructionSave: Record<string, unknown> | null = null;
|
||||
const instructionPreparationKey = createHash("sha256").update(JSON.stringify({
|
||||
adapterType: agent.adapterType, adapterConfig: agent.adapterConfig, runtimeConfig: agent.runtimeConfig, sessionConfigMetadata,
|
||||
workspace: nativeSessionWorkspaceScope({
|
||||
binding: { runId: run.id, executionWorkspaceId: persistedExecutionWorkspace?.id ?? run.id },
|
||||
workspace: executionWorkspace,
|
||||
}), cwd: executionWorkspace.cwd,
|
||||
})).digest("hex");
|
||||
const recordInstructionSave = async (saved: NonNullable<Awaited<ReturnType<typeof instructionCopies.get>>>) => {
|
||||
const receipt = parseObject(saved.receipt);
|
||||
const storageWarning = readNonEmptyString(receipt.storageWarning);
|
||||
@@ -9057,15 +9081,24 @@ export function heartbeatService(
|
||||
};
|
||||
const collectStoppedInstructions = async () => {
|
||||
if (!instructionCopy) return;
|
||||
let saved = await instructionCopies.collectStopped({ companyId: agent.companyId, runId: run.id, target: executionTarget });
|
||||
// Capture before disposal. Exhausted bounded collection leaves a durable
|
||||
// explicit loss report, never a claim that missing bytes were saved.
|
||||
while (saved?.state === "pending_collection" && saved.attempts < 3) {
|
||||
saved = await instructionCopies.collectStopped({ companyId: agent.companyId, runId: run.id, target: executionTarget });
|
||||
}
|
||||
const saved = await collectStoppedInstructionCopyWithRetries(() => instructionCopies.collectStopped({
|
||||
companyId: agent.companyId, runId: run.id, target: executionTarget,
|
||||
}));
|
||||
if (!saved) return;
|
||||
if (saved.state !== "superseded") await recordInstructionSave(saved);
|
||||
};
|
||||
const nativeInstructionWorkingCopy = () => instructionCopy ? {
|
||||
...(isAgentDirectoryCopy(instructionCopy) ? { runId: run.id, preparationKey: instructionPreparationKey } : {}),
|
||||
root: instructionCopy.executionRoot,
|
||||
...(instructionCopy.receipt?.warm === true ? { checkpointWarm: async () => {
|
||||
const saved = await instructionCopies.checkpointWarm({ companyId: agent.companyId, runId: run.id, target: executionTarget });
|
||||
if (saved) await recordInstructionSave(saved);
|
||||
return saved?.state === "warm_saved" && saved.errorCode === null;
|
||||
} } : {}),
|
||||
hasChanges: () => instructionCopies.hasChanges({ companyId: agent.companyId, runId: run.id, target: executionTarget }),
|
||||
collectStopped: collectStoppedInstructions,
|
||||
retirementFailed: async () => { await instructionCopies.reportRetirementUnconfirmed(agent.companyId, run.id); },
|
||||
} : undefined;
|
||||
if (managedAiRuntime && aiBinding) {
|
||||
try { await assertManagedAiProjectAuth({ ...resolvedConfig, cwd: executionWorkspace.cwd }, aiBinding.provider, executionTarget); }
|
||||
catch { throw new ConfigurationIncompleteFailure("Project authentication conflicts with this agent’s managed AI connection", { configurationIncomplete: { reason: "ai_connection_incompatible", actionUrl: `/agents/${agent.id}/runtime` } }); }
|
||||
@@ -9913,9 +9946,9 @@ export function heartbeatService(
|
||||
try {
|
||||
// Missing contract fields on a restored session mean the deployed
|
||||
// legacy format. New sessions opt into whole-directory persistence.
|
||||
const priorFileRun = taskSessionForRun?.lastRunId
|
||||
const priorFileRun = taskSession?.lastRunId
|
||||
? await db.select({ profile: heartbeatRuns.runnerProfileJson }).from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.id, taskSessionForRun.lastRunId), eq(heartbeatRuns.companyId, agent.companyId), eq(heartbeatRuns.agentId, agent.id))).then(rows => rows[0])
|
||||
eq(heartbeatRuns.id, taskSession.lastRunId), eq(heartbeatRuns.companyId, agent.companyId), eq(heartbeatRuns.agentId, agent.id))).then(rows => rows[0])
|
||||
: null;
|
||||
const savedFileInput = parseObject(parseObject(run.runnerProfileJson).nativeExecutionInput);
|
||||
const priorFileInput = Object.keys(savedFileInput).length ? savedFileInput : parseObject(parseObject(priorFileRun?.profile).nativeExecutionInput);
|
||||
@@ -9931,6 +9964,21 @@ export function heartbeatService(
|
||||
canReuse: () => instructionCopies.canReuseWarm(agent.companyId, agent.id, taskSessionForRun!.lastRunId!),
|
||||
});
|
||||
}
|
||||
if (!warmFiles && nativeRuntimeResolution.kind === "native" && priorWorkingCopy.kind === "agent_files" && taskSession) {
|
||||
releaseWarmInstructionPreparation = await claimWarmNativeInstructionCopy({
|
||||
priorExecution: parseNativeExecutionInput(priorFileInput), companyId: agent.companyId, agentId: agent.id,
|
||||
executionWorkspaceId: persistedExecutionWorkspace?.id ?? run.id, workspace: executionWorkspace,
|
||||
environmentId: executionTarget?.environmentId ?? null, runId: run.id, preparationKey: instructionPreparationKey, forceRetirement: !taskSessionForRun,
|
||||
adopt: async previousRunId => {
|
||||
instructionCopy = await instructionCopies.adopt({ companyId: agent.companyId, agentId: agent.id,
|
||||
runId: run.id, previousRunId, target: executionTarget, cwd: executionWorkspace.cwd, allowRetirementHandoff: true });
|
||||
return nativeInstructionWorkingCopy() ?? null;
|
||||
},
|
||||
});
|
||||
// A collection-only handoff was retired. Re-enter normal
|
||||
// preparation instead of composing a root already collected.
|
||||
if (!releaseWarmInstructionPreparation) instructionCopy = null;
|
||||
}
|
||||
const prepareInstructions = (reuseRunId?: string) => instructionCopies.prepare({
|
||||
companyId: agent.companyId, agentId: agent.id, runId: run.id,
|
||||
target: executionTarget, cwd: executionWorkspace.cwd,
|
||||
@@ -9942,7 +9990,7 @@ export function heartbeatService(
|
||||
},
|
||||
});
|
||||
try {
|
||||
instructionCopy = await prepareInstructions(nativeInstructionReservation?.reuseRunId);
|
||||
instructionCopy ??= await prepareInstructions(nativeInstructionReservation?.reuseRunId);
|
||||
} catch (error) {
|
||||
if (!(error instanceof AgentDirectoryReuseInvalidatedError)) throw error;
|
||||
// prepare has released its canonical lock. Retirement can now
|
||||
@@ -11157,17 +11205,7 @@ export function heartbeatService(
|
||||
},
|
||||
onLog,
|
||||
onEvent: onAdapterEvent,
|
||||
instructionWorkingCopy: instructionCopy ? {
|
||||
runId: run.id,
|
||||
root: instructionCopy.executionRoot,
|
||||
...(instructionCopy.receipt?.warm === true ? { checkpointWarm: async () => {
|
||||
const saved = await instructionCopies.checkpointWarm({ companyId: agent.companyId, runId: run.id, target: executionTarget });
|
||||
if (saved) await recordInstructionSave(saved);
|
||||
return saved?.state === "warm_saved" && saved.errorCode === null;
|
||||
} } : {}),
|
||||
hasChanges: () => instructionCopies.hasChanges({ companyId: agent.companyId, runId: run.id, target: executionTarget }),
|
||||
collectStopped: collectStoppedInstructions,
|
||||
} : undefined,
|
||||
instructionWorkingCopy: nativeInstructionWorkingCopy(),
|
||||
|
||||
onUsage: async receipt => { await usageRecorder.capture(receipt); },
|
||||
preparationSpans: nativeRunnerPreparationSpans,
|
||||
@@ -13113,6 +13151,7 @@ export function heartbeatService(
|
||||
}
|
||||
// A retained or unverified process stays above this release boundary.
|
||||
// If no stopped-copy capture occurred, preserve an explicit loss report.
|
||||
await releaseWarmInstructionPreparation?.();
|
||||
const uncapturedInstructions = await instructionCopies.reportUnavailable(run.companyId, run.id);
|
||||
if (uncapturedInstructions?.state === "unavailable") {
|
||||
await appendRunEvent(run, { eventType: "instruction_save", stream: "system", level: "warn",
|
||||
|
||||
@@ -89,18 +89,20 @@ describe("hot-restart path compatibility", () => {
|
||||
).not.toBeNull();
|
||||
});
|
||||
|
||||
it("reads Linux process start time from proc metadata", async () => {
|
||||
it("reads Linux process birth from kernel ticks rather than proc directory metadata", async () => {
|
||||
await expect(
|
||||
readProcessStartedAt(123, {
|
||||
platform: "linux",
|
||||
stat: async (target) => {
|
||||
expect(target).toBe("/proc/123");
|
||||
return {
|
||||
ctimeMs: Date.parse("2026-08-01T01:00:00.123Z"),
|
||||
};
|
||||
linuxProcessStart: {
|
||||
clockTicksPerSecond: 100,
|
||||
readFile: (target) => target === "/proc/stat"
|
||||
? `btime ${Date.parse("2026-08-01T01:00:00.000Z") / 1000}\n`
|
||||
: target === "/proc/123/stat"
|
||||
? `123 (runner) S ${Array(18).fill("0").join(" ")} 123\n`
|
||||
: (() => { throw new Error("Unexpected proc read"); })(),
|
||||
},
|
||||
}),
|
||||
).resolves.toBe("2026-08-01T01:00:00.123Z");
|
||||
).resolves.toBe("2026-08-01T01:00:01.230Z");
|
||||
});
|
||||
|
||||
it("reads macOS process start time through ps", async () => {
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { execFile } from "node:child_process";
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { readLinuxProcessStartedAt, type LinuxProcessStartOptions } from "../vendor/paperclip-runner/index.js";
|
||||
import {
|
||||
resolvePaperclipHomeDir,
|
||||
resolvePaperclipInstanceId,
|
||||
@@ -14,7 +15,6 @@ const HOT_RESTART_LOCK_STALE_MS = 30_000;
|
||||
const HOT_RESTART_LOCK_TIMEOUT_MS = 10_000;
|
||||
|
||||
type ProcessCommandRunner = (command: string, args: string[]) => Promise<string>;
|
||||
type ProcessStatReader = (target: string) => Promise<{ ctimeMs: number }>;
|
||||
|
||||
export type HotRestartIntentRun = {
|
||||
runId: string;
|
||||
@@ -178,17 +178,15 @@ export async function readProcessStartedAt(
|
||||
pid: number,
|
||||
options: {
|
||||
platform?: NodeJS.Platform;
|
||||
stat?: ProcessStatReader;
|
||||
linuxProcessStart?: LinuxProcessStartOptions;
|
||||
runCommand?: ProcessCommandRunner;
|
||||
} = {},
|
||||
) {
|
||||
const platform = options.platform ?? process.platform;
|
||||
const stat = options.stat ?? fs.stat;
|
||||
const runCommand = options.runCommand ?? runProcessCommand;
|
||||
|
||||
if (platform === "linux") {
|
||||
const processStat = await stat(`/proc/${pid}`);
|
||||
return new Date(processStat.ctimeMs).toISOString();
|
||||
return readLinuxProcessStartedAt(pid, options.linuxProcessStart);
|
||||
}
|
||||
|
||||
if (["darwin", "freebsd", "openbsd", "aix", "sunos"].includes(platform)) {
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { normalizeEscapedLineBreaks } from "@paperclipai/shared/validators/text";
|
||||
import { activeIssueInteractionCondition, historicalQuestionCondition } from "./issue-question-context.js";
|
||||
import {
|
||||
currentContinuationOrigins,
|
||||
@@ -1667,6 +1668,7 @@ function resolveRequestItemVerdictSubmissions(args: {
|
||||
|
||||
function normalizeQuestionAnswers(args: {
|
||||
questions: AskUserQuestionsInteraction["payload"]["questions"];
|
||||
textQuestionIds?: ReadonlySet<string>;
|
||||
answers: RespondIssueThreadInteraction["answers"];
|
||||
}) {
|
||||
const questionById = new Map(
|
||||
@@ -1701,7 +1703,11 @@ function normalizeQuestionAnswers(args: {
|
||||
);
|
||||
}
|
||||
|
||||
const otherText = answer.otherText?.trim() ?? "";
|
||||
// Canonical text fields may contain code or intentional whitespace. The
|
||||
// legacy custom-answer path keeps its historical newline/trim behavior.
|
||||
const otherText = args.textQuestionIds?.has(answer.questionId)
|
||||
? answer.otherText ?? ""
|
||||
: normalizeEscapedLineBreaks(answer.otherText ?? "").trim();
|
||||
answerByQuestionId.set(answer.questionId, {
|
||||
questionId: answer.questionId,
|
||||
optionIds: uniqueOptionIds,
|
||||
@@ -1713,7 +1719,7 @@ function normalizeQuestionAnswers(args: {
|
||||
const answer = answerByQuestionId.get(question.id);
|
||||
if (
|
||||
question.required &&
|
||||
(!answer || (answer.optionIds.length === 0 && !answer.otherText))
|
||||
(!answer || (answer.optionIds.length === 0 && !answer.otherText?.trim()))
|
||||
) {
|
||||
throw unprocessable(`Question ${question.id} requires an answer`);
|
||||
}
|
||||
@@ -4957,6 +4963,9 @@ export function issueThreadInteractionService(
|
||||
) as AskUserQuestionsInteraction;
|
||||
const normalizedAnswers = normalizeQuestionAnswers({
|
||||
questions: interaction.payload.questions,
|
||||
textQuestionIds: new Set((interaction.payload.questionSet?.questions ?? [])
|
||||
.filter((question) => question.answerMode === "text")
|
||||
.map((question) => question.id)),
|
||||
answers: input.answers,
|
||||
});
|
||||
if (interaction.payload.questionSet) {
|
||||
|
||||
@@ -8,7 +8,7 @@ const provider = { kind: "acpx", agent: "copilot", model: "exact-model", permiss
|
||||
const authorize = (value: unknown) => ({ [ACPX_QUALIFICATION_ENV]: JSON.stringify(value) });
|
||||
describe("host ACPX qualification admission", () => {
|
||||
afterEach(() => vi.unstubAllEnvs());
|
||||
it.each(["copilot", "pi"])("admits %s through agent validation and native input only for the exact host pair", (agent) => {
|
||||
it.each(["copilot"])("admits %s through agent validation and native input only for the exact host pair", (agent) => {
|
||||
const config = { provider: "acpx", acpxAgent: agent, model: "exact-model" };
|
||||
vi.stubEnv(ACPX_QUALIFICATION_ENV, undefined);
|
||||
expect(() => resolvePaperclipRunnerProviderProfile(config)).toThrow(expect.objectContaining({ code: "paperclip_runner_acpx_agent_unavailable" }));
|
||||
@@ -19,6 +19,21 @@ describe("host ACPX qualification admission", () => {
|
||||
expect(() => resolvePaperclipRunnerProviderProfile({ ...config, model: "other-model" })).toThrow(expect.objectContaining({ code: "paperclip_runner_acpx_qualification_invalid" }));
|
||||
expect(() => resolvePaperclipRunnerProviderProfile({ ...config, model: "" })).toThrow(expect.objectContaining({ code: "paperclip_runner_acpx_model_required" }));
|
||||
});
|
||||
it("admits only the exact Pi model without host authorization and preserves permission modes", () => {
|
||||
const config = { provider: "acpx", acpxAgent: "pi", model: "openrouter/deepseek/deepseek-v4-flash-0731" };
|
||||
for (const authorization of [undefined, "malformed unrelated candidate authorization"]) {
|
||||
vi.stubEnv(ACPX_QUALIFICATION_ENV, authorization);
|
||||
expect(resolvePaperclipRunnerProviderProfile(config)).toMatchObject(config);
|
||||
for (const acpxPermissionMode of [undefined, "approve-all", "approve-paperclip", "approve-reads", "deny-all"]) {
|
||||
expect(resolvePaperclipRunnerNativeProviderInput({ backend: "acpx_runtime", adapterConfig: { ...config, acpxPermissionMode } }))
|
||||
.toMatchObject({ ...config, acpxPermissionMode: acpxPermissionMode ?? "approve-all" });
|
||||
}
|
||||
for (const model of [undefined, ""]) {
|
||||
expect(() => resolvePaperclipRunnerProviderProfile({ ...config, model }))
|
||||
.toThrow(expect.objectContaining({ code: "paperclip_runner_acpx_model_required" }));
|
||||
}
|
||||
}
|
||||
});
|
||||
it("keeps normal candidate execution closed and admits only the exact operator pair", () => {
|
||||
expect(resolveAcpxQualification(provider, {})).toBeUndefined();
|
||||
expect(resolveAcpxQualification(provider, authorize([{ agent: "copilot", model: "exact-model" }]))).toBe("copilot");
|
||||
@@ -43,7 +58,7 @@ describe("host ACPX qualification admission", () => {
|
||||
it("does not alter existing qualified providers", () => {
|
||||
expect(resolveAcpxQualification({ ...provider, agent: "codex" } as typeof provider, authorize([]))).toBeUndefined();
|
||||
});
|
||||
it.each(["claude", "codex", "grok", "cursor"])("keeps %s model selection open to native verification", (acpxAgent) => {
|
||||
it.each(["claude", "codex", "grok", "cursor", "pi"])("keeps %s model selection open to native verification", (acpxAgent) => {
|
||||
const adapterConfig = { provider: "acpx", acpxAgent, model: "explicit-new-model" };
|
||||
expect(resolvePaperclipRunnerProviderProfile(adapterConfig))
|
||||
.toMatchObject({ acpxAgent, model: "explicit-new-model" });
|
||||
@@ -68,3 +83,9 @@ describe("host ACPX qualification admission", () => {
|
||||
expect(source).not.toContain("resolveAcpxQualification(input.execution.provider, effectiveRunnerEnvironment)");
|
||||
});
|
||||
});
|
||||
|
||||
it.each([undefined, "off", "low", "high", "max"] as const)("validates saved Pi thinking %s and projects it explicitly", piThinkingLevel => {
|
||||
const adapterConfig = { provider: "acpx", acpxAgent: "pi", model: "openrouter/deepseek/deepseek-v4-flash-0731", piThinkingLevel };
|
||||
expect(resolvePaperclipRunnerNativeProviderInput({ backend: "acpx_runtime", adapterConfig })).toMatchObject({ piThinkingLevel: piThinkingLevel ?? "low" });
|
||||
for (const alias of ["medium", "minimal", "xhigh"]) expect(() => resolvePaperclipRunnerProviderProfile({ ...adapterConfig, piThinkingLevel: alias })).toThrow(expect.objectContaining({ code: "paperclip_runner_pi_thinking_invalid" }));
|
||||
});
|
||||
@@ -3,6 +3,13 @@ import { inheritNativeRunnerAdapterConfig } from "./native-agent-runtime-inherit
|
||||
import { resolvePaperclipRunnerProviderProfile } from "./provider-profile.js";
|
||||
|
||||
describe("native hire runtime inheritance", () => {
|
||||
it("preserves Cursor's selected mode without its live session or credentials", () => {
|
||||
expect(inheritNativeRunnerAdapterConfig({
|
||||
provider: "acpx", acpxAgent: "cursor", acpxSessionMode: "plan", model: "exact-cursor-model",
|
||||
runtimeSessionId: "parent-session", env: { CURSOR_API_KEY: "parent-secret" },
|
||||
})).toEqual({ provider: "acpx", acpxAgent: "cursor", acpxSessionMode: "plan", model: "exact-cursor-model" });
|
||||
});
|
||||
|
||||
it.each([
|
||||
["claude_managed", "managedProfileId", "managedAgentsRetentionAcknowledged"],
|
||||
["aws_agentcore", "agentCoreProfileId", "agentCoreRetentionAcknowledged"],
|
||||
@@ -40,3 +47,7 @@ describe("native hire runtime inheritance", () => {
|
||||
.not.toHaveProperty("managedProfileId");
|
||||
});
|
||||
});
|
||||
|
||||
it("preserves Pi thinking configuration without live identity or credentials", () => {
|
||||
expect(inheritNativeRunnerAdapterConfig({ provider: "acpx", acpxAgent: "pi", piThinkingLevel: "low", runtimeSessionId: "prior", env: { OPENROUTER_API_KEY: "canary" } })).toEqual({ provider: "acpx", acpxAgent: "pi", piThinkingLevel: "low" });
|
||||
});
|
||||
@@ -14,6 +14,7 @@ export const INHERITABLE_NATIVE_RUNNER_CONFIG_KEYS = [
|
||||
"opencodePermissionMode",
|
||||
"acpxPermissionMode",
|
||||
"acpxSessionMode",
|
||||
"piThinkingLevel",
|
||||
"lifecycleMode",
|
||||
"modelReasoningEffort",
|
||||
"maxIterations",
|
||||
|
||||
@@ -12,6 +12,23 @@ describe("explicit file output requirements", () => {
|
||||
"Make a file but do not send it to anyone else.",
|
||||
"Export a summary of this PDF as CSV.",
|
||||
"Create no temporary files; export the results as CSV.",
|
||||
"Write report.pdf. This is an internal verification file, not a deliverable. Also export the results as CSV.",
|
||||
"Write a downloadable report.pdf. This is personal memory, not a task deliverable.",
|
||||
"Attempt native write to report.txt with content requested.",
|
||||
"Write report.pdf and checklist.md. This is an internal assertion file, not a deliverable.",
|
||||
"Write report.pdf; write internal-proof.txt. This is an internal verification file, not a deliverable.",
|
||||
"Write report.pdf and attempt native write to /outside/probe.txt; this negative test must be denied.",
|
||||
"Attempt native write to /outside/probe.json and write report.pdf; this negative test must be denied.",
|
||||
"Write report.txt and attach it. This is an internal verification file, not a deliverable.",
|
||||
"Attach it.",
|
||||
"Write report.txt and return it as an attachment. This is an internal verification file, not a deliverable.",
|
||||
"Write report.txt and send it to me. This is an internal verification file, not a deliverable.",
|
||||
"Write report.txt. This is an internal verification file, not a deliverable. Send it to me.",
|
||||
"Write report.txt and provide it to me. This is an internal verification file, not a deliverable.",
|
||||
"Write report.txt and give me it. This is an internal verification file, not a deliverable.",
|
||||
"Write report.txt and return it. This is an internal verification file, not a deliverable.",
|
||||
"Write report.txt and send it as an attachment in your response. This is an internal verification file, not a deliverable.",
|
||||
"Write report.txt. This is an internal verification file, not a deliverable. Send it as a download link in your response.",
|
||||
])("recognizes an explicit output request: %s", objective => {
|
||||
expect(explicitlyRequestsFileOutput(objective)).toBe(true);
|
||||
});
|
||||
@@ -31,6 +48,20 @@ describe("explicit file output requirements", () => {
|
||||
"Create no files.",
|
||||
"Generate no attachments and answer in chat.",
|
||||
"Write a reply without any files.",
|
||||
"Use native write/read file tools for this task, not bash or the instructions API.",
|
||||
"Write a memory entry to memory/pi-native.txt inside the registered AGENT_HOME. This is personal memory, not a task deliverable.",
|
||||
"Use native write to copy those exact bytes into pi-agent-memory-proof.txt in the task workspace. This is an internal assertion file, not a deliverable.",
|
||||
"Before finishing, attempt native write exactly once to /outside/pi-unassigned.txt with content forbidden. This intentionally unassigned root must be denied.",
|
||||
"Write internal-proof.txt; then check it exists. This is an internal verification file, not a deliverable.",
|
||||
"Attempt native write to /outside/probe.txt and create no files. This negative test must be denied.",
|
||||
"Write internal-proof.txt; do not attach it. This is an internal verification file, not a deliverable.",
|
||||
"Write a reply and return it in chat.",
|
||||
"Write internal-proof.txt and return it in chat. This is an internal verification file, not a deliverable.",
|
||||
"Write internal-proof.txt and send it in chat. This is an internal verification file, not a deliverable.",
|
||||
"Write internal-proof.txt. This is an internal verification file, not a deliverable. Return it inline.",
|
||||
"Write internal-proof.txt and send it as a code block in your response. This is an internal verification file, not a deliverable.",
|
||||
"Write internal-proof.txt and return it in my reply. This is an internal verification file, not a deliverable.",
|
||||
"Write internal-proof.txt and provide it as plain text. This is an internal verification file, not a deliverable.",
|
||||
])("does not require a file for a text or source-review request: %s", objective => {
|
||||
expect(explicitlyRequestsFileOutput(objective)).toBe(false);
|
||||
});
|
||||
|
||||
@@ -63,26 +63,59 @@ async function hasCurrentPublicationReceipt(db: Db, companyId: string, receipts:
|
||||
* an output requirement or erase a user's request for a file.
|
||||
*/
|
||||
export function explicitlyRequestsFileOutput(objective: string): boolean {
|
||||
return objective.split(/(?:[.!?](?:\s|$)|\n|[;,]|\bbut\b)/iu).some(clause => {
|
||||
const sentences = objective.replaceAll("\\_", "_").split(/(?:[.!?](?:\s|$)|\n)/iu);
|
||||
let precedingFileOutput = false;
|
||||
return sentences.some((sentence, index) => {
|
||||
const file = /\b(?:files?|attachments?|downloads?|pdf|spreadsheets?|workbooks?|slide decks?|powerpoints?|docx|xlsx|csv)\b|\b[^\s/]+\.(?:md|txt|pdf|docx?|xlsx?|csv|pptx?|png|jpe?g|svg|zip)\b/giu;
|
||||
const create = /\b(?:create|make|write|save|export|attach|send|generate|produce|prepare|provide|give|return|build)\b/iu.exec(clause);
|
||||
if (create && /\b(?:do not|don't|never|no need to)\s*$/iu.test(clause.slice(0, create.index))) return false;
|
||||
const output = create ? clause.slice(create.index + create[0].length) : "";
|
||||
const fileObject = [...output.matchAll(file)].some(match => {
|
||||
const prefix = output.slice(0, match.index);
|
||||
const suffix = output.slice(match.index + match[0].length);
|
||||
// "Create no files" is a prohibition, even though it contains a creation
|
||||
// verb. Negate this object only; another explicit output can still count.
|
||||
if (/\b(?:no|zero|without(?:\s+any)?)\s+(?:(?:new|temporary|downloadable|attached|additional)\s+)*$/iu.test(prefix)) return false;
|
||||
// "Write a summary of this PDF" names input, not a requested file.
|
||||
// Explicit export destinations still count after such input references.
|
||||
const destination = /\b(?:as|into|to)\s+(?:(?:a|an|the|new|separate|markdown|word|excel)\s+)*$/iu.test(prefix);
|
||||
if (!destination && /\b(?:of|about|on|from|using|for|with)\b/iu.test(prefix)) return false;
|
||||
if (/^files?$/iu.test(match[0]) && /^\s+(?:permissions?|systems?|formats?|names?|paths?|types?|sizes?|descriptors?)\b/iu.test(suffix)) return false;
|
||||
return true;
|
||||
const outputs = sentence.split(/(?:[;,]|\bbut\b)/iu).flatMap(clause => {
|
||||
const creates = [...clause.matchAll(/\b(?:create|make|write|save|export|attach|send|generate|produce|prepare|provide|give|return|build)\b/giu)];
|
||||
return creates.flatMap((create, createIndex) => {
|
||||
const before = clause.slice(0, create.index);
|
||||
if (/\b(?:do not|don't|never|no need to)\s*$/iu.test(before)) return [];
|
||||
// Bind each object to its own verb. A denied write or "create no files"
|
||||
// cannot suppress a separate requested report in this same clause.
|
||||
const output = clause.slice(create.index + create[0].length, creates[createIndex + 1]?.index);
|
||||
const objects = [...output.matchAll(file)].filter(match => {
|
||||
const prefix = output.slice(0, match.index);
|
||||
const suffix = output.slice(match.index + match[0].length);
|
||||
// "Create no files" is a prohibition, even though it contains a creation
|
||||
// verb. Negate this object only; another explicit output can still count.
|
||||
if (/\b(?:no|zero|without(?:\s+any)?)\s+(?:(?:new|temporary|downloadable|attached|additional)\s+)*$/iu.test(prefix)) return false;
|
||||
// "Write a summary of this PDF" names input, not a requested file.
|
||||
// Explicit export destinations still count after such input references.
|
||||
const destination = /\b(?:as|into|to)\s+(?:(?:a|an|the|new|separate|markdown|word|excel)\s+)*$/iu.test(prefix);
|
||||
if (!destination && /\b(?:of|about|on|from|using|for|with)\b/iu.test(prefix)) return false;
|
||||
if (/^files?$/iu.test(match[0]) && /^\s+(?:tools?|permissions?|systems?|formats?|names?|paths?|types?|sizes?|descriptors?)\b/iu.test(suffix)) return false;
|
||||
return true;
|
||||
});
|
||||
// An explicit attachment/export request can refer to the file by
|
||||
// pronoun. It still requires publication when its name is omitted.
|
||||
const referencesOutput = /^(?:attach|export|send|provide|give|return)$/iu.test(create[0])
|
||||
&& /^\s+(?:me\s+)?(?:it|them|this|that)\b/iu.test(output);
|
||||
const referencedAttachment = referencesOutput && /^(?:attach|export)$/iu.test(create[0]);
|
||||
const inline = /\b(?:inline|(?:in|within|inside|as|into)\s+(?:(?:a|an|the|my|your|our|final|plain|markdown|chat|fenced)\s+)*(?:chat|response|reply|message|comment|text|code block)|(?:its|the) contents)\b/iu.test(output);
|
||||
const downloadable = referencedAttachment || (!/\b(?:no|without)\s+(?:downloadable|attached)/iu.test(output)
|
||||
&& /\b(?:downloadable|attached)\s+(?:file|report|document|checklist|draft)\b/iu.test(output));
|
||||
const publication = /\b(?:downloadable|attached|attach|export|send|provide|return|give)\b/iu.test(create[0] + output);
|
||||
const deniedAttempt = create[0].toLowerCase() === "write" && objects.length === 1
|
||||
&& /\b(?:attempt|try)\s+(?:the\s+)?native\s*$/iu.test(before)
|
||||
&& /\b(?:must be denied|denial is (?:the )?expected|(?:this|the) negative test)\b/iu.test(objective);
|
||||
if (objects.length === 0 && !downloadable && /^\s+(?:no|zero|without)\b/iu.test(output)) return [];
|
||||
return [{ objects: objects.length, downloadable, publication, deniedAttempt,
|
||||
referencesOutput, publicationReference: referencesOutput && !inline }];
|
||||
});
|
||||
});
|
||||
// "This ..." qualifies a single requested file in the preceding sentence,
|
||||
// even when a later clause checks it. Multiple files cannot share this
|
||||
// exception and separate report requests still require publication.
|
||||
const internal = outputs.reduce((count, output) => count + output.objects + Number(output.downloadable && output.objects === 0), 0) === 1
|
||||
&& /^\s*This is (?:an? )?(?:personal memory|internal (?:assertion|verification) file)\b[^.!?]*\bnot a (?:task )?deliverable\b/iu.test(sentences[index + 1] ?? "");
|
||||
return outputs.some(output => {
|
||||
const publicationReference = output.publicationReference && precedingFileOutput;
|
||||
if (!output.referencesOutput) precedingFileOutput = output.objects > 0;
|
||||
return (output.objects > 0 || output.downloadable || publicationReference)
|
||||
&& (output.publication || (!internal && !output.deniedAttempt));
|
||||
});
|
||||
return fileObject ||
|
||||
(!/\b(?:no|without)\s+(?:downloadable|attached)/iu.test(clause) && /\b(?:downloadable|attached)\s+(?:file|report|document|checklist|draft)\b/iu.test(clause));
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -585,3 +585,52 @@ describe("native completion references", () => {
|
||||
expect(input.completionContract.contract).toEqual(args.completionContract.contract);
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
describe("Cursor mode native execution projection", () => {
|
||||
function fixture(acpxSessionMode?: "agent" | "plan" | "ask") {
|
||||
return {
|
||||
companyId: "company-1", runId: "run-1", agentId: "agent-1",
|
||||
issue: { id: "issue-1", identifier: "MODE-1", title: "Review a plan", description: null, workMode: "standard" },
|
||||
taskPrompt: "Review the project.",
|
||||
workspace: { id: "workspace-1", cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null },
|
||||
normalizedSessionId: null, provider: "acpx" as const, acpxAgent: "cursor" as const,
|
||||
model: "exact-cursor-model", acpxSessionMode, acpxPermissionMode: "deny-all" as const,
|
||||
completionContract: { id: "contract-1", sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1", contract: { revision: "1", objective: "Review", criteria: [{ id: "output", requirement: "Review" }] } },
|
||||
runtimeContext: nativeRuntimeContextFixture(),
|
||||
};
|
||||
}
|
||||
it.each([undefined, "agent", "plan", "ask"] as const)("preserves mode %s independently of permissions and task planning", mode => {
|
||||
const result = buildNativeExecutionInput(fixture(mode));
|
||||
expect(result.provider).toMatchObject({ kind: "acpx", agent: "cursor", mode: mode ?? "agent", permissionMode: "deny-all" });
|
||||
expect(result.executionMode).toBe("default");
|
||||
expect(result.task.workMode).toBe("standard");
|
||||
});
|
||||
it("rejects a Cursor mode attached to another harness", () => {
|
||||
expect(() => buildNativeExecutionInput({ ...fixture("plan"), acpxAgent: "copilot" })).toThrow("only for Cursor");
|
||||
});
|
||||
});
|
||||
|
||||
describe("Pi thinking native execution projection", () => {
|
||||
function fixture(piThinkingLevel?: "off" | "low" | "high" | "max") {
|
||||
return {
|
||||
companyId: "company-1", runId: "run-1", agentId: "agent-1",
|
||||
issue: { id: "issue-1", identifier: "MODE-1", title: "Review a plan", description: null, workMode: "standard" },
|
||||
taskPrompt: "Review the project.",
|
||||
workspace: { id: "workspace-1", cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null },
|
||||
normalizedSessionId: null, provider: "acpx" as const, acpxAgent: "pi" as const,
|
||||
model: "openrouter/deepseek/deepseek-v4-flash-0731", piThinkingLevel, acpxPermissionMode: "deny-all" as const,
|
||||
completionContract: { id: "contract-1", sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1", contract: { revision: "1", objective: "Review", criteria: [{ id: "output", requirement: "Review" }] } },
|
||||
runtimeContext: nativeRuntimeContextFixture(),
|
||||
};
|
||||
}
|
||||
it.each([undefined, "off", "low", "high", "max"] as const)("preserves mode %s independently of permissions and task planning", mode => {
|
||||
const result = buildNativeExecutionInput(fixture(mode));
|
||||
expect(result.provider).toMatchObject({ kind: "acpx", agent: "pi", piThinkingLevel: mode ?? "low", permissionMode: "deny-all" });
|
||||
expect(result.executionMode).toBe("default");
|
||||
expect(result.task.workMode).toBe("standard");
|
||||
});
|
||||
it("rejects a Pi thinking level attached to another harness", () => {
|
||||
expect(() => buildNativeExecutionInput({ ...fixture("low"), acpxAgent: "copilot" })).toThrow("only for Pi");
|
||||
});
|
||||
});
|
||||
@@ -1,5 +1,5 @@
|
||||
import type { PaperclipTurnContext } from "@paperclipai/adapter-utils/server-utils";
|
||||
import { resolvePaperclipRunnerCursorMode } from "@paperclipai/adapter-utils";
|
||||
import { resolvePaperclipRunnerCursorMode, resolvePaperclipRunnerPiThinkingLevel } from "@paperclipai/adapter-utils";
|
||||
import { createHash } from "node:crypto";
|
||||
import { buildNativeContinuationPrompt } from "./native-continuation.js";
|
||||
import type {
|
||||
@@ -77,6 +77,7 @@ export interface BuildNativeExecutionInput {
|
||||
opencodePermissionMode?: NativeOpenCodePermissionMode;
|
||||
acpxPermissionMode?: NativeAcpxPermissionMode;
|
||||
acpxSessionMode?: "agent" | "plan" | "ask";
|
||||
piThinkingLevel?: "off" | "low" | "high" | "max";
|
||||
model?: string | null;
|
||||
managedProfile?: Extract<
|
||||
NativeExecutionInputV5["provider"],
|
||||
@@ -113,6 +114,7 @@ export function buildNativeExecutionInput(input: BuildNativeExecutionInput): Nat
|
||||
throw new Error("native_execution_input_invalid: issue work mode must be standard, planning, or ask");
|
||||
}
|
||||
const mode = resolvePaperclipRunnerCursorMode(input.provider, input.acpxAgent, input.acpxSessionMode);
|
||||
const piThinkingLevel = resolvePaperclipRunnerPiThinkingLevel(input.provider, input.acpxAgent, input.piThinkingLevel);
|
||||
const executionMode = input.executionMode
|
||||
?? (input.issue.workMode === "planning" ? "plan" : "default");
|
||||
const acpxProfile = input.provider === "acpx"
|
||||
@@ -276,6 +278,7 @@ export function buildNativeExecutionInput(input: BuildNativeExecutionInput): Nat
|
||||
model: input.model,
|
||||
permissionMode: input.acpxPermissionMode ?? "approve-all",
|
||||
...(mode === undefined ? {} : { mode }),
|
||||
...(piThinkingLevel === undefined ? {} : { piThinkingLevel }),
|
||||
profile: {
|
||||
driverKind: acpxProfile!.driverKind,
|
||||
protocolVersion: acpxProfile!.protocolVersion,
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
import { randomBytes, randomUUID, timingSafeEqual } from "node:crypto";
|
||||
import { createServer } from "node:http";
|
||||
import path from "node:path";
|
||||
import {
|
||||
cleanupGitHubOperationLaunchers,
|
||||
githubOperationLauncherDirectory,
|
||||
prepareGitHubOperationLaunchers,
|
||||
startAdapterExecutionTargetPaperclipBridge,
|
||||
} from "@paperclipai/adapter-utils/execution-target";
|
||||
@@ -98,7 +98,7 @@ export async function createNativeGitHubAccess(input: {
|
||||
bridge = await startBridge({
|
||||
...location,
|
||||
runtimeRootDir: input.target?.kind === "remote"
|
||||
? path.posix.join(input.target.remoteCwd, ".paperclip-runtime", "github", location.runId)
|
||||
? githubOperationLauncherDirectory(location)
|
||||
: null,
|
||||
adapterKey: "native-github",
|
||||
hostApiToken: token,
|
||||
|
||||
@@ -23,6 +23,7 @@ import {
|
||||
nativeRunFinalizations,
|
||||
} from "@paperclipai/db";
|
||||
import type { PrpEvent } from "@paperclipai/paperclip-runner";
|
||||
import { respondIssueThreadInteractionSchema } from "@paperclipai/shared";
|
||||
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
@@ -442,6 +443,54 @@ describeEmbeddedPostgres("native question bridge", () => {
|
||||
release();
|
||||
});
|
||||
|
||||
it.each(["legacy", "canonical_select"])("keeps %s custom-answer normalization and rejects blank public answers", async (mode) => {
|
||||
await seed();
|
||||
const event = runtimeRequestEvent();
|
||||
const input = (event.payload.request as { input: { questions: Record<string, unknown>[] } }).input;
|
||||
input.questions[0]!.customAnswer = { enabled: true };
|
||||
const interaction = await projectNativeRuntimeRequest({ db, binding: binding(), event });
|
||||
if (mode === "legacy") {
|
||||
const payload = { ...interaction!.payload };
|
||||
if ("questionSet" in payload) delete payload.questionSet;
|
||||
await db.update(issueThreadInteractions).set({ payload }).where(eq(issueThreadInteractions.id, interaction!.id));
|
||||
}
|
||||
const service = issueThreadInteractionService(db);
|
||||
await expect(service.answerQuestions({ id: issueId, companyId, status: "in_progress" }, interaction!.id,
|
||||
respondIssueThreadInteractionSchema.parse({ answers: [{ questionId: "color", optionIds: [], otherText: " \n " }] }), { userId: "operator-1" },
|
||||
)).rejects.toThrow(/requires an answer/);
|
||||
const answered = await service.answerQuestions({ id: issueId, companyId, status: "in_progress" }, interaction!.id,
|
||||
respondIssueThreadInteractionSchema.parse({ answers: [{ questionId: "color", optionIds: [], otherText: " Extra\\nline " }] }), { userId: "operator-1" },
|
||||
);
|
||||
expect(answered.result).toMatchObject({ answers: [{ questionId: "color", otherText: "Extra\nline" }] });
|
||||
});
|
||||
|
||||
it("persists initial text without resolving and delivers only an explicit edited answer", async () => {
|
||||
await seed();
|
||||
const initialText = " Editable draft\n漢字\n";
|
||||
const event = runtimeRequestEvent();
|
||||
const request = event.payload.request as Record<string, unknown>;
|
||||
request.input = { schema: "paperclip.question_set.v1", questions: [{ id: "draft", prompt: "Edit", required: true, answerMode: "text", initialText }] };
|
||||
const interaction = await projectNativeRuntimeRequest({ db, binding: binding(), event });
|
||||
expect(interaction).toMatchObject({ status: "pending", payload: { questionSet: { questions: [{ initialText }] } } });
|
||||
const [stored] = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interaction!.id));
|
||||
expect(stored).toMatchObject({ status: "pending", result: null, payload: { questionSet: { questions: [{ initialText }] } } });
|
||||
const queueCommand = vi.fn(() => ({ commandId: "edited", controllerSeq: 1 }));
|
||||
const release = registerNativeQuestionCommandTarget({ binding: { companyId, issueId, runId, agentId }, queueCommand });
|
||||
try {
|
||||
await flushNativeQuestionResponses(db, runId);
|
||||
expect(queueCommand).not.toHaveBeenCalled();
|
||||
const edited = "\n Operator 漢字 edited\n\nLiteral \\n stays literal. \n";
|
||||
await issueThreadInteractionService(db).answerQuestions(
|
||||
{ id: issueId, companyId, status: "in_progress" }, interaction!.id,
|
||||
respondIssueThreadInteractionSchema.parse({ answers: [{ questionId: "draft", optionIds: [], otherText: edited }] }), { userId: "operator-1" },
|
||||
);
|
||||
await flushNativeQuestionResponses(db, runId);
|
||||
expect(queueCommand).toHaveBeenCalledExactlyOnceWith("request.resolve", {
|
||||
requestId: "request-1", response: { schema: "paperclip.question_response.v1", answers: { draft: { text: edited } } },
|
||||
}, `question_${interaction!.id}`);
|
||||
} finally { release(); }
|
||||
});
|
||||
|
||||
it.each(["succeeded", "failed", "cancelled", "timed_out"])("delivers a historical answer exactly once after a %s native run", async status => {
|
||||
await seed();
|
||||
await db.update(issues).set({ conversationAgentId: agentId, conversationUserId: "operator-1", conversationState: "active", conversationSessionGeneration: 1, executionRunId: null }).where(eq(issues.id, issueId));
|
||||
|
||||
@@ -11,7 +11,7 @@ import { conversationNativeDecision, isConversation } from "../agent-conversatio
|
||||
import { issueTreeControlService } from "../issue-tree-control.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { preserveNativeWorkspaceExportLease } from "./native-workspace-export-resume.js";
|
||||
import { and, eq, inArray, isNotNull, isNull, or, sql } from "drizzle-orm";
|
||||
import { and, desc, eq, inArray, isNotNull, isNull, or, sql } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import {
|
||||
approvals,
|
||||
@@ -20,6 +20,7 @@ import {
|
||||
heartbeatRuns,
|
||||
heartbeatRunEvents,
|
||||
issueApprovals,
|
||||
issueComments,
|
||||
issueRecoveryActions,
|
||||
issueThreadInteractions,
|
||||
issues,
|
||||
@@ -62,7 +63,11 @@ import {
|
||||
import { logger } from "../../middleware/logger.js";
|
||||
import {
|
||||
CHAT_RUN_PRESENTATION_AUTHORIZATION_REASON,
|
||||
findHeartbeatRunCompletionComment,
|
||||
isExternalChatPresentationContext,
|
||||
readCompletedAssistantMessageCandidate,
|
||||
resolveHeartbeatRunResponse,
|
||||
selectHeartbeatRunFinalAgentMessage,
|
||||
} from "../heartbeat-run-summary.js";
|
||||
import { resolveChatRunPresentationAuthorizationReason } from "../chat-run-publications.js";
|
||||
import {
|
||||
@@ -940,13 +945,6 @@ export async function repairCommittedNativeChatResponse(
|
||||
!accepted ||
|
||||
!decision ||
|
||||
result.schema !== "paperclip.run_result.v1" ||
|
||||
result.reportedWorkDisposition !== "yielded" ||
|
||||
record(result.continuation).kind !== "response_wake" ||
|
||||
!Array.isArray(result.attentionRequests) ||
|
||||
result.attentionRequests.length > 0 ||
|
||||
terminal.runTerminalState !== "succeeded" ||
|
||||
terminal.turnTerminalState !== "completed" ||
|
||||
terminal.reportedWorkDisposition !== "yielded" ||
|
||||
!(await acceptedResponseDigestMatches(tx, run, accepted))
|
||||
)
|
||||
return false;
|
||||
@@ -954,6 +952,70 @@ export async function repairCommittedNativeChatResponse(
|
||||
// presentation contract, including selected attachments. Do not bypass it.
|
||||
if (record(decision.decisionJson).externalChatReviewPresentation)
|
||||
return false;
|
||||
// Copyback can be owned by reconciliation before the live heartbeat
|
||||
// reaches its presentation step. Recover a completed internal task's
|
||||
// exact final reply from the same accepted turn, without rerunning work
|
||||
// or granting external-chat publication authority.
|
||||
if (
|
||||
!isConversation(issue) &&
|
||||
run.status === "succeeded" &&
|
||||
record(run.contextSnapshot).skipIssueComment !== true &&
|
||||
!isExternalChatPresentationContext(run.contextSnapshot) &&
|
||||
result.reportedWorkDisposition === "done" &&
|
||||
terminal.runTerminalState === "succeeded" &&
|
||||
terminal.turnTerminalState === "completed" &&
|
||||
terminal.reportedWorkDisposition === "done" &&
|
||||
(await resolveChatRunPresentationAuthorizationReason(tx, input)) === "internal_agent_write"
|
||||
) {
|
||||
const rows = await tx.select({ seq: heartbeatRunEvents.seq, payload: heartbeatRunEvents.payload })
|
||||
.from(heartbeatRunEvents).where(and(
|
||||
eq(heartbeatRunEvents.companyId, input.companyId),
|
||||
eq(heartbeatRunEvents.runId, run.id),
|
||||
eq(heartbeatRunEvents.eventType, "item.completed"),
|
||||
sql`${heartbeatRunEvents.payload}->'prpEvent'->>'turnId' = ${accepted.turnId}`,
|
||||
sql`${heartbeatRunEvents.payload} #>> '{prpEvent,payload,kind}' = 'agentMessage'`,
|
||||
sql`${heartbeatRunEvents.payload} #>> '{prpEvent,payload,channel}' = 'final'`,
|
||||
)).orderBy(desc(heartbeatRunEvents.seq)).limit(200);
|
||||
const finalAgentMessage = selectHeartbeatRunFinalAgentMessage({
|
||||
candidates: rows.flatMap((row) => {
|
||||
const candidate = readCompletedAssistantMessageCandidate({ seq: row.seq, prpEvent: record(row.payload).prpEvent });
|
||||
return candidate ? [candidate] : [];
|
||||
}),
|
||||
});
|
||||
if (!finalAgentMessage) return false;
|
||||
const comments = await tx.select({ id: issueComments.id, body: issueComments.body })
|
||||
.from(issueComments).where(and(
|
||||
eq(issueComments.companyId, input.companyId),
|
||||
eq(issueComments.issueId, input.issueId),
|
||||
eq(issueComments.createdByRunId, run.id),
|
||||
)).orderBy(desc(issueComments.createdAt), desc(issueComments.id));
|
||||
const resolved = resolveHeartbeatRunResponse({
|
||||
resultJson: { ...record(run.resultJson), nativeResult: result },
|
||||
existingComment: findHeartbeatRunCompletionComment(comments, run.resultJson),
|
||||
finalAgentMessage,
|
||||
});
|
||||
if (!resolved.text || resolved.decision.commentAction !== "create") return false;
|
||||
const comment = await issueService(db).addComment(input.issueId, resolved.text,
|
||||
{ agentId: run.agentId, runId: run.id },
|
||||
{ authorizationReason: "internal_agent_write", completionReply: true }, tx);
|
||||
await tx.update(heartbeatRuns).set({
|
||||
resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({
|
||||
presentationDecision: { ...resolved.decision, commentId: comment.id,
|
||||
reasonCodes: [...resolved.decision.reasonCodes, "committed_task_response_recovered"] },
|
||||
})}::jsonb`, updatedAt: new Date(),
|
||||
}).where(eq(heartbeatRuns.id, run.id));
|
||||
agentId = run.agentId;
|
||||
return true;
|
||||
}
|
||||
if (
|
||||
result.reportedWorkDisposition !== "yielded" ||
|
||||
record(result.continuation).kind !== "response_wake" ||
|
||||
!Array.isArray(result.attentionRequests) ||
|
||||
result.attentionRequests.length > 0 ||
|
||||
terminal.runTerminalState !== "succeeded" ||
|
||||
terminal.turnTerminalState !== "completed" ||
|
||||
terminal.reportedWorkDisposition !== "yielded"
|
||||
) return false;
|
||||
await authorizeCommittedChatResponse(db, tx, {
|
||||
...input,
|
||||
agentId: run.agentId,
|
||||
|
||||
@@ -8,6 +8,8 @@ import {
|
||||
link,
|
||||
mkdir,
|
||||
readFile,
|
||||
readdir,
|
||||
rm,
|
||||
symlink,
|
||||
unlink,
|
||||
writeFile,
|
||||
@@ -39,6 +41,7 @@ import { issueService } from "../issues.js";
|
||||
import { findHeartbeatRunCompletionComment, resolveHeartbeatRunResponse } from "../heartbeat-run-summary.js";
|
||||
import {
|
||||
renderNativeRunnerStagedAttachmentPrompt,
|
||||
stageNativeRunnerAttachmentBytes,
|
||||
stageNativeRunnerWakeAttachments,
|
||||
} from "./native-runner-file-handoff.js";
|
||||
import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js";
|
||||
@@ -195,6 +198,91 @@ describe("native runner file handoff", () => {
|
||||
};
|
||||
}
|
||||
|
||||
async function stageEmptyWake(root: string, boundCompanyId = companyId) {
|
||||
await db.update(heartbeatRuns)
|
||||
.set({ contextSnapshot: { issueId } })
|
||||
.where(eq(heartbeatRuns.id, runId));
|
||||
return stageNativeRunnerWakeAttachments({
|
||||
db,
|
||||
binding: {
|
||||
companyId: boundCompanyId, issueId, runId, agentId,
|
||||
workspaceRoot: root, executionTargetKind: "local",
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
it("leaves an empty workspace unchanged for an empty wake and permits later explicit staging", async () => {
|
||||
const root = await mkdtemp(path.join(temporaryRoot, "empty-wake-"));
|
||||
try {
|
||||
const wake = await stageEmptyWake(root);
|
||||
expect(wake.attachments).toEqual([]);
|
||||
await wake.cleanup();
|
||||
await expect(readdir(root)).resolves.toEqual([]);
|
||||
|
||||
const explicit = await stageNativeRunnerAttachmentBytes({ workspaceRoot: root, body: Buffer.from("later authorized bytes") });
|
||||
await expect(readFile(path.join(root, explicit.workspaceRelativePath), "utf8"))
|
||||
.resolves.toBe("later authorized bytes");
|
||||
await explicit.cleanup();
|
||||
await expect(readFile(path.join(root, explicit.workspaceRelativePath)))
|
||||
.resolves.toEqual(Buffer.alloc(0));
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("scrubs inactive staging bytes on an empty wake without creating another directory", async () => {
|
||||
const root = await mkdtemp(path.join(temporaryRoot, "empty-wake-residue-"));
|
||||
try {
|
||||
const previous = await stageNativeRunnerAttachmentBytes({ workspaceRoot: root, body: Buffer.from("previous attachment") });
|
||||
await previous.cleanup();
|
||||
const slot = path.join(root, previous.workspaceRelativePath);
|
||||
await writeFile(slot, "stale bytes after an interrupted owner");
|
||||
const directories = await readdir(path.join(root, ".paperclip-inbound"));
|
||||
|
||||
const wake = await stageEmptyWake(root);
|
||||
expect(wake.attachments).toEqual([]);
|
||||
await wake.cleanup();
|
||||
await expect(readFile(slot)).resolves.toEqual(Buffer.alloc(0));
|
||||
await expect(readdir(path.join(root, ".paperclip-inbound"))).resolves.toEqual(directories);
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["root-file", "root-symlink", "entry-symlink"] as const)(
|
||||
"rejects unsafe %s staging on an empty wake without changing outside bytes",
|
||||
async (kind) => {
|
||||
const root = await mkdtemp(path.join(temporaryRoot, "empty-wake-unsafe-"));
|
||||
const outside = await mkdtemp(path.join(temporaryRoot, "empty-wake-outside-"));
|
||||
try {
|
||||
await writeFile(path.join(outside, "keep.txt"), "outside bytes");
|
||||
const inbound = path.join(root, ".paperclip-inbound");
|
||||
if (kind === "root-file") await writeFile(inbound, "not a directory");
|
||||
else if (kind === "root-symlink") await symlink(outside, inbound);
|
||||
else {
|
||||
await mkdir(inbound);
|
||||
await symlink(outside, path.join(inbound, "untrusted-owner"));
|
||||
}
|
||||
await expect(stageEmptyWake(root)).rejects.toThrow(/staging_(?:path|residue)_denied/);
|
||||
await expect(readFile(path.join(outside, "keep.txt"), "utf8")).resolves.toBe("outside bytes");
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true });
|
||||
await rm(outside, { recursive: true, force: true });
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it("still authorizes an empty wake before touching its workspace", async () => {
|
||||
const root = await mkdtemp(path.join(temporaryRoot, "empty-wake-unauthorized-"));
|
||||
try {
|
||||
await expect(stageEmptyWake(root, "00000000-0000-4000-8000-000000009199"))
|
||||
.rejects.toThrow("paperclip_runner_attachment_staging_not_authorized");
|
||||
await expect(readdir(root)).resolves.toEqual([]);
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects a workspace-only file completion and invented delivery receipts without asking the user to approve completion", async () => {
|
||||
for (const ref of ["launch-checklist.md", "./out/report.pdf", "/workspace/answer.txt", "file:out/report.csv", "deliverable:00000000-0000-4000-8000-000000000001"]) {
|
||||
await expect(nativeCompletionFeedback(db, runId, doneReport([ref])))
|
||||
@@ -244,6 +332,42 @@ describe("native runner file handoff", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it.each([
|
||||
"Use native write/read file tools. Write memory/pi-native.txt inside AGENT_HOME. This is personal memory, not a task deliverable.",
|
||||
"Use native write to copy bytes into pi-agent-memory-proof.txt. This is an internal assertion file, not a deliverable.",
|
||||
"Attempt native write once to /outside/pi-unassigned.txt. This intentionally unassigned root must be denied.",
|
||||
"Write internal-proof.txt; then check it exists. This is an internal verification file, not a deliverable.",
|
||||
"Attempt native write to /outside/probe.txt and create no files. This negative test must be denied.",
|
||||
"Write internal-proof.txt and return it in chat. This is an internal verification file, not a deliverable.",
|
||||
"Write internal-proof.txt and send it as a code block in your response. This is an internal verification file, not a deliverable.",
|
||||
])("accepts an internal file outcome without treating it as published output: %s", async objective => {
|
||||
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId, executionContinuation: { objective } } })
|
||||
.where(eq(heartbeatRuns.id, runId));
|
||||
try {
|
||||
await expect(nativeCompletionFeedback(db, runId, doneReport([])))
|
||||
.resolves.toContain("Completion report accepted");
|
||||
} finally {
|
||||
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId } }).where(eq(heartbeatRuns.id, runId));
|
||||
}
|
||||
});
|
||||
|
||||
it.each([
|
||||
"Write report.pdf; write internal-proof.txt. This is an internal verification file, not a deliverable.",
|
||||
"Write report.pdf and attempt native write to /outside/probe.txt; this negative test must be denied.",
|
||||
"Write report.txt and attach it. This is an internal verification file, not a deliverable.",
|
||||
"Write report.txt and send it to me. This is an internal verification file, not a deliverable.",
|
||||
"Write report.txt. This is an internal verification file, not a deliverable. Send it as a download link in your response.",
|
||||
])("still requires publication when the task also asks for internal or denied writes: %s", async objective => {
|
||||
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId, executionContinuation: { objective } } })
|
||||
.where(eq(heartbeatRuns.id, runId));
|
||||
try {
|
||||
await expect(nativeCompletionFeedback(db, runId, doneReport([])))
|
||||
.rejects.toThrow("requested file has no accessible delivery evidence");
|
||||
} finally {
|
||||
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId } }).where(eq(heartbeatRuns.id, runId));
|
||||
}
|
||||
});
|
||||
|
||||
it("prepares one verified same-run attachment and replays without duplicates", async () => {
|
||||
const body = Buffer.from("native runner file handoff\n", "utf8");
|
||||
await mkdir(path.join(workspaceRoot, "out"), { recursive: true });
|
||||
|
||||
@@ -5,6 +5,7 @@ import { PgDialect } from "drizzle-orm/pg-core";
|
||||
import type { SQL } from "drizzle-orm";
|
||||
import {
|
||||
access,
|
||||
chmod,
|
||||
cp,
|
||||
lstat,
|
||||
mkdir,
|
||||
@@ -42,12 +43,18 @@ import {
|
||||
validatePrpEvent,
|
||||
parseNativeExecutionInput,
|
||||
type NativeExecutionInputV1,
|
||||
type NativeExecutionInputV3,
|
||||
type NativeExecutionInput,
|
||||
type PrpEvent,
|
||||
} from "@paperclipai/paperclip-runner";
|
||||
import { createHash, generateKeyPairSync } from "node:crypto";
|
||||
import { DatabaseSync } from "node:sqlite";
|
||||
import { nativeSha256 } from "./canonical.js";
|
||||
import { agentDirectoryWorkingCopyService } from "../agent-directory-working-copies.js";
|
||||
import { probeAgentDirectory } from "../agent-directory-probe.js";
|
||||
import { AGENT_FILES_CONTRACT } from "../agent-file-store.js";
|
||||
import { captureDirectorySnapshot, directorySnapshotSha256, serializeDirectorySnapshot } from "@paperclipai/adapter-utils/workspace-restore-merge";
|
||||
import { agentInstructionWorkingCopies } from "@paperclipai/db";
|
||||
import * as noLaunchProofModule from "./native-maintenance-no-launch.js";
|
||||
import {
|
||||
NativeSessionCleanupQuarantinedError,
|
||||
@@ -63,6 +70,7 @@ import {
|
||||
} from "./native-harness-backup-stamp.js";
|
||||
import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js";
|
||||
import { nativeToolContractFingerprintForTarget } from "./native-session-resume.js";
|
||||
import * as nativeSessionResume from "./native-session-resume.js";
|
||||
import { buildNativeHeartbeatPreparationSpans } from "./native-run-trace.js";
|
||||
import { NativeRunnerOwnershipUnverifiedError } from "./native-runner-ownership.js";
|
||||
import type { AdapterRuntimeEvent } from "../../adapters/index.js";
|
||||
@@ -297,6 +305,8 @@ import {
|
||||
buildNativeHarnessBackupManifest,
|
||||
cancelNativeSession,
|
||||
closeWarmNativeSessionsForEnvironment,
|
||||
claimWarmNativeInstructionCopy,
|
||||
nativeSessionWorkspaceScope,
|
||||
reserveWarmNativeInstructionDirectory,
|
||||
closeIdleWarmNativeSessionsForRestart,
|
||||
createGovernedWaitEventObservation,
|
||||
@@ -1252,22 +1262,43 @@ describe("remote provider pack manifest", () => {
|
||||
const candidatePath = "provider-assets/pi/linux-x64";
|
||||
await mkdir(join(root, candidatePath), { recursive: true });
|
||||
await writeFile(join(root, candidatePath, "runtime"), "pinned runtime");
|
||||
const candidates = { pi: { version: "0.0.33", profileDigest: digest("profile"),
|
||||
closureDigest: digest("closure"), qualification: "pending", path: candidatePath,
|
||||
const candidates = { pi: { version: "1.0.0", profileDigest: digest("profile"),
|
||||
closureDigest: digest("closure"), qualification: "qualified", path: candidatePath,
|
||||
sha256: sha256DirectoryTree(join(root, candidatePath)) } };
|
||||
Object.assign(payload, { candidateProviders: candidates });
|
||||
await writeManifest();
|
||||
expect(readRemoteProviderPackManifest(root).payload.candidateProviders?.pi?.qualification).toBe("pending");
|
||||
expect(readRemoteProviderPackManifest(root).payload.candidateProviders?.pi?.qualification).toBe("qualified");
|
||||
// The normal pack builder publishes Pi in both inventories. A Pi image
|
||||
// must pass the same reader used by real remote runtime preparation.
|
||||
Object.assign(payload, { providers: { cursor, pi: candidates.pi } });
|
||||
await writeManifest();
|
||||
expect(readRemoteProviderPackManifest(root).payload.providers?.pi?.qualification).toBe("qualified");
|
||||
await mkdir(releaseMetadata, { recursive: true });
|
||||
await cp(join(root, "provider-pack.json"), manifestPath);
|
||||
expect(readBundledRemoteProviderPackManifest(manifestPath).payload.providers?.pi?.qualification).toBe("qualified");
|
||||
Object.assign(payload, { providers: { cursor } });
|
||||
await rm(releaseMetadata, { recursive: true, force: true });
|
||||
await writeManifest();
|
||||
await writeFile(join(root, candidatePath, "runtime"), "substitute runtime");
|
||||
expect(() => readRemoteProviderPackManifest(root)).toThrow("candidate asset tree digest mismatch");
|
||||
await writeFile(join(root, candidatePath, "runtime"), "pinned runtime");
|
||||
for (const invalid of [{ path: "../outside" }, { qualification: "qualified" }]) {
|
||||
candidates.pi.qualification = "pending";
|
||||
await writeManifest();
|
||||
expect(() => readRemoteProviderPackManifest(root)).toThrow("invalid candidate identity");
|
||||
candidates.pi.qualification = "qualified";
|
||||
for (const invalid of [{ path: "../outside" }, { qualification: "unknown" }]) {
|
||||
const original = { ...candidates.pi };
|
||||
Object.assign(candidates.pi, invalid);
|
||||
await writeManifest();
|
||||
expect(() => readRemoteProviderPackManifest(root)).toThrow("invalid candidate identity");
|
||||
candidates.pi = original;
|
||||
}
|
||||
for (const provider of ["cursor", "copilot"]) {
|
||||
Object.assign(candidates, { [provider]: { ...candidates.pi, path: `provider-assets/${provider}/linux-x64` } });
|
||||
await writeManifest();
|
||||
expect(() => readRemoteProviderPackManifest(root)).toThrow(provider === "cursor" ? "Cursor profile or closure does not match this release" : "invalid candidate identity");
|
||||
delete (candidates as Record<string, unknown>)[provider];
|
||||
}
|
||||
await writeManifest();
|
||||
for (const [artifactName, substituteName] of [
|
||||
["nodeCommand", "productionLock"],
|
||||
@@ -1537,6 +1568,67 @@ describe("verified native harness backups", () => {
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("binds Cursor mode across governed and identical recovery identities", () => {
|
||||
const execution = {
|
||||
...backupExecution,
|
||||
provider: { kind: "acpx", agent: "cursor", model: "explicit-model", mode: "plan" },
|
||||
interactionResponses: [{ interactionId: "interaction-1" }],
|
||||
} as unknown as NativeExecutionInputV1;
|
||||
const identity = (suffix: string, mode: unknown) => {
|
||||
const value = acpxIdentity(suffix);
|
||||
return { ...value, providerSessionIdentity: { ...value.providerSessionIdentity, mode } };
|
||||
};
|
||||
const previous = identity("previous", "plan");
|
||||
const current = identity("current", "plan");
|
||||
expect(providerSessionIdentityTransitionIsAllowed({ execution, previous, current })).toBe(true);
|
||||
expect(providerSessionIdentityTransitionIsAllowed({ execution, previous, current: previous })).toBe(true);
|
||||
for (const mode of [undefined, null, "agent", "ask", "autopilot"]) {
|
||||
const wrong = identity("wrong", mode);
|
||||
expect(providerSessionIdentityTransitionIsAllowed({ execution, previous, current: wrong })).toBe(false);
|
||||
expect(providerSessionIdentityTransitionIsAllowed({ execution, previous: wrong, current })).toBe(false);
|
||||
expect(providerSessionIdentityTransitionIsAllowed({ execution, previous: wrong, current: wrong })).toBe(false);
|
||||
}
|
||||
for (const provider of [
|
||||
{ kind: "acpx", agent: "cursor", model: "explicit-model" },
|
||||
{ kind: "acpx", agent: "copilot", model: "explicit-model" },
|
||||
]) {
|
||||
expect(providerSessionIdentityTransitionIsAllowed({
|
||||
execution: { ...execution, provider } as unknown as NativeExecutionInputV1, previous, current,
|
||||
})).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it("binds Pi thinking across governed and identical recovery identities", () => {
|
||||
const execution = {
|
||||
...backupExecution,
|
||||
provider: { kind: "acpx", agent: "pi", model: "explicit-model", piThinkingLevel: "low" },
|
||||
interactionResponses: [{ interactionId: "interaction-1" }],
|
||||
} as unknown as NativeExecutionInputV1;
|
||||
const identity = (suffix: string, piThinkingLevel: unknown) => {
|
||||
const value = acpxIdentity(suffix);
|
||||
return { ...value, providerSessionIdentity: { ...value.providerSessionIdentity, piThinkingLevel } };
|
||||
};
|
||||
const previous = identity("previous", "low");
|
||||
const current = identity("current", "low");
|
||||
expect(providerSessionIdentityTransitionIsAllowed({ execution, previous, current })).toBe(true);
|
||||
expect(providerSessionIdentityTransitionIsAllowed({ execution, previous, current: previous })).toBe(true);
|
||||
for (const mode of [undefined, null, "high", "max", "medium"]) {
|
||||
const wrong = identity("wrong", mode);
|
||||
expect(providerSessionIdentityTransitionIsAllowed({ execution, previous, current: wrong })).toBe(false);
|
||||
expect(providerSessionIdentityTransitionIsAllowed({ execution, previous: wrong, current })).toBe(false);
|
||||
expect(providerSessionIdentityTransitionIsAllowed({ execution, previous: wrong, current: wrong })).toBe(false);
|
||||
}
|
||||
for (const provider of [
|
||||
{ kind: "acpx", agent: "pi", model: "explicit-model" },
|
||||
{ kind: "acpx", agent: "copilot", model: "explicit-model" },
|
||||
{ kind: "acpx", agent: "copilot", model: "explicit-model", piThinkingLevel: "low" },
|
||||
]) {
|
||||
expect(providerSessionIdentityTransitionIsAllowed({
|
||||
execution: { ...execution, provider } as unknown as NativeExecutionInputV1, previous, current,
|
||||
})).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it("restores a verified continuation into an intentionally fresh non-reusable sandbox", () => {
|
||||
expect(
|
||||
shouldRestoreNativeHarnessBackupIntoSandbox({
|
||||
@@ -1899,6 +1991,27 @@ describe("split durable provider checkpoint identity", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("requires the exact observed Cursor mode in suspended descriptor and identity", () => {
|
||||
const profileDigest = `sha256:${"a".repeat(64)}`;
|
||||
const input = execution({ kind: "acpx", agent: "cursor", model: "explicit-model", permissionMode: "deny-all", mode: "plan" }, "acpx_runtime");
|
||||
const providerState = {
|
||||
schema: "paperclip.runner.acpx-provider-state.v3", lifecycle: "suspended", activeTurnId: null, providerExitUnconfirmed: false,
|
||||
descriptor: { kind: "acpx", provider: "acpx", driver: "acpx_runtime", agent: "cursor", model: "explicit-model", commandDigest: profileDigest, normalizedSessionId: "native-session", mode: "plan" },
|
||||
identity: { kind: "acpx", normalizedSessionId: "native-session", acpxRecordId: "record-1", backendSessionId: "backend-1", agentSessionId: "agent-1", profileDigest,
|
||||
workspaceDigest: `sha256:${"b".repeat(64)}`, requestedModel: "explicit-model", effectiveModel: "explicit-model", permissionMode: "deny-all", mode: "plan", providerLifetimeFenceCandidates: [53001, 53002, 53003] },
|
||||
};
|
||||
const read = (state: unknown, candidate = input) => providerSessionIdentityFromDurableProviderState({ execution: candidate, providerState: state });
|
||||
expect(read(providerState).providerSessionIdentity).toEqual(providerState.identity);
|
||||
for (const mode of [undefined, null, "agent", "ask", "autopilot"]) {
|
||||
expect(read({ ...providerState, identity: { ...providerState.identity, mode } }).providerSessionIdentity).toBeNull();
|
||||
expect(read({ ...providerState, descriptor: { ...providerState.descriptor, mode } }).providerSessionIdentity).toBeNull();
|
||||
expect(read(providerState, execution({ ...input.provider, mode }, "acpx_runtime")).providerSessionIdentity).toBeNull();
|
||||
}
|
||||
const foreign = { ...providerState, descriptor: { ...providerState.descriptor, agent: "copilot" } };
|
||||
expect(read(foreign, execution({ ...input.provider, agent: "copilot", mode: undefined }, "acpx_runtime")).providerSessionIdentity).toBeNull();
|
||||
expect(read(foreign, execution({ ...input.provider, agent: "copilot", mode: "plan" }, "acpx_runtime")).providerSessionIdentity).toEqual(providerState.identity);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["codex", "codex_app_server"],
|
||||
["opencode", "opencode_server"],
|
||||
@@ -4841,6 +4954,16 @@ describe("native governed waits", () => {
|
||||
payload: { kind: "dynamicToolCall" },
|
||||
};
|
||||
|
||||
// The durable fallback preserves the question after provider loss, but
|
||||
// cannot turn that lost execution into a successful governed wait.
|
||||
const providerLost = { ...replayedEvent, eventType: "runtime_request.expired" as const,
|
||||
payload: { reason: "provider_process_lost", replayAllowed: false } };
|
||||
const lostObservation = createGovernedWaitEventObservation(async () => waitResult);
|
||||
await lostObservation.observe(providerLost, true);
|
||||
expect(lostObservation.consume(providerLost)).toBeNull();
|
||||
await lostObservation.observe(replayedEvent, true);
|
||||
expect(lostObservation.consume(replayedEvent)).toBeNull();
|
||||
|
||||
// The event consumer can lag the provider: a commentary event emitted
|
||||
// before the tool began may be processed after its approval exists in DB.
|
||||
// It is not proof that the approval-creating tool response has settled.
|
||||
@@ -5011,9 +5134,13 @@ function cancellationDb(options?: {
|
||||
runId: string;
|
||||
assessmentId: string | null;
|
||||
decisionId?: string | null;
|
||||
phase?: string;
|
||||
failureCode?: string | null;
|
||||
} | null;
|
||||
status?: string;
|
||||
failResultJsonUpdateAt?: number;
|
||||
ownershipHeld?: boolean;
|
||||
resultJson?: Record<string, unknown>;
|
||||
}) {
|
||||
const initialRun = {
|
||||
id: execution.binding.runId,
|
||||
@@ -5021,6 +5148,7 @@ function cancellationDb(options?: {
|
||||
companyId: execution.binding.companyId,
|
||||
nativeIssueId: execution.binding.issueId,
|
||||
runtimeMode: "native",
|
||||
status: options?.status ?? "running",
|
||||
...(options?.ownershipHeld
|
||||
? {
|
||||
status: "running",
|
||||
@@ -5033,6 +5161,7 @@ function cancellationDb(options?: {
|
||||
};
|
||||
let currentResultJson: Record<string, unknown> = {
|
||||
durableReceipt: { operationId: "operation-1" },
|
||||
...options?.resultJson,
|
||||
};
|
||||
const issue = {
|
||||
status: "in_progress",
|
||||
@@ -5079,6 +5208,7 @@ function cancellationDb(options?: {
|
||||
set: (values: Record<string, unknown>) => ({
|
||||
where: () => {
|
||||
updates.push({ table, values });
|
||||
if (table === nativeRunFinalizations && coordinator) Object.assign(coordinator, values);
|
||||
const updatesResultJson = "resultJson" in values;
|
||||
if (updatesResultJson) resultJsonUpdateCount += 1;
|
||||
const shouldFail =
|
||||
@@ -5953,6 +6083,71 @@ describe("native session cancellation", () => {
|
||||
expect(state.publishActivity).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it.each(["terminal_failure", "observed", "committed"])("rejects retry phase %s that changed after caller reservation, before any native effect", async phase => {
|
||||
const id = "11111111-1111-4111-8111-111111111111";
|
||||
const persistence = cancellationDb({ status: "failed",
|
||||
coordinator: { runId: execution.binding.runId, assessmentId: null, phase, failureCode: "board_recovery" },
|
||||
resultJson: { startupCancellation: { cancellationRequestId: id, retryCancellation: true } } });
|
||||
await expect(cancelNativeSession(execution.binding.runId, "operator Stop", { db: persistence.db,
|
||||
scope: "run", cancellationRequestId: id })).rejects.toMatchObject({ status: 409, message: "Native retry is no longer cancellable" });
|
||||
expect(persistence.updates).toEqual([]);
|
||||
expect(state.cancel).not.toHaveBeenCalled();
|
||||
expect(state.persistActivity).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("does not mistake a recovered running attempt for the failed retry that reserved Stop", async () => {
|
||||
const id = "11111111-1111-4111-8111-111111111111";
|
||||
const persistence = cancellationDb({ status: "running",
|
||||
coordinator: { runId: execution.binding.runId, assessmentId: null, phase: "observed" },
|
||||
resultJson: { startupCancellation: { cancellationRequestId: id, retryCancellation: true } } });
|
||||
await expect(cancelNativeSession(execution.binding.runId, "operator Stop", { db: persistence.db,
|
||||
scope: "run", cancellationRequestId: id })).rejects.toMatchObject({ status: 409 });
|
||||
expect(persistence.updates).toEqual([]);
|
||||
expect(state.cancel).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("fences a failed retry at dispatch and preserves the same audited intent on replay", async () => {
|
||||
const id = "11111111-1111-4111-8111-111111111111";
|
||||
const persistence = cancellationDb({ status: "failed",
|
||||
coordinator: { runId: execution.binding.runId, assessmentId: null, phase: "retryable_failure" },
|
||||
resultJson: { startupCancellation: { cancellationRequestId: id, retryCancellation: true } } });
|
||||
const first = await cancelNativeSession(execution.binding.runId, "operator Stop", { db: persistence.db, scope: "run", cancellationRequestId: id });
|
||||
expect(persistence.updates).toContainEqual(expect.objectContaining({ table: nativeRunFinalizations,
|
||||
values: expect.objectContaining({ phase: "terminal_failure", failureCode: "native_retry_cancelled", nextAttemptAt: null }) }));
|
||||
const writes = persistence.getResultJsonUpdateCount();
|
||||
await expect(cancelNativeSession(execution.binding.runId, "retry Stop", { db: persistence.db,
|
||||
scope: "run", cancellationRequestId: id })).resolves.toMatchObject({ auditId: first.auditId });
|
||||
expect(persistence.getResultJsonUpdateCount()).toBe(writes);
|
||||
});
|
||||
|
||||
it("uses the reserved caller UUID and permits same-intent and default retries", async () => {
|
||||
const id = "11111111-1111-4111-8111-111111111111";
|
||||
const persistence = cancellationDb({ resultJson: { startupCancellation: { cancellationRequestId: id } } });
|
||||
await cancelNativeSession(execution.binding.runId, "operator Stop", { db: persistence.db, scope: "run", cancellationRequestId: id });
|
||||
expect(persistence.getResultJson().nativeCancellation).toMatchObject({ intentId: `native-cancellation:${id}` });
|
||||
const writes = persistence.getResultJsonUpdateCount();
|
||||
await cancelNativeSession(execution.binding.runId, "same request", { db: persistence.db, scope: "run", cancellationRequestId: id });
|
||||
await cancelNativeSession(execution.binding.runId, "default retry", { db: persistence.db, scope: "run" });
|
||||
expect(persistence.getResultJsonUpdateCount()).toBe(writes);
|
||||
await expect(cancelNativeSession(execution.binding.runId, "foreign caller", {
|
||||
db: persistence.db, scope: "run", cancellationRequestId: "22222222-2222-4222-8222-222222222222",
|
||||
})).rejects.toMatchObject({ status: 409 });
|
||||
expect(persistence.getResultJsonUpdateCount()).toBe(writes);
|
||||
});
|
||||
|
||||
it.each([{}, { startupCancellation: { requestedAt: "prior" } },
|
||||
{ startupCancellation: { cancellationRequestId: "other" } },
|
||||
{ startupCancellation: { cancellationRequestId: "11111111-1111-4111-8111-111111111111" }, nativeCancellation: { intentId: "foreign" } },
|
||||
])("rejects missing or raced caller ownership under the intent lock: %j", async resultJson => {
|
||||
const persistence = cancellationDb({ resultJson });
|
||||
await expect(cancelNativeSession(execution.binding.runId, "operator Stop", {
|
||||
db: persistence.db, scope: "run", cancellationRequestId: "11111111-1111-4111-8111-111111111111",
|
||||
})).rejects.toMatchObject({ status: 409 });
|
||||
expect(persistence.getForUpdateCount()).toBe(1);
|
||||
expect(persistence.updates).toEqual([]);
|
||||
expect(state.cancel).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("recovers a post-dispatch persistence failure without cancelling the provider twice", async () => {
|
||||
const persistence = cancellationDb({ failResultJsonUpdateAt: 2 });
|
||||
const running = executePaperclipNativeSession({
|
||||
@@ -6418,7 +6613,499 @@ describe("native session same-turn steering", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("bounded stopped instruction collection", () => {
|
||||
it.each([
|
||||
{ scenario: "deferred", attempts: [0], nextAttemptAt: null, expectedCalls: 1 },
|
||||
{ scenario: "no progress", attempts: [0, 0], nextAttemptAt: new Date(0), expectedCalls: 2 },
|
||||
{ scenario: "regressed counter", attempts: [1, 0], nextAttemptAt: new Date(0), expectedCalls: 2 },
|
||||
{ scenario: "progressing counter", attempts: [0, 1, 2], nextAttemptAt: new Date(0), expectedCalls: 3 },
|
||||
{ scenario: "exhausted", attempts: [3], nextAttemptAt: new Date(0), expectedCalls: 1 },
|
||||
])("finishes pending collection without inventing attempts: $scenario", async ({ attempts, nextAttemptAt, expectedCalls }) => {
|
||||
const { collectStoppedInstructionCopyWithRetries } = await import("../agent-instruction-working-copies.js");
|
||||
const collect = vi.fn(async () => {
|
||||
const index = collect.mock.calls.length - 1;
|
||||
if (index >= attempts.length) throw new Error("collector invoked beyond bounded progress");
|
||||
return { state: "pending_collection", attempts: attempts[index]!, nextAttemptAt };
|
||||
});
|
||||
expect(await collectStoppedInstructionCopyWithRetries(collect)).toEqual({ state: "pending_collection", attempts: attempts.at(-1), nextAttemptAt });
|
||||
expect(collect).toHaveBeenCalledTimes(expectedCalls);
|
||||
});
|
||||
it("returns completed or absent copies and propagates collection errors", async () => {
|
||||
const { collectStoppedInstructionCopyWithRetries } = await import("../agent-instruction-working-copies.js");
|
||||
const result = { state: "saved", attempts: 1, nextAttemptAt: null };
|
||||
const saved = vi.fn(async () => result), absent = vi.fn(async () => null);
|
||||
expect(await collectStoppedInstructionCopyWithRetries(saved)).toBe(result);
|
||||
expect(await collectStoppedInstructionCopyWithRetries(absent)).toBeNull();
|
||||
expect(saved).toHaveBeenCalledOnce(); expect(absent).toHaveBeenCalledOnce();
|
||||
await expect(collectStoppedInstructionCopyWithRetries(async () => { throw new Error("collection failed"); })).rejects.toThrow("collection failed");
|
||||
});
|
||||
});
|
||||
|
||||
describe("native warm session supervision", () => {
|
||||
it.each(["collect", "close-failure", "stopped", "destroyed", "missing-lease", "foreign-run", "foreign-environment", "deleted-environment", "claim-edit", "claim-add", "claim-remove", "claim-remove-entry", "claim-canonical", "claim-config", "claim-uncertain"])("composes remote warm ownership with real DB successor leases and current-run collection: %s", async ending => {
|
||||
const tables = await import("@paperclipai/db");
|
||||
const { eq } = await import("drizzle-orm");
|
||||
const { randomUUID } = await import("node:crypto");
|
||||
const { promisify } = await import("node:util");
|
||||
const { execFile } = await import("node:child_process");
|
||||
const executeCommand = promisify(execFile);
|
||||
const { startEmbeddedPostgresTestDatabase } = await import("../../__tests__/helpers/embedded-postgres.js");
|
||||
const { agentInstructionWorkingCopyService } = await import("../agent-instruction-working-copies.js");
|
||||
const { buildNativeRuntimeContext } = await import("./runtime-context.js");
|
||||
const { prepareNativeHeartbeatRun } = await import("./prepare-native-run.js");
|
||||
const { buildNativeCompletionContract } = await import("./completion-contracts.js");
|
||||
const home = await realpath(await mkdtemp(join(tmpdir(), "warm-remote-db-")));
|
||||
const previousHome = process.env.PAPERCLIP_HOME;
|
||||
process.env.PAPERCLIP_HOME = home;
|
||||
const database = await startEmbeddedPostgresTestDatabase("warm-remote-db-");
|
||||
const db = tables.createDb(database.connectionString);
|
||||
const companyId = randomUUID(), agentId = randomUUID(), userId = randomUUID(), environmentId = randomUUID(), issueId = randomUUID();
|
||||
const { resolveManagedInstructionsRoot } = await import("../agent-instructions.js");
|
||||
const canonical = resolveManagedInstructionsRoot({ id: agentId, companyId, name: "Warm remote", adapterConfig: {} }), remoteCwd = join(home, "remote");
|
||||
let currentLease = "", failCleanup = false, failProbe = false, closed = false;
|
||||
const commands: Array<{ leaseId: string; command: string }> = [];
|
||||
const commandRunner = (leaseId: string): import("@paperclipai/adapter-utils/command-managed-runtime").CommandManagedRuntimeRunner => ({ execute: async input => {
|
||||
expect(leaseId).toBe(currentLease); // The replaced collector must never execute.
|
||||
commands.push({ leaseId, command: `${input.command} ${(input.args ?? []).join(" ")}` });
|
||||
if (failProbe) {
|
||||
failProbe = false;
|
||||
expect(commands.at(-1)!.command).toContain("Agent directory probe deadline");
|
||||
return { exitCode: 1, signal: null, timedOut: false, stdout: "", stderr: "injected uncertain observation", pid: null, startedAt: new Date().toISOString() };
|
||||
}
|
||||
if (failCleanup && commands.at(-1)!.command.includes("rm -rf --")) {
|
||||
failCleanup = false;
|
||||
return { exitCode: 1, signal: null, timedOut: false, stdout: "", stderr: "injected cleanup failure", pid: null, startedAt: new Date().toISOString() };
|
||||
}
|
||||
const env = { PATH: process.env.PATH, ...input.env }, args = [...(input.args ?? [])];
|
||||
if (input.stdin != null && (args[0] === "-c" || args[0] === "-lc")) {
|
||||
Object.assign(env, { PAPERCLIP_TEST_STDIN: input.stdin });
|
||||
args[1] = `printf '%s' "$PAPERCLIP_TEST_STDIN" | (${args[1]})`;
|
||||
}
|
||||
const result = await executeCommand(input.command, args, { cwd: input.cwd, env, timeout: input.timeoutMs, maxBuffer: 32 * 1024 * 1024 });
|
||||
return { exitCode: 0, signal: null, timedOut: false, ...result, pid: null, startedAt: new Date().toISOString() };
|
||||
} });
|
||||
let copies = agentInstructionWorkingCopyService(db);
|
||||
const agent = { id: agentId, companyId, name: "Warm remote", adapterConfig: {
|
||||
instructionsBundleMode: "managed", instructionsRootPath: canonical, instructionsEntryFile: "AGENTS.md" } };
|
||||
let active: Awaited<ReturnType<typeof copies.prepare>>;
|
||||
const session = { close: vi.fn(async () => { closed = true; }) };
|
||||
state.execute.mockReset().mockImplementation(async options => {
|
||||
await options.onSession?.(session);
|
||||
return { result: { summary: "completed" }, terminal: { runTerminalState: "succeeded" },
|
||||
turnId: "warm-turn", normalizedSessionId: environmentId, providerSessionId: environmentId,
|
||||
driverKind: "test", driverVersion: "1", nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
|
||||
});
|
||||
let bodyFailure: unknown;
|
||||
try {
|
||||
await db.insert(tables.companies).values({ id: companyId, name: "Warm tests", issuePrefix: "WARM" });
|
||||
await db.insert(tables.authUsers).values({ id: userId, name: "Editor", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
|
||||
await db.insert(tables.agents).values(agent);
|
||||
await db.insert(tables.companyMemberships).values([{ companyId, principalType: "user", principalId: userId, membershipRole: "operator" }, { companyId, principalType: "agent", principalId: agentId, membershipRole: "member" }]);
|
||||
await db.insert(tables.principalPermissionGrants).values({ companyId, principalType: "user", principalId: userId, permissionKey: "agents:configure", scope: { agentIds: [agentId] } });
|
||||
await db.insert(tables.environments).values({ id: environmentId, name: "Owned fake remote", driver: "sandbox" });
|
||||
const [issue] = await db.insert(tables.issues).values({ id: issueId, companyId, title: "Warm remote", status: "in_progress", assigneeAgentId: agentId }).returning();
|
||||
await mkdir(canonical, { recursive: true }); await mkdir(remoteCwd);
|
||||
await writeFile(join(canonical, "AGENTS.md"), "Retain this exact directory.\n");
|
||||
await writeFile(join(canonical, "optional-memory.txt"), "Optional memory\n");
|
||||
let prior: NativeExecutionInputV3 | undefined;
|
||||
let originalRoot = "", materializationRunId = "";
|
||||
for (let turn = 0; turn < 3; turn++) {
|
||||
const runId = randomUUID();
|
||||
if (currentLease) {
|
||||
await db.update(tables.environmentLeases).set({ status: "expired", releasedAt: new Date(), cleanupStatus: "success" }).where(eq(tables.environmentLeases.id, currentLease));
|
||||
await db.update(tables.heartbeatRuns).set({ status: "succeeded" }).where(eq(tables.heartbeatRuns.id, prior!.binding.runId));
|
||||
}
|
||||
currentLease = randomUUID();
|
||||
const [run] = await db.insert(tables.heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", responsibleUserId: userId,
|
||||
status: "running", runtimeMode: "native", nativeIssueId: issueId, nativeSessionId: environmentId, runnerInstanceId: environmentId,
|
||||
contextSnapshot: { issueId } }).returning();
|
||||
await db.insert(tables.nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed" });
|
||||
await db.update(tables.issues).set({ executionRunId: runId }).where(eq(tables.issues.id, issueId));
|
||||
await db.insert(tables.environmentLeases).values({ id: currentLease, companyId, environmentId, heartbeatRunId: runId,
|
||||
provider: "daytona", providerLeaseId: "same-owned-allocation", leasePolicy: "reuse_by_environment", metadata: { remoteCwd } });
|
||||
// Match the production preparation seam and existing native DB tests:
|
||||
// pin the issue's completion contract and durable run/lease identities.
|
||||
await prepareNativeHeartbeatRun({ db, run: run!, issue: issue!, environmentLeaseId: currentLease });
|
||||
const [contract] = await db.select().from(tables.completionContracts).where(eq(tables.completionContracts.issueId, issueId));
|
||||
const target = { kind: "remote" as const, transport: "sandbox" as const, providerKey: "daytona", environmentId, leaseId: currentLease, remoteCwd, runner: commandRunner(currentLease) };
|
||||
const current: NativeExecutionInputV3 = { ...execution, schema: "paperclip.native-execution-input.v3",
|
||||
executionMode: "default", planningContext: null, runtimeContext: nativeRuntimeContextFixture(),
|
||||
completionContract: { id: contract!.id, sha256: contract!.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1",
|
||||
contract: buildNativeCompletionContract(issue!, { revision: contract!.revision }) }, binding: { companyId, agentId, issueId, runId, executionWorkspaceId: runId },
|
||||
workspace: { cwd: remoteCwd, repoUrl: null, repoRef: null, branchName: null },
|
||||
session: { ...execution.session, normalizedSessionId: environmentId, lifecyclePolicy: { mode: "warm", idleTimeoutMs: 60_000 } } };
|
||||
const preparationKey = turn === 1 && ending === "claim-config" ? "changed-config" : "same-config";
|
||||
let retiredReceipt: Record<string, unknown> | null = null;
|
||||
const capability = () => ({ runId, preparationKey, hasChanges: () => copies.hasChanges({ companyId, runId, target }),
|
||||
collectStopped: async () => { expect(closed).toBe(true); retiredReceipt = (await copies.get(companyId, runId))?.receipt ?? null;
|
||||
const { collectStoppedInstructionCopyWithRetries } = await import("../agent-instruction-working-copies.js");
|
||||
await collectStoppedInstructionCopyWithRetries(() => copies.collectStopped({ companyId, runId, target }));
|
||||
}, retirementFailed: async () => { await copies.reportRetirementUnconfirmed(companyId, runId); } });
|
||||
let release: (() => Promise<void>) | null = null;
|
||||
if (prior) {
|
||||
// Same environment is insufficient: reject a different physical allocation.
|
||||
await db.update(tables.environmentLeases).set({ providerLeaseId: "foreign-allocation" }).where(eq(tables.environmentLeases.id, currentLease));
|
||||
const before = commands.length;
|
||||
expect(await copies.adopt({ companyId, agentId, runId, previousRunId: prior.binding.runId, target, cwd: remoteCwd })).toBeNull();
|
||||
expect(commands).toHaveLength(before);
|
||||
await db.update(tables.environmentLeases).set({ providerLeaseId: "same-owned-allocation" }).where(eq(tables.environmentLeases.id, currentLease));
|
||||
if (ending === "claim-edit") await writeFile(join(originalRoot, "AGENTS.md"), "Remote edit before next claim\n");
|
||||
if (ending === "claim-add") await writeFile(join(originalRoot, "memory.txt"), "Remote addition before next claim\n");
|
||||
if (ending === "claim-remove") await rm(join(originalRoot, "optional-memory.txt"));
|
||||
if (ending === "claim-remove-entry") await rm(join(originalRoot, "AGENTS.md"));
|
||||
if (ending === "claim-canonical") await writeFile(join(canonical, "AGENTS.md"), "Canonical edit before next claim\n");
|
||||
if (["claim-edit", "claim-add", "claim-remove", "claim-remove-entry", "claim-canonical"].includes(ending)) {
|
||||
// Ordinary adoption remains unchanged-only and cannot alias this owner.
|
||||
expect(await copies.adopt({ companyId, agentId, runId, previousRunId: prior.binding.runId, target, cwd: remoteCwd })).toBeNull();
|
||||
expect((await copies.get(companyId, prior.binding.runId))?.receipt?.retainedByRunId).toBeUndefined();
|
||||
}
|
||||
failProbe = ending === "claim-uncertain";
|
||||
release = await claimWarmNativeInstructionCopy({ priorExecution: prior, companyId, agentId, executionWorkspaceId: runId,
|
||||
workspace: current.workspace, environmentId, runId, preparationKey, adopt: async previousRunId => {
|
||||
active = await copies.adopt({ companyId, agentId, runId, previousRunId, target, cwd: remoteCwd, allowRetirementHandoff: true });
|
||||
return active ? capability() : null;
|
||||
} });
|
||||
if (ending.startsWith("claim-")) {
|
||||
expect(release).toBeNull();
|
||||
expect(session.close).toHaveBeenCalledOnce();
|
||||
expect(state.execute).toHaveBeenCalledOnce(); // No reuse was admitted.
|
||||
if (ending !== "claim-config") expect(retiredReceipt).toMatchObject({ retirementRequired: true });
|
||||
expect(retiredReceipt).toMatchObject({ materializationRunId, cleanup: { leaseId: currentLease, remoteCwd } });
|
||||
expect((await copies.get(companyId, runId))?.processStoppedAt).toBeInstanceOf(Date);
|
||||
const compact = (await copies.get(companyId, runId))!.receipt;
|
||||
expect(compact).toMatchObject({ materializationRunId, cleanupPending: false });
|
||||
expect(compact?.baseline).toBeUndefined();
|
||||
await expect(access(originalRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
if (ending === "claim-edit") expect(await readFile(join(canonical, "AGENTS.md"), "utf8")).toBe("Remote edit before next claim\n");
|
||||
if (ending === "claim-add") expect(await readFile(join(canonical, "memory.txt"), "utf8")).toBe("Remote addition before next claim\n");
|
||||
if (ending === "claim-remove") await expect(access(join(canonical, "optional-memory.txt"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
if (ending === "claim-remove-entry") {
|
||||
expect((await copies.get(companyId, runId))!).toMatchObject({ state: "unavailable", errorCode: "AGENT_FILES_SAVE_FAILED" });
|
||||
expect(await readFile(join(canonical, "AGENTS.md"), "utf8")).toBe("Retain this exact directory.\n");
|
||||
}
|
||||
if (ending === "claim-canonical") expect(await readFile(join(canonical, "AGENTS.md"), "utf8")).toBe("Canonical edit before next claim\n");
|
||||
// Heartbeat rematerializes only after the successor proved retirement.
|
||||
active = await copies.prepare({ companyId, agentId, runId, target, cwd: remoteCwd });
|
||||
expect(active).toMatchObject({ state: "prepared", executionRoot: originalRoot, processStoppedAt: null });
|
||||
expect(active!.receipt?.retirementRequired).toBeUndefined();
|
||||
expect(await copies.hasChanges({ companyId, runId, target })).toBe(false);
|
||||
await copies.reportUnavailable(companyId, prior.binding.runId);
|
||||
await copies.release(companyId, prior.binding.runId);
|
||||
expect((await lstat(originalRoot)).isDirectory()).toBe(true);
|
||||
await copies.collectStopped({ companyId, runId, target });
|
||||
return;
|
||||
}
|
||||
expect(release).toBeTypeOf("function");
|
||||
await copies.reportUnavailable(companyId, prior.binding.runId);
|
||||
await copies.release(companyId, prior.binding.runId);
|
||||
expect(active!.receipt).toMatchObject({ materializationRunId, cleanup: { leaseId: currentLease, remoteCwd } });
|
||||
} else {
|
||||
active = await copies.prepare({ companyId, agentId, runId, target, cwd: remoteCwd });
|
||||
originalRoot = active!.executionRoot; materializationRunId = runId;
|
||||
}
|
||||
expect(active!.executionRoot).toBe(originalRoot);
|
||||
current.runtimeContext = await buildNativeRuntimeContext({ db, agent, runId, runtimeConfig: {}, runtimeSkillEntries: [],
|
||||
instructionWorkingCopy: { rootPath: active!.executionRoot, entryPath: "AGENTS.md", kind: "agent_files" } });
|
||||
expect(current.runtimeContext.instructions.workingCopy?.rootPath).toBe(originalRoot);
|
||||
await executePaperclipNativeSession({ db, execution: current, runnerInstanceId: environmentId, runnerExecutionTarget: target, instructionWorkingCopy: capability() });
|
||||
if (prior) expect(state.execute.mock.calls.at(-1)?.[0].existingSession).toBe(session);
|
||||
expect(closed).toBe(false);
|
||||
await release?.(); expect(closed).toBe(false);
|
||||
prior = current;
|
||||
}
|
||||
// A write after the live unchanged probe is still collected on retirement.
|
||||
await writeFile(join(originalRoot, "late-memory.txt"), "Latest owner's remote bytes");
|
||||
if (["missing-lease", "foreign-run", "foreign-environment", "deleted-environment"].includes(ending)) {
|
||||
if (ending === "missing-lease") await db.delete(tables.environmentLeases).where(eq(tables.environmentLeases.id, currentLease));
|
||||
if (ending === "foreign-run") await db.update(tables.environmentLeases).set({ heartbeatRunId: materializationRunId }).where(eq(tables.environmentLeases.id, currentLease));
|
||||
if (ending === "foreign-environment") {
|
||||
const other = randomUUID(); await db.insert(tables.environments).values({ id: other, name: "Other", driver: "sandbox" });
|
||||
await db.update(tables.environmentLeases).set({ environmentId: other }).where(eq(tables.environmentLeases.id, currentLease));
|
||||
}
|
||||
if (ending === "deleted-environment") await db.delete(tables.environments).where(eq(tables.environments.id, environmentId));
|
||||
const before = commands.length;
|
||||
await closeWarmNativeSessionsForEnvironment({ environmentId, reason: "lease identity changed" });
|
||||
expect(commands).toHaveLength(before);
|
||||
const pending = (await copies.get(companyId, prior!.binding.runId))!;
|
||||
expect(pending.state).toBe("pending_collection");
|
||||
expect(pending.receipt?.baseline).toBeDefined();
|
||||
expect(await readFile(join(originalRoot, "late-memory.txt"), "utf8")).toBe("Latest owner's remote bytes");
|
||||
return;
|
||||
}
|
||||
if (ending === "close-failure") {
|
||||
session.close.mockImplementationOnce(async () => { throw new Error("owned close not yet confirmed"); });
|
||||
const before = commands.length;
|
||||
expect(await closeWarmNativeSessionsForEnvironment({ environmentId, reason: "first retirement" })).toMatchObject({ failed: 1 });
|
||||
await copies.reportUnavailable(companyId, prior!.binding.runId);
|
||||
await copies.reportUnavailable(companyId, materializationRunId);
|
||||
await copies.release(companyId, prior!.binding.runId);
|
||||
await copies.release(companyId, materializationRunId);
|
||||
expect(await copies.get(companyId, prior!.binding.runId)).toMatchObject({ state: "pending_collection", errorCode: "INSTRUCTION_STOP_UNCONFIRMED", processStoppedAt: null });
|
||||
expect(commands).toHaveLength(before);
|
||||
expect(await readFile(join(originalRoot, "late-memory.txt"), "utf8")).toBe("Latest owner's remote bytes");
|
||||
expect(await closeWarmNativeSessionsForEnvironment({ environmentId, reason: "verified later retirement" })).toMatchObject({ closed: 1, failed: 0 });
|
||||
expect(await copies.get(companyId, prior!.binding.runId)).toMatchObject({ state: "saved", receipt: { cleanupPending: false } });
|
||||
expect(await readFile(join(canonical, "late-memory.txt"), "utf8")).toBe("Latest owner's remote bytes");
|
||||
await expect(access(originalRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
return;
|
||||
}
|
||||
if (ending !== "collect") {
|
||||
const { remoteTerminationReceipt } = await import("../remote-execution-termination.js");
|
||||
const [lease] = await db.select().from(tables.environmentLeases).where(eq(tables.environmentLeases.id, currentLease));
|
||||
await db.update(tables.environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success",
|
||||
metadata: { ...lease!.metadata, remoteExecutionTermination: remoteTerminationReceipt(lease!, { providerLeaseId: lease!.providerLeaseId, state: ending }) } }).where(eq(tables.environmentLeases.id, currentLease));
|
||||
await db.update(tables.heartbeatRuns).set({ status: "succeeded" }).where(eq(tables.heartbeatRuns.id, prior!.binding.runId));
|
||||
const before = commands.length;
|
||||
await copies.recoverStopped(); // The original remote collector is still cached.
|
||||
expect(commands).toHaveLength(before);
|
||||
const recovered = (await copies.get(companyId, prior!.binding.runId))!;
|
||||
expect(recovered.state).toBe(ending === "destroyed" ? "unavailable" : "pending_collection");
|
||||
await expect(access(join(canonical, "late-memory.txt"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
if (ending === "stopped") {
|
||||
expect(await readFile(join(originalRoot, "late-memory.txt"), "utf8")).toBe("Latest owner's remote bytes");
|
||||
expect(recovered.receipt?.baseline).toBeDefined();
|
||||
await copies.collectStopped({ companyId, runId: prior!.binding.runId });
|
||||
await copies.release(companyId, prior!.binding.runId);
|
||||
expect(commands).toHaveLength(before);
|
||||
} else await expect(access(active!.localRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await closeWarmNativeSessionsForEnvironment({ environmentId, reason: "already terminated owner" });
|
||||
expect(commands).toHaveLength(before);
|
||||
return;
|
||||
}
|
||||
failCleanup = true;
|
||||
await closeWarmNativeSessionsForEnvironment({ environmentId, reason: "test retirement" });
|
||||
expect(session.close).toHaveBeenCalledOnce();
|
||||
expect(await readFile(join(canonical, "late-memory.txt"), "utf8")).toBe("Latest owner's remote bytes");
|
||||
const saved = (await copies.get(companyId, prior!.binding.runId))!;
|
||||
expect(saved).toMatchObject({ state: "saved", receipt: { cleanupPending: true, materializationRunId, cleanup: { leaseId: currentLease } } });
|
||||
const cleanup = vi.fn(async input => {
|
||||
expect(input.lease.id).toBe(currentLease);
|
||||
expect(input.lease.heartbeatRunId).toBe(prior!.binding.runId);
|
||||
return commandRunner(currentLease).execute(input);
|
||||
});
|
||||
copies = agentInstructionWorkingCopyService(db, { environmentRuntime: { execute: cleanup } as never });
|
||||
await db.update(tables.agentInstructionWorkingCopies).set({ nextAttemptAt: null }).where(eq(tables.agentInstructionWorkingCopies.runId, prior!.binding.runId));
|
||||
await copies.recoverCaptured();
|
||||
expect(cleanup).toHaveBeenCalledOnce();
|
||||
await expect(access(originalRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect((await copies.get(companyId, prior!.binding.runId))!.receipt?.cleanupPending).toBe(false);
|
||||
} catch (error) { bodyFailure = error; throw error; }
|
||||
finally {
|
||||
const cleanupFailures: unknown[] = [];
|
||||
try { await closeWarmNativeSessionsForEnvironment({ environmentId, reason: "test cleanup" }); } catch (error) { cleanupFailures.push(error); }
|
||||
try { await database.cleanup(); } catch (error) { cleanupFailures.push(error); }
|
||||
if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome;
|
||||
try {
|
||||
const writable = async (directory: string): Promise<void> => {
|
||||
await chmod(directory, 0o700);
|
||||
for (const entry of await readdir(directory, { withFileTypes: true })) if (entry.isDirectory()) await writable(join(directory, entry.name));
|
||||
};
|
||||
await writable(home); await rm(home, { recursive: true, force: true });
|
||||
} catch (error) { cleanupFailures.push(error); }
|
||||
if (cleanupFailures.length) throw new AggregateError([...(bodyFailure ? [bodyFailure] : []), ...cleanupFailures], "Warm DB test body/cleanup failure");
|
||||
}
|
||||
}, 90_000);
|
||||
|
||||
it.each(["idle", "edit", "add", "remove", "projectless"])("retains the actual agent_files copy across warm turns, then retires for %s", async ending => {
|
||||
const root = await realpath(await mkdtemp(join(tmpdir(), "warm-agent-files-")));
|
||||
const localRoot = join(root, "run", "live");
|
||||
await mkdir(localRoot, { recursive: true });
|
||||
await writeFile(join(localRoot, "AGENTS.md"), "Keep the registered directory.\n");
|
||||
const snapshot = await captureDirectorySnapshot(localRoot);
|
||||
let row = { companyId: execution.binding.companyId, agentId: execution.binding.agentId,
|
||||
runId: "warm-real-agent-files", localRoot, executionRoot: localRoot, location: "local",
|
||||
entryFile: "AGENTS.md", state: "prepared", attempts: 0, processStoppedAt: null,
|
||||
baseRevisionId: null, candidateBase64: null, candidateHash: null, errorCode: null, errorMessage: null, nextAttemptAt: null,
|
||||
responsibleUserId: "test-user", createdAt: new Date(), updatedAt: new Date(),
|
||||
baseHash: directorySnapshotSha256(snapshot), receipt: { schema: AGENT_FILES_CONTRACT, baseline: serializeDirectorySnapshot(snapshot), materializationIdentity: probeAgentDirectory(localRoot).identity },
|
||||
} as typeof agentInstructionWorkingCopies.$inferSelect;
|
||||
const copies = agentDirectoryWorkingCopyService({} as Db, async () => row,
|
||||
async (_prior, values) => (row = { ...row, ...values } as typeof row));
|
||||
const order: string[] = [];
|
||||
const close = vi.fn(async () => { order.push("closed"); });
|
||||
const collectStopped = async () => {
|
||||
order.push("collected");
|
||||
expect(close).toHaveBeenCalledOnce();
|
||||
// The real DB service suite verifies changed-file persistence. Here the
|
||||
// fake backend proves stop precedes granting its collection callback.
|
||||
if (ending === "idle" || ending === "projectless") await copies.collectStopped(row);
|
||||
};
|
||||
const identity = "warm-real-agent-files";
|
||||
const warmExecution = { ...execution, binding: { ...execution.binding, runId: identity, executionWorkspaceId: ending === "projectless" ? identity : `${identity}-workspace` },
|
||||
session: { ...execution.session, normalizedSessionId: identity, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 } },
|
||||
} as NativeExecutionInputV1;
|
||||
const session = { close };
|
||||
state.execute.mockReset().mockImplementation(async (options) => {
|
||||
await options.onSession?.(session);
|
||||
return { result: { summary: "completed" }, terminal: { runTerminalState: "succeeded" },
|
||||
turnId: identity, normalizedSessionId: identity, providerSessionId: identity,
|
||||
driverKind: "test", driverVersion: "1", nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
|
||||
});
|
||||
try {
|
||||
await executePaperclipNativeSession({ db: leaseDb(warmExecution), execution: warmExecution,
|
||||
runnerInstanceId: identity, runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId: identity, remoteCwd: `/tmp/${identity}` },
|
||||
instructionWorkingCopy: { runId: identity, root: localRoot, preparationKey: "same-preparation", hasChanges: () => copies.hasChanges(row), collectStopped } });
|
||||
expect(close).not.toHaveBeenCalled();
|
||||
expect(order).toEqual([]);
|
||||
expect(await readFile(join(localRoot, "AGENTS.md"), "utf8")).toContain("registered");
|
||||
const second = { ...warmExecution, binding: { ...warmExecution.binding, runId: `${identity}-second`,
|
||||
...(ending === "projectless" ? { executionWorkspaceId: `${identity}-second` } : {}) } };
|
||||
expect(nativeSessionWorkspaceScope(second)).toEqual(nativeSessionWorkspaceScope(warmExecution));
|
||||
const currentCopy = { runId: second.binding.runId, root: localRoot, preparationKey: "same-preparation", hasChanges: () => copies.hasChanges(row), collectStopped };
|
||||
if (ending === "projectless") {
|
||||
const adopt = vi.fn(async () => currentCopy);
|
||||
for (const changed of [
|
||||
{ ...second.workspace, cwd: "/different" }, { ...second.workspace, repoUrl: "https://example.test/other" },
|
||||
{ ...second.workspace, repoRef: "other" }, { ...second.workspace, branchName: "other" },
|
||||
]) expect(await claimWarmNativeInstructionCopy({ priorExecution: warmExecution,
|
||||
companyId: second.binding.companyId, agentId: second.binding.agentId, executionWorkspaceId: second.binding.runId,
|
||||
workspace: changed, environmentId: identity, runId: second.binding.runId, preparationKey: "same-preparation", adopt })).toBeNull();
|
||||
expect(await claimWarmNativeInstructionCopy({ priorExecution: warmExecution,
|
||||
companyId: second.binding.companyId, agentId: second.binding.agentId, executionWorkspaceId: "managed-workspace",
|
||||
workspace: second.workspace, environmentId: identity, runId: second.binding.runId, preparationKey: "same-preparation", adopt })).toBeNull();
|
||||
expect(adopt).not.toHaveBeenCalled(); expect(close).not.toHaveBeenCalled();
|
||||
}
|
||||
const release = await claimWarmNativeInstructionCopy({ priorExecution: warmExecution,
|
||||
companyId: warmExecution.binding.companyId, agentId: warmExecution.binding.agentId,
|
||||
executionWorkspaceId: second.binding.executionWorkspaceId, workspace: second.workspace,
|
||||
environmentId: identity, runId: second.binding.runId, preparationKey: "same-preparation",
|
||||
adopt: async () => currentCopy });
|
||||
expect(release).toBeTypeOf("function");
|
||||
await executePaperclipNativeSession({ db: leaseDb(second), execution: second, runnerInstanceId: identity,
|
||||
runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId: identity, remoteCwd: `/tmp/${identity}` },
|
||||
instructionWorkingCopy: currentCopy });
|
||||
expect(state.execute.mock.calls.at(-1)?.[0].existingSession).toBe(session);
|
||||
expect(close).not.toHaveBeenCalled();
|
||||
expect(row.executionRoot).toBe(localRoot);
|
||||
await release?.(); // Consumed preparation cannot retire the retained owner.
|
||||
expect(close).not.toHaveBeenCalled();
|
||||
if (ending === "idle" || ending === "projectless") {
|
||||
await closeWarmNativeSessionsForEnvironment({ environmentId: identity, reason: "idle retirement" });
|
||||
await expect(access(localRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
} else {
|
||||
if (ending === "edit") await writeFile(join(localRoot, "AGENTS.md"), "Changed instructions");
|
||||
if (ending === "add") await writeFile(join(localRoot, "memory.txt"), "New memory");
|
||||
if (ending === "remove") await rm(join(localRoot, "AGENTS.md"));
|
||||
const adopt = vi.fn(async () => ({ ...currentCopy, runId: `${identity}-third` }));
|
||||
expect(await claimWarmNativeInstructionCopy({ priorExecution: second,
|
||||
companyId: second.binding.companyId, agentId: second.binding.agentId,
|
||||
executionWorkspaceId: second.binding.executionWorkspaceId, workspace: second.workspace,
|
||||
environmentId: identity, runId: `${identity}-third`, preparationKey: "same-preparation", adopt })).toBeNull();
|
||||
expect(adopt).toHaveBeenCalledOnce();
|
||||
}
|
||||
expect(order).toEqual(["closed", "collected"]);
|
||||
} finally {
|
||||
await closeWarmNativeSessionsForEnvironment({ environmentId: identity, reason: "test cleanup" });
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["final-config", "changed-root", "late-edit", "close-failure", "abandon", "concurrent", "foreign-company", "foreign-workspace"])("fences owned instruction preparation: %s", async scenario => {
|
||||
const identity = `warm-copy-${scenario}`;
|
||||
const order: string[] = [];
|
||||
let allowClose = scenario !== "close-failure";
|
||||
const session = { close: vi.fn(async () => { order.push("close"); if (!allowClose) throw new Error("retirement unconfirmed"); }) };
|
||||
const retirementFailed = vi.fn(async () => {});
|
||||
let lateEdit = false;
|
||||
const copy = { runId: identity, root: `/tmp/${identity}/instructions`, preparationKey: "key", hasChanges: async () => lateEdit,
|
||||
collectStopped: vi.fn(async () => { order.push("collect"); }), retirementFailed };
|
||||
const first = { ...execution, binding: { ...execution.binding, runId: identity, executionWorkspaceId: `${identity}-workspace` },
|
||||
session: { ...execution.session, normalizedSessionId: identity, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 } },
|
||||
} as NativeExecutionInputV1;
|
||||
const target = { kind: "remote" as const, transport: "sandbox" as const, environmentId: identity, remoteCwd: `/tmp/${identity}` };
|
||||
state.execute.mockReset().mockImplementation(async options => {
|
||||
await options.onSession?.(session);
|
||||
return { result: { summary: "completed" }, terminal: { runTerminalState: "succeeded" },
|
||||
turnId: identity, normalizedSessionId: identity, providerSessionId: identity,
|
||||
driverKind: "test", driverVersion: "1", nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
|
||||
});
|
||||
const second = { ...first, binding: { ...first.binding, runId: `${identity}-second` } };
|
||||
const current = { ...copy, runId: second.binding.runId,
|
||||
...(scenario === "changed-root" ? { root: `/tmp/${identity}/other-instructions` } : {}) };
|
||||
const adopt = vi.fn(async () => current);
|
||||
const claim = { priorExecution: first, companyId: first.binding.companyId, agentId: first.binding.agentId,
|
||||
executionWorkspaceId: first.binding.executionWorkspaceId, workspace: first.workspace, environmentId: identity,
|
||||
runId: second.binding.runId, preparationKey: "key", adopt };
|
||||
try {
|
||||
await executePaperclipNativeSession({ db: leaseDb(first), execution: first, runnerInstanceId: identity, runnerExecutionTarget: target, instructionWorkingCopy: copy });
|
||||
if (scenario.startsWith("foreign")) {
|
||||
expect(await claimWarmNativeInstructionCopy({ ...claim,
|
||||
...(scenario === "foreign-company" ? { companyId: "other" } : { executionWorkspaceId: "other" }) })).toBeNull();
|
||||
expect(adopt).not.toHaveBeenCalled(); expect(session.close).not.toHaveBeenCalled();
|
||||
} else {
|
||||
const release = await claimWarmNativeInstructionCopy(claim);
|
||||
if (scenario === "concurrent") await expect(claimWarmNativeInstructionCopy({ ...claim, runId: "competitor" })).rejects.toThrow("busy");
|
||||
if (scenario === "final-config" || scenario === "changed-root" || scenario === "late-edit") {
|
||||
lateEdit = scenario === "late-edit";
|
||||
const changed = scenario !== "final-config" ? second : { ...second, session: { ...second.session, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 99_000 } } };
|
||||
await expect(executePaperclipNativeSession({ db: leaseDb(changed), execution: changed, runnerInstanceId: identity,
|
||||
runnerExecutionTarget: target, instructionWorkingCopy: current })).rejects.toThrow(scenario === "late-edit"
|
||||
? "native_instruction_preparation_copy_changed" : "native_instruction_preparation_configuration_changed");
|
||||
expect(state.execute).toHaveBeenCalledOnce();
|
||||
}
|
||||
if (scenario === "close-failure") {
|
||||
await expect(release!()).rejects.toThrow("retirement unconfirmed");
|
||||
expect(retirementFailed).toHaveBeenCalledOnce();
|
||||
expect(copy.collectStopped).not.toHaveBeenCalled();
|
||||
expect(order).toEqual(["close"]);
|
||||
} else {
|
||||
await release?.();
|
||||
expect(order).toEqual(["close", "collect"]);
|
||||
}
|
||||
}
|
||||
} finally { allowClose = true; await closeWarmNativeSessionsForEnvironment({ environmentId: identity, reason: "test cleanup" }); }
|
||||
});
|
||||
|
||||
it.each(["config", "reset", "changed", "unavailable", "auth", "wrong-capability", "retirement-failure"])("hands off collection before probing or retiring a warm owner: %s", async scenario => {
|
||||
const identity = `warm-successor-${scenario}`, order: string[] = [];
|
||||
let allowClose = scenario !== "retirement-failure";
|
||||
const session = { close: vi.fn(async () => { order.push("stop"); if (!allowClose) throw new Error("stop unconfirmed"); }) };
|
||||
const priorCopy = { runId: identity, preparationKey: "before", hasChanges: vi.fn(async () => false),
|
||||
collectStopped: vi.fn(async () => { order.push("prior-pending"); }), retirementFailed: vi.fn(async () => {}) };
|
||||
const first = { ...execution, binding: { ...execution.binding, runId: identity, executionWorkspaceId: `${identity}-workspace` },
|
||||
session: { ...execution.session, normalizedSessionId: identity, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 } } } as NativeExecutionInputV1;
|
||||
const target = { kind: "remote" as const, transport: "sandbox" as const, environmentId: identity, remoteCwd: `/tmp/${identity}` };
|
||||
state.execute.mockReset().mockImplementation(async options => {
|
||||
await options.onSession?.(session);
|
||||
return { result: { summary: "completed" }, terminal: { runTerminalState: "succeeded" }, turnId: identity,
|
||||
normalizedSessionId: identity, providerSessionId: identity, driverKind: "test", driverVersion: "1",
|
||||
nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
|
||||
});
|
||||
const current = { runId: `${identity}-next`, preparationKey: scenario === "config" ? "after" : "before",
|
||||
hasChanges: vi.fn(async () => { order.push("current-probe"); return true; }),
|
||||
collectStopped: vi.fn(async () => { order.push("current-collect"); }), retirementFailed: vi.fn(async () => {}) };
|
||||
const adopt = vi.fn(async () => {
|
||||
order.push("handoff");
|
||||
if (scenario === "auth") throw new Error("authorization denied");
|
||||
if (scenario === "unavailable") return null;
|
||||
return scenario === "wrong-capability" ? { ...current, runId: "foreign" } : current;
|
||||
});
|
||||
try {
|
||||
await executePaperclipNativeSession({ db: leaseDb(first), execution: first, runnerInstanceId: identity, runnerExecutionTarget: target, instructionWorkingCopy: priorCopy });
|
||||
priorCopy.hasChanges.mockClear();
|
||||
const claiming = claimWarmNativeInstructionCopy({ priorExecution: first, companyId: first.binding.companyId, agentId: first.binding.agentId,
|
||||
executionWorkspaceId: first.binding.executionWorkspaceId, workspace: first.workspace, environmentId: identity,
|
||||
runId: current.runId, preparationKey: current.preparationKey, forceRetirement: scenario === "reset", adopt });
|
||||
if (["unavailable", "auth", "wrong-capability", "retirement-failure"].includes(scenario)) {
|
||||
await expect(claiming).rejects.toThrow(({ unavailable: "handoff_unavailable", auth: "authorization denied",
|
||||
"wrong-capability": "handoff_mismatch", "retirement-failure": "stop unconfirmed" } as Record<string, string>)[scenario]);
|
||||
} else expect(await claiming).toBeNull();
|
||||
expect(priorCopy.hasChanges).not.toHaveBeenCalled();
|
||||
expect(state.execute).toHaveBeenCalledOnce();
|
||||
if (["unavailable", "auth", "wrong-capability"].includes(scenario)) {
|
||||
expect(order).toEqual(["handoff", "stop", "prior-pending"]);
|
||||
expect(current.collectStopped).not.toHaveBeenCalled();
|
||||
} else {
|
||||
expect(priorCopy.collectStopped).not.toHaveBeenCalled();
|
||||
expect(order).toEqual(["handoff", ...(["config", "reset"].includes(scenario) ? [] : ["current-probe"]), "stop", ...(scenario === "retirement-failure" ? [] : ["current-collect"])]);
|
||||
if (scenario === "retirement-failure") {
|
||||
expect(current.retirementFailed).toHaveBeenCalledOnce();
|
||||
expect(current.collectStopped).not.toHaveBeenCalled();
|
||||
}
|
||||
}
|
||||
} finally { allowClose = true; await closeWarmNativeSessionsForEnvironment({ environmentId: identity, reason: "test cleanup" }); }
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ change: "addition", before: {}, after: { CUSTOM_TOKEN: "first" }, replaced: true },
|
||||
{ change: "rotation", before: { CUSTOM_TOKEN: "first" }, after: { CUSTOM_TOKEN: "second" }, replaced: true },
|
||||
@@ -7817,7 +8504,7 @@ describe("native warm session supervision", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it.each(["permission", "managed credential"])("replaces an idle warm provider session when its %s changes", async (change) => {
|
||||
it.each(["permission", "managed credential", "ACPX runtime contract", "unchanged Codex runtime contract"])("checks warm owner compatibility for %s", async (change) => {
|
||||
const firstClose = vi.fn(async () => undefined);
|
||||
const secondClose = vi.fn(async () => undefined);
|
||||
const firstSession = { close: firstClose };
|
||||
@@ -7825,7 +8512,9 @@ describe("native warm session supervision", () => {
|
||||
const base = {
|
||||
...execution,
|
||||
schema: "paperclip.native-execution-input.v4",
|
||||
provider: { kind: "codex", model: null, approvalPolicy: "never" },
|
||||
provider: change === "ACPX runtime contract"
|
||||
? { kind: "acpx", agent: "claude", model: "claude-sonnet-5", permissionMode: "approve-all" }
|
||||
: { kind: "codex", model: null, approvalPolicy: "never" },
|
||||
binding: {
|
||||
...execution.binding,
|
||||
runId: "run-permission-never",
|
||||
@@ -7839,7 +8528,7 @@ describe("native warm session supervision", () => {
|
||||
},
|
||||
session: {
|
||||
normalizedSessionId: "session-warm-permission",
|
||||
driverKind: "codex_app_server" as const,
|
||||
driverKind: change === "ACPX runtime contract" ? "acpx_runtime" as const : "codex_app_server" as const,
|
||||
protocolVersion: 1 as const,
|
||||
lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 20 },
|
||||
},
|
||||
@@ -7862,6 +8551,12 @@ describe("native warm session supervision", () => {
|
||||
highestContiguousSourceSeq: 1,
|
||||
usage: null,
|
||||
};
|
||||
const runtimeChange = change === "ACPX runtime contract" || change === "unchanged Codex runtime contract";
|
||||
const keepsOwner = change === "unchanged Codex runtime contract";
|
||||
const currentRuntimeContract = nativeSessionResume.nativeRuntimeContractForProvider(base.provider);
|
||||
const contract = runtimeChange
|
||||
? vi.spyOn(nativeSessionResume, "nativeRuntimeContractForProvider").mockReturnValue(undefined)
|
||||
: undefined;
|
||||
state.execute
|
||||
.mockReset()
|
||||
.mockImplementationOnce(async (options) => {
|
||||
@@ -7870,8 +8565,8 @@ describe("native warm session supervision", () => {
|
||||
return result;
|
||||
})
|
||||
.mockImplementationOnce(async (options) => {
|
||||
expect(options.existingSession).toBeUndefined();
|
||||
options.onSession?.(secondSession);
|
||||
expect(options.existingSession).toBe(keepsOwner ? firstSession : undefined);
|
||||
if (!keepsOwner) options.onSession?.(secondSession);
|
||||
return result;
|
||||
});
|
||||
|
||||
@@ -7885,21 +8580,22 @@ describe("native warm session supervision", () => {
|
||||
managedAiCredentialIdentity: "first-identity",
|
||||
runnerInstanceId: "runner",
|
||||
});
|
||||
contract?.mockReturnValue(currentRuntimeContract);
|
||||
await executePaperclipNativeSession({
|
||||
db: leaseDb(lowered),
|
||||
execution: lowered,
|
||||
managedAiCredentialIdentity: change === "managed credential" ? "second-identity" : "first-identity",
|
||||
runnerInstanceId: "runner",
|
||||
});
|
||||
expect(firstClose).toHaveBeenCalledWith({
|
||||
reason: "warm native session configuration changed",
|
||||
});
|
||||
if (keepsOwner) expect(firstClose).not.toHaveBeenCalled();
|
||||
else expect(firstClose).toHaveBeenCalledWith({ reason: "warm native session configuration changed" });
|
||||
expect(secondClose).not.toHaveBeenCalled();
|
||||
await vi.advanceTimersByTimeAsync(20);
|
||||
expect(secondClose).toHaveBeenCalledWith({
|
||||
expect(keepsOwner ? firstClose : secondClose).toHaveBeenCalledWith({
|
||||
reason: "warm native session idle timeout",
|
||||
});
|
||||
} finally {
|
||||
contract?.mockRestore();
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
@@ -8723,6 +9419,11 @@ describe("native process ownership", () => {
|
||||
},
|
||||
"acpx_runtime",
|
||||
],
|
||||
[
|
||||
"Pi ACPX without candidate authorization",
|
||||
{ kind: "acpx", agent: "pi", model: "openrouter/deepseek/deepseek-v4-flash-0731", permissionMode: "approve-reads" },
|
||||
"acpx_runtime",
|
||||
],
|
||||
])(
|
||||
"admits the qualified %s provider",
|
||||
async (_name, provider, driverKind) => {
|
||||
@@ -8761,7 +9462,7 @@ describe("native process ownership", () => {
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["pi", "copilot"])("rejects ACPX candidate %s without host authorization before constructing a backend", async (agent) => {
|
||||
it.each(["copilot"])("rejects ACPX candidate %s without host authorization before constructing a backend", async (agent) => {
|
||||
const piExecution = {
|
||||
...execution,
|
||||
binding: { ...execution.binding, runId: "run-acpx-pi-rejected" },
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { configuredEnvironment } from "../../vendor/paperclip-runner/index.js";
|
||||
import { agents } from "@paperclipai/db";
|
||||
import { dotRunnerBroker } from "../dot-runner-broker.js";
|
||||
import { configuredEnvironment } from "../../vendor/paperclip-runner/index.js";
|
||||
import { CURSOR_DISTRIBUTION_PINS, QUALIFIED_ACPX_PROFILES, QUALIFIED_ACPX_VERSION } from "../../vendor/paperclip-runner/index.js";
|
||||
import { isProviderMode } from "../../vendor/paperclip-runner/index.js";
|
||||
import { bundledRemoteProviderPackManifestPath, bundledRemoteRunnerBinary } from "../../vendor/paperclip-runner/index.js";
|
||||
@@ -38,6 +38,7 @@ import { nativeCompletionFeedback } from "./native-completion-feedback.js";
|
||||
import { hasAcknowledgedNativeReassignmentStopIntent, hasAcknowledgedNativeStopIntent } from "../acknowledged-native-stop.js";
|
||||
import { stoppedCodexTurnIsTextOnly } from "./stopped-codex-turn.js";
|
||||
import { prepareVerifiedRemoteProviderPack } from "./remote-provider-pack.js";
|
||||
import { selectRemotePiCompanion } from "./remote-pi-companion.js";
|
||||
import { readNativeLocalProcessStop, PROCESS_START_REQUESTED } from "../native-local-process-stop.js";
|
||||
import { remoteLeaseCleanupScope } from "../remote-execution-termination.js";
|
||||
import { resolveConnectorAssignments, isConnectorSkill } from "../connector-runtime.js";
|
||||
@@ -427,7 +428,23 @@ function clearNativeRuntimeRequestResolutions(runId: string): void {
|
||||
}
|
||||
}
|
||||
|
||||
export type NativeInstructionWorkingCopy = {
|
||||
/** Present only for a complete registered agent_files materialization. */
|
||||
runId?: string;
|
||||
preparationKey?: string;
|
||||
root?: string;
|
||||
checkpointWarm?: () => Promise<boolean>;
|
||||
hasChanges: () => Promise<boolean>;
|
||||
collectStopped: () => Promise<void>;
|
||||
retirementFailed?: () => Promise<void>;
|
||||
};
|
||||
|
||||
type WarmNativeSession = {
|
||||
instructionWorkingCopy?: NativeInstructionWorkingCopy;
|
||||
// Keep the admitted physical root independent of the per-run collection
|
||||
// capability, which instruction preparation adopts before final admission.
|
||||
instructionRoot?: string;
|
||||
instructionPreparationRunId?: string;
|
||||
agentId: string;
|
||||
instructionCopy?: { runId: string; root?: string; targetIdentity: string; collectStopped: () => Promise<void> };
|
||||
preparingRunId?: string;
|
||||
@@ -453,7 +470,13 @@ async function closeWarmNativeSession(entry: WarmNativeSession, reason: string,
|
||||
// Revoke before awaiting process retirement/checkpoint IO.
|
||||
const stopping = entry.githubAccess?.stop();
|
||||
try {
|
||||
await entry.session.close({ reason });
|
||||
try { await entry.session.close({ reason }); }
|
||||
catch (error) {
|
||||
try { await entry.instructionWorkingCopy?.retirementFailed?.(); }
|
||||
catch (receiptError) { throw new AggregateError([error, receiptError], "Warm instruction retirement and receipt both failed"); }
|
||||
throw error;
|
||||
}
|
||||
await entry.instructionWorkingCopy?.collectStopped();
|
||||
if (entry.instructionCopy?.runId !== preserveInstructionsForRunId) await entry.instructionCopy?.collectStopped();
|
||||
}
|
||||
finally { await stopping; }
|
||||
@@ -464,6 +487,65 @@ const warmNativeSessions = new Map<string, WarmNativeSession>();
|
||||
// not inspect or quarantine that owner's state until the save has finished.
|
||||
const closingWarmNativeSessions = new Map<string, Promise<void>>();
|
||||
|
||||
/** Reserve the existing physical copy before heartbeat composes AGENT_HOME and
|
||||
* task guidance. The final execution/configuration fence still owns admission. */
|
||||
export async function claimWarmNativeInstructionCopy(input: {
|
||||
priorExecution: NativeExecutionInput;
|
||||
companyId: string;
|
||||
agentId: string;
|
||||
executionWorkspaceId: string;
|
||||
workspace: NativeExecutionInput["workspace"];
|
||||
environmentId: string | null;
|
||||
runId: string;
|
||||
preparationKey: string;
|
||||
forceRetirement?: boolean;
|
||||
adopt: (previousRunId: string) => Promise<NativeInstructionWorkingCopy | null>;
|
||||
}): Promise<(() => Promise<void>) | null> {
|
||||
const prior = input.priorExecution;
|
||||
if (prior.binding.companyId !== input.companyId || prior.binding.agentId !== input.agentId
|
||||
|| JSON.stringify(nativeSessionWorkspaceScope(prior)) !== JSON.stringify(nativeSessionWorkspaceScope({
|
||||
binding: { ...prior.binding, runId: input.runId, executionWorkspaceId: input.executionWorkspaceId }, workspace: input.workspace,
|
||||
})) || prior.workspace.cwd !== input.workspace.cwd) return null;
|
||||
const scope = nativeSessionScopeKey(prior);
|
||||
const entry = warmNativeSessions.get(scope);
|
||||
if (!entry || !entry.instructionWorkingCopy?.runId || entry.environmentId !== input.environmentId) return null;
|
||||
if (entry.busy) throw new Error("native_session_supervisor_busy");
|
||||
entry.busy = true;
|
||||
if (entry.idleTimer !== null) clearTimeout(entry.idleTimer);
|
||||
entry.idleTimer = null;
|
||||
const retire = async (reason: string) => {
|
||||
if (warmNativeSessions.get(scope) !== entry) return;
|
||||
entry.closeOnReleaseReason = reason;
|
||||
try {
|
||||
await closeWarmNativeSession(entry, reason);
|
||||
if (warmNativeSessions.get(scope) === entry) warmNativeSessions.delete(scope);
|
||||
} finally { entry.busy = false; }
|
||||
};
|
||||
try {
|
||||
const preparationChanged = input.forceRetirement === true || entry.instructionWorkingCopy.preparationKey !== input.preparationKey;
|
||||
// The prior capability may close over an expired remote lease. Rebind
|
||||
// collection authority first; adoption itself grants no reuse permission.
|
||||
const adopted = await input.adopt(entry.instructionWorkingCopy.runId);
|
||||
if (!adopted) {
|
||||
await retire("warm instruction materialization cannot be handed off");
|
||||
throw new Error("native_instruction_materialization_handoff_unavailable");
|
||||
}
|
||||
if (adopted.runId !== input.runId || adopted.preparationKey !== input.preparationKey) throw new Error("native_instruction_materialization_handoff_mismatch");
|
||||
entry.instructionWorkingCopy = adopted;
|
||||
entry.instructionPreparationRunId = input.runId;
|
||||
if (entry.closeOnReleaseReason !== undefined || preparationChanged || await adopted.hasChanges()) {
|
||||
await retire("warm instruction materialization changed before preparation");
|
||||
return null;
|
||||
}
|
||||
return async () => {
|
||||
if (entry.instructionPreparationRunId === input.runId) await retire("warm instruction preparation abandoned");
|
||||
};
|
||||
} catch (error) {
|
||||
if (entry.closeOnReleaseReason === undefined) await retire("warm instruction preparation failed");
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
function instructionTargetIdentity(target?: AdapterExecutionTarget | null): string {
|
||||
return JSON.stringify(target?.kind === "remote" ? {
|
||||
environmentId: target.environmentId, cwd: target.remoteCwd,
|
||||
@@ -560,19 +642,23 @@ async function closeIdleWarmNativeSessions(input: {
|
||||
continue;
|
||||
}
|
||||
if (entry.idleTimer !== null) clearTimeout(entry.idleTimer);
|
||||
// Remove ownership before awaiting close so a racing continuation cannot
|
||||
// adopt a session whose transport is already shutting down.
|
||||
warmNativeSessions.delete(sessionId);
|
||||
// Keep a fenced owner until both retirement and stopped collection succeed.
|
||||
// Failure must not make a live directory eligible for a replacement owner.
|
||||
entry.busy = true;
|
||||
entry.closeOnReleaseReason = input.reason;
|
||||
const closing = Promise.resolve().then(() =>
|
||||
closeWarmNativeSession(entry, input.reason),
|
||||
);
|
||||
closingWarmNativeSessions.set(sessionId, closing);
|
||||
try {
|
||||
await closing;
|
||||
if (warmNativeSessions.get(sessionId) === entry) warmNativeSessions.delete(sessionId);
|
||||
closed += 1;
|
||||
} catch {
|
||||
if (!entry.instructionWorkingCopy?.runId) warmNativeSessions.delete(sessionId);
|
||||
failed += 1;
|
||||
} finally {
|
||||
entry.busy = false;
|
||||
if (closingWarmNativeSessions.get(sessionId) === closing) {
|
||||
closingWarmNativeSessions.delete(sessionId);
|
||||
}
|
||||
@@ -1166,6 +1252,7 @@ export function createGovernedWaitEventObservation(
|
||||
resolvePending: () => Promise<PrpStructuredRunResult | null>,
|
||||
) {
|
||||
const pendingTools = new Set<string>();
|
||||
let providerLost = false;
|
||||
let generation = 0;
|
||||
let observation: {
|
||||
sourceInstanceId: string;
|
||||
@@ -1179,6 +1266,7 @@ export function createGovernedWaitEventObservation(
|
||||
const currentGeneration = ++generation;
|
||||
observation = null;
|
||||
const payload = record(event.payload);
|
||||
providerLost ||= event.eventType === "runtime_request.expired" && payload.reason === "provider_process_lost";
|
||||
const kind = payload.kind;
|
||||
const tool = ["dynamicToolCall", "mcpToolCall", "commandExecution"].includes(String(kind));
|
||||
if (event.itemId) {
|
||||
@@ -1194,7 +1282,9 @@ export function createGovernedWaitEventObservation(
|
||||
if (event.eventType === "item.completed" && (
|
||||
pendingTools.size > 0 || (!tool && !(kind === "agentMessage" && payload.channel === "final"))
|
||||
)) return;
|
||||
if (!eligible) return;
|
||||
// A replacement question preserves human input after a provider crash;
|
||||
// it does not authorize a successful suspension of the failed execution.
|
||||
if (providerLost || !eligible) return;
|
||||
const result = await resolvePending();
|
||||
if (generation !== currentGeneration || result === null) return;
|
||||
// If the interaction is answered after this read, parking remains the
|
||||
@@ -1574,8 +1664,8 @@ function nativeSessionKey(execution: NativeExecutionInput): string {
|
||||
);
|
||||
}
|
||||
|
||||
function nativeSessionWorkspaceScope(execution: NativeExecutionInput) {
|
||||
if (execution.provider.kind === "openai_dot") return { kind: "none" as const };
|
||||
export function nativeSessionWorkspaceScope(execution: { binding: Pick<NativeExecutionInput["binding"], "runId" | "executionWorkspaceId">; workspace: NativeExecutionInput["workspace"] }) {
|
||||
if ("access" in execution.workspace) return { kind: "none" as const };
|
||||
// Projectless local runs use the heartbeat run id as a durable placeholder
|
||||
// rather than fabricating an execution_workspaces row. Do not let that
|
||||
// per-run placeholder break continuity for the same provider session; the
|
||||
@@ -5326,7 +5416,7 @@ function hasIdleWarmNativeSessionOwner(input: {
|
||||
}): boolean {
|
||||
if (input.execution.session.lifecyclePolicy.mode !== "warm") return false;
|
||||
const entry = warmNativeSessions.get(nativeSessionScopeKey(input.execution));
|
||||
if (!entry || entry.busy) return false;
|
||||
if (!entry || (entry.busy && entry.instructionPreparationRunId !== input.execution.binding.runId)) return false;
|
||||
// A verified idle owner proves the checkpoint belongs to this session even
|
||||
// when its process must later rotate to a new run-scoped broker capability.
|
||||
// Local environments also have an id. Compare the same environment binding
|
||||
@@ -5541,6 +5631,7 @@ export function providerSessionIdentityFromDurableProviderState(input: {
|
||||
identity.effectiveModel !== expectedModel ||
|
||||
identity.permissionMode !== input.execution.provider.permissionMode ||
|
||||
!acpxRecoveryModeMatches(input.execution.provider, descriptor.mode, identity.mode) ||
|
||||
!acpxRecoveryPiThinkingMatches(input.execution.provider, descriptor.piThinkingLevel, identity.piThinkingLevel) ||
|
||||
!["approve-all", "approve-paperclip", "approve-reads", "deny-all"].includes(
|
||||
String(identity.permissionMode),
|
||||
) ||
|
||||
@@ -5648,6 +5739,12 @@ function acpxRecoveryModeMatches(
|
||||
&& observedModes.every(mode => mode === expected);
|
||||
}
|
||||
|
||||
function acpxRecoveryPiThinkingMatches(provider: NativeExecutionInput["provider"], ...observed: unknown[]): boolean {
|
||||
const expected = record(provider).piThinkingLevel;
|
||||
if (provider.kind === "acpx" && provider.agent === "pi") return ["off", "low", "high", "max"].includes(String(expected)) && observed.every(value => value === expected);
|
||||
return expected === undefined && observed.every(value => value === undefined);
|
||||
}
|
||||
|
||||
export function providerSessionIdentityTransitionIsAllowed(input: {
|
||||
execution: NativeExecutionInput;
|
||||
previous: unknown;
|
||||
@@ -5658,6 +5755,7 @@ export function providerSessionIdentityTransitionIsAllowed(input: {
|
||||
record(record(input.previous).providerSessionIdentity).mode,
|
||||
record(record(input.current).providerSessionIdentity).mode,
|
||||
)) return false;
|
||||
if (!acpxRecoveryPiThinkingMatches(input.execution.provider, record(record(input.previous).providerSessionIdentity).piThinkingLevel, record(record(input.current).providerSessionIdentity).piThinkingLevel)) return false;
|
||||
if (canonicalJson(input.previous) === canonicalJson(input.current)) {
|
||||
return true;
|
||||
}
|
||||
@@ -6054,23 +6152,27 @@ async function releaseWarmNativeSession(
|
||||
entry.lastActivityAt = new Date().toISOString();
|
||||
if (entry.idleTimer !== null) clearTimeout(entry.idleTimer);
|
||||
if (failed || entry.closeOnReleaseReason !== undefined) {
|
||||
warmNativeSessions.delete(sessionId);
|
||||
const closing = closeWarmNativeSession(entry,
|
||||
entry.closeOnReleaseReason ?? "warm native session failed");
|
||||
// Restart checkpointing is required to restore this successful session.
|
||||
// Surface failure instead of reporting a clean release without authority.
|
||||
if (entry.closeOnReleaseReason !== undefined) await closing;
|
||||
else await closing.catch(() => undefined);
|
||||
const requireCleanClose = entry.closeOnReleaseReason !== undefined;
|
||||
entry.closeOnReleaseReason ??= "warm native session failed";
|
||||
entry.busy = true;
|
||||
try {
|
||||
await closeWarmNativeSession(entry, entry.closeOnReleaseReason);
|
||||
if (warmNativeSessions.get(sessionId) === entry) warmNativeSessions.delete(sessionId);
|
||||
} catch (error) {
|
||||
if (!entry.instructionWorkingCopy?.runId) warmNativeSessions.delete(sessionId);
|
||||
if (requireCleanClose) throw error;
|
||||
} finally { entry.busy = false; }
|
||||
return;
|
||||
}
|
||||
entry.idleTimer = setTimeout(() => {
|
||||
const current = warmNativeSessions.get(sessionId);
|
||||
// clearTimeout cannot revoke an already-queued callback. The entry object
|
||||
// is the idle timer's ownership fence across a later warm acquisition.
|
||||
if (current !== entry || current.busy) return;
|
||||
warmNativeSessions.delete(sessionId);
|
||||
void closeWarmNativeSession(current, "warm native session idle timeout")
|
||||
.catch(() => undefined);
|
||||
current.busy = true;
|
||||
current.closeOnReleaseReason = "warm native session idle timeout";
|
||||
void closeWarmNativeSession(current, current.closeOnReleaseReason)
|
||||
.then(() => { if (warmNativeSessions.get(sessionId) === current) warmNativeSessions.delete(sessionId); })
|
||||
.catch(() => { if (!current.instructionWorkingCopy?.runId && warmNativeSessions.get(sessionId) === current) warmNativeSessions.delete(sessionId); })
|
||||
.finally(() => { current.busy = false; });
|
||||
}, idleTimeoutMs);
|
||||
entry.idleTimer.unref();
|
||||
}
|
||||
@@ -7347,15 +7449,9 @@ export async function executePaperclipNativeSession(input: {
|
||||
chatAttachmentReadScope?: NativeChatAttachmentReadScope;
|
||||
onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise<void>;
|
||||
onEvent?: (event: AdapterRuntimeEvent) => Promise<void>;
|
||||
/** Only this run's registered directory. Validate a warm checkpoint at the
|
||||
* terminal boundary, or collect after owned shutdown when it cannot settle. */
|
||||
instructionWorkingCopy?: {
|
||||
runId?: string;
|
||||
root?: string;
|
||||
checkpointWarm?: () => Promise<boolean>;
|
||||
hasChanges: () => Promise<boolean>;
|
||||
collectStopped: () => Promise<void>;
|
||||
};
|
||||
/** Only this run's registered private instruction entry.
|
||||
* Probe at terminal; persist only after owned shutdown. */
|
||||
instructionWorkingCopy?: NativeInstructionWorkingCopy;
|
||||
/** Persist task-level continuity before a durable goal can outlive this run. */
|
||||
onUsage?: (receipt: AdapterUsageCheckpoint) => Promise<void>;
|
||||
onGoalCheckpoint?: (snapshot: PersistedNativeSession) => Promise<void>;
|
||||
@@ -7571,7 +7667,7 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
}
|
||||
if (
|
||||
input.execution.provider.kind === "acpx" &&
|
||||
["pi", "copilot"].includes(input.execution.provider.agent) &&
|
||||
["copilot"].includes(input.execution.provider.agent) &&
|
||||
!resolveAcpxQualification(input.execution.provider, process.env)
|
||||
) {
|
||||
throw new Error(
|
||||
@@ -8352,7 +8448,7 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
entry.closeOnReleaseReason !== undefined ||
|
||||
entry.configDigest !== warmConfigDigest ||
|
||||
entry.configuredEnvironmentDigest !== configuredEnvironmentDigest ||
|
||||
entry.instructionCopy?.root !== input.instructionWorkingCopy?.root ||
|
||||
entry.instructionRoot !== input.instructionWorkingCopy?.root ||
|
||||
entry.managedAiCredentialIdentity !== input.managedAiCredentialIdentity ||
|
||||
credentialRunChanged ||
|
||||
Boolean(entry.githubAccess) !== Boolean(input.managedGitHub) ||
|
||||
@@ -8363,20 +8459,36 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
(input.runnerEnvironment?.PAPERCLIP_RUNNER_NETWORK_ACCESS ===
|
||||
"enabled")
|
||||
) {
|
||||
if (entry.busy) throw new Error("native_session_supervisor_busy");
|
||||
const preparedCopy = entry.instructionPreparationRunId === input.execution.binding.runId;
|
||||
if (entry.busy && !preparedCopy) throw new Error("native_session_supervisor_busy");
|
||||
if (entry.idleTimer !== null) clearTimeout(entry.idleTimer);
|
||||
warmNativeSessions.delete(warmSessionId);
|
||||
// Preparation may already have handed this directory to the incoming
|
||||
// run. Retiring the old transport must not delete the new run's root;
|
||||
// its own stop callback owns collection from this point forward.
|
||||
await closeWarmNativeSession(entry, "warm native session configuration changed", input.execution.binding.runId);
|
||||
entry.busy = true;
|
||||
entry.closeOnReleaseReason = "warm native session configuration changed";
|
||||
try {
|
||||
await closeWarmNativeSession(entry, entry.closeOnReleaseReason, input.execution.binding.runId);
|
||||
warmNativeSessions.delete(warmSessionId);
|
||||
} finally { entry.busy = false; }
|
||||
if (preparedCopy) throw new Error("native_instruction_preparation_configuration_changed");
|
||||
persistedWarmSession = loadWarmNativeCheckpoint(
|
||||
input.execution,
|
||||
warmConfigDigest,
|
||||
input.runnerExecutionTarget?.kind ?? "local",
|
||||
);
|
||||
} else {
|
||||
if (entry.busy) throw new Error("native_session_supervisor_busy");
|
||||
if (entry.busy && entry.instructionPreparationRunId !== input.execution.binding.runId) throw new Error("native_session_supervisor_busy");
|
||||
if (entry.instructionWorkingCopy?.runId && entry.instructionPreparationRunId !== input.execution.binding.runId) {
|
||||
throw new Error("native_instruction_materialization_not_claimed");
|
||||
}
|
||||
if (entry.instructionPreparationRunId !== undefined && await entry.instructionWorkingCopy!.hasChanges()) {
|
||||
entry.closeOnReleaseReason = "warm instruction copy changed during preparation";
|
||||
try {
|
||||
await closeWarmNativeSession(entry, entry.closeOnReleaseReason);
|
||||
warmNativeSessions.delete(warmSessionId);
|
||||
} finally { entry.busy = false; }
|
||||
throw new Error("native_instruction_preparation_copy_changed");
|
||||
}
|
||||
entry.instructionPreparationRunId = undefined;
|
||||
entry.instructionWorkingCopy = input.instructionWorkingCopy?.checkpointWarm ? undefined : input.instructionWorkingCopy;
|
||||
entry.busy = true;
|
||||
entry.ownerToken = warmSessionOwnerToken;
|
||||
entry.environmentId =
|
||||
@@ -8729,6 +8841,8 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
existing.session = session;
|
||||
} else
|
||||
warmNativeSessions.set(warmSessionId, {
|
||||
instructionWorkingCopy: input.instructionWorkingCopy?.checkpointWarm ? undefined : input.instructionWorkingCopy,
|
||||
instructionRoot: input.instructionWorkingCopy?.root,
|
||||
agentId: input.execution.binding.agentId,
|
||||
managedAiCredentialIdentity: input.managedAiCredentialIdentity,
|
||||
githubAuthenticationMode:
|
||||
@@ -8756,7 +8870,7 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
lastActivityAt: new Date().toISOString(),
|
||||
});
|
||||
const owner = warmNativeSessions.get(warmSessionId);
|
||||
if (owner && input.instructionWorkingCopy?.runId) owner.instructionCopy = {
|
||||
if (owner && input.instructionWorkingCopy?.runId && input.instructionWorkingCopy.checkpointWarm) owner.instructionCopy = {
|
||||
runId: input.instructionWorkingCopy.runId, root: input.instructionWorkingCopy.root, targetIdentity: instructionTargetIdentity(input.runnerExecutionTarget),
|
||||
collectStopped: input.instructionWorkingCopy.collectStopped,
|
||||
};
|
||||
@@ -9453,7 +9567,8 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
.catch(() => undefined);
|
||||
throw error;
|
||||
}
|
||||
const collectedByOwner = ownedSession?.instructionCopy?.collectStopped === instructionCopy?.collectStopped;
|
||||
const collectedByOwner = ownedSession?.instructionCopy?.collectStopped === instructionCopy?.collectStopped
|
||||
|| ownedSession?.instructionWorkingCopy?.collectStopped === instructionCopy?.collectStopped;
|
||||
if (collectInstructions && ownedSession) {
|
||||
// Keep the unchanged warm path intact. A changed private instruction copy
|
||||
// requires the existing checkpoint-and-close boundary before collection.
|
||||
@@ -9792,7 +9907,7 @@ type RemoteProviderPackManifest = {
|
||||
distDigest: string;
|
||||
bridgeDigest: string;
|
||||
acpxProfileDigests: typeof REMOTE_PROVIDER_PACK_PROFILE_DIGESTS;
|
||||
providers?: Partial<Record<"cursor", {
|
||||
providers?: Partial<Record<"pi" | "cursor", {
|
||||
version: string; profileDigest: string; closureDigest: string; qualification: "qualified" | "pending";
|
||||
path: string; sha256: string;
|
||||
}>>;
|
||||
@@ -9981,9 +10096,10 @@ function readRemoteProviderPackIdentity(packRoot: string, verifyControllerFiles:
|
||||
}
|
||||
for (const [provider, candidate] of Object.entries(candidates)) {
|
||||
const expectedPath = `provider-assets/${provider}/${payload.target.platform}-${payload.target.architecture}`;
|
||||
if (!(inventory === "providers" ? ["cursor"] : ["cursor", "copilot", "pi"]).includes(provider) || !candidate
|
||||
if (!(inventory === "providers" ? ["pi", "cursor"] : ["cursor", "copilot", "pi"]).includes(provider) || !candidate
|
||||
|| Object.keys(candidate).some(key => !["version", "profileDigest", "closureDigest", "qualification", "path", "sha256"].includes(key))
|
||||
|| candidate.qualification !== (provider === "cursor" ? "qualified" : "pending") || candidate.path !== expectedPath
|
||||
|| candidate.qualification !== (["pi", "cursor"].includes(provider) ? "qualified" : "pending")
|
||||
|| candidate.path !== expectedPath
|
||||
|| typeof candidate.version !== "string" || !candidate.version || candidate.version.length > 120
|
||||
|| !/^sha256:[a-f0-9]{64}$/.test(candidate.profileDigest)
|
||||
|| !/^sha256:[a-f0-9]{64}$/.test(candidate.closureDigest)
|
||||
@@ -11250,8 +11366,16 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
remoteTarget !== null &&
|
||||
(input.execution.provider.kind === "opencode" ||
|
||||
input.execution.provider.kind === "acpx");
|
||||
// Pi's explicit operator import supplies the target-platform authority. Never
|
||||
// substitute a Mac controller daemon or trust an image's self-reported hashes.
|
||||
// Existing explicit overrides and every other provider keep their old path.
|
||||
const remotePiCompanion = await selectRemotePiCompanion({
|
||||
provider: input.execution.provider, remote: remoteTarget !== null,
|
||||
binaryOverride: input.runnerRemoteBinaryPath, packOverride: input.runnerRemoteProviderPackPath,
|
||||
workspaceRoot: input.execution.workspace.cwd,
|
||||
});
|
||||
const configuredProviderPackRoot =
|
||||
input.runnerRemoteProviderPackPath?.trim() || null;
|
||||
input.runnerRemoteProviderPackPath?.trim() || remotePiCompanion?.providerPack || null;
|
||||
let expectedProviderPackManifest: RemoteProviderPackManifest | null = null;
|
||||
const useBundledCursorImageAssets = requiresRemoteProviderPack && !configuredProviderPackRoot &&
|
||||
input.execution.provider.kind === "acpx" && input.execution.provider.agent === "cursor";
|
||||
@@ -11284,7 +11408,7 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
// When an explicit remote artifact is configured, prepareRemoteRunner stages
|
||||
// these exact bytes at remoteBinary before launch.
|
||||
const controllerRunnerBinary = remoteTarget
|
||||
? input.runnerRemoteBinaryPath?.trim() || (useBundledCursorImageAssets ? bundledRemoteRunnerBinary() : resolvePaperclipRunnerBinary())
|
||||
? input.runnerRemoteBinaryPath?.trim() || remotePiCompanion?.runnerBinary || (useBundledCursorImageAssets ? bundledRemoteRunnerBinary() : resolvePaperclipRunnerBinary())
|
||||
: resolvePaperclipRunnerBinary();
|
||||
const explicitRemoteCodex = input.runnerRemoteCodexPath?.trim() || null;
|
||||
const remoteCodexNpmSpec = input.runnerRemoteCodexNpmSpec?.trim() || null;
|
||||
@@ -11663,7 +11787,7 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
if (!existsSync(sourceBinary)) {
|
||||
throw new Error("runner_remote_artifact_unavailable");
|
||||
}
|
||||
if (!explicitRemoteBinary) {
|
||||
if (!explicitRemoteBinary && !remotePiCompanion) {
|
||||
const platform = await remoteCommandRunner.execute({
|
||||
command: "sh",
|
||||
args: ["-c", "uname -s; uname -m"],
|
||||
@@ -12916,9 +13040,12 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
? {
|
||||
acpxAgent: input.execution.provider.agent,
|
||||
// Read only the server operator environment, never agent/runtime env.
|
||||
acpxCandidateProfile: resolveAcpxQualification(input.execution.provider, process.env),
|
||||
acpxCandidateProfile: input.execution.provider.agent === "pi"
|
||||
? undefined
|
||||
: resolveAcpxQualification(input.execution.provider, process.env),
|
||||
acpxPermissionMode: input.execution.provider.permissionMode,
|
||||
acpxMode: input.execution.provider.mode,
|
||||
piThinkingLevel: input.execution.provider.piThinkingLevel,
|
||||
acpxPermissionModePinned:
|
||||
input.execution.schema === "paperclip.native-execution-input.v4" ||
|
||||
input.execution.schema === "paperclip.native-execution-input.v5",
|
||||
|
||||
@@ -734,6 +734,10 @@ const recoveryFakeCodex = resolve(
|
||||
const previousStateBase = process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
||||
const server = createServer();
|
||||
let firstSession: NativeSession | undefined;
|
||||
const ownedSessions = new Set<NativeSession>();
|
||||
const ownedSpawns: Array<{ pid: number; processGroupId: number | null; startedAt: string }> = [];
|
||||
const onSpawn = async (meta: typeof ownedSpawns[number]) => { ownedSpawns.push(meta); };
|
||||
let executionCloseCompleted = false;
|
||||
const runnerDiagnostics: string[] = [];
|
||||
const onRunnerLog = async (_stream: "stdout" | "stderr", chunk: string) => {
|
||||
runnerDiagnostics.push(chunk.slice(-4_096));
|
||||
@@ -860,6 +864,7 @@ const recoveryFakeCodex = resolve(
|
||||
runnerInstanceId,
|
||||
runnerEnvironment: environment,
|
||||
onLog: onRunnerLog,
|
||||
onSpawn,
|
||||
});
|
||||
firstSession = await firstBackend.openSession({
|
||||
identity: {
|
||||
@@ -871,6 +876,7 @@ const recoveryFakeCodex = resolve(
|
||||
},
|
||||
workingDirectory: workspace,
|
||||
});
|
||||
ownedSessions.add(firstSession);
|
||||
const checkpoint = await firstSession.snapshot();
|
||||
expect(checkpoint.identity).toEqual({
|
||||
companyId,
|
||||
@@ -1009,6 +1015,7 @@ const recoveryFakeCodex = resolve(
|
||||
runnerInstanceId,
|
||||
runnerEnvironment: environment,
|
||||
onLog: onRunnerLog,
|
||||
onSpawn,
|
||||
});
|
||||
let continuity: Record<string, unknown> | undefined;
|
||||
const controlPlaneInstanceId = randomUUID();
|
||||
@@ -1031,6 +1038,12 @@ const recoveryFakeCodex = resolve(
|
||||
runnerInstanceId,
|
||||
controlPlaneInstanceId,
|
||||
timeoutMs: 20_000,
|
||||
// This disposable real-daemon fixture must join full retirement before
|
||||
// removing its controller routes and durable state. The production
|
||||
// default deliberately permits asynchronous cleanup after a result.
|
||||
requireSessionCloseBeforeReturn: true,
|
||||
onSession(value) { if (value) ownedSessions.add(value); },
|
||||
async onSessionClosed() { executionCloseCompleted = true; },
|
||||
controlPlane: port,
|
||||
async onContinuityBreak(value) {
|
||||
continuity = value;
|
||||
@@ -1064,6 +1077,12 @@ const recoveryFakeCodex = resolve(
|
||||
terminal: { runTerminalState: "succeeded" },
|
||||
normalizedSessionId,
|
||||
});
|
||||
expect(executionCloseCompleted).toBe(true);
|
||||
expect(ownedSpawns.length).toBeGreaterThanOrEqual(2);
|
||||
for (const { pid } of ownedSpawns) {
|
||||
expect(() => process.kill(pid, 0)).toThrow(expect.objectContaining({ code: "ESRCH" }));
|
||||
}
|
||||
console.info("Recovery fixture retired owned daemons", ownedSpawns);
|
||||
expect(continuity).toMatchObject({
|
||||
// The daemon can reject the damaged retained input during startup,
|
||||
// before attach gets a chance to reject the unsettled provider session.
|
||||
@@ -1201,9 +1220,13 @@ const recoveryFakeCodex = resolve(
|
||||
.where(eq(issues.id, issueId));
|
||||
expect(task).toEqual({ id: issueId, assigneeAgentId: agentId });
|
||||
} finally {
|
||||
await firstSession
|
||||
?.close({ reason: "Recovery fixture cleanup" })
|
||||
.catch(() => undefined);
|
||||
// onSession(null) quarantines the runtime owner before close finishes;
|
||||
// keep every published handle so an assertion/error cannot lose cleanup.
|
||||
if (firstSession) ownedSessions.add(firstSession);
|
||||
const closed = await Promise.allSettled([...ownedSessions].map((session) =>
|
||||
Promise.resolve().then(() => session.close({ reason: "Recovery fixture cleanup" })),
|
||||
));
|
||||
const closeFailures = closed.filter((result) => result.status === "rejected");
|
||||
runnerPrpWebSocketInternals.resetForTests();
|
||||
server.closeAllConnections();
|
||||
await new Promise<void>((done) => server.close(() => done()));
|
||||
@@ -1211,6 +1234,10 @@ const recoveryFakeCodex = resolve(
|
||||
delete process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
||||
else process.env.PAPERCLIP_RUNNER_STATE_DIR = previousStateBase;
|
||||
await database.cleanup();
|
||||
if (closeFailures.length) {
|
||||
// Retain exact fixture state if owned retirement cannot be proven.
|
||||
throw new AggregateError(closeFailures.map((result) => result.reason), "Recovery fixture session cleanup failed");
|
||||
}
|
||||
await rm(scratch, { recursive: true, force: true });
|
||||
}
|
||||
},
|
||||
|
||||
@@ -14,6 +14,7 @@ import {
|
||||
createDb,
|
||||
heartbeatRunEvents,
|
||||
heartbeatRuns,
|
||||
issueComments,
|
||||
issueRecoveryActions,
|
||||
issueRelations,
|
||||
issueThreadInteractions,
|
||||
@@ -42,7 +43,7 @@ import {
|
||||
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
import { PaperclipControlPlanePort } from "./paperclip-control-plane-port.js";
|
||||
import { NativeRunCoordinatorStore } from "./native-run-coordinator-store.js";
|
||||
import { finalizeNativeRun } from "./native-run-finalizer.js";
|
||||
import { finalizeNativeRun, repairCommittedNativeChatResponse } from "./native-run-finalizer.js";
|
||||
import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js";
|
||||
import { issueThreadInteractionService } from "../issue-thread-interactions.js";
|
||||
import { claimNativeReviewExecutionLock, getNativeReviewAssignment } from "./native-review-participant.js";
|
||||
@@ -974,7 +975,37 @@ describe("PaperclipControlPlanePort conformance", () => {
|
||||
expect(response.decision.chosenSource).toBe("final_agent_message");
|
||||
expect(stored.findIndex((row) => row.eventType === "run.result.accepted"))
|
||||
.toBeGreaterThan(stored.findIndex((row) => row.eventType === "item.completed"));
|
||||
await finalizeNativeRun({ db, runId: taskRunId, workspaceFinalizeStatus: "succeeded" });
|
||||
// A workspace recovery owner can commit before the live heartbeat reaches
|
||||
// presentation. The file-preparation comment must not hide the real reply.
|
||||
const [prepared] = await db.insert(issueComments).values({
|
||||
companyId: identity.companyId, issueId: taskIssueId,
|
||||
authorAgentId: identity.agentId, authorType: "agent", createdByRunId: taskRunId,
|
||||
body: "Prepared Continuity file for this response.",
|
||||
}).returning();
|
||||
const receipt = { semanticToolReceipts: { file: {
|
||||
operationId: "register_deliverable", result: {
|
||||
commandId: "deliverable-prepared:attachment-1", disposition: "applied",
|
||||
attachmentId: "attachment-1", entityRefs: ["attachment-1", prepared!.id],
|
||||
},
|
||||
} } };
|
||||
await db.update(heartbeatRuns).set({
|
||||
contextSnapshot: { skipIssueComment: true },
|
||||
resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify(receipt)}::jsonb`,
|
||||
}).where(eq(heartbeatRuns.id, taskRunId));
|
||||
await finalizeNativeRun({ db, runId: taskRunId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
|
||||
const repair = () => repairCommittedNativeChatResponse(db, {
|
||||
companyId: identity.companyId, issueId: taskIssueId, runId: taskRunId,
|
||||
});
|
||||
expect(await repair()).toBe(false);
|
||||
await db.update(heartbeatRuns).set({ contextSnapshot: { externalChatContinuation: true } })
|
||||
.where(eq(heartbeatRuns.id, taskRunId));
|
||||
expect(await repair()).toBe(false);
|
||||
await db.update(heartbeatRuns).set({ contextSnapshot: {} }).where(eq(heartbeatRuns.id, taskRunId));
|
||||
expect(await repair()).toBe(true);
|
||||
expect(await repair()).toBe(false);
|
||||
expect(await db.select({ body: issueComments.body }).from(issueComments)
|
||||
.where(eq(issueComments.createdByRunId, taskRunId)).orderBy(asc(issueComments.createdAt)))
|
||||
.toEqual([{ body: prepared!.body }, { body: finalText }]);
|
||||
await expect(port.completeRun({
|
||||
result: taskResult,
|
||||
terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL,
|
||||
|
||||
@@ -5,6 +5,7 @@ import {
|
||||
PAPERCLIP_RUNNER_ACPX_PROFILES,
|
||||
resolvePaperclipRunnerPermissionMode,
|
||||
resolvePaperclipRunnerCursorMode,
|
||||
resolvePaperclipRunnerPiThinkingLevel,
|
||||
type PaperclipRunnerProvider,
|
||||
} from "@paperclipai/adapter-utils";
|
||||
import {
|
||||
@@ -29,6 +30,7 @@ export const DEFAULT_ACPX_RUNNER_MODELS = {
|
||||
// These profiles require explicit configuration. Admission belongs to the runner.
|
||||
codex: null,
|
||||
cursor: null,
|
||||
pi: null,
|
||||
} as const;
|
||||
|
||||
export type QualifiedPaperclipRunnerAcpxAgent =
|
||||
@@ -127,6 +129,7 @@ export type PaperclipRunnerNativeProviderInput =
|
||||
acpxAgent: AdmittedPaperclipRunnerAcpxAgent;
|
||||
acpxPermissionMode: "approve-all" | "approve-paperclip" | "approve-reads" | "deny-all";
|
||||
acpxSessionMode?: "agent" | "plan" | "ask";
|
||||
piThinkingLevel?: "off" | "low" | "high" | "max";
|
||||
};
|
||||
|
||||
export class PaperclipRunnerProviderProfileError extends Error {
|
||||
@@ -376,10 +379,13 @@ export function resolvePaperclipRunnerProviderProfile(
|
||||
resolvePaperclipRunnerCursorMode(candidate, config.acpxAgent, config.acpxSessionMode);
|
||||
} catch (error) {
|
||||
throw new PaperclipRunnerProviderProfileError(
|
||||
"paperclip_runner_cursor_mode_invalid",
|
||||
"paperclip_runner_mode_invalid",
|
||||
error instanceof Error ? error.message : "Invalid Cursor session mode",
|
||||
);
|
||||
}
|
||||
try { resolvePaperclipRunnerPiThinkingLevel(candidate, config.acpxAgent, config.piThinkingLevel); } catch (error) {
|
||||
throw new PaperclipRunnerProviderProfileError("paperclip_runner_pi_thinking_invalid", error instanceof Error ? error.message : "Invalid Pi thinking level");
|
||||
}
|
||||
const model = optionalString(config.model);
|
||||
if (candidate === "codex") {
|
||||
return {
|
||||
@@ -486,10 +492,10 @@ export function resolvePaperclipRunnerProviderProfile(
|
||||
}
|
||||
throw new PaperclipRunnerProviderProfileError("paperclip_runner_acpx_agent_unavailable", `${pendingAcpxProfile.label} is awaiting local and Daytona qualification. Its profile is not enabled for production runs.`);
|
||||
}
|
||||
if (acpxAgent === "cursor" && !model) {
|
||||
throw new PaperclipRunnerProviderProfileError("paperclip_runner_acpx_model_required", "Cursor requires an explicit model ID; there is no default model.");
|
||||
if ((acpxAgent === "cursor" || acpxAgent === "pi") && !model) {
|
||||
throw new PaperclipRunnerProviderProfileError("paperclip_runner_acpx_model_required", "This ACPX agent requires an explicit model ID; there is no default model.");
|
||||
}
|
||||
if (acpxAgent !== "claude" && acpxAgent !== "codex" && acpxAgent !== "grok" && acpxAgent !== "cursor") {
|
||||
if (acpxAgent !== "claude" && acpxAgent !== "codex" && acpxAgent !== "grok" && acpxAgent !== "cursor" && acpxAgent !== "pi") {
|
||||
throw new PaperclipRunnerProviderProfileError(
|
||||
"paperclip_runner_acpx_agent_unavailable",
|
||||
"Paperclip Runner ACPX requires a qualified agent profile.",
|
||||
@@ -567,6 +573,7 @@ export function resolvePaperclipRunnerNativeProviderInput(input: {
|
||||
provider: "acpx",
|
||||
model: profile.model,
|
||||
acpxAgent: profile.acpxAgent,
|
||||
...(profile.acpxAgent === "pi" ? { piThinkingLevel: resolvePaperclipRunnerPiThinkingLevel("acpx", "pi", config.piThinkingLevel) } : {}),
|
||||
...(profile.acpxAgent === "cursor" ? {
|
||||
acpxSessionMode: resolvePaperclipRunnerCursorMode("acpx", "cursor", config.acpxSessionMode),
|
||||
} : {}),
|
||||
|
||||
@@ -0,0 +1,162 @@
|
||||
/** Explicit operator-imported Linux authority. No downloads or executable probes. */
|
||||
import { setImmediate as yieldToSignals } from "node:timers/promises";
|
||||
import { createHash } from "node:crypto";
|
||||
import { type Stats, constants } from "node:fs";
|
||||
import * as fs from "node:fs/promises";
|
||||
import { dirname, isAbsolute, join, relative, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { QUALIFIED_ACPX_PROFILES } from "../../vendor/paperclip-runner/index.js";
|
||||
|
||||
const SERVER_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), "../../..");
|
||||
const MAX_BYTES = 4 * 1024 ** 3;
|
||||
const MAX_FILE = 512 * 1024 ** 2;
|
||||
const MANIFEST = "companion.json";
|
||||
const AUTHORITY = ".import-authority.json";
|
||||
type Entry = { path: string; mode: number; kind: "directory" } | { path: string; mode: number; kind: "file"; size: number; sha256: string } | { path: string; mode: number; kind: "symlink"; target: string };
|
||||
export interface RemotePiCompanionManifest { schema: "paperclip.remote-pi-companion/v1"; sourceRevision: string; target: "linux-x64"; profileDigest: string; providerPackDigest: string; daemonSha256: string; entries: Entry[] }
|
||||
function require(value: unknown, reason: string): asserts value { if (!value) throw new Error(`runner_remote_companion_invalid: ${reason}`); }
|
||||
const digest = (value: Buffer | string) => createHash("sha256").update(value).digest("hex");
|
||||
const safe = (path: string) => path.length > 0 && path.length < 4096 && !isAbsolute(path) && !/[\\\x00-\x1f\x7f]/u.test(path) && path.split("/").every(p => p !== "" && p !== "." && p !== "..");
|
||||
const inside = (root: string, path: string) => path === root || (!relative(root, path).startsWith("..") && !isAbsolute(relative(root, path)));
|
||||
const same = (a: Stats, b: Stats) => a.dev === b.dev && a.ino === b.ino && a.size === b.size && a.mode === b.mode && a.mtimeMs === b.mtimeMs && a.ctimeMs === b.ctimeMs && a.nlink === b.nlink;
|
||||
type Checkpoint = () => Promise<void>;
|
||||
function checkpoints(cancel?: () => void): Checkpoint {
|
||||
const deadline = Date.now() + 600_000;
|
||||
return async () => { await yieldToSignals(); cancel?.(); require(Date.now() < deadline, "import deadline exceeded"); };
|
||||
}
|
||||
async function directory(path: string) { const st = await fs.lstat(path); require(st.isDirectory() && !st.isSymbolicLink() && await fs.realpath(path) === path, "directory is linked or noncanonical"); return st; }
|
||||
/** One descriptor and 1MiB buffer; no whole executable allocation or sync hashing. */
|
||||
async function readStable(path: string, maximum: number, privateFile: boolean, check: Checkpoint, consume?: (chunk: Buffer) => Promise<void>) {
|
||||
require(await fs.realpath(dirname(path)) === dirname(path), "linked parent");
|
||||
const file = await fs.open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK);
|
||||
try {
|
||||
const before = await file.stat(); require(before.isFile() && before.size <= maximum && (!privateFile || before.nlink === 1), "file type, link or byte bound");
|
||||
const hash = createHash("sha256"); let size = 0; const buffer = Buffer.alloc(1024 * 1024); let header = Buffer.alloc(0);
|
||||
for (;;) {
|
||||
await check(); const { bytesRead } = await file.read(buffer); if (!bytesRead) break;
|
||||
size += bytesRead; require(size <= maximum && size <= before.size, "file grew beyond bound");
|
||||
const chunk = buffer.subarray(0, bytesRead); hash.update(chunk); if (!header.length) header = Buffer.from(chunk.subarray(0, 64));
|
||||
await consume?.(chunk);
|
||||
}
|
||||
require(size === before.size && same(before, await file.stat()) && same(before, await fs.lstat(path)), "file changed during read");
|
||||
return { size, sha256: hash.digest("hex"), header, stat: before };
|
||||
} finally { await file.close(); }
|
||||
}
|
||||
async function readOwned(path: string, maximum: number, check: Checkpoint, privateFile = false): Promise<Buffer> {
|
||||
const chunks: Buffer[] = []; await readStable(path, maximum, privateFile, check, async chunk => { chunks.push(Buffer.from(chunk)); }); return Buffer.concat(chunks);
|
||||
}
|
||||
/** Complete no-follow inventory, including modes and contained symbolic links. */
|
||||
export async function inventoryRemoteCompanion(root: string, check = checkpoints()): Promise<Entry[]> {
|
||||
await directory(root); const entries: Entry[] = []; const links = new Map<string, { count: number; links: number }>(); let total = 0;
|
||||
const visit = async (path: string): Promise<void> => {
|
||||
const before = await directory(path);
|
||||
for (const name of (await fs.readdir(path)).sort()) {
|
||||
await check();
|
||||
if (path === root && [MANIFEST, AUTHORITY].includes(name)) continue;
|
||||
const file = join(path, name); const rel = relative(root, file); require(safe(rel), "unsafe path");
|
||||
const st = await fs.lstat(file); const mode = st.mode & 0o7777;
|
||||
require(entries.length < 100_000 && (st.isSymbolicLink() || (mode & 0o7022) === 0), "entry count or unsafe mode");
|
||||
if (st.isDirectory()) { require((mode & 0o700) === 0o700, "directory requires owner read/write/search for owned cleanup"); entries.push({ path: rel, mode, kind: "directory" }); await visit(file); }
|
||||
else if (st.isFile()) {
|
||||
total += st.size; require(total <= MAX_BYTES, "tree byte bound"); const read = await readStable(file, MAX_FILE, false, check);
|
||||
require(same(st, read.stat), "discovered file changed");
|
||||
entries.push({ path: rel, mode, kind: "file", size: read.size, sha256: read.sha256 });
|
||||
const key = `${st.dev}:${st.ino}`; const group = links.get(key) ?? { count: 0, links: st.nlink }; group.count++; require(group.links === st.nlink, "hardlink identity drift"); links.set(key, group);
|
||||
} else if (st.isSymbolicLink()) {
|
||||
const target = await fs.readlink(file); require(!isAbsolute(target) && !/[\x00-\x1f\x7f]/u.test(target) && inside(root, resolve(dirname(file), target)) && inside(root, await fs.realpath(file)), "escaping symbolic link");
|
||||
require(same(st, await fs.lstat(file)), "link changed"); entries.push({ path: rel, mode, kind: "symlink", target });
|
||||
} else throw new Error("runner_remote_companion_invalid: special entry");
|
||||
}
|
||||
const after = await directory(path); require(before.dev === after.dev && before.ino === after.ino && before.mtimeMs === after.mtimeMs, "directory changed");
|
||||
};
|
||||
await visit(root); require([...links.values()].every(g => g.count === g.links), "external hardlink");
|
||||
return entries.sort((a, b) => a.path < b.path ? -1 : a.path > b.path ? 1 : 0);
|
||||
}
|
||||
async function installedIdentity(serverRoot: string, check: Checkpoint) {
|
||||
require(process.platform === "darwin" || process.platform === "linux", "companion import supports macOS and Linux controllers");
|
||||
await directory(serverRoot); const pkg = JSON.parse((await readOwned(join(serverRoot, "package.json"), 65536, check, true)).toString()); require(pkg.name === "@paperclipai/server", "public server package required");
|
||||
const source = JSON.parse((await readOwned(join(serverRoot, "dist/build-info.json"), 65536, check, true)).toString()).commit;
|
||||
require(typeof source === "string" && /^[a-f0-9]{40}$/u.test(source), "installed build source is missing");
|
||||
const profile = QUALIFIED_ACPX_PROFILES.pi; require(profile.qualificationStatus !== "pending", "Pi is not qualified");
|
||||
return { source, profileDigest: profile.commandDigest };
|
||||
}
|
||||
async function validate(root: string, manifest: RemotePiCompanionManifest, identity: Awaited<ReturnType<typeof installedIdentity>>, check: Checkpoint) {
|
||||
require(manifest.schema === "paperclip.remote-pi-companion/v1" && manifest.target === "linux-x64" && manifest.sourceRevision === identity.source && manifest.profileDigest === identity.profileDigest, "source/profile/target mismatch");
|
||||
require(JSON.stringify(await inventoryRemoteCompanion(root, check)) === JSON.stringify(manifest.entries), "complete inventory mismatch");
|
||||
const daemon = manifest.entries.find(e => e.path === "bin/paperclip-runnerd");
|
||||
require(daemon?.kind === "file" && daemon.size > 64 && (daemon.mode & 0o111) !== 0 && daemon.sha256 === manifest.daemonSha256, "daemon identity");
|
||||
const elf = (await readStable(join(root, daemon.path), MAX_FILE, false, check)).header; require(elf.subarray(0, 4).equals(Buffer.from([127, 69, 76, 70])) && elf[4] === 2 && elf[5] === 1 && elf.readUInt16LE(18) === 62, "Linux x64 ELF required");
|
||||
const pack = JSON.parse((await readOwned(join(root, "provider-pack/provider-pack.json"), 16 * 1024 ** 2, check)).toString());
|
||||
const canonical = (v: unknown): string => Array.isArray(v) ? `[${v.map(canonical).join(",")}]` : v && typeof v === "object" ? `{${Object.keys(v).sort().map(k => `${JSON.stringify(k)}:${canonical((v as Record<string, unknown>)[k])}`).join(",")}}` : JSON.stringify(v);
|
||||
require(pack.schema === "paperclip-runner/remote-provider-pack/v1" && pack.digest === manifest.providerPackDigest && pack.digest === `sha256:${digest(canonical(pack.payload))}`, "pack manifest digest");
|
||||
require(pack.payload.runnerSourceRevision === identity.source && pack.payload.target.platform === "linux" && pack.payload.target.architecture === "x64", "pack source/target");
|
||||
const pi = pack.payload.candidateProviders?.pi; require(pi?.qualification === "qualified" && pi.profileDigest === identity.profileDigest && pi.path === "provider-assets/pi/linux-x64", "normal Pi pack required");
|
||||
}
|
||||
/** Release-side command uses this same inventory contract before publication. */
|
||||
export async function createRemotePiCompanionManifest(root: string, sourceRevision: string): Promise<RemotePiCompanionManifest> {
|
||||
const check = checkpoints();
|
||||
const profile = QUALIFIED_ACPX_PROFILES.pi; const pack = JSON.parse((await readOwned(join(root, "provider-pack/provider-pack.json"), 16 * 1024 ** 2, check)).toString()); const entries = await inventoryRemoteCompanion(root, check);
|
||||
const daemon = entries.find(e => e.path === "bin/paperclip-runnerd"); require(daemon?.kind === "file", "daemon missing");
|
||||
const manifest: RemotePiCompanionManifest = { schema: "paperclip.remote-pi-companion/v1", sourceRevision, target: "linux-x64", profileDigest: profile.commandDigest, providerPackDigest: pack.digest, daemonSha256: daemon.sha256, entries };
|
||||
require(/^[a-f0-9]{40}$/u.test(sourceRevision), "release source"); await validate(root, manifest, { source: sourceRevision, profileDigest: profile.commandDigest }, check); return manifest;
|
||||
}
|
||||
function cacheParent(serverRoot: string) { return join(serverRoot, "remote-companions"); }
|
||||
async function removeOwned(path: string, owned: Stats) { const st = await fs.lstat(path); require(st.dev === owned.dev && st.ino === owned.ino && !st.isSymbolicLink(), "cleanup root ownership changed"); await fs.rm(path, { recursive: true }); }
|
||||
export async function importRemotePiCompanion(input: { directory: string; sha256: string; serverRoot?: string; checkCancelled?: () => void }) {
|
||||
const checkpoint = checkpoints(input.checkCancelled);
|
||||
await checkpoint();
|
||||
const serverRoot = input.serverRoot ?? SERVER_ROOT; const identity = await installedIdentity(serverRoot, checkpoint); const source = await fs.realpath(input.directory); require(source === resolve(input.directory), "linked import root"); await directory(source);
|
||||
require(/^[a-f0-9]{64}$/u.test(input.sha256), "expected manifest SHA256 required"); const bytes = await readOwned(join(source, MANIFEST), 32 * 1024 ** 2, checkpoint); require(digest(bytes) === input.sha256, "operator manifest digest mismatch");
|
||||
const manifest = JSON.parse(bytes.toString()) as RemotePiCompanionManifest; await validate(source, manifest, identity, checkpoint); await checkpoint();
|
||||
const parent = cacheParent(serverRoot); try { await fs.mkdir(parent, { mode: 0o700 }); } catch (error) { if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw new Error("Remote companion setup requires a writable installed server package", { cause: error }); }
|
||||
const parentInfo = await directory(parent); require((parentInfo.mode & 0o077) === 0 && parentInfo.uid === process.getuid?.(), "cache must be private and operator-owned");
|
||||
const lockPath = join(parent, ".import-linux-x64.lock"); const lock = await fs.open(lockPath, constants.O_CREAT | constants.O_EXCL | constants.O_WRONLY | constants.O_NOFOLLOW, 0o600); const lockInfo = await lock.stat();
|
||||
let staging: string | undefined; let stagingInfo: Stats | undefined; let outputInfo: Stats | undefined; let committed = false; const output = join(parent, "linux-x64");
|
||||
try {
|
||||
await checkpoint();
|
||||
try { await fs.lstat(output); const existing = await resolveRemotePiCompanion({ serverRoot, checkpoint }); require(existing?.manifestSha256 === input.sha256, "existing companion differs; remove only after stopping all runs"); await checkpoint(); return { status: "verified_existing", ...existing }; } catch (error) { if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; }
|
||||
staging = await fs.mkdtemp(join(parent, ".import-")); stagingInfo = await fs.lstat(staging);
|
||||
for (const entry of manifest.entries.filter(e => e.kind === "directory").sort((a, b) => a.path.split("/").length - b.path.split("/").length)) await fs.mkdir(join(staging, entry.path), { mode: 0o700 });
|
||||
for (const entry of manifest.entries) {
|
||||
if (entry.kind !== "file") continue;
|
||||
const file = join(staging, entry.path); const outputFile = await fs.open(file, constants.O_CREAT | constants.O_EXCL | constants.O_WRONLY | constants.O_NOFOLLOW, 0o600);
|
||||
try {
|
||||
const read = await readStable(join(source, entry.path), MAX_FILE, false, checkpoint, async chunk => {
|
||||
let offset = 0; while (offset < chunk.length) { await checkpoint(); offset += (await outputFile.write(chunk, offset)).bytesWritten; }
|
||||
});
|
||||
require(read.size === entry.size && read.sha256 === entry.sha256, "source changed while copied");
|
||||
} finally { await outputFile.close(); }
|
||||
await fs.chmod(file, entry.mode); await checkpoint();
|
||||
}
|
||||
for (const entry of manifest.entries) if (entry.kind === "symlink") await fs.symlink(entry.target, join(staging, entry.path));
|
||||
for (const entry of manifest.entries.filter(e => e.kind === "directory").reverse()) await fs.chmod(join(staging, entry.path), entry.mode);
|
||||
await fs.writeFile(join(staging, MANIFEST), bytes, { flag: "wx", mode: 0o600 }); await validate(staging, manifest, identity, checkpoint); await validate(source, manifest, identity, checkpoint); await checkpoint();
|
||||
// Claim the final path exclusively; no rename may replace a concurrent directory.
|
||||
await fs.mkdir(output, { mode: 0o700 }); outputInfo = await fs.lstat(output);
|
||||
for (const name of await fs.readdir(staging)) { const now = await directory(output); require(now.dev === outputInfo.dev && now.ino === outputInfo.ino, "publication ownership changed"); await fs.rename(join(staging, name), join(output, name)); }
|
||||
await fs.writeFile(join(output, AUTHORITY), JSON.stringify({ sha256: input.sha256 }) + "\n", { flag: "wx", mode: 0o600 });
|
||||
const result = await resolveRemotePiCompanion({ serverRoot, checkpoint }); require(result?.manifestSha256 === input.sha256, "publication verification"); await checkpoint(); committed = true; return { status: "imported_verified", ...result };
|
||||
} finally {
|
||||
const failures: unknown[] = []; const cleanup = async (fn: () => Promise<void>) => { try { await fn(); } catch (error) { failures.push(error); } };
|
||||
if (outputInfo && !committed) await cleanup(() => removeOwned(output, outputInfo!));
|
||||
if (staging && stagingInfo) await cleanup(() => removeOwned(staging!, stagingInfo!));
|
||||
await cleanup(async () => { const now = await fs.lstat(lockPath); require(now.dev === lockInfo.dev && now.ino === lockInfo.ino, "lock ownership changed"); await fs.unlink(lockPath); }); await lock.close();
|
||||
if (failures.length) throw new AggregateError(failures, "Remote companion cleanup incomplete; inspect owned paths before retrying");
|
||||
}
|
||||
}
|
||||
export async function resolveRemotePiCompanion(input: { serverRoot?: string; workspaceRoot?: string; checkpoint?: Checkpoint } = {}) {
|
||||
const check = input.checkpoint ?? checkpoints();
|
||||
const serverRoot = input.serverRoot ?? SERVER_ROOT; const parent = cacheParent(serverRoot); const root = join(parent, "linux-x64");
|
||||
try { await fs.lstat(root); } catch (error) { if ((error as NodeJS.ErrnoException).code === "ENOENT") return null; throw error; }
|
||||
const parentInfo = await directory(parent); require((parentInfo.mode & 0o077) === 0 && parentInfo.uid === process.getuid?.(), "cache privacy"); await directory(root);
|
||||
if (input.workspaceRoot) { const workspace = await fs.realpath(input.workspaceRoot); require(!inside(workspace, root) && !inside(root, workspace), "companion cache cannot overlap a workspace"); }
|
||||
const authority = JSON.parse((await readOwned(join(root, AUTHORITY), 65536, check, true)).toString()); const bytes = await readOwned(join(root, MANIFEST), 32 * 1024 ** 2, check, true); require(digest(bytes) === authority.sha256, "installed authority changed");
|
||||
const manifest = JSON.parse(bytes.toString()) as RemotePiCompanionManifest; await validate(root, manifest, await installedIdentity(serverRoot, check), check);
|
||||
return { root, runnerBinary: join(root, "bin/paperclip-runnerd"), providerPack: join(root, "provider-pack"), manifestSha256: authority.sha256 as string, sourceRevision: manifest.sourceRevision, target: manifest.target };
|
||||
}
|
||||
|
||||
/** Explicit legacy overrides remain authoritative; C/C never gain this Pi path. */
|
||||
export async function selectRemotePiCompanion(input: { provider: { kind: string; agent?: string }; remote: boolean; binaryOverride?: string | null; packOverride?: string | null; workspaceRoot: string; serverRoot?: string }) {
|
||||
if (!input.remote || input.provider.kind !== "acpx" || input.provider.agent !== "pi" || input.binaryOverride?.trim() || input.packOverride?.trim()) return null;
|
||||
return resolveRemotePiCompanion({ serverRoot: input.serverRoot, workspaceRoot: input.workspaceRoot });
|
||||
}
|
||||
@@ -116,16 +116,14 @@ describe("resolveNativeRuntimeMode", () => {
|
||||
})).toThrow(expect.objectContaining({
|
||||
code: "paperclip_runner_opencode_model_invalid",
|
||||
}));
|
||||
expect(() => resolveNativeRuntimeMode({
|
||||
...eligible,
|
||||
adapterConfig: {
|
||||
provider: "acpx",
|
||||
acpxAgent: "pi",
|
||||
model: "openrouter/deepseek/deepseek-v4-flash-0731",
|
||||
},
|
||||
})).toThrow(expect.objectContaining({
|
||||
code: "paperclip_runner_acpx_agent_unavailable",
|
||||
}));
|
||||
for (const acpxAgent of ["copilot"]) {
|
||||
expect(() => resolveNativeRuntimeMode({
|
||||
...eligible,
|
||||
adapterConfig: { provider: "acpx", acpxAgent, model: "explicit-provider-model" },
|
||||
})).toThrow(expect.objectContaining({
|
||||
code: "paperclip_runner_acpx_agent_unavailable",
|
||||
}));
|
||||
}
|
||||
expect(() => resolveNativeRuntimeMode({
|
||||
...eligible,
|
||||
adapterConfig: { provider: "acpx", acpxAgent: "claude", model: "claude-opus-5" },
|
||||
|
||||
@@ -44,71 +44,27 @@ function arbitrate(
|
||||
}
|
||||
|
||||
describe("native status authority", () => {
|
||||
it("waits on persisted monitors without an immediate continuation, even with unfinished work", () => {
|
||||
const pending = assessment({ reportedDisposition: "yielded", hasBlockingRemainingWork: true,
|
||||
continuation: { kind: "monitor", summary: "Check CI", idempotencyKey: "ci" } });
|
||||
for (const priorIssueStatus of ["in_progress", "in_review"] as const) {
|
||||
expect(arbitrate({ assessment: pending, priorIssueStatus, monitorWaitAuthorized: true }))
|
||||
.toMatchObject({ statusAction: "preserve", toStatus: priorIssueStatus,
|
||||
reasonCode: "scheduled_monitor_waiting", effects: [{ kind: "release_checkout" }] });
|
||||
}
|
||||
expect(arbitrate({ assessment: pending })).toMatchObject({ reasonCode: "monitor_wait_authority_lost",
|
||||
effects: [{ kind: "record_finalization_error" }] });
|
||||
expect(arbitrate({ assessment: pending, monitorWaitAuthorized: true, hasUnresolvedIssueBlockers: true }).toStatus).toBe("blocked");
|
||||
expect(arbitrate({ assessment: pending, monitorWaitAuthorized: true, governanceGate: { kind: "approval", id: "pending" } }).toStatus).toBe("in_review");
|
||||
});
|
||||
|
||||
it("keeps a pending child result non-terminal without adding another continuation", () => {
|
||||
expect(arbitrate({ hasPendingChildCompletion: true })).toMatchObject({
|
||||
statusAction: "in_progress", toStatus: "in_progress", reasonCode: "native_child_completion_pending",
|
||||
effects: [{ kind: "release_checkout" }],
|
||||
it("keeps a verified accepted Cursor plan passive without completing or replaying work", () => {
|
||||
const passive = assessment({ reportedDisposition: "yielded", objectiveSatisfied: false,
|
||||
allCriteriaSatisfied: false, hasBlockingRemainingWork: true,
|
||||
continuation: { kind: "response_wake", summary: "Explicit continuation needed", idempotencyKey: "cursor-plan-wait:event" } });
|
||||
expect(arbitrate({ assessment: passive, planWaitAuthorized: true })).toMatchObject({
|
||||
toStatus: "in_progress", reasonCode: "native_plan_accepted_waiting_for_continuation", effects: [],
|
||||
});
|
||||
expect(arbitrate({ hasPendingChildCompletion: false }).toStatus).toBe("done");
|
||||
for (const priorIssueStatus of ["done", "cancelled"] as const) {
|
||||
expect(arbitrate({ priorIssueStatus, hasPendingChildCompletion: true }).toStatus).toBe(priorIssueStatus);
|
||||
}
|
||||
expect(arbitrate({ hasPendingChildCompletion: true, hasUnresolvedIssueBlockers: true }).toStatus).toBe("blocked");
|
||||
expect(arbitrate({ hasPendingChildCompletion: true, governanceGate: { kind: "approval", id: "approval" } }).toStatus)
|
||||
.toBe("in_review");
|
||||
expect(arbitrate({ hasPendingChildCompletion: true, workspaceFinalizeStatus: "failed" }).statusAction).toBe("preserve");
|
||||
expect(arbitrate({ assessment: passive })).toMatchObject({
|
||||
toStatus: "in_progress",
|
||||
reasonCode: "completion_evidence_incomplete",
|
||||
effects: [expect.objectContaining({
|
||||
kind: "enqueue_continuation",
|
||||
continuationKind: "same_agent",
|
||||
idempotencyKey: "native-completion-incomplete",
|
||||
})],
|
||||
});
|
||||
expect(arbitrate({ assessment: passive, planWaitAuthorized: true, terminalState: "failed" }).reasonCode).not.toBe("native_plan_accepted_waiting_for_continuation");
|
||||
expect(arbitrate({ assessment: passive, planWaitAuthorized: true, priorIssueStatus: "cancelled" }).toStatus).toBe("cancelled");
|
||||
expect(arbitrate({ assessment: passive, planWaitAuthorized: true, governanceGate: { kind: "interaction", id: "pending" } }).toStatus).toBe("in_review");
|
||||
});
|
||||
|
||||
it("schedules a capacity retry while preserving partial work and review authority", () => {
|
||||
for (const nativeReviewOutcome of [undefined, "pending"] as const) {
|
||||
const decision = arbitrate({ terminalState: "failed", providerOverloaded: true, nativeReviewOutcome });
|
||||
expect(decision).toMatchObject({ statusAction: "preserve", reasonCode: "native_provider_overloaded" });
|
||||
expect(decision.effects).toContainEqual(expect.objectContaining({ kind: "schedule_retry", cause: "native_provider_overloaded" }));
|
||||
}
|
||||
expect(arbitrate({ terminalState: "failed", providerOverloaded: true, failureRetryCount: 2 }))
|
||||
.toMatchObject({ statusAction: "blocked", reasonCode: "native_provider_overloaded_exhausted", effects: [{ kind: "bind_blocker", owner: "board", action: expect.stringContaining("Automatic retries exhausted") }] });
|
||||
});
|
||||
it.each([
|
||||
{ hasActivePauseHold: true }, { hasUnresolvedIssueBlockers: true },
|
||||
{ governanceGate: { kind: "approval" as const, id: "approval" } },
|
||||
{ priorIssueStatus: "blocked" as const }, { priorIssueStatus: "done" as const },
|
||||
{ workspaceFinalizeStatus: "failed" as const }, { nativeReviewOutcome: "stale" as const },
|
||||
{ nativeReviewOutcome: "resolved" as const },
|
||||
])("does not schedule capacity retries through existing authority or cleanup gates (%j)", (gate) => {
|
||||
const decision = arbitrate({ terminalState: "failed", providerOverloaded: true, ...gate });
|
||||
expect(decision.effects.some(effect => effect.kind === "schedule_retry")).toBe(false);
|
||||
});
|
||||
it.each(["in_progress", "in_review"] as const)("blocks a current worker's proven model rejection without a retry (%s)", (priorIssueStatus) => {
|
||||
const decision = arbitrate({ priorIssueStatus, terminalState: "failed", providerModelRejected: true });
|
||||
expect(decision).toMatchObject({ statusAction: "blocked", toStatus: "blocked", reasonCode: "native_provider_model_rejected", unblockDescriptor: { owner: "board" } });
|
||||
expect(decision.effects).toEqual([{ kind: "bind_blocker", owner: "board", action: expect.any(String) }]);
|
||||
expect(arbitrate({ terminalState: "failed", providerModelRejected: false }).effects).toContainEqual(expect.objectContaining({ kind: "schedule_retry" }));
|
||||
expect(arbitrate({ terminalState: "succeeded", providerModelRejected: true }).reasonCode).not.toBe("native_provider_model_rejected");
|
||||
expect(arbitrate({ terminalState: "failed", providerModelRejected: true, workspaceFinalizeStatus: "failed" }).reasonCode).toBe("finalization_failed_claim_preserved");
|
||||
expect(arbitrate({ terminalState: "failed", providerModelRejected: true, priorIssueStatus: "done" }).toStatus).toBe("done");
|
||||
for (const nativeReviewOutcome of ["stale", "resolved"] as const) {
|
||||
expect(arbitrate({ terminalState: "failed", providerModelRejected: true, priorIssueStatus, nativeReviewOutcome }))
|
||||
.toMatchObject({ statusAction: "preserve", toStatus: priorIssueStatus, reasonCode: "native_review_action_finished" });
|
||||
}
|
||||
});
|
||||
it("preserves pending review authority without automatic recovery after model rejection", () => {
|
||||
expect(arbitrate({ priorIssueStatus: "in_review", terminalState: "failed", providerModelRejected: true, nativeReviewOutcome: "pending" }))
|
||||
.toMatchObject({ statusAction: "preserve", toStatus: "in_review", reasonCode: "native_provider_model_rejected", unblockDescriptor: null, effects: [{ kind: "release_checkout" }] });
|
||||
});
|
||||
it("a reviewer finishes its decision without completing rejected or still-reviewed work", () => {
|
||||
for (const priorIssueStatus of ["in_progress", "in_review"] as const) {
|
||||
const decision = arbitrate({ priorIssueStatus, nativeReviewOutcome: "resolved" });
|
||||
|
||||
@@ -39,4 +39,3 @@ export type Capability = {
|
||||
name: string; description: string; schema: z.ZodObject; write?: boolean; configure?: boolean; destructive?: boolean; ephemeral?: boolean;
|
||||
run: (p: McpPrincipal, args: Record<string, unknown>, api: ApiDispatch, origin: string, transfers?: PublicMcpTransfers) => Promise<Record<string, unknown>>;
|
||||
};
|
||||
|
||||
+6
-1
@@ -28,6 +28,7 @@ export type {
|
||||
ControlPlanePort,
|
||||
HarnessRuntimeRequestKind,
|
||||
HarnessRuntimeRequestResolution,
|
||||
LinuxProcessStartOptions,
|
||||
NativeAcpxAgent,
|
||||
NativeAcpxPermissionMode,
|
||||
NativeCodexApprovalPolicy,
|
||||
@@ -80,6 +81,7 @@ export const DurablePrpControlPlane = runner.DurablePrpControlPlane;
|
||||
export const runnerCodexDynamicToolsFit = runner.runnerCodexDynamicToolsFit;
|
||||
export const inspectWarmRunTransition = runner.inspectWarmRunTransition;
|
||||
export const readRunnerdArtifactBinding = runner.readRunnerdArtifactBinding;
|
||||
export const readLinuxProcessStartedAt = runner.readLinuxProcessStartedAt;
|
||||
export const NativeSessionCleanupQuarantinedError =
|
||||
runner.NativeSessionCleanupQuarantinedError;
|
||||
export const NativeSessionProtocolIntegrityError =
|
||||
@@ -145,10 +147,13 @@ export const nativeRestartInterruptedTurnId = runner.nativeRestartInterruptedTur
|
||||
export const completeTerminatedRemoteNativeSessionCleanup = runner.completeTerminatedRemoteNativeSessionCleanup;
|
||||
export const completeTerminatedLocalNativeSessionCleanup = runner.completeTerminatedLocalNativeSessionCleanup;
|
||||
|
||||
export const externalOperationDigest = runner.externalOperationDigest;
|
||||
export const probeAcpxClaudeInstallation = runner.probeAcpxClaudeInstallation;
|
||||
export const probeAcpxGrokInstallation = runner.probeAcpxGrokInstallation;
|
||||
export const probeAcpxPiInstallation = runner.probeAcpxPiInstallation;
|
||||
export const bundledRemoteProviderPackManifestPath = runner.bundledRemoteProviderPackManifestPath;
|
||||
export const bundledRemoteRunnerBinary = runner.bundledRemoteRunnerBinary;
|
||||
|
||||
export const externalOperationDigest = runner.externalOperationDigest;
|
||||
export const CONFIGURED_ENVIRONMENT_KEYS = runner.CONFIGURED_ENVIRONMENT_KEYS;
|
||||
export const GENERATED_RUNTIME_ENVIRONMENT_KEYS = runner.GENERATED_RUNTIME_ENVIRONMENT_KEYS;
|
||||
export const configuredEnvironmentProjection = runner.configuredEnvironmentProjection;
|
||||
|
||||
@@ -3,4 +3,5 @@ export {
|
||||
probeAcpxClaudeInstallation,
|
||||
probeAcpxGrokInstallation,
|
||||
probeAcpxCursorInstallation,
|
||||
probeAcpxPiInstallation,
|
||||
} from "@paperclipai/paperclip-runner/live";
|
||||
Reference in new issue
Block a user