mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-24 18:19:05 +02:00
## Thinking Path > - Paperclip manages AI agents and the tools they may use. > - Connection setup separates provider preference from permission to use a tool. > - The first native connection baseline could not exercise its intended decisions. > - The browser used mutable task titles, and the provider fixture already granted access. > - Native provider-choice instructions also disagreed with the preferred question format. Schema rejection gave no field guidance. > - This pull request repairs those test preconditions and native guidance, then fixes restart/approval defects exposed by the corrected baseline. It also restores missing OpenCode tool-error evidence. > - The benefit is an inspectable baseline before any further instruction reduction. ## Linked Issues or Issue Description Refs #15407. The original 15-cell baseline remains 0 PASS / 15 FAIL. Ten cells stopped on stale titles, two Codex cells had schema denials, two OpenCode cells used already-granted tools, and one Claude cell returned no native result. No intended user decisions were submitted. The exact invalid Codex field and underlying Claude failure cause remain unknown. [Original campaign](https://github.com/paperclipai/paperclip/actions/runs/37562577199) · [Original report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37562577199-1/index.html) ## What Changed - Match the browser's task route and visible identifier instead of a title the agent can change. - Start native provider-choice fixtures with no agent tool access. Verify the public effective-access records. - In the positive case, select Arcade, then grant its exact HubSpot tool through the real access card. Require both saved decisions and exactly one observed call. - Return a canonical `providerQuestionSet` for native input and retain the equivalent legacy `providerQuestion`. - Keep invalid input rejected. Return bounded schema locations and required field names without submitted values. - Preserve Claude's exact session/content identity while allowing authenticated registered instruction-copy paths to rotate on a new run. - Reject duplicate approval reports for an existing exact tool-action card before they create another human review. - Wait for a recorded service-approval continuation within the existing deadline; retain missing or failed continuation grades. - Forward OpenCode tool activity through the runner facade, preserving bounded errors and execution-part identity without inventing host-call joins or exposing arguments. - Preserve original grades, costs, scope limits and diagnoses in the dated repair report. ## Verification - Eval typecheck passes. Support suite: 1,801 PASS, one intentional skip; Node checks: 128 PASS. - Connection/schema tests: 51 PASS. Real-server public fixture setup: one PASS with zero providers. - Browser support regression: five PASS, including renamed and wrong tasks. - Focused Rust safe-feedback test: one PASS. - Repository typecheck and build pass before the latest master replay. Post-replay connection/shared/real-server fixture checks: 52 PASS; eval typecheck passes. The browser review fix additionally passes all five browser checks and seven suite checks. - The full local repository run was interrupted incomplete after about 45 minutes, with five integration failures retained. All five pass in a separate targeted invocation (1,250 unrelated tests skipped). No full local-suite pass or root cause for the initial local failures is claimed. - Corrected frozen source `162cc90fdabe7f505b88ae095044531b82784c92`: **10 PASS / 5 FAIL** across the [passing Codex canary](https://github.com/paperclipai/paperclip/actions/runs/37575158761) and [remaining 14 cells](https://github.com/paperclipai/paperclip/actions/runs/37576261807). The canary passes all 17 checks. Claude's two provider-choice continuations fail on restart, Claude service approval exposes an early evaluator rejection, Codex service approval creates a duplicate approval, and OpenCode provider-second times out after both decisions with no HubSpot call. No original result is regraded. - Final ledgers count 31 actual runs: 27 succeeded, two failed, two cancelled during cleanup. All 15 cleanup/budget checks pass. The late Claude continuation is absent from its earlier workflow snapshot; it remains in the result/API/final ledger. Original evidence retains 279 hashes. Recorded LLM subtotal $0.04553787 is incomplete billing, not actual total cost; local runtime is unmetered. - New repair regressions reproduce the Claude attach failure, duplicate approval acceptance and dropped OpenCode tool events before their respective fixes. Nine Rust attachment checks, 127 ACPX host/adapter tests, 33 completion/control-plane checks, nine eval deadline tests, 59 OpenCode proxy/driver tests, one Rust tool-error/redaction check, and TypeScript/Rust composer parity pass. Eval typecheck, repository typecheck and build pass. Existing support coverage is 1,802 PASS plus 128 Node PASS, one intentional support skip; two additional deadline tests also pass. - New-source full CI/review and live canaries are pending. The next bounded selection is Claude provider-decline, Codex service-approve and one OpenCode provider-second diagnostic with repaired event evidence. No broader campaign or instruction-reduction qualification is claimed. - Initial corrected campaign [37574251834](https://github.com/paperclipai/paperclip/actions/runs/37574251834) was cancelled during shared build after review found the breadcrumb whitespace assumption. Its matrix job has zero steps and no provider execution. The real adjacent-span browser regression now reproduces the old failure and passes after the fix. ## Risks - The corrected baseline remains 10/15. The new restart/approval fixes require live qualification; OpenCode evidence forwarding does not itself establish or fix its prior behavioral failure. - The positive provider case now expects three runs, including separate access approval. Its new results are distinct from the original invalid fixture. - The old Claude missing-result cause and rejected Codex field are unknown. These repairs do not retroactively explain or erase either failure. - Path rotation must preserve prompt, custom instruction, skill/content identity and protected provider settings; regression checks reject stale or changed content. No connection authorization, JSON schema, budget, cleanup, or final-result requirement is relaxed. Historical Everyday prompts and gateway setup remain unchanged. ## Model Used OpenAI Codex, GPT-6. The exact deployment variant and context window are not exposed in this session. Used repository inspection, code editing, test execution and retained-evidence analysis. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
151 lines
7.8 KiB
TypeScript
151 lines
7.8 KiB
TypeScript
import { describe, expect, it } from "vitest";
|
|
|
|
import { enqueueOpenCodeProxyInput } from "./opencode-proxy-input.js";
|
|
|
|
describe("OpenCode proxy input sequencing", () => {
|
|
it("drains requests in order before EOF shutdown", async () => {
|
|
let releaseFirst!: () => void;
|
|
const firstBlocked = new Promise<void>((resolve) => {
|
|
releaseFirst = resolve;
|
|
});
|
|
const events: string[] = [];
|
|
let pending = Promise.resolve();
|
|
pending = enqueueOpenCodeProxyInput(
|
|
pending,
|
|
async () => {
|
|
events.push("initialize:start");
|
|
await firstBlocked;
|
|
events.push("initialize:complete");
|
|
},
|
|
() => events.push("initialize:error"),
|
|
);
|
|
pending = enqueueOpenCodeProxyInput(
|
|
pending,
|
|
async () => {
|
|
events.push("thread:start");
|
|
},
|
|
() => events.push("thread:error"),
|
|
);
|
|
const shutdown = pending.then(() => {
|
|
events.push("shutdown");
|
|
});
|
|
|
|
await Promise.resolve();
|
|
expect(events).toEqual(["initialize:start"]);
|
|
releaseFirst();
|
|
await shutdown;
|
|
expect(events).toEqual([
|
|
"initialize:start",
|
|
"initialize:complete",
|
|
"thread:start",
|
|
"shutdown",
|
|
]);
|
|
});
|
|
|
|
it("reports a failed queued request without rejecting the drain", async () => {
|
|
const events: string[] = [];
|
|
let pending = Promise.resolve();
|
|
pending = enqueueOpenCodeProxyInput(
|
|
pending,
|
|
async () => {
|
|
throw new Error("initialize failed");
|
|
},
|
|
(error) => events.push((error as Error).message),
|
|
);
|
|
|
|
await pending;
|
|
expect(events).toEqual(["initialize failed"]);
|
|
});
|
|
});
|
|
|
|
it.each([true, false])("settles a real proxy interrupt with controller feedback or EOF (feedback: %s)", async respond => {
|
|
const { execFileSync, spawn } = await import("node:child_process");
|
|
const { chmod, mkdir, mkdtemp, readFile, readdir, rm, writeFile } = await import("node:fs/promises");
|
|
const { closeSync, openSync } = await import("node:fs");
|
|
const { randomUUID } = await import("node:crypto");
|
|
const { tmpdir } = await import("node:os");
|
|
const { join, resolve } = await import("node:path");
|
|
const { createInterface } = await import("node:readline");
|
|
const root = await mkdtemp(join(tmpdir(), "opencode-proxy-interrupt-feedback-"));
|
|
const launchRoot = join(root, `.paperclip-verified-executable-${randomUUID().replaceAll("-", "")}`);
|
|
await mkdir(launchRoot, { mode: 0o700 });
|
|
await chmod(launchRoot, 0o700);
|
|
const executable = join(launchRoot, "launch");
|
|
const fixture = resolve("test/fixtures/fake-opencode-server.mjs");
|
|
execFileSync("cc", ["-x", "c", "-o", executable, "-"], { input: `#include <unistd.h>\n#include <stdlib.h>\nint main(int argc, char **argv) { char **args = calloc(argc + 2, sizeof(char *)); args[0] = ${JSON.stringify(process.execPath)}; args[1] = ${JSON.stringify(fixture)}; for (int i = 1; i < argc; i++) args[i + 1] = argv[i]; execv(args[0], args); return 127; }` });
|
|
if (process.platform === "darwin") {
|
|
execFileSync("codesign", ["--force", "--sign", "-", executable]);
|
|
execFileSync("codesign", ["--verify", "--strict", executable]);
|
|
}
|
|
await chmod(executable, 0o500);
|
|
const proxy = join(root, "proxy.cjs");
|
|
await writeFile(proxy, execFileSync(process.execPath, ["--input-type=module", "-e", `import { bundleVerifiedProviderEntrypoints } from "./scripts/build-verified-provider-entrypoints.mjs"; const entries = await bundleVerifiedProviderEntrypoints({ write: false }); process.stdout.write(entries.find(({ entrypoint }) => entrypoint.name === "opencode-app-server-proxy").verifiedResult.outputFiles[0].contents);`], { maxBuffer: 16 * 1024 * 1024 }));
|
|
const fd = process.platform === "linux" ? openSync(executable, "r") : undefined;
|
|
const runtime = join(root, "runtime");
|
|
const child = spawn(process.execPath, [proxy, "--paperclip-trusted-opencode-executable", fd === undefined ? executable : "/proc/self/fd/3"], {
|
|
env: { ...process.env, PAPERCLIP_OPENCODE_RUNTIME_DIR: runtime, OPENROUTER_API_KEY: "fixture-key" },
|
|
stdio: fd === undefined ? ["pipe", "pipe", "pipe"] : ["pipe", "pipe", "pipe", fd],
|
|
detached: process.platform !== "win32",
|
|
});
|
|
if (fd !== undefined) closeSync(fd);
|
|
const messages: Array<{ id?: string | number; method?: string; params?: Record<string, unknown>; result?: unknown; error?: unknown }> = [];
|
|
const input = createInterface({ input: child.stdout! });
|
|
input.on("line", line => messages.push(JSON.parse(line)));
|
|
child.stderr!.resume();
|
|
let exited = false;
|
|
const exit = new Promise<void>(resolve => child.once("exit", () => { exited = true; resolve(); }));
|
|
const send = (message: unknown) => child.stdin!.write(`${JSON.stringify(message)}\n`);
|
|
const response = async (id: number) => {
|
|
await expect.poll(() => messages.find(message => message.id === id), { timeout: 5_000 }).toBeDefined();
|
|
const value = messages.find(message => message.id === id)!;
|
|
expect(value.error).toBeUndefined();
|
|
return value.result;
|
|
};
|
|
try {
|
|
send({ id: 1, method: "initialize", params: {} });
|
|
await response(1);
|
|
send({ id: 2, method: "thread/start", params: { cwd: root, model: "openrouter/deepseek/deepseek-v4-flash-0731", completionContract: { revision: "proxy-v1", criterionIds: ["objective"] }, conversationMode: "prepared" } });
|
|
await response(2);
|
|
send({ id: 3, method: "turn/start", params: { input: [{ type: "text", text: JSON.stringify({ schema: "paperclip.native-model-envelope.v3", task: { prompt: "invalid-tool-feedback completion-feedback" }, completionContract: { revision: "proxy-v1", criteria: [{ id: "objective", requirement: "Complete the fixture." }] } }) }] } });
|
|
await response(3);
|
|
await expect.poll(() => messages.find(message => message.method === "item/tool/call"), { timeout: 5_000 }).toBeDefined();
|
|
const call = messages.find(message => message.method === "item/tool/call")!;
|
|
expect(call.params!.tool).toBe("paperclip_finish");
|
|
const toolFrames = messages.filter(message => (message.params?.item as { id?: string } | undefined)?.id === "part-invalid");
|
|
expect(toolFrames.map(message => message.method)).toEqual(["item/started", "item/updated", "item/completed"]);
|
|
expect(toolFrames.at(-1)?.params).toMatchObject({
|
|
threadId: call.params!.threadId, turnId: call.params!.turnId,
|
|
item: { id: "part-invalid", type: "builtinToolCall", tool: "invalid", status: "failed", output: "Tool not found: fixture_missing" },
|
|
});
|
|
const feedback = "Accepted. Include [Saved document](/PAP/issues/PAP-1#document-plan).";
|
|
// The interrupt is ahead of the response on stdin. Commands stay serial,
|
|
// but their awaited response must not queue behind the command itself.
|
|
send({ id: 4, method: "turn/interrupt", params: { turnId: call.params!.turnId } });
|
|
if (respond) {
|
|
send({ id: call.id, result: { success: true, contentItems: [{ type: "inputText", text: feedback }] } });
|
|
} else {
|
|
child.stdin!.end();
|
|
}
|
|
expect(await response(4)).toBe(true);
|
|
if (respond) {
|
|
const sessions = (await readdir(runtime, { withFileTypes: true })).filter(entry => entry.isDirectory());
|
|
await expect.poll(async () => JSON.parse(await readFile(join(runtime, sessions[0]!.name, "data/fake-completion-feedback.json"), "utf8"))).toMatchObject([
|
|
{ result: { content: [{ text: expect.stringContaining(feedback) }] } },
|
|
]);
|
|
child.stdin!.end();
|
|
}
|
|
await expect.poll(() => exited, { timeout: 5_000 }).toBe(true);
|
|
await exit;
|
|
expect(messages.filter(message => message.method === "paperclip/runResult")).toHaveLength(respond ? 1 : 0);
|
|
} finally {
|
|
child.kill("SIGTERM");
|
|
if (!exited) {
|
|
await Promise.race([exit, new Promise(resolve => setTimeout(resolve, 1_000))]);
|
|
if (!exited && child.pid) process.kill(process.platform === "win32" ? child.pid : -child.pid, "SIGKILL");
|
|
await exit;
|
|
}
|
|
input.close();
|
|
await rm(root, { recursive: true, force: true });
|
|
}
|
|
}, 30_000);
|