From cc67d4e1d83396e249e3ac27c3f224d432b5385b Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 2 Oct 2026 22:07:14 -0500 Subject: [PATCH 01/37] fix: preserve steering and recover stopped task conversations (#15015) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task conversation must let a user guide a running agent and resume stopped work. > - The active run owns its input protocol, even when the user changes the next model or effort. > - Queue delivery waits for a provider receipt, which must be able to persist during the request. > - A stopped startup also needs a clear user action that passes normal task admission. > - This pull request fixes steering delivery, makes queue actions immediate, and restores explicit continuation. > - The benefit is a responsive conversation that can recover without losing saved input. ## Linked Issues or Issue Description **What happened?** A queued message could change from Steer to Interrupt while a native run prepared. A steer request could wait on its own database lock and fail to deliver. A stopped startup could then leave the conversation without a working Retry or message continuation. Interrupt also waited for the server and showed a toast. **Expected behavior** The active run keeps its input protocol. Steer delivers input to that run. Steer and Interrupt clear the submitted queue rows and show the input in the conversation immediately. Failed delivery restores the latest queue with an inline error. An eligible stopped run offers Retry, and authenticated user input can start a fresh turn through normal task admission. **Steps to reproduce** 1. Start a task with a native Paperclip Runner. 2. Change the selected model or effort while that run prepares. 3. Queue a message and press Steer. 4. Observe the provider receipt and queue state during the request. 5. Stop a startup before its provider process begins, then try Retry or send a new message. 6. Repeat queued delivery with a legacy runner and press Interrupt. **Paperclip version or commit** Reproduced on the parent of this branch, `59c07ede7`. **Deployment mode** Authenticated private deployment. The fixes also cover local task conversations. Related work: Refs #12834, Refs #13354, Refs #13275. The open refactor in #13160 moves the same queue route; it does not fix the receipt lock or stopped-run continuation addressed here. ## What Changed - Select queue behavior from the active run's immutable dispatch and runtime resolution. - Leave the run row unlocked during provider acknowledgement, then lock and read it before merging the receipt. - Retain queued input if the target run stops during that wait. Keep inline delivery errors visible after empty queue updates. - Permit exact Retry and authenticated continuation after verified native startup cancellation. Preserve pause, approval, budget, ownership, and process-stop gates. - Carry undelivered native queue input into a fresh turn once the old execution is confirmed stopped. - Show Steer and Interrupt input in the conversation and clear submitted composer rows immediately. Restore the latest queue inline on failure. Remove delivery toasts. - Keep optimistic delivery stable across stale polls, empty queues, and paginated history. Preserve classic Interrupt error handling. - Document recovery and optimistic delivery behavior. Add regression tests across server, shared queue projection, and UI boundaries. ## Verification - Red-green regression tests reproduced the queue protocol, receipt lock, stopped-startup continuation, and optimistic delivery failures. - The focused server route, continuation, queue, and runner boundary suites passed during implementation. - The queue-route suite passes with 78 tests. The three complete conversation UI suites pass with 347 tests. - UI typecheck, production build, and `pnpm check:token-gates` pass. - Workspace `pnpm -r typecheck` and `pnpm build` pass. The local monolithic `pnpm test:run` is still running; remote CI verifies the complete suite on the latest commit. - All CI gates pass on `bd9031ad56abfcde13d13a13488c1b9217c2fd3a`, including the full test shards, runner verification, browser E2E, typecheck, release registry, and canary dry run. - Greptile reports 5/5 for that commit. Both review threads are resolved. ## Risks This changes queue display and explicit continuation admission. The UI must restore rejected delivery without losing other-session edits. The server must preserve concurrent provider result updates and must not resume a process whose stop is uncertain. Focused tests cover these boundaries. This change has no database migration. ## Model Used OpenAI Codex, an agent based on GPT-6. The exact runtime model ID and context window are not exposed in this session. Used reasoning, repository tools, code execution, and browser inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/SPEC-implementation.md | 8 +- doc/execution-semantics.md | 24 ++- .../shared/src/types/execution-projection.ts | 2 + .../issue-queued-comments-routes.test.ts | 76 ++++++++ .../adapters/queued-comment-postgres.ts | 4 +- .../application/queued-comment-ports.ts | 2 + server/src/routes/agents.ts | 8 +- server/src/routes/issues.ts | 31 +++- .../src/services/cancelled-native-startup.ts | 13 ++ server/src/services/execution-blocker.ts | 7 +- .../explicit-native-continuation.test.ts | 72 ++++++++ .../services/explicit-native-continuation.ts | 19 +- server/src/services/heartbeat.ts | 8 +- .../issue-queued-comment-queue.test.ts | 16 ++ .../services/issue-queued-comment-queue.ts | 13 +- .../native-runtime/native-session-executor.ts | 5 +- .../ExecutionBlockerNotice.test.tsx | 15 ++ ui/src/components/ExecutionBlockerNotice.tsx | 4 +- ui/src/components/TaskChatThread.test.tsx | 22 +++ ui/src/components/TaskChatThread.tsx | 22 ++- .../task-chat/TaskChatQueuedMessages.test.tsx | 58 +++++- .../task-chat/TaskChatQueuedMessages.tsx | 34 ++-- ui/src/lib/issue-queued-comment-queue.test.ts | 12 ++ ui/src/lib/issue-queued-comment-queue.ts | 9 +- ui/src/pages/IssueDetail.test.tsx | 106 +++++++++-- ui/src/pages/IssueDetail.tsx | 166 +++++++++++++----- 26 files changed, 645 insertions(+), 111 deletions(-) diff --git a/doc/SPEC-implementation.md b/doc/SPEC-implementation.md index f6bda971fab..821c0486fa1 100644 --- a/doc/SPEC-implementation.md +++ b/doc/SPEC-implementation.md @@ -1702,12 +1702,18 @@ Confirmed project creation appears as a durable card in the shared task transcri ### User continuation after execution recovery stops -An authenticated user message or an exact failed-run Retry can start a fresh +An authenticated user message, a validated undelivered native message queue, or an exact failed-run Retry can start a fresh native or legacy conversation turn once the prior execution is confirmed stopped. Retain the source history and uncertain action outcomes; do not replay tool calls or reset the failed incident's automatic retry budget. Existing pause, approval, budget, ownership, and dependency gates remain in effect. See `doc/execution-semantics.md` for admission and stop-proof requirements. +The task recovery notice offers Retry for eligible failures and verified native +startup cancellations, with failed attempts explained inline. Preparing native +turns keep the Steer label. Steer and Interrupt immediately move the submitted +messages from the composer queue into the conversation while delivery proceeds. +Provider acknowledgement remains authoritative; failed delivery restores the +latest queue with an inline error. Neither action produces a toast. ### Managed AI authentication diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 05b50745ec8..d1130d31819 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -486,7 +486,7 @@ A terminal native failure can retain a result accepted before checkpoint or clea A new user message can continue a terminal native run whose process fields were cleared before local stop receipts existed. Admission must verify the exact run, runner, workspace, and provider session in the retained suspended state, with no active provider turn, pending tool call, or undelivered output. Missing or mismatched state keeps the hold. A later recorded process launch also keeps the hold until its stop is verified. Normal assignment, decision, controller, environment cleanup, and active-run gates still apply. The message starts one fresh conversation turn; it does not replay the failed run, reset its recovery budget, or certify unknown action outcomes. -The task thread exposes the existing guarded Retry action for failed or timed-out legacy conversation runs. Where the server supports an explicit new attempt after a stopped legacy conversation, the thread must not hide that action solely because the old run still has a recovery-needed projection. Native and process recovery holds, pending decisions, active execution, and other retry gates remain in force. When a gate hides Retry, the thread says the message is preserved instead of promising an unavailable action. This presentation change does not rewrite historical outcomes or certify prior actions. +The task thread exposes the guarded Retry action for failed or timed-out conversation runs and native preparation cancelled before provider startup when the never-started proof is verified. The server projects this eligibility on the recovery notice and rechecks it on Retry. Cleanup quarantine retains its inspection path, and non-conversation reconciliation gates remain enforced. Pending decisions, active execution, pause, budget, dependency, and ownership gates remain in force. A refused Retry reports its reason inline and remains available for another attempt. A valid recovery action must name: @@ -1003,7 +1003,7 @@ For native conversations, an authenticated user message sent after the previous Local recovery records a server-authored stop receipt before it clears a verified absent process identity. A new execution request invalidates that receipt before any process can spawn; recording a new process identity also invalidates it. Missing process IDs without a receipt still block admission. Remote execution continues to require termination receipts for every lease. -If cleanup or another execution gate is still pending, the message stays in its existing queue receipt. Startup and periodic scheduling reconsider up to 50 due receipts per pass, at most once per 30 seconds per receipt, without calling a model or resetting recovery attempts. Cleanup callbacks use the same admission path. The issue lock prevents concurrent workers from delivering an adopted or discarded receipt again. The queued-message area shows the current wait reason. Pauses, approvals, budgets, ownership, and external chat authorization remain enforced. A message sent before the run finished does not grant new post-stop authority. +If cleanup or another execution gate is still pending, the message stays in its existing queue receipt. Startup and periodic scheduling reconsider up to 50 due receipts per pass, at most once per 30 seconds per receipt, without calling a model or resetting recovery attempts. Cleanup callbacks use the same admission path. The issue lock prevents concurrent workers from delivering an adopted or discarded receipt again. The queued-message area shows the current wait reason. Pauses, approvals, budgets, ownership, and external chat authorization remain enforced. For a failed native conversation, saved authenticated messages that have never been delivered can request a fresh turn after verified stop even when submitted before the failure finished. Admission validates and adopts their exact queue receipt, order, and original authors. This does not replay the failed request or undo an operator cancellation. Historical legacy interruption holds for conversational adapters no longer block new messages or Resume. Automatic classification uses the server-owned adapter identity saved atomically at run claim, the saved adapter invocation, or the continuation policy, never the agent’s current adapter settings. Missing historical adapter evidence retains the automatic hold; an explicit user continuation can retire it after proving the predecessor stopped. A terminal row with a live predecessor process, an unreleased environment lease, or failed/pending cleanup still blocks actual admission and Resume; a release timestamp alone does not prove cleanup succeeded. Retry scheduling can happen before cleanup, but grants no execution authority. Recovery folds their obsolete no-replay bookkeeping without changing task ownership, status, or automatically waking old work. The audit trail remains readable. Native integrity and ownership holds, and non-conversational adapter holds, remain enforced. @@ -1045,7 +1045,7 @@ unknown, and they do not gain automatic replay eligibility. Admission validates the persisted comment's author, task, and time against every held predecessor. Retry validates the selected failed run's company, task, and agent and preserves that run's identity through admission and history loading. -Duplicate Retry requests adopt the same successor. An agent-authored comment, an old queued request, or a generic +Duplicate Retry requests adopt the same successor. Only a validated, unconsumed human queue receipt can authorize saved-message continuation after a native failure; arbitrary historical input, an agent-authored comment, or a generic system wake cannot release a hold. The source task keeps its assignee. Process ownership, active controllers, cleanup leases, pause, approval, budget, and normal execution gates still apply. Dependency-blocked interaction mode remains limited @@ -1322,8 +1322,22 @@ execution recovery holds still apply. Unconfirmed cleanup does not start work. The active session advertises steering only when its driver supports it. A transport method that rejects steering does not grant that capability. The -queued-message control remains mounted until the server accepts a steer request, -so a rejected last-row action keeps its message and visible error. +queued-message controller remains mounted across an empty queue, preserving +pending delivery and inline errors. Steer immediately moves the selected message +into the conversation; Interrupt immediately moves all submitted queued messages. +Stale queue polls do not restore those rows while delivery is pending or accepted. +Rejected delivery restores the latest queue and shows an inline error without a +toast. The provider acknowledgement remains the authoritative delivery receipt; +optimistic display never grants permission or records successful delivery. A preparing +Paperclip Runner retains the Steer label while steering is unavailable; the +initial legacy database default and an empty queue snapshot cannot select +Interrupt. Model and effort changes apply to the next turn and do not change +the active turn's steering target. While awaiting steering acknowledgement, the +queue route holds the task and queue locks but leaves the run row available for +PRP event persistence. It locks and rereads the run after acknowledgement before +merging the receipt, preserving concurrent provider updates. +If the run stops during that wait, the request fails with a stale-target error +and retains the queued input for continuation. ### Preserve work across handoff and deliver requested files diff --git a/packages/shared/src/types/execution-projection.ts b/packages/shared/src/types/execution-projection.ts index 60090799e9a..02a92849144 100644 --- a/packages/shared/src/types/execution-projection.ts +++ b/packages/shared/src/types/execution-projection.ts @@ -9,6 +9,8 @@ export interface ExecutionBlocker { runError?: string | null; /** Candidate only: admission must still verify termination and all gates. */ canContinue?: boolean; + /** Explicit Retry candidate; the server rechecks stop proof and execution gates. */ + canRetry?: boolean; savedMessageCount?: number; } diff --git a/server/src/__tests__/issue-queued-comments-routes.test.ts b/server/src/__tests__/issue-queued-comments-routes.test.ts index 96e73d597ce..552cab5c486 100644 --- a/server/src/__tests__/issue-queued-comments-routes.test.ts +++ b/server/src/__tests__/issue-queued-comments-routes.test.ts @@ -30,6 +30,7 @@ import { issueService } from "../services/issues.js"; import { issueThreadInteractionService } from "../services/issue-thread-interactions.js"; import { remoteTerminationReceipt } from "../services/remote-execution-termination.js"; import { initializeRunIdentity, reconcileSteeredIdentity } from "../services/run-identity.js"; +import { appendHeartbeatRunEvent } from "../services/heartbeat-run-events.js"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase, @@ -1449,6 +1450,22 @@ describeEmbeddedPostgres("issue queued-comment routes", () => { }); }); + it.each(["paperclip_runner", "codex_local"])("keeps Steer during native preparation even after agent settings change to %s", async adapterType => { + const seeded = await seedQueue(); + const steeringCalls = steerNativeSessionMock.mock.calls.length; + await db.update(heartbeatRuns).set({ runtimeMode: "legacy", runtimeModeResolvedAt: null, + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } }, + }).where(eq(heartbeatRuns.id, seeded.runId)); + await db.update(agents).set({ adapterType }).where(eq(agents.id, seeded.agentId)); + const client = app(seeded.companyId); + const initial = await request(client).get(`/api/issues/${seeded.issueId}/queued-comments`).expect(200); + expect(initial.body).toMatchObject({ protocol: "paperclip_runner_v1", steeringDisposition: "temporarily_unavailable" }); + const edited = await request(client).patch(`/api/issues/${seeded.issueId}/queued-comments/${seeded.commentIds[0]}`) + .send({ queueId: seeded.wakeId, revision: initial.body.revision, body: "Updated direction" }).expect(200); + expect(edited.body).toMatchObject({ protocol: "paperclip_runner_v1", steeringDisposition: "temporarily_unavailable" }); + expect(steerNativeSessionMock.mock.calls).toHaveLength(steeringCalls); + }); + it("serializes discard against queued-run claim", async () => { const seeded = await seedQueue(); await db.delete(issueComments).where(eq(issueComments.id, seeded.commentIds[1])); @@ -1579,6 +1596,65 @@ describeEmbeddedPostgres("issue queued-comment routes", () => { }); }); + it("allows the runner to persist its acknowledgement while steering, preserving concurrent run updates", async () => { + const seeded = await seedQueue(); + await seedDispatchIdentity(seeded); + const originalProfile = { adapterDispatch: { adapterType: "paperclip_runner" }, + nativeExecutionInput: { provider: { kind: "codex", model: "gpt-5.6-sol", reasoningEffort: "medium" } } }; + await db.update(heartbeatRuns).set({ runnerProfileJson: originalProfile }).where(eq(heartbeatRuns.id, seeded.runId)); + await db.update(issues).set({ assigneeAdapterOverrides: { model: "gpt-6-sol", reasoningEffort: "xhigh" } }) + .where(eq(issues.id, seeded.issueId)); + steerNativeSessionMock.mockImplementationOnce(async () => { + await db.transaction(async tx => { + await tx.execute(sql`set local lock_timeout = '1000ms'`); + await appendHeartbeatRunEvent(tx as unknown as typeof db, { + companyId: seeded.companyId, agentId: seeded.agentId, runId: seeded.runId, + eventType: "item.completed", payload: { kind: "steering_acknowledgement" }, + }); + await tx.update(heartbeatRuns).set({ resultJson: { concurrentProviderReceipt: "preserve" } }) + .where(eq(heartbeatRuns.id, seeded.runId)); + }); + return { turnId: "turn-before-model-change" }; + }); + const initial = await request(app(seeded.companyId)).get(`/api/issues/${seeded.issueId}/queued-comments`); + const steered = await request(app(seeded.companyId)) + .post(`/api/issues/${seeded.issueId}/queued-comments/${seeded.commentIds[0]}/steer`) + .send({ queueId: seeded.wakeId, targetRunId: seeded.runId, revision: initial.body.revision }); + expect(steered.status, JSON.stringify(steered.body)).toBe(200); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)); + expect(run.status).toBe("running"); + expect(run.runnerProfileJson).toEqual(originalProfile); + expect(steerNativeSessionMock).toHaveBeenCalledWith(expect.objectContaining({ runId: seeded.runId, + message: "First queued message", correlationId: seeded.commentIds[0] })); + expect(run.resultJson).toMatchObject({ concurrentProviderReceipt: "preserve", queuedSteeringAcknowledgements: { + [seeded.commentIds[0]]: { status: "acknowledged", turnId: "turn-before-model-change" }, + } }); + expect(steered.body.entries.map((entry: any) => entry.comment.id)).toEqual([seeded.commentIds[1]]); + }); + + it.each(["cancelled", "failed", "succeeded"] as const)("preserves queued input when the run stops during acknowledgement (%s)", async status => { + const seeded = await seedQueue(); + await seedDispatchIdentity(seeded); + steerNativeSessionMock.mockImplementationOnce(async () => { + await db.update(heartbeatRuns).set({ status, finishedAt: new Date(), resultJson: { terminalReceipt: "preserve" } }) + .where(eq(heartbeatRuns.id, seeded.runId)); + return { turnId: "turn-that-stopped" }; + }); + const initial = await request(app(seeded.companyId)).get(`/api/issues/${seeded.issueId}/queued-comments`); + const response = await request(app(seeded.companyId)) + .post(`/api/issues/${seeded.issueId}/queued-comments/${seeded.commentIds[0]}/steer`) + .send({ queueId: seeded.wakeId, targetRunId: seeded.runId, revision: initial.body.revision }); + expect(response.status).toBe(409); + expect(response.body.details).toMatchObject({ code: "queued_comment_stale_target" }); + const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, seeded.wakeId)); + expect(wake.status).toBe("deferred_issue_execution"); + expect((wake.payload as any)._paperclipWakeContext.wakeCommentIds).toEqual(seeded.commentIds); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)); + expect(run.resultJson).toEqual({ terminalReceipt: "preserve" }); + const activity = await db.select().from(activityLog).where(eq(activityLog.action, "issue.queued_comment_steered")); + expect(activity).toHaveLength(0); + }); + it("keeps the identity pending after a steering timeout, then reconciles it on a later acknowledgement", async () => { const seeded = await seedQueue(); await seedDispatchIdentity(seeded); diff --git a/server/src/modules/wake-queue/adapters/queued-comment-postgres.ts b/server/src/modules/wake-queue/adapters/queued-comment-postgres.ts index 375a99eb633..2588636e4bc 100644 --- a/server/src/modules/wake-queue/adapters/queued-comment-postgres.ts +++ b/server/src/modules/wake-queue/adapters/queued-comment-postgres.ts @@ -31,7 +31,9 @@ function toWakeRow(row: WakeRow): QueuedCommentWakeRow { } function toRunRow(row: RunRow): QueuedCommentRunRow { - return { id: row.id, status: row.status, runtimeMode: row.runtimeMode, contextSnapshot: parseObject(row.contextSnapshot) }; + return { id: row.id, status: row.status, runtimeMode: row.runtimeMode, + runtimeModeResolvedAt: row.runtimeModeResolvedAt, runnerProfileJson: row.runnerProfileJson, + contextSnapshot: parseObject(row.contextSnapshot) }; } export type QueuedCommentQueuePostgresAdapterDeps = { diff --git a/server/src/modules/wake-queue/application/queued-comment-ports.ts b/server/src/modules/wake-queue/application/queued-comment-ports.ts index a30d0c6bc02..d3a4f48f01c 100644 --- a/server/src/modules/wake-queue/application/queued-comment-ports.ts +++ b/server/src/modules/wake-queue/application/queued-comment-ports.ts @@ -61,6 +61,8 @@ export type QueuedCommentRunRow = { id: string; status: string; runtimeMode: string | null; + runtimeModeResolvedAt?: Date | null; + runnerProfileJson?: Record | null; contextSnapshot: Record; }; diff --git a/server/src/routes/agents.ts b/server/src/routes/agents.ts index c3f1b8a422c..1fc61ee9e11 100644 --- a/server/src/routes/agents.ts +++ b/server/src/routes/agents.ts @@ -13,7 +13,7 @@ import { assertAiConnectionCreateAccess, canInstallSharedAiConnectionForNewAgent import { isAiConnectionCompatible } from "@paperclipai/shared"; import { applyConnectorSkills, resolveConnectorAssignments, annotateConnectorSkills, isConnectorSkill } from "../services/connector-runtime.js"; import { getExecutionBlocker } from "../services/execution-blocker.js"; -import { canContinueCancelledRun } from "../services/run-cancellation.js"; +import { canRetryStoppedRun } from "../services/cancelled-native-startup.js"; import { paperclipRunnerTransitionConfig, normalizeLegacyRunnerProvider, isPaperclipRunnerProvider } from "@paperclipai/adapter-utils"; import { executionProjectionForRun, executionProjectionsForRuns } from "../services/execution-projection.js"; import { selectDashboardRunIds } from "../services/dashboard-run-selection.js"; @@ -6006,14 +6006,14 @@ export function agentRoutes( ) { throw notFound("Failed run not found"); } - if (!["failed", "timed_out"].includes(failedRun.status) && !canContinueCancelledRun(failedRun)) { - throw conflict("Only a failed run or verified unexpected cancellation can start a new attempt."); - } if (failedRun.runtimeMode === "native" && failedRun.errorCode === "native_session_cleanup_quarantined") { throw conflict("The stopped native session requires cleanup and reconciliation before a new attempt.", { code: "native_session_cleanup_quarantined", }); } + if (!(await canRetryStoppedRun(db, failedRun))) { + throw conflict("Only a failed run or a verified eligible cancellation can start a new attempt."); + } const failedContext = asRecord(failedRun.contextSnapshot) ?? {}; const issueId = typeof failedContext.issueId === "string" diff --git a/server/src/routes/issues.ts b/server/src/routes/issues.ts index 828cfef4703..8b1543a5385 100644 --- a/server/src/routes/issues.ts +++ b/server/src/routes/issues.ts @@ -7064,6 +7064,8 @@ export function issueRoutes( queueId: string; targetRunId?: string; allowStoppedTarget?: boolean; + /** Steering must allow PRP event ingestion to update the run before ACK. */ + lockRun?: boolean; }) { const [currentIssue] = await input.tx .select() @@ -7145,8 +7147,8 @@ export function issueRoutes( state === "deferred" ? (input.targetRunId ?? input.issue.executionRunId ?? null) : null; - const activeRun = activeRunId - ? await input.tx + const activeRunQuery = activeRunId + ? input.tx .select() .from(heartbeatRuns) .where( @@ -7156,9 +7158,11 @@ export function issueRoutes( input.allowStoppedTarget ? undefined : eq(heartbeatRuns.status, "running"), ), ) - .for("update") - .limit(1) - .then((rows) => rows[0] ?? null) + .$dynamic() + : null; + const activeRun = activeRunQuery + ? await (input.lockRun === false ? activeRunQuery : activeRunQuery.for("update")) + .limit(1).then(rows => rows[0] ?? null) : null; if (input.targetRunId) { const runContext = readObject(activeRun?.contextSnapshot); @@ -15577,7 +15581,6 @@ export function issueRoutes( eq(heartbeatRuns.agentId, retryWake.agentId), ), ) - .for("update") .limit(1) .then((rows) => rows[0] ?? null) : null; @@ -15615,6 +15618,7 @@ export function issueRoutes( actor, queueId: req.body.queueId, targetRunId: req.body.targetRunId, + lockRun: false, }); if (!locked.activeRun) { throw conflict("The queued message targets a stale run", { @@ -15680,6 +15684,17 @@ export function issueRoutes( ? () => reconcileSteeredIdentity(db, steeringIdentity) : undefined, })); + // PRP ingestion allocates event sequences on this row. Locking it + // while awaiting the provider prevents its durable ACK from arriving. + // Re-read under lock after ACK so concurrent run receipts are retained. + const [acknowledgedRun] = await tx.select().from(heartbeatRuns) + .where(eq(heartbeatRuns.id, locked.activeRun.id)).for("update"); + if (!acknowledgedRun || acknowledgedRun.status !== "running") { + throw conflict("The queued message targets a stopped run. Your message is still queued.", { + code: "queued_comment_stale_target", + }); + } + const acknowledgedResult = readObject(acknowledgedRun.resultJson); if (steeringIdentity) await acceptSteeredIdentity(tx, steeringIdentity); acknowledgedTurnId = acknowledgement.turnId; @@ -15711,9 +15726,9 @@ export function issueRoutes( .update(heartbeatRuns) .set({ resultJson: { - ...runResult, + ...acknowledgedResult, queuedSteeringAcknowledgements: { - ...acknowledgements, + ...readObject(acknowledgedResult.queuedSteeringAcknowledgements), [commentId]: { status: "acknowledged", queueId: req.body.queueId, diff --git a/server/src/services/cancelled-native-startup.ts b/server/src/services/cancelled-native-startup.ts index 6be8792dda6..db3f082a88a 100644 --- a/server/src/services/cancelled-native-startup.ts +++ b/server/src/services/cancelled-native-startup.ts @@ -3,10 +3,23 @@ import { environmentLeases, heartbeatRunEvents, heartbeatRuns, nativeRunFinaliza import { claimedAdapterType } from "./conversation-continuation.js"; import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-local-process-stop.js"; import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js"; +import { canContinueCancelledRun } from "./run-cancellation.js"; type Run = typeof heartbeatRuns.$inferSelect; type Coordinator = typeof nativeRunFinalizations.$inferSelect; +/** Candidate for explicit user Retry only, never automatic replay after Stop. */ +export async function canRetryStoppedRun(db: Db, run: Run): Promise { + if (run.errorCode === "native_session_cleanup_quarantined") return false; + if (["failed", "timed_out"].includes(run.status) || canContinueCancelledRun(run)) return true; + if (run.status !== "cancelled" || !run.finishedAt || run.processPid || run.processGroupId || + run.processStartedAt || run.sessionIdAfter) return false; + const [coordinator] = await db.select().from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.companyId, run.companyId), eq(nativeRunFinalizations.runId, run.id), + )); + return isCancelledNativeStartup(db, run, coordinator); +} + /** Caller holds the coordinator and run locks when using this proof to admit * work. Attempt zero is a durable never-claimed receipt: every native executor * commits its first claim before it can start or attach a provider. */ diff --git a/server/src/services/execution-blocker.ts b/server/src/services/execution-blocker.ts index 325b9b9d4f5..2d70f70add9 100644 --- a/server/src/services/execution-blocker.ts +++ b/server/src/services/execution-blocker.ts @@ -3,6 +3,7 @@ import { conversationRecoveryActionPredicate, getConversationOwnershipBlocker } import { z } from "zod"; import { agentWakeupRequests, chatConversations, chatEndpoints, heartbeatRuns, issueComments, issues, issueRecoveryActions, toolConnections, type Db } from "@paperclipai/db"; import { canContinueCancelledRun, readRunCancellation } from "./run-cancellation.js"; +import { canRetryStoppedRun } from "./cancelled-native-startup.js"; import { queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js"; import { EXECUTION_RECONCILIATION_CAUSES, type ExecutionBlocker } from "@paperclipai/shared"; @@ -83,7 +84,9 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st : run?.error); const eligibleContinuation = Boolean(run && conversation?.assigneeAgentId === run.agentId && !["done", "cancelled"].includes(conversation.status) && canContinueCancelledRun(run)); - const [chatBinding] = eligibleContinuation ? await db.select({ + const canRetry = Boolean(run && conversation?.assigneeAgentId === run.agentId && + !["done", "cancelled"].includes(conversation.status) && await canRetryStoppedRun(db, run)); + const [chatBinding] = eligibleContinuation || canRetry ? await db.select({ state: chatConversations.state, endpointStatus: chatEndpoints.status, connectionStatus: toolConnections.status, connectionEnabled: toolConnections.enabled, }).from(chatConversations).leftJoin(chatEndpoints, and( @@ -94,6 +97,7 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st eq(chatConversations.companyId, companyId), eq(chatConversations.issueId, issueId), )).limit(1) : []; let nextAction = action.nextAction; + if (canRetry && !eligibleContinuation) nextAction += " Try again or send a new message to continue once the previous execution has stopped."; if (chatBinding) { if (chatBinding.state === "endpoint_removed" || !chatBinding.endpointStatus || chatBinding.endpointStatus === "archived") { nextAction = "This chat connection was removed. Inspect the stopped run and create a new task to continue the work."; @@ -117,6 +121,7 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st runStatus: run?.status ?? null, runError: runError?.slice(0, 1024) ?? null, canContinue: eligibleContinuation && !chatBinding, + canRetry: canRetry && !chatBinding, savedMessageCount: saved.length, }; } diff --git a/server/src/services/explicit-native-continuation.test.ts b/server/src/services/explicit-native-continuation.test.ts index 830c3ff2645..b535771c0f0 100644 --- a/server/src/services/explicit-native-continuation.test.ts +++ b/server/src/services/explicit-native-continuation.test.ts @@ -1075,6 +1075,78 @@ const support = await getEmbeddedPostgresTestSupport(); expect(coordinator.attempt).toBe(3); expect(coordinator.failureDetail?.replacementDenied).toBe("explicit_user_continuation"); }); + + it.each(["ready", "live_process", "pause", "delivered", "mixed_authors", "last_delivered", "operator_stop"])( + "automatically delivers an unconsumed native message sent before the failed run stopped, respecting %s", async gate => { + const f = await seed(); + await db.update(issueComments).set({ createdAt: new Date("2026-09-11T09:59:00Z") }).where(eq(issueComments.id, f.commentId)); + await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }); + let queuedIds = [f.commentId]; + let expectedIds = queuedIds; + if (["mixed_authors", "last_delivered"].includes(gate)) { + const secondId = randomUUID(); + await db.insert(issueComments).values({ id: secondId, companyId: f.companyId, issueId: f.issueId, + authorType: "user", authorUserId: "second-author", body: "Also preserve the existing work.", + createdAt: new Date("2026-09-11T09:59:01Z") }); + queuedIds = [f.commentId, secondId]; + expectedIds = gate === "last_delivered" ? [f.commentId] : queuedIds; + if (gate === "last_delivered") await db.update(heartbeatRuns).set({ + contextSnapshot: { issueId: f.issueId, wakeCommentIds: [secondId] }, + }).where(eq(heartbeatRuns.id, f.sourceRunId)); + } + if (gate === "live_process") await db.update(heartbeatRuns).set({ processPid: process.pid }).where(eq(heartbeatRuns.id, f.sourceRunId)); + if (gate === "operator_stop") await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "operator_interrupted" }).where(eq(heartbeatRuns.id, f.sourceRunId)); + if (gate === "delivered") await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, + wakeCommentIds: queuedIds } }).where(eq(heartbeatRuns.id, f.sourceRunId)); + if (gate === "pause") { + const holdId = randomUUID(); + await db.insert(issueTreeHolds).values({ id: holdId, companyId: f.companyId, rootIssueId: f.issueId, mode: "pause", status: "active" }); + await db.insert(issueTreeHoldMembers).values({ companyId: f.companyId, holdId, issueId: f.issueId, depth: 0, issueTitle: "Deploy", issueStatus: "blocked" }); + } + const queueId = randomUUID(); + await db.insert(agentWakeupRequests).values({ id: queueId, companyId: f.companyId, agentId: f.agentId, + source: "automation", status: "deferred_issue_execution", reason: "issue_commented", + requestedByActorType: "user", requestedByActorId: "board", updatedAt: new Date(0), + payload: { issueId: f.issueId, commentId: queuedIds.at(-1), executionWait: { reason: "execution_recovery" }, + _paperclipWakeContext: { issueId: f.issueId, wakeReason: "issue_commented", wakeCommentIds: queuedIds, wakeCommentId: queuedIds.at(-1) } }, + }); + const service = heartbeatService(db); + await service.resumeExecutionWaitComments(); + await service.resumeExecutionWaitComments(); + const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, queueId)); + if (!["ready", "mixed_authors", "last_delivered"].includes(gate)) { + expect(receipt.status).toBe("deferred_issue_execution"); + expect(await getExecutionBlocker(db, f.companyId, f.issueId)).not.toBeNull(); + expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")))).toHaveLength(0); + return; + } + expect(receipt.status).toBe("coalesced"); + const [successor] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, receipt.runId!)); + expect(successor.contextSnapshot).toMatchObject({ wakeCommentIds: expectedIds, + explicitUserContinuation: { previousRunId: f.sourceRunId, commentId: expectedIds.at(-1) } }); + expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull(); + const [prior] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId)); + expect(prior.status).toBe("failed"); + }); + + it.each(["native", "before_selection"])("offers and admits an exact Retry for native preparation cancelled before provider startup: %s", async phase => { + const f = await seedCancelledStartup(); + if (phase === "before_selection") { + await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId)); + await db.update(heartbeatRuns).set({ runtimeMode: "legacy", runtimeModeResolvedAt: null, nativeIssueId: null, + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } }, + resultJson: { startupCancellation: { beforeNativeSelection: true }, startupPreparationSettledAt: new Date().toISOString() }, + }).where(eq(heartbeatRuns.id, f.sourceRunId)); + } + const blocker = await getExecutionBlocker(db, f.companyId, f.issueId); + expect(blocker).toMatchObject({ runId: f.sourceRunId, canRetry: true }); + await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }); + const service = heartbeatService(db); + const successor = await service.wakeup(f.agentId, { source: "on_demand", triggerDetail: "manual", reason: "retry_failed_run", + failedRunId: f.sourceRunId, requestedByActorType: "user", requestedByActorId: "board", payload: { issueId: f.issueId } }); + expect(successor).toMatchObject({ status: "queued", retryOfRunId: f.sourceRunId }); + expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull(); + }); it.each(["failed", "succeeded_result", "rejected_result", "still_committing", "controller", "successor", "live_process"])( "requires a fully committed failed result and verified stop for a new turn (%s)", async scenario => { const f = await seed(), contractId = randomUUID(), resultId = randomUUID(); diff --git a/server/src/services/explicit-native-continuation.ts b/server/src/services/explicit-native-continuation.ts index eb47212a0dd..4dd475b43ab 100644 --- a/server/src/services/explicit-native-continuation.ts +++ b/server/src/services/explicit-native-continuation.ts @@ -76,7 +76,7 @@ export async function admitExplicitNativeContinuation(input: { failedRunId?: string | null; /** Server-recorded board intent to send an existing legacy message queue. */ queuedCommentInterruptId?: string; - /** Internal delivery of an unconsumed, user-authored legacy queue entry. */ + /** Internal delivery of an unconsumed, user-authored saved queue entry. */ queuedCommentRequestId?: string; dryRun?: boolean; onBlocked?: (reason: string, message: string) => void; @@ -114,10 +114,14 @@ export async function admitExplicitNativeContinuation(input: { !savedQueue.idempotencyKey?.startsWith("chat-inbound:") && queuedCommentIdsFromWakePayload(savedQueue.payload).includes(commentId)); if (input.queuedCommentRequestId && !queuedRequest) return null; + let partiallyDeliveredQueue = false; if (queuedRequest) { const ids = queuedCommentIdsFromWakePayload(savedQueue!.payload); const undelivered = await undeliveredLegacyUserCommentIds(db, companyId, issueId, agentId, ids); - if (undelivered.length !== ids.length) return null; + // A partly consumed queue still owns its remaining input. Admission and + // adoption both recheck the durable queue and discard already delivered IDs. + if (undelivered.at(-1) !== commentId) return null; + partiallyDeliveredQueue = undelivered.length !== ids.length; } const [comment] = retry || response ? [] : await db.select().from(issueComments).where(and( eq(issueComments.companyId, companyId), eq(issueComments.issueId, issueId), @@ -165,16 +169,17 @@ export async function admitExplicitNativeContinuation(input: { const unusedAdmission = run.status === "cancelled" && !run.startedAt && run.errorCode === "execution_reconciliation_required" && !run.processPid && !run.processGroupId && !run.nativeSessionId; + if (partiallyDeliveredQueue && run.runtimeMode !== "native" && !unusedAdmission) return null; const legacyUserTurn = run.runtimeMode === "legacy" && action.cause === "legacy_execution_requires_reconciliation" && isConversationAdapter(agent.adapterType); if ((queuedInterrupt || queuedRequest) && !legacyUserTurn && !unusedAdmission && + !(queuedRequest && run.runtimeMode === "native" && + (run.status !== "cancelled" || authorizedAt > run.finishedAt)) && !(queuedInterrupt && response?.source.requiresFreshSession && run.runtimeMode === "native")) return null; // Saved input is a request for a new turn, never permission to undo an // operator Stop or redeliver a message already consumed by this run. - if (retry && run.status === "cancelled" && !canContinueCancelledRun(run)) return blocked("cancelled_by_operator", "Inspect the stopped run and send a new message to continue."); - if (queuedRequest && !queuedInterrupt && ((run.status === "cancelled" && !unusedAdmission && !canContinueCancelledRun(run)) || - run.contextSnapshot?.wakeCommentId === commentId || + if (queuedRequest && !queuedInterrupt && (run.contextSnapshot?.wakeCommentId === commentId || (Array.isArray(run.contextSnapshot?.wakeCommentIds) && run.contextSnapshot.wakeCommentIds.includes(commentId)))) return null; if (legacyUserTurn) { const historicalAdapter = await historicalAdapterType(db, run); @@ -200,6 +205,10 @@ export async function admitExplicitNativeContinuation(input: { return blocked("workspace_repair_required", "Verify safe workspace staging or repair before continuing. Your message is saved."); } const cancelledStartup = await isCancelledNativeStartup(db, run, coordinator); + if (queuedRequest && !queuedInterrupt && run.status === "cancelled" && !unusedAdmission && + !canContinueCancelledRun(run) && !(cancelledStartup && authorizedAt > run.finishedAt!)) return null; + if (retry && run.status === "cancelled" && !canContinueCancelledRun(run) && !cancelledStartup) + return blocked("cancelled_by_operator", "Inspect the stopped run and send a new message to continue."); if (cancelledStartup) cancelledStartupIds.add(run.id); if (run.runtimeMode !== "native" && !unusedAdmission && !legacyUserTurn && !cancelledStartup) return null; // A provider failure can finish the normal result/assessment commit path. diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 1c1e7c64c27..02162d05c83 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -33,6 +33,7 @@ import { completeTerminatedRemoteNativeSessionCleanup } from "../vendor/papercli import { hasRemoteTerminationReceipt, remoteExecutionHasStopped, remoteTerminationReceipt, stoppedRemoteCleanupScopes } from "./remote-execution-termination.js"; import { applyConnectorSkills, prepareConnectorSkillDelivery, resolveConnectorAssignments } from "./connector-runtime.js"; import { admitExplicitNativeContinuation, undeliveredLegacyUserCommentIds } from "./explicit-native-continuation.js"; +import { canRetryStoppedRun } from "./cancelled-native-startup.js"; import { connectionIntentService } from "./connection-intents.js"; import { managedAiSessionFingerprintConfig, prepareManagedAiRuntime, assertManagedAiProjectAuth, stripAiAuthBindings, isAiConnectionBusy, AI_AUTH_ENV_KEYS } from "./ai-connection-runtime.js"; import { aiConnectionBindingSchema } from "@paperclipai/shared"; @@ -10413,7 +10414,7 @@ export function heartbeatService( let commentId = deriveCommentId(context, payload); let requestedByActorId = wake.requestedByActorId; const reason = readNonEmptyString(context.wakeReason) ?? wake.reason; - if (stoppedNativeContinuation) { + if (run.runtimeMode === "native") { const ids = await undeliveredLegacyUserCommentIds(db, run.companyId, issueId, run.agentId, queuedCommentIdsFromWakePayload(payload)); if (!ids.length) continue; @@ -10443,6 +10444,7 @@ export function heartbeatService( const admitted = await admitExplicitNativeContinuation({ db, companyId: run.companyId, issueId, agentId: run.agentId, actorType: wake.requestedByActorType, actorId: requestedByActorId, reason, commentId, successorRunId: randomUUID(), dryRun: true, + ...(run.runtimeMode === "native" ? { queuedCommentRequestId: wake.id } : {}), onBlocked: (reason, message) => { wait = { reason, message }; }, }); if (!admitted) { @@ -10460,7 +10462,7 @@ export function heartbeatService( await enqueueWakeup(run.agentId, { source: wake.source as WakeupOptions["source"], triggerDetail: (wake.triggerDetail ?? undefined) as WakeupOptions["triggerDetail"], reason, payload, contextSnapshot: context, requestedByActorType: "user", requestedByActorId, - ...(stoppedNativeContinuation ? { queuedCommentRequestId: wake.id } : {}), + ...(run.runtimeMode === "native" ? { queuedCommentRequestId: wake.id } : {}), idempotencyKey: `remote-stop-comment:${run.id}:${wake.id}` }, wake.id); break; } @@ -26901,7 +26903,7 @@ export function heartbeatService( if (opts.requestedByActorType !== "user" || !opts.requestedByActorId || reason !== "retry_failed_run" || source !== "on_demand" || triggerDetail !== "manual" || !failed || failed.companyId !== agent.companyId || failed.agentId !== agentId || - (!["failed", "timed_out"].includes(failed.status) && !canContinueCancelledRun(failed)) || + !(await canRetryStoppedRun(db, failed)) || (failed.nativeIssueId ?? readNonEmptyString(failed.contextSnapshot?.issueId)) !== issueId) { throw conflict("The selected failed run cannot be retried for this task."); } diff --git a/server/src/services/issue-queued-comment-queue.test.ts b/server/src/services/issue-queued-comment-queue.test.ts index 854cf986f76..4659954ed1f 100644 --- a/server/src/services/issue-queued-comment-queue.test.ts +++ b/server/src/services/issue-queued-comment-queue.test.ts @@ -67,6 +67,22 @@ describe("decideQueuedCommentQueueSteering", () => { expect(decision).toEqual({ protocol: "paperclip_runner_v1", kind: "probe", steeringRunId: "run-1" }); }); + + it("keeps a preparing Paperclip Runner on the steering protocol until runtime selection finishes", () => { + expect(decideQueuedCommentQueueSteering({ state: "deferred", queueRunRuntimeMode: null, + activeRun: { id: "preparing-run", runtimeMode: "legacy", runtimeModeResolvedAt: null, + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } } }, + assignedAgentAdapterType: "paperclip_runner", queuedCommentCount: 1, + })).toEqual({ protocol: "paperclip_runner_v1", kind: "temporarily_unavailable" }); + }); + + it("preserves the historical adapter when mutable agent settings change", () => { + expect(decideQueuedCommentQueueSteering({ state: "deferred", queueRunRuntimeMode: null, + activeRun: { id: "legacy-run", runtimeMode: "legacy", runtimeModeResolvedAt: null, + runnerProfileJson: { adapterDispatch: { adapterType: "codex_local" } } }, + assignedAgentAdapterType: "paperclip_runner", queuedCommentCount: 1, + })).toEqual({ protocol: "legacy", kind: "unsupported" }); + }); }); describe("buildQueuedCommentQueueSnapshot entry permissions", () => { diff --git a/server/src/services/issue-queued-comment-queue.ts b/server/src/services/issue-queued-comment-queue.ts index 8a31139d873..7e3b5224686 100644 --- a/server/src/services/issue-queued-comment-queue.ts +++ b/server/src/services/issue-queued-comment-queue.ts @@ -128,11 +128,16 @@ export function decideQueuedCommentQueueSteering(facts: { /** The queued run's own runtime mode. Read only when `state` is `"queued"`. */ queueRunRuntimeMode: string | null; /** The currently running turn, if any. Read only when `state` is `"deferred"`. */ - activeRun: { id: string; runtimeMode: string | null } | null; + activeRun: { id: string; runtimeMode: string | null; runtimeModeResolvedAt?: Date | null; runnerProfileJson?: Record | null } | null; assignedAgentAdapterType: string | null; queuedCommentCount: number; }): QueuedCommentQueueSteeringDecision { - const persistedRuntimeMode = + // The default legacy value is not a selection receipt during preparation. + // Use the run's immutable dispatch choice, never the agent's mutable settings. + const preparingNative = facts.state === "deferred" && facts.activeRun?.runtimeMode === "legacy" && + facts.activeRun.runtimeModeResolvedAt === null && + record(facts.activeRun.runnerProfileJson?.adapterDispatch).adapterType === "paperclip_runner"; + const persistedRuntimeMode = preparingNative ? null : facts.state === "queued" ? facts.queueRunRuntimeMode : facts.state === "deferred" @@ -140,7 +145,7 @@ export function decideQueuedCommentQueueSteering(facts: { : null; const protocol: QueuedCommentQueueProtocol = - persistedRuntimeMode === "native" + preparingNative || persistedRuntimeMode === "native" || (persistedRuntimeMode === null && facts.assignedAgentAdapterType === "paperclip_runner") ? "paperclip_runner_v1" : "legacy"; @@ -150,7 +155,7 @@ export function decideQueuedCommentQueueSteering(facts: { } const steeringRun = facts.state === "deferred" ? facts.activeRun : null; - if (!steeringRun || facts.queuedCommentCount === 0) { + if (!steeringRun || preparingNative || facts.queuedCommentCount === 0) { return { protocol, kind: "temporarily_unavailable" }; } diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 11c1e5256e4..148a5ebab3d 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -6510,8 +6510,9 @@ export async function steerNativeSession(input: { steeringDeliveries.delete(deliveryKey); }); } - // Do not await the persistence callback here: the route holds the run lock - // until acknowledgement. After a timeout this callback can acquire that lock. + // The route still serializes queue mutations on the task until acknowledgement. + // Reconciliation can acquire that task lock after success or a timeout; never + // join it from the provider's acknowledgement path. if (input.onAcknowledged) void delivery.then(input.onAcknowledged).catch(() => undefined); let timeout: ReturnType | null = null; diff --git a/ui/src/components/ExecutionBlockerNotice.test.tsx b/ui/src/components/ExecutionBlockerNotice.test.tsx index 9627098a37d..3683c76883a 100644 --- a/ui/src/components/ExecutionBlockerNotice.test.tsx +++ b/ui/src/components/ExecutionBlockerNotice.test.tsx @@ -106,6 +106,21 @@ describe("stopped task recovery notice", () => { expect(agentsApi.retryFailedRun).toHaveBeenCalledWith("agent", "failed-run", "company"); expect(onRetried).toHaveBeenCalledOnce(); }); + + it.each(["native_continuation_requires_reconciliation", "uncertain_external_action"])("offers Retry for a server-admitted native failure: %s", async cause => { + await act(async () => root.render( + + )); + const button = container.querySelector("button"); + expect(button?.textContent).toBe("Retry"); + vi.mocked(agentsApi.retryFailedRun).mockResolvedValue({} as never); + await act(async () => button!.click()); + expect(agentsApi.retryFailedRun).toHaveBeenCalledWith("agent", "failed-run", "company"); + }); it("shows a failed Retry in the same container and allows another attempt", async () => { vi.mocked(agentsApi.retryFailedRun).mockRejectedValue(new Error("Environment cleanup is still running.")); await act(async () => container.querySelector("button")!.click()); diff --git a/ui/src/components/ExecutionBlockerNotice.tsx b/ui/src/components/ExecutionBlockerNotice.tsx index d80d11b6a36..175e3d9caca 100644 --- a/ui/src/components/ExecutionBlockerNotice.tsx +++ b/ui/src/components/ExecutionBlockerNotice.tsx @@ -45,8 +45,8 @@ export function ExecutionBlockerNotice({ companyId, issueId, blocker, onRetried Inspect run )} - {!requiresInspection && blocker.agentId && blocker.runId && - ((blocker.cause === "legacy_execution_requires_reconciliation" && failedRun) || blocker.canContinue) && ( + {(!requiresInspection || blocker.canRetry) && blocker.agentId && blocker.runId && + ((blocker.cause === "legacy_execution_requires_reconciliation" && failedRun) || blocker.canContinue || blocker.canRetry) && ( diff --git a/ui/src/components/TaskChatThread.test.tsx b/ui/src/components/TaskChatThread.test.tsx index c960d3e9c7f..f4d0f046ab1 100644 --- a/ui/src/components/TaskChatThread.test.tsx +++ b/ui/src/components/TaskChatThread.test.tsx @@ -3675,6 +3675,28 @@ describe("TaskChatThread Paperclip Runner queue", () => { expect(occurrenceCount(queuedComment.body)).toBe(1); }); + it.each(["steer", "interrupt"] as const)("preserves inline %s errors after the optimistic last row clears", async action => { + let rejectDelivery!: (error: Error) => void; + const delivery = new Promise((_, reject) => { rejectDelivery = reject; }); + const actionQueue = { ...queue, protocol: action === "interrupt" ? "legacy" as const : queue.protocol }; + const props = { comments: [queuedComment], onAdd: async () => {}, queuedCommentQueue: actionQueue, + onSteerQueuedComment: () => delivery, onInterruptQueued: () => delivery }; + render(); + await act(async () => { + container.querySelector(`[data-testid="task-chat-queued-${action}-queued-prp-1"]`)!.click(); + }); + render(); + expect(container.querySelector('[data-testid="task-chat-queued-messages"]')).toBeNull(); + await act(async () => { + rejectDelivery(new Error("Connection lost")); + await delivery.catch(() => undefined); + }); + await act(async () => { render(); }); + expect(container.textContent).toContain(action === "steer" + ? "Couldn’t steer. Message is still queued." : "Couldn’t interrupt. Message is still queued."); + expect(occurrenceCount(queuedComment.body)).toBe(1); + }); + it.each(["run-1", null])("keeps legacy queued delivery available with target %s", (targetRunId) => { const onInterruptQueued = vi.fn(async () => {}); render( diff --git a/ui/src/components/TaskChatThread.tsx b/ui/src/components/TaskChatThread.tsx index 45800935a12..1c409845c90 100644 --- a/ui/src/components/TaskChatThread.tsx +++ b/ui/src/components/TaskChatThread.tsx @@ -5,7 +5,7 @@ import { } from "@/lib/issue-detail-performance"; import { hasWorkspaceRestoreFailure } from "@paperclipai/shared"; import { workspaceRestoreMarkerDetail } from "@/lib/workspace-restore-marker"; -import type { ActivityEvent, TaskBrowser } from "@paperclipai/shared"; +import type { ActivityEvent, IssueQueuedCommentQueue, TaskBrowser } from "@paperclipai/shared"; import { useProjectCreatedItems } from "@/hooks/useProjectCreatedItems"; import { skillCreatedItems } from "@/components/task-chat/skill-created-items"; import { requiresExecutionReconciliation } from "@paperclipai/shared"; @@ -633,6 +633,19 @@ export function TaskChatThread(props: TaskChatThreadProps) { queuedCommentQueue && queuedCommentQueue.entries.length > 0 ? queuedCommentQueue : null; + const emptyQueuedCommentQueue = useMemo( + () => ({ + issueId: issueId ?? "", + queueId: null, + state: null, + targetRunId: null, + revision: "empty", + protocol: "paperclip_runner_v1", + steeringDisposition: "temporarily_unavailable", + entries: [], + }), + [issueId], + ); const queuedCommentIds = useMemo( () => new Set( @@ -3092,9 +3105,10 @@ export function TaskChatThread(props: TaskChatThreadProps) { className="relative isolate flex flex-col" data-testid="task-chat-composer-stack" > - {queuedMessageQueue && !composerPause ? ( + {!composerPause ? ( { if (!onReorderQueuedComments) @@ -3110,7 +3124,7 @@ export function TaskChatThread(props: TaskChatThreadProps) { await onSteerQueuedComment(commentId, revision); }} onInterrupt={ - onInterruptQueued && queuedMessageQueue.queueId + onInterruptQueued && queuedMessageQueue?.queueId ? async () => { await onInterruptQueued( queuedMessageQueue.targetRunId, diff --git a/ui/src/components/task-chat/TaskChatQueuedMessages.test.tsx b/ui/src/components/task-chat/TaskChatQueuedMessages.test.tsx index 8595554a717..ab2828867cc 100644 --- a/ui/src/components/task-chat/TaskChatQueuedMessages.test.tsx +++ b/ui/src/components/task-chat/TaskChatQueuedMessages.test.tsx @@ -12,10 +12,12 @@ import { function deferred() { let resolve!: (value: T) => void; - const promise = new Promise((innerResolve) => { + let reject!: (error: Error) => void; + const promise = new Promise((innerResolve, innerReject) => { resolve = innerResolve; + reject = innerReject; }); - return { promise, resolve }; + return { promise, resolve, reject }; } const queue: IssueQueuedCommentQueue = { @@ -138,7 +140,7 @@ describe("TaskChatQueuedMessages", () => { expect(next?.map((entry) => entry.position)).toEqual([0, 1]); }); - it("promotes only the selected steering row immediately", async () => { + it("removes the selected steering row before the provider acknowledges it", async () => { const acknowledgement = deferred(); const props = render({ onSteer: vi.fn().mockReturnValue(acknowledgement.promise), @@ -157,18 +159,63 @@ describe("TaskChatQueuedMessages", () => { '[data-testid="task-chat-queued-message-comment-1"]', ), ).toBeNull(); + expect(container.textContent).toContain("Steering queued message."); expect( container.querySelector( '[data-testid="task-chat-queued-message-comment-2"]', ), ).not.toBeNull(); + await act(async () => { + render({ ...props, queue: { ...queue, entries: [...queue.entries] } }); + }); + expect(container.querySelector('[data-testid="task-chat-queued-message-comment-1"]')).toBeNull(); + await act(async () => { acknowledgement.resolve(); await acknowledgement.promise; }); }); + it("clears the entire interrupt queue immediately and restores it inline on failure", async () => { + let rejectInterrupt!: (error: Error) => void; + const interrupted = new Promise((_, reject) => { rejectInterrupt = reject; }); + const props = render({ queue: { ...queue, protocol: "legacy", steeringDisposition: "unsupported" }, + onInterrupt: vi.fn().mockReturnValue(interrupted) }); + await act(async () => { + container.querySelector('[data-testid="task-chat-queued-interrupt-comment-1"]')!.click(); + }); + expect(props.onInterrupt).toHaveBeenCalledOnce(); + expect(container.querySelector('[data-testid="task-chat-queued-messages"]')).toBeNull(); + await act(async () => { rejectInterrupt(new Error("Connection lost")); await interrupted.catch(() => undefined); }); + expect(container.querySelectorAll('[data-testid^="task-chat-queued-message-"]')).toHaveLength(2); + expect(container.textContent).toContain("Couldn’t interrupt. Message is still queued."); + }); + + it.each(["steer", "interrupt"] as const)("shows a delivery error even when the queue becomes empty while %s is pending", async action => { + const delivery = deferred(); + const props = render({ queue: { ...queue, protocol: action === "interrupt" ? "legacy" : "paperclip_runner_v1" }, + onSteer: vi.fn().mockReturnValue(delivery.promise), onInterrupt: vi.fn().mockReturnValue(delivery.promise) }); + await act(async () => { container.querySelector(`[data-testid="task-chat-queued-${action}-comment-1"]`)!.click(); }); + await act(async () => { render({ ...props, queue: { ...props.queue, revision: "empty-poll", entries: [] } }); }); + await act(async () => { delivery.reject(new Error("Connection lost")); await delivery.promise.catch(() => undefined); }); + const error = action === "steer" ? "Couldn’t steer." : "Couldn’t interrupt."; + expect([...container.querySelectorAll('[role="status"]')].some(status => status.textContent?.includes(error))).toBe(true); + expect(container.querySelector('[data-testid="task-chat-queued-message-comment-1"]')).toBeNull(); + }); + + it("does not restore a message discarded elsewhere while steering was pending", async () => { + let rejectSteer!: (error: Error) => void; + const delivery = new Promise((_, reject) => { rejectSteer = reject; }); + const props = render({ onSteer: vi.fn().mockReturnValue(delivery) }); + await act(async () => { container.querySelector('[data-testid="task-chat-queued-steer-comment-1"]')!.click(); }); + await act(async () => { render({ ...props, queue: { ...queue, revision: "rev-2", entries: [queue.entries[1]] } }); }); + await act(async () => { rejectSteer(new Error("Queue changed")); await delivery.catch(() => undefined); }); + expect(container.querySelector('[data-testid="task-chat-queued-message-comment-1"]')).toBeNull(); + expect(container.querySelector('[data-testid="task-chat-queued-message-comment-2"]')).not.toBeNull(); + expect(container.textContent).toContain("Couldn’t steer."); + }); + it("keeps a row queued when steering fails and announces the retryable state", async () => { render({ onSteer: vi.fn().mockRejectedValue(new Error("steering_timeout")), @@ -344,9 +391,6 @@ describe("TaskChatQueuedMessages", () => { container.querySelector( '[data-testid="task-chat-queued-message-comment-1"]', ), - ).not.toBeNull(); - expect(container.textContent).toContain( - "Queued messages will be sent when the previous run has stopped.", - ); + ).toBeNull(); }); }); diff --git a/ui/src/components/task-chat/TaskChatQueuedMessages.tsx b/ui/src/components/task-chat/TaskChatQueuedMessages.tsx index 2df4b95c99b..a365a560677 100644 --- a/ui/src/components/task-chat/TaskChatQueuedMessages.tsx +++ b/ui/src/components/task-chat/TaskChatQueuedMessages.tsx @@ -1,4 +1,4 @@ -import { useEffect, useMemo, useState } from "react"; +import { useEffect, useMemo, useRef, useState } from "react"; import { DndContext, KeyboardSensor, @@ -232,6 +232,9 @@ export function TaskChatQueuedMessages({ onDiscard, }: TaskChatQueuedMessagesProps) { const [entries, setEntries] = useState(queue.entries); + const latestQueue = useRef(queue); + latestQueue.current = queue; + const optimisticDeliveryIds = useRef(new Set()); const [pending, setPending] = useState<{ commentId: string; action: Exclude; @@ -247,7 +250,10 @@ export function TaskChatQueuedMessages({ ); useEffect(() => { - setEntries(queue.entries); + // Polling can return the old queue while delivery is still in flight. + setEntries(queue.entries.filter( + (entry) => !optimisticDeliveryIds.current.has(entry.comment.id), + )); }, [queue.entries, queue.revision]); const ids = useMemo( @@ -309,7 +315,15 @@ export function TaskChatQueuedMessages({ ) { return; } - const previous = entries; + const deliveredIds = action === "interrupt" + ? entries.map((entry) => entry.comment.id) + : action === "steer" ? [commentId] : []; + for (const id of deliveredIds) optimisticDeliveryIds.current.add(id); + if (deliveredIds.length) { + setEntries((current) => current.filter( + (entry) => !optimisticDeliveryIds.current.has(entry.comment.id), + )); + } setPending({ commentId, action }); setVisibleError(null); setAnnouncement( @@ -319,11 +333,6 @@ export function TaskChatQueuedMessages({ ? "Sending queued messages." : "Discarding queued message.", ); - if (action === "steer") { - setEntries((current) => - current.filter((entry) => entry.comment.id !== commentId), - ); - } try { if (action === "steer") await onSteer(commentId, queue.revision); else if (action === "interrupt") await onInterrupt?.(); @@ -341,7 +350,12 @@ export function TaskChatQueuedMessages({ : "Queued message discarded.", ); } catch (error) { - if (action === "steer") setEntries(previous); + for (const id of deliveredIds) optimisticDeliveryIds.current.delete(id); + if (deliveredIds.length) { + setEntries(latestQueue.current.entries.filter( + (entry) => !optimisticDeliveryIds.current.has(entry.comment.id), + )); + } setAnnouncement(""); const code = queueActionErrorCode(error); setVisibleError( @@ -360,7 +374,7 @@ export function TaskChatQueuedMessages({ } } - if (entries.length === 0) return null; + if (entries.length === 0 && !visibleError) return null; return (
{ }); }); + it("keeps native steering while an empty queue snapshot predates the submitted message", () => { + const authoritativeQueue = normalizeIssueQueuedCommentQueue({ + protocol: "legacy", entries: [], + }, "issue-1"); + const queue = mergePendingIssueQueuedComments({ + issueId: "issue-1", authoritativeQueue, + pendingComments: [{ comment: comment("optimistic-1", "Use the new model next turn"), targetRunId: "run-native" }], + fallbackProtocol: "paperclip_runner_v1", + }); + expect(queue).toMatchObject({ protocol: "paperclip_runner_v1", steeringDisposition: "temporarily_unavailable" }); + }); + it("deduplicates acknowledged entries and restores the authoritative queue identity", () => { const pending = comment("comment-1", "Use three seconds instead"); const authoritativeQueue = normalizeIssueQueuedCommentQueue( diff --git a/ui/src/lib/issue-queued-comment-queue.ts b/ui/src/lib/issue-queued-comment-queue.ts index 8c214f4177c..9cae6860d8a 100644 --- a/ui/src/lib/issue-queued-comment-queue.ts +++ b/ui/src/lib/issue-queued-comment-queue.ts @@ -134,8 +134,11 @@ export function mergePendingIssueQueuedComments(params: { const fallbackTargetRunId = params.pendingComments.find((entry) => entry.targetRunId)?.targetRunId ?? null; - const protocol = - params.authoritativeQueue?.protocol ?? params.fallbackProtocol; + // An empty snapshot predating this submission does not own its runtime. + // Use the active run until the server acknowledges an actual queue. + const protocol = authoritativeOwnsQueue + ? params.authoritativeQueue!.protocol + : params.fallbackProtocol; const targetRunId = authoritativeOwnsQueue ? (params.authoritativeQueue?.targetRunId ?? null) : fallbackTargetRunId; @@ -151,7 +154,7 @@ export function mergePendingIssueQueuedComments(params: { revision: params.authoritativeQueue?.revision ?? "awaiting-server", protocol, steeringDisposition: - params.authoritativeQueue?.steeringDisposition ?? + (authoritativeOwnsQueue ? params.authoritativeQueue?.steeringDisposition : undefined) ?? (protocol === "paperclip_runner_v1" && targetRunId ? "temporarily_unavailable" : "unsupported"), diff --git a/ui/src/pages/IssueDetail.test.tsx b/ui/src/pages/IssueDetail.test.tsx index 057c8570fb6..526f8dc9677 100644 --- a/ui/src/pages/IssueDetail.test.tsx +++ b/ui/src/pages/IssueDetail.test.tsx @@ -755,10 +755,12 @@ function createTooltipRoot(container: Element): Root { function createDeferred() { let resolve!: (value: T) => void; - const promise = new Promise((innerResolve) => { + let reject!: (error: Error) => void; + const promise = new Promise((innerResolve, innerReject) => { resolve = innerResolve; + reject = innerReject; }); - return { promise, resolve }; + return { promise, resolve, reject }; } function createIssue(overrides: Partial = {}): Issue { @@ -3925,11 +3927,11 @@ describe("IssueDetail", () => { expect(mockHeartbeatsApi.cancel).not.toHaveBeenCalled(); }); - it("projects a native follow-up into the steering well before the post resolves", async () => { + it.each(["native", "legacy"] as const)("projects a native follow-up into the steering well before the post resolves with %s persisted mode", async (runtimeMode) => { const postedComment = createDeferred(); const activeRun = { id: "run-native", - runtimeMode: "native" as const, + runtimeMode, status: "running", invocationSource: "issue", triggerDetail: null, @@ -3961,6 +3963,7 @@ describe("IssueDetail", () => { state: null, targetRunId: null, entries: [], + protocol: "legacy", steeringDisposition: "temporarily_unavailable", }), ); @@ -5875,11 +5878,11 @@ describe("IssueDetail", () => { }, ); - it("keeps the last queued message mounted until steering is acknowledged so rejection stays visible", async () => { + it.each([true, false])("optimistically promotes the last steering message and restores it without a toast on rejection (history loaded: %s)", async historyLoaded => { const queue = createQueuedCommentQueue(); mockIssuesApi.get.mockResolvedValue(createIssue({ status: "in_progress", assigneeAgentId: "agent-1", executionRunId: "run-active-1" })); mockAgentsApi.list.mockResolvedValue([createAgent({ adapterType: "paperclip_runner" })]); - mockIssuesApi.listComments.mockResolvedValue([queue.entries[0].comment]); + mockIssuesApi.listComments.mockResolvedValue(historyLoaded ? [queue.entries[0].comment] : []); mockIssuesApi.getQueuedComments.mockResolvedValue(queue); mockHeartbeatsApi.activeRunForIssue.mockResolvedValue({ id: "run-active-1", runtimeMode: "native", status: "running", invocationSource: "issue", @@ -5900,13 +5903,18 @@ describe("IssueDetail", () => { }); let pending!: Promise; await act(async () => { pending = props.onSteerQueuedComment("queued-comment-1", queue.revision).catch(error => error); }); - // The child owns its pending/error state. Unmounting it here loses any later error. const whilePending = mockIssueChatThreadRender.mock.calls.at(-1)?.[0] as Props; - expect(whilePending.queuedCommentQueue?.entries).toHaveLength(1); + expect(whilePending.queuedCommentQueue).toBeNull(); + expect(mockIssueChatThreadRender.mock.calls.at(-1)?.[0]?.comments).toContainEqual(expect.objectContaining({ + id: "queued-comment-1", steeredIntoRunId: "run-active-1", queueState: undefined, + })); const failure = new ApiError("This runner does not support steering", 409, { code: "steering_unsupported" }); await act(async () => { rejectSteer(failure); await pending; }); expect(await pending).toBe(failure); - expect((mockIssueChatThreadRender.mock.calls.at(-1)?.[0] as Props).queuedCommentQueue?.entries).toHaveLength(1); + await waitForAssertion(() => { + expect((mockIssueChatThreadRender.mock.calls.at(-1)?.[0] as Props).queuedCommentQueue?.entries).toHaveLength(1); + }); + expect(mockPushToast).not.toHaveBeenCalled(); }); it("promotes a steered message immediately while its durable timeline position refreshes", async () => { @@ -5975,14 +5983,16 @@ describe("IssueDetail", () => { mockActivityApi.forIssue.mockReturnValue(activityRefresh); let steeringPromise!: Promise; + let deliverySettled = false; await act(async () => { - steeringPromise = steer("queued-comment-1", queue.revision); + steeringPromise = steer("queued-comment-1", queue.revision).then(() => { deliverySettled = true; }); await Promise.resolve(); }); await waitForAssertion(() => { expect(mockIssuesApi.steerQueuedComment).toHaveBeenCalled(); expect(mockActivityApi.forIssue.mock.calls.length).toBeGreaterThan(1); expect(mockIssueChatThreadRender.mock.calls.at(-1)?.[0]?.queuedCommentQueue).toBeNull(); + expect(deliverySettled).toBe(true); }); const whileRefreshing = mockIssueChatThreadRender.mock.calls.at( @@ -6045,6 +6055,82 @@ describe("IssueDetail", () => { }); }); + it.each([ + ["success", true], ["failure", true], ["success", false], ["failure", false], + ] as const)("optimistically promotes the interrupt queue with no toast on %s (history loaded: %s)", async (outcome, historyLoaded) => { + const queue = createQueuedCommentQueue({ protocol: "legacy", steeringDisposition: "unsupported" }); + queue.entries.push({ ...queue.entries[0], position: 1, comment: createIssueComment({ id: "queued-comment-2", body: "Second direction" }) }); + mockIssuesApi.get.mockResolvedValue(createIssue({ status: "in_progress", assigneeAgentId: "agent-1", executionRunId: "run-active-1" })); + mockAgentsApi.list.mockResolvedValue([createAgent({ adapterType: "codex_local" })]); + mockIssuesApi.listComments.mockResolvedValue(historyLoaded ? queue.entries.map(entry => entry.comment) : []); + mockIssuesApi.getQueuedComments.mockResolvedValue(queue); + mockHeartbeatsApi.activeRunForIssue.mockResolvedValue({ + id: "run-active-1", runtimeMode: "legacy", status: "running", invocationSource: "issue", + startedAt: "2026-04-21T00:00:00.000Z", finishedAt: null, createdAt: "2026-04-21T00:00:00.000Z", + agentId: "agent-1", agentName: "Coder", adapterType: "codex_local", issueId: "issue-1", + }); + const delivery = createDeferred(); + mockIssuesApi.interruptQueuedComments.mockReturnValue(delivery.promise); + await act(async () => { root.render(); }); + type Props = { onInterruptQueued: (runId: string | null) => Promise; queuedCommentQueue: IssueQueuedCommentQueue | null; comments: Array<{ id: string; queueState?: string; conversationAnchorAt?: string }> }; + let props!: Props; + await waitForAssertion(() => { + props = mockIssueChatThreadRender.mock.calls.at(-1)?.[0] as Props; + expect(props.queuedCommentQueue?.entries).toHaveLength(2); + }); + let pending!: Promise; + await act(async () => { pending = props.onInterruptQueued("run-active-1").catch(error => error); }); + const whilePending = mockIssueChatThreadRender.mock.calls.at(-1)?.[0] as Props; + expect(whilePending.queuedCommentQueue).toBeNull(); + for (const entry of queue.entries) expect(whilePending.comments).toContainEqual(expect.objectContaining({ + id: entry.comment.id, queueState: undefined, conversationAnchorAt: expect.any(String), + })); + expect(mockPushToast).not.toHaveBeenCalled(); + await act(async () => { + if (outcome === "failure") delivery.reject(new Error("Connection lost")); + else delivery.resolve(createQueuedCommentQueue({ entries: [] })); + await pending; + }); + await waitForAssertion(() => { + const final = mockIssueChatThreadRender.mock.calls.at(-1)?.[0] as Props; + if (outcome === "failure") expect(final.queuedCommentQueue?.entries).toHaveLength(2); + else expect(final.queuedCommentQueue).toBeNull(); + }); + expect(mockPushToast).not.toHaveBeenCalled(); + }); + + it("restores a failed classic interrupt inline without a toast or an unhandled rejection", async () => { + mockInstanceSettingsApi.getExperimental.mockResolvedValue({ enableStreamlinedUi: true, enableClassicTaskInterface: true }); + mockIssuesApi.get.mockResolvedValue(createIssue({ status: "in_progress", assigneeAgentId: "agent-1", executionRunId: "run-active-1" })); + mockAgentsApi.list.mockResolvedValue([createAgent({ adapterType: "codex_local" })]); + mockIssuesApi.listComments.mockResolvedValue([createIssueComment({ id: "queued-comment-1", body: "Next direction", createdAt: new Date("2026-04-21T00:00:05.000Z") })]); + mockHeartbeatsApi.activeRunForIssue.mockResolvedValue({ + id: "run-active-1", runtimeMode: "legacy", status: "running", invocationSource: "issue", + startedAt: "2026-04-21T00:00:00.000Z", finishedAt: null, createdAt: "2026-04-21T00:00:00.000Z", + agentId: "agent-1", agentName: "Coder", adapterType: "codex_local", issueId: "issue-1", + }); + mockIssuesApi.getQueuedComments.mockResolvedValue(createQueuedCommentQueue({ protocol: "legacy", steeringDisposition: "unsupported" })); + const delivery = createDeferred(); + mockIssuesApi.interruptQueuedComments.mockReturnValue(delivery.promise); + await act(async () => { root.render(); }); + type Props = { onInterruptQueued: (runId: string | null) => Promise; comments: Array<{ id: string; queueState?: string }> }; + let props!: Props; + await waitForAssertion(() => { + props = mockIssueChatThreadRender.mock.calls.at(-1)?.[0] as Props; + expect(props.comments).toContainEqual(expect.objectContaining({ id: "queued-comment-1", queueState: "queued" })); + }); + let pending!: Promise; + await act(async () => { pending = props.onInterruptQueued("run-active-1").catch(error => error); }); + expect(mockIssueChatThreadRender.mock.calls.at(-1)?.[0]?.comments).toContainEqual(expect.objectContaining({ id: "queued-comment-1", queueState: undefined })); + await act(async () => { delivery.reject(new Error("Connection lost")); await pending; }); + expect(await pending).toBeUndefined(); + await waitForAssertion(() => { + expect(container.textContent).toContain("Couldn’t interrupt. Message is still queued."); + expect(mockIssueChatThreadRender.mock.calls.at(-1)?.[0]?.comments).toContainEqual(expect.objectContaining({ id: "queued-comment-1", queueState: "queued" })); + }); + expect(mockPushToast).not.toHaveBeenCalled(); + }); + it("keeps an unacknowledged queue visible while withholding server controls", async () => { mockIssuesApi.get.mockResolvedValue( createIssue({ diff --git a/ui/src/pages/IssueDetail.tsx b/ui/src/pages/IssueDetail.tsx index dff1a70b223..87fb4dc718f 100644 --- a/ui/src/pages/IssueDetail.tsx +++ b/ui/src/pages/IssueDetail.tsx @@ -1536,15 +1536,30 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ const [localSteeringPlacements, setLocalSteeringPlacements] = useState< ReadonlyMap< string, - { targetRunId: string; anchorAt: string; sequence: number } + { + targetRunId: string; + anchorAt: string; + sequence: number; + comment?: IssueDetailComment; + } > >(() => new Map()); + const [localInterruptPlacements, setLocalInterruptPlacements] = useState< + ReadonlyMap< + string, + { anchorAt: string; sequence: number; comment?: IssueDetailComment } + > + >(() => new Map()); + const [classicQueuedDeliveryError, setClassicQueuedDeliveryError] = + useState(null); useEffect(() => { setConsumedQueuedCommentIds(new Set()); setDiscardedQueuedCommentIds(new Set()); }, [issueId]); useEffect(() => { setLocalSteeringPlacements(new Map()); + setLocalInterruptPlacements(new Map()); + setClassicQueuedDeliveryError(null); }, [issueId]); const hasLiveRuns = liveRunCount > 0 || !!resolvedActiveRun; const { @@ -1757,7 +1772,20 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ } } - const projectedComments = comments.map((comment) => { + // The queue can contain a saved message outside the loaded history page. + // Keep its optimistic bubble until the canonical comment is available. + const commentsToProject = [...comments]; + const loadedCommentIds = new Set(comments.map((comment) => comment.id)); + for (const placement of [ + ...localSteeringPlacements.values(), + ...localInterruptPlacements.values(), + ]) { + if (placement.comment && !loadedCommentIds.has(placement.comment.id)) { + commentsToProject.push(placement.comment); + loadedCommentIds.add(placement.comment.id); + } + } + const projectedComments = commentsToProject.map((comment) => { const activityMeta = runMetaByCommentId.get(comment.id); // Internal run finalization can persist a reply without a separate // comment_added activity row. Its durable authoring run is stronger @@ -1985,6 +2013,7 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ interactions, liveRunIds, localSteeringPlacements, + localInterruptPlacements, locallyQueuedCommentRunIds, queuedCommentReason, resolvedActivity, @@ -2021,8 +2050,8 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ return targetRunId ? [{ comment, targetRunId }] : []; }); const fallbackProtocol = - liveRuntimeRun?.runtimeMode === "native" && - liveRuntimeRun.adapterType === "paperclip_runner" + liveRuntimeRun?.adapterType === "paperclip_runner" || + (!liveRuntimeRun && assigneeUsesPaperclipRunner) ? "paperclip_runner_v1" : "legacy"; return mergePendingIssueQueuedComments({ @@ -2038,6 +2067,7 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ discardedQueuedCommentIds, issueId, liveRuntimeRun, + assigneeUsesPaperclipRunner, locallyQueuedCommentRunIds, queuedCommentQueueEnabled, ]); @@ -2049,6 +2079,12 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ return [ { ...comment, + ...(!comment.consumedByRunId && localInterruptPlacements.has(comment.id) + ? { + conversationAnchorAt: localInterruptPlacements.get(comment.id)!.anchorAt, + conversationAnchorSequence: localInterruptPlacements.get(comment.id)!.sequence, + } + : {}), clientStatus: undefined, queueState: undefined, queueTargetRunId: null, @@ -2056,7 +2092,12 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ }, ]; }), - [commentsWithRunMeta, consumedQueuedCommentIds, discardedQueuedCommentIds], + [ + commentsWithRunMeta, + consumedQueuedCommentIds, + discardedQueuedCommentIds, + localInterruptPlacements, + ], ); const storeQueuedCommentQueue = useCallback( @@ -2146,6 +2187,23 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ throw new Error( "The queued message no longer has an active run target.", ); + const anchorAt = new Date().toISOString(); + setLocalSteeringPlacements((current) => { + const next = new Map(current); + const sequence = [...current.values()].filter( + (placement) => placement.targetRunId === targetRunId, + ).length; + next.set(commentId, { + targetRunId, + anchorAt, + sequence, + comment: effectiveQueuedCommentQueue?.entries.find( + (entry) => entry.comment.id === commentId, + )?.comment, + }); + return next; + }); + setConsumedQueuedCommentIds((current) => new Set(current).add(commentId)); try { const nextQueue = await issuesApi.steerQueuedComment( issueId, @@ -2156,23 +2214,10 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ revision, }, ); - // Keep the queue component mounted until the server accepts steering: - // its pending/error state must survive a rejected last-row action. - const anchorAt = new Date().toISOString(); - setLocalSteeringPlacements((current) => { - const next = new Map(current); - const sequence = [...current.values()].filter( - (placement) => placement.targetRunId === targetRunId, - ).length; - next.set(commentId, { targetRunId, anchorAt, sequence }); - return next; - }); - setConsumedQueuedCommentIds((current) => new Set(current).add(commentId)); - // The local steering placement already promoted the message into the - // active turn. Refresh its durable acknowledgement before publishing - // the returned queue so the local and server anchors hand off without a - // bubble-to-queue-to-bubble jump. - await Promise.all([ + // The local placement bridges the durable receipt refresh. Release the + // control as soon as delivery is acknowledged, even on a slow refetch. + storeQueuedCommentQueue(nextQueue); + void Promise.all([ queryClient.invalidateQueries({ queryKey: queryKeys.issues.comments(issueId), }), @@ -2180,7 +2225,6 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ queryKey: queryKeys.issues.activity(issueQueryRef), }), ]); - storeQueuedCommentQueue(nextQueue); } catch (error) { setConsumedQueuedCommentIds((current) => { const next = new Set(current); @@ -2196,8 +2240,7 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ } }, [ - effectiveQueuedCommentQueue?.queueId, - effectiveQueuedCommentQueue?.targetRunId, + effectiveQueuedCommentQueue, issueId, issueQueryRef, queryClient, @@ -2206,6 +2249,53 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ ], ); + const interruptQueuedComments = useCallback( + async (runId: string | null) => { + const queuedComments = effectiveQueuedCommentQueue?.targetRunId === runId + ? effectiveQueuedCommentQueue.entries.map((entry) => entry.comment) + : commentsWithRunMeta.filter( + (comment) => comment.queueState === "queued" && comment.queueTargetRunId === runId, + ); + const ids = queuedComments.map((comment) => comment.id); + const anchorAt = new Date().toISOString(); + setClassicQueuedDeliveryError(null); + setConsumedQueuedCommentIds((current) => new Set([...current, ...ids])); + setLocalInterruptPlacements((current) => { + const next = new Map(current); + queuedComments.forEach((comment, sequence) => { + next.set(comment.id, { anchorAt, sequence, comment }); + }); + return next; + }); + try { + await onInterruptQueued(runId); + } catch (error) { + setConsumedQueuedCommentIds((current) => { + const next = new Set(current); + for (const id of ids) next.delete(id); + return next; + }); + setLocalInterruptPlacements((current) => { + const next = new Map(current); + for (const id of ids) next.delete(id); + return next; + }); + if (classicTaskInterfaceEnabled) { + setClassicQueuedDeliveryError("Couldn’t interrupt. Message is still queued."); + return; + } + await refreshQueueAfterConflict(error); + } + }, + [ + effectiveQueuedCommentQueue, + commentsWithRunMeta, + onInterruptQueued, + refreshQueueAfterConflict, + classicTaskInterfaceEnabled, + ], + ); + const discardQueuedComment = useCallback( async (commentId: string, revision: string) => { const queueId = effectiveQueuedCommentQueue?.queueId; @@ -2445,7 +2535,7 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ onReviewConversation={onReviewConversation} imageUploadHandler={onImageUpload} onAttachImage={onAttachImage} - onInterruptQueued={onInterruptQueued} + onInterruptQueued={interruptQueuedComments} queuedCommentQueue={effectiveQueuedCommentQueue} onEditQueuedComment={editQueuedComment} onReorderQueuedComments={reorderQueuedComments} @@ -2495,7 +2585,14 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ tryAgainNoLiveExecutionPathPending={ tryAgainNoLiveExecutionPathPending } - footer={footer} + footer={classicQueuedDeliveryError ? ( + <> +

+ {classicQueuedDeliveryError} +

+ {footer} + + ) : footer} externalReferences={externalReferences} linkCaseReferences={linkCaseReferences} /> @@ -5110,23 +5207,10 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS onSuccess: () => { invalidateIssueDetail(); invalidateIssueRunState(); - pushToast({ - title: "Interrupt requested", - body: "Queued messages will be sent when the previous run has stopped.", - tone: "success", - }); }, - onError: (err) => { + onError: () => { invalidateIssueDetail(); invalidateIssueRunState(); - pushToast({ - title: "Interrupt failed", - body: - err instanceof Error - ? err.message - : "Unable to interrupt the active run", - tone: "error", - }); }, }); From 94f6f3eb4725b702f850fb7b98a99fab2b1b7a04 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 2 Oct 2026 23:08:05 -0500 Subject: [PATCH 02/37] fix: recover historical interrupted native preparation (#15020) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task conversation must accept new user input after an interrupted startup. > - Native startup begins with a legacy preparation row before runtime selection. > - Older builds did not retain the startup cancellation receipt on that row. > - The immutable adapter claim and expired controller can still prove that no provider started. > - This pull request uses that narrow proof for explicit Retry and saved user input. > - The benefit is a conversation that recovers after an upgrade without repeating old work. ## Linked Issues or Issue Description Refs #15015. Related #13293 covers retained process evidence after provider startup. This change covers interrupted preparation before native runtime selection. **What happened?** After an upgrade, a task stopped during native preparation can still show automatic recovery stopped. A new user message saves but does not start. Retry is absent because the historical run has no cancellation receipt. **Expected behavior** Offer Retry and admit new user input when immutable run evidence proves that no provider started and cleanup is complete. Preserve incomplete or contradictory evidence as a recovery hold. **Steps to reproduce** 1. Retain a cancelled run with the Paperclip Runner adapter claim, an unresolved runtime, and the preparing stage. 2. Keep its old controller boot ID and expired lease. Retain no native identity, coordinator, result, process identity, or provider events. 3. Upgrade from a build that did not save the startup cancellation receipt. 4. Send a new user message or select Retry. Confirm that one fresh turn starts. 5. Repeat with an active controller, a provider launch, or unfinished cleanup. Confirm that execution stays held. **Paperclip version or commit** Reproduced on `cc67d4e1d` with a historical interrupted preparation row. **Deployment mode** Authenticated private self-hosted server, built from source. ## What Changed - Recognize historical interrupted native preparation from immutable run evidence and an expired controller from another server boot. - Reject adapter invocation evidence in the startup proof. Keep process and environment cleanup checks. - Apply the same proof to saved user messages in the bounded recovery worker. - Recheck saved-message eligibility under the existing task and run locks. Keep normal ownership, decision, pause, and budget gates. - Add regression tests for Retry, a new message, concurrent saved-message recovery, and contradictory evidence. Document the compatibility rule. ## Verification - Red: Retry, new-message recovery, and saved-message recovery fail on the parent commit. - Green: 185 continuation tests, 335 process-recovery tests, and 78 queued-comment route tests pass. Two additional red regressions cover partially delivered saved queues and pass after the admission fix. - Workspace `pnpm -r typecheck` and `pnpm build` pass. - Final head `48db53f4f`: all 54 CI checks pass; two optional Storybook checks skip. Greptile is 5/5 with zero unresolved threads. - The local monolithic `pnpm test:run` was stopped after CI passed. One unrelated workspace case failed in that long run; all three workspace reconciliation cases pass in isolation. No complete local monolithic pass is claimed. - After merge, deploy the exact merged commit and verify recovery through the normal task composer. ## Risks Historical compatibility could grant a new turn without enough startup evidence. The proof requires an immutable native adapter claim, an unresolved preparing stage, no result or native identity, an expired controller from another server boot, and no invocation or provider evidence. Environment cleanup remains mandatory. The fix does not replay old input or change historical run results. No schema or dependency change. ## Model Used OpenAI Codex, an agent based on GPT-6. The exact runtime model ID and context window are not exposed in this session. Used reasoning, repository tools, code execution, and browser inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/execution-semantics.md | 7 ++ .../src/services/cancelled-native-startup.ts | 16 ++- .../explicit-native-continuation.test.ts | 98 +++++++++++++++++++ .../services/explicit-native-continuation.ts | 11 ++- server/src/services/heartbeat.ts | 16 ++- 5 files changed, 135 insertions(+), 13 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index d1130d31819..326da73f0bd 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1156,6 +1156,13 @@ controller, lease, or result). It also checks for contradictory launch/process evidence and verifies local cleanup or exact remote termination receipts. The preparer must have finished or its startup lease must have expired. A missing PID alone does not establish this proof. +For older interrupted preparation rows without a cancellation receipt, the +immutable Paperclip Runner adapter claim, unresolved runtime, and preparing stage +must agree. The old controller must belong to another server boot and its lease +must have expired. No native identity, coordinator, result, adapter invocation, +provider event, or process-launch evidence may exist. Environment cleanup still +requires the same receipts. This historical proof permits explicit Retry or a +newer saved user message; it does not replay the cancelled input. The existing bounded saved-message worker rechecks this proof after restart. Admission atomically settles an unclaimed coordinator and admits one fresh turn, diff --git a/server/src/services/cancelled-native-startup.ts b/server/src/services/cancelled-native-startup.ts index db3f082a88a..ef7fb126d5f 100644 --- a/server/src/services/cancelled-native-startup.ts +++ b/server/src/services/cancelled-native-startup.ts @@ -4,6 +4,7 @@ import { claimedAdapterType } from "./conversation-continuation.js"; import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-local-process-stop.js"; import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js"; import { canContinueCancelledRun } from "./run-cancellation.js"; +import { legacyControllerBootId } from "./legacy-controller-lease.js"; type Run = typeof heartbeatRuns.$inferSelect; type Coordinator = typeof nativeRunFinalizations.$inferSelect; @@ -27,7 +28,16 @@ export async function isCancelledNativeStartup(db: Db, run: Run, coordinator: Co if (run.status !== "cancelled" || !run.finishedAt || run.processPid || run.processGroupId || run.processStartedAt || run.sessionIdAfter) return false; const cancellation = run.resultJson?.startupCancellation as Record | undefined; - const beforeSelection = run.runtimeMode === "legacy" && !run.runtimeModeResolvedAt && + // Older builds did not retain the cancellation fence. Their immutable + // native-adapter claim and unresolved preparation stage still prove that + // provider dispatch did not begin. Require an expired owner from another + // server boot; neither a missing PID nor mutable agent settings is proof. + const historicalBeforeSelection = run.runtimeMode === "legacy" && !run.runtimeModeResolvedAt && + run.executionStage === "preparing" && !run.nativeIssueId && !run.nativeSessionId && !coordinator && + claimedAdapterType(run) === "paperclip_runner" && run.errorCode === "operator_interrupted" && + run.resultJson === null && Boolean(run.controllerBootId && run.controllerBootId !== legacyControllerBootId && + run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt <= new Date()); + const beforeSelection = historicalBeforeSelection || run.runtimeMode === "legacy" && !run.runtimeModeResolvedAt && !run.nativeSessionId && !coordinator && claimedAdapterType(run) === "paperclip_runner" && cancellation?.beforeNativeSelection === true; const neverClaimed = run.runtimeMode === "native" && coordinator && @@ -42,7 +52,7 @@ export async function isCancelledNativeStartup(db: Db, run: Run, coordinator: Co const leases = await db.select().from(environmentLeases).where(and( eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id), )); - if ((!settled && leases.length === 0) || leases.some(lease => + if ((!settled && leases.length === 0 && !historicalBeforeSelection) || leases.some(lease => lease.provider === "local" ? !lease.releasedAt || lease.status === "pending_cleanup" || lease.cleanupStatus === "failed" : !hasRemoteTerminationReceipt(lease))) return false; @@ -51,7 +61,7 @@ export async function isCancelledNativeStartup(db: Db, run: Run, coordinator: Co const [execution] = await db.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents).where(and( eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id), or(isNotNull(heartbeatRunEvents.sourceEventId), - inArray(heartbeatRunEvents.eventType, [PROCESS_START_REQUESTED, PROCESS_IDENTITY_RECORDED, + inArray(heartbeatRunEvents.eventType, ["adapter.invoke", PROCESS_START_REQUESTED, PROCESS_IDENTITY_RECORDED, "harness.ready", "session.started", "session.resumed", "session.updated", "turn.started", "provider.event", "provider.rpc_result", "tool.execution.started"])), )).limit(1); diff --git a/server/src/services/explicit-native-continuation.test.ts b/server/src/services/explicit-native-continuation.test.ts index b535771c0f0..31b0cb0f4a8 100644 --- a/server/src/services/explicit-native-continuation.test.ts +++ b/server/src/services/explicit-native-continuation.test.ts @@ -1,3 +1,4 @@ +import { legacyControllerBootId } from "./legacy-controller-lease.js"; import { instanceSettingsService } from "./instance-settings.js"; import { createPostgresWakeQueueAdapter } from "../modules/wake-queue/adapters/postgres.js"; import { createReleaseIssueExecution } from "../modules/wake-queue/application/use-cases.js"; @@ -597,6 +598,103 @@ const support = await getEmbeddedPostgresTestSupport(); expect(await admit(f)).toMatchObject({ previousRunId: f.sourceRunId }); }); + async function seedHistoricalCancelledPreparation() { + const f = await seedCancelledStartup(); + await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId)); + await db.delete(environmentLeases).where(eq(environmentLeases.heartbeatRunId, f.sourceRunId)); + await db.update(heartbeatRuns).set({ runtimeMode: "legacy", runtimeModeResolvedAt: null, + nativeIssueId: null, nativeSessionId: null, executionStage: "preparing", errorCode: "operator_interrupted", + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } }, resultJson: null, + }).where(eq(heartbeatRuns.id, f.sourceRunId)); + await db.update(issueRecoveryActions).set({ cause: "legacy_execution_requires_reconciliation" }) + .where(eq(issueRecoveryActions.sourceIssueId, f.issueId)); + return f; + } + + it.each(["message", "retry"])("recovers historical native preparation through an explicit %s", async kind => { + const f = await seedHistoricalCancelledPreparation(); + expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ canRetry: true }); + await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }); + const successor = await heartbeatService(db).wakeup(f.agentId, { + source: "on_demand", triggerDetail: "manual", reason: kind === "retry" ? "retry_failed_run" : "issue_commented", + ...(kind === "retry" ? { failedRunId: f.sourceRunId } : {}), requestedByActorType: "user", requestedByActorId: "board", + payload: { issueId: f.issueId, ...(kind === "message" ? { commentId: f.commentId } : {}) }, + contextSnapshot: { issueId: f.issueId, ...(kind === "message" ? { wakeCommentId: f.commentId } : {}) }, + }); + expect(successor).toMatchObject({ status: "queued", contextSnapshot: { forceFreshSession: true, + previousRunId: f.sourceRunId, explicitUserContinuation: { commentId: kind === "message" ? f.commentId : null } } }); + expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull(); + const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId)); + expect(source).toMatchObject({ status: "cancelled", runtimeMode: "legacy", resultJson: null }); + }); + + it.each(["fresh", "earlier_delivered", "last_delivered"])("resumes saved input after historical native preparation expires across restart exactly once (%s)", async kind => { + const f = await seedHistoricalCancelledPreparation(); + await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }); + await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60_000) }) + .where(eq(heartbeatRuns.id, f.sourceRunId)); + await heartbeatService(db).wakeup(f.agentId, { source: "automation", reason: "issue_commented", + requestedByActorType: "user", requestedByActorId: "board", payload: { issueId: f.issueId, commentId: f.commentId }, + contextSnapshot: { issueId: f.issueId, wakeCommentId: f.commentId } }); + const [waiting] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId)); + expect(waiting.status).toBe("deferred_issue_execution"); + if (kind !== "fresh") { + const deliveredId = randomUUID(); + await db.insert(issueComments).values({ id: deliveredId, companyId: f.companyId, issueId: f.issueId, + authorType: "user", authorUserId: "another-author", body: "Already handled", + createdAt: new Date(kind === "earlier_delivered" ? "2026-09-11T10:30:00Z" : "2026-09-11T11:30:00Z") }); + await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "succeeded", + startedAt: new Date("2026-09-11T12:00:00Z"), finishedAt: new Date("2026-09-11T12:01:00Z"), + contextSnapshot: { issueId: f.issueId, wakeCommentIds: [deliveredId] } }); + await db.update(agentWakeupRequests).set({ payload: { ...waiting.payload, + _paperclipWakeContext: { wakeCommentIds: kind === "earlier_delivered" ? [deliveredId, f.commentId] : [f.commentId, deliveredId] } }, + }).where(eq(agentWakeupRequests.id, waiting.id)); + } + await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(0) }) + .where(eq(heartbeatRuns.id, f.sourceRunId)); + await db.update(agentWakeupRequests).set({ updatedAt: new Date(0) }).where(eq(agentWakeupRequests.id, waiting.id)); + await Promise.all([heartbeatService(db).resumeExecutionWaitComments(), heartbeatService(db).resumeExecutionWaitComments()]); + const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued"))); + expect(successors).toHaveLength(1); + expect(successors[0].contextSnapshot).toMatchObject({ wakeCommentIds: [f.commentId], + explicitUserContinuation: { commentId: f.commentId }, forceFreshSession: true }); + const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, waiting.id)); + expect(receipt).toMatchObject({ status: "coalesced", runId: successors[0].id }); + expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull(); + }); + + it.each(["resolved", "dispatching", "adapter", "missing_boot", "current_boot", "missing_lease", "live_lease", "receipt", "launch", "invoked", "coordinator", "cleanup", "remote"])( + "holds historical native preparation with contradictory or incomplete %s evidence", async kind => { + const f = await seedHistoricalCancelledPreparation(); + const patch = kind === "resolved" ? { runtimeModeResolvedAt: new Date() } + : kind === "dispatching" ? { executionStage: "dispatching" } + : kind === "adapter" ? { runnerProfileJson: { adapterDispatch: { adapterType: "process" } } } + : kind === "missing_boot" ? { controllerBootId: null } + : kind === "current_boot" ? { controllerBootId: legacyControllerBootId } + : kind === "missing_lease" ? { controllerLeaseExpiresAt: null } + : kind === "live_lease" ? { controllerLeaseExpiresAt: new Date(Date.now() + 60_000) } + : kind === "receipt" ? { resultJson: { startupCancellation: { beforeNativeSelection: false } } } : {}; + if (Object.keys(patch).length) await db.update(heartbeatRuns).set(patch).where(eq(heartbeatRuns.id, f.sourceRunId)); + if (kind === "launch" || kind === "invoked") await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, + agentId: f.agentId, runId: f.sourceRunId, seq: 1, + eventType: kind === "launch" ? PROCESS_START_REQUESTED : "adapter.invoke", payload: { adapterType: "paperclip_runner" } }); + if (kind === "coordinator") { + await db.update(heartbeatRuns).set({ nativeIssueId: f.issueId }).where(eq(heartbeatRuns.id, f.sourceRunId)); + await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, + runId: f.sourceRunId, issueId: f.issueId, phase: "observed", attempt: 0 }); + } + if (kind === "cleanup" || kind === "remote") await db.insert(environmentLeases).values({ companyId: f.companyId, + heartbeatRunId: f.sourceRunId, provider: kind === "remote" ? "daytona" : "local", + providerLeaseId: kind === "remote" ? "unverified" : null, leasePolicy: "ephemeral", + status: "pending_cleanup", cleanupStatus: "failed" }); + expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ canRetry: false }); + expect(await admit(f)).toBeNull(); + // Later tests exercise the global cleanup sweep with their own retry + // counters. Do not leave this negative fixture as another cleanup target. + await db.delete(environmentLeases).where(eq(environmentLeases.heartbeatRunId, f.sourceRunId)); + }, + ); + it.each(["attempt", "generation", "controller", "lease", "process", "launch", "provider", "cleanup", "remote", "preparing", "closed", "reassigned"])( "retains cancellation safeguards with %s evidence", async kind => { const f = await seedCancelledStartup(); diff --git a/server/src/services/explicit-native-continuation.ts b/server/src/services/explicit-native-continuation.ts index 4dd475b43ab..bf1f66530fa 100644 --- a/server/src/services/explicit-native-continuation.ts +++ b/server/src/services/explicit-native-continuation.ts @@ -169,14 +169,9 @@ export async function admitExplicitNativeContinuation(input: { const unusedAdmission = run.status === "cancelled" && !run.startedAt && run.errorCode === "execution_reconciliation_required" && !run.processPid && !run.processGroupId && !run.nativeSessionId; - if (partiallyDeliveredQueue && run.runtimeMode !== "native" && !unusedAdmission) return null; const legacyUserTurn = run.runtimeMode === "legacy" && action.cause === "legacy_execution_requires_reconciliation" && isConversationAdapter(agent.adapterType); - if ((queuedInterrupt || queuedRequest) && !legacyUserTurn && !unusedAdmission && - !(queuedRequest && run.runtimeMode === "native" && - (run.status !== "cancelled" || authorizedAt > run.finishedAt)) && - !(queuedInterrupt && response?.source.requiresFreshSession && run.runtimeMode === "native")) return null; // Saved input is a request for a new turn, never permission to undo an // operator Stop or redeliver a message already consumed by this run. if (queuedRequest && !queuedInterrupt && (run.contextSnapshot?.wakeCommentId === commentId || @@ -205,6 +200,12 @@ export async function admitExplicitNativeContinuation(input: { return blocked("workspace_repair_required", "Verify safe workspace staging or repair before continuing. Your message is saved."); } const cancelledStartup = await isCancelledNativeStartup(db, run, coordinator); + if (partiallyDeliveredQueue && run.runtimeMode !== "native" && !unusedAdmission && !cancelledStartup) return null; + if ((queuedInterrupt || queuedRequest) && !legacyUserTurn && !unusedAdmission && + !(queuedRequest && run.runtimeMode === "native" && + (run.status !== "cancelled" || authorizedAt > run.finishedAt!)) && + !(queuedRequest && cancelledStartup && authorizedAt > run.finishedAt!) && + !(queuedInterrupt && response?.source.requiresFreshSession && run.runtimeMode === "native")) return null; if (queuedRequest && !queuedInterrupt && run.status === "cancelled" && !unusedAdmission && !canContinueCancelledRun(run) && !(cancelledStartup && authorizedAt > run.finishedAt!)) return null; if (retry && run.status === "cancelled" && !canContinueCancelledRun(run) && !cancelledStartup) diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 02162d05c83..88f73ea4c6d 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -33,7 +33,7 @@ import { completeTerminatedRemoteNativeSessionCleanup } from "../vendor/papercli import { hasRemoteTerminationReceipt, remoteExecutionHasStopped, remoteTerminationReceipt, stoppedRemoteCleanupScopes } from "./remote-execution-termination.js"; import { applyConnectorSkills, prepareConnectorSkillDelivery, resolveConnectorAssignments } from "./connector-runtime.js"; import { admitExplicitNativeContinuation, undeliveredLegacyUserCommentIds } from "./explicit-native-continuation.js"; -import { canRetryStoppedRun } from "./cancelled-native-startup.js"; +import { canRetryStoppedRun, isCancelledNativeStartup } from "./cancelled-native-startup.js"; import { connectionIntentService } from "./connection-intents.js"; import { managedAiSessionFingerprintConfig, prepareManagedAiRuntime, assertManagedAiProjectAuth, stripAiAuthBindings, isAiConnectionBusy, AI_AUTH_ENV_KEYS } from "./ai-connection-runtime.js"; import { aiConnectionBindingSchema } from "@paperclipai/shared"; @@ -10373,7 +10373,13 @@ export function heartbeatService( async function resumeRemoteStopComments(run: typeof heartbeatRuns.$inferSelect, requestId?: string) { if (!isHeartbeatRunTerminalStatus(run.status) || adapterExecutionControls.has(run.id)) return; - if (run.runtimeMode !== "native" && + const [preparationCoordinator] = run.runtimeMode === "legacy" && run.status === "cancelled" && !run.runtimeModeResolvedAt + ? await db.select().from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.companyId, run.companyId), eq(nativeRunFinalizations.runId, run.id), + )) : []; + const cancelledPreparation = run.runtimeMode === "legacy" && + await isCancelledNativeStartup(db, run, preparationCoordinator); + if (run.runtimeMode !== "native" && !cancelledPreparation && parseObject(run.resultJson?.startupCancellation).beforeNativeSelection !== true && !(await remoteExecutionHasStopped(db, run.companyId, run.id))) return; const issueId = run.nativeIssueId ?? (typeof run.contextSnapshot?.issueId === "string" ? run.contextSnapshot.issueId : null); @@ -10414,7 +10420,7 @@ export function heartbeatService( let commentId = deriveCommentId(context, payload); let requestedByActorId = wake.requestedByActorId; const reason = readNonEmptyString(context.wakeReason) ?? wake.reason; - if (run.runtimeMode === "native") { + if (run.runtimeMode === "native" || cancelledPreparation) { const ids = await undeliveredLegacyUserCommentIds(db, run.companyId, issueId, run.agentId, queuedCommentIdsFromWakePayload(payload)); if (!ids.length) continue; @@ -10444,7 +10450,7 @@ export function heartbeatService( const admitted = await admitExplicitNativeContinuation({ db, companyId: run.companyId, issueId, agentId: run.agentId, actorType: wake.requestedByActorType, actorId: requestedByActorId, reason, commentId, successorRunId: randomUUID(), dryRun: true, - ...(run.runtimeMode === "native" ? { queuedCommentRequestId: wake.id } : {}), + ...((run.runtimeMode === "native" || cancelledPreparation) ? { queuedCommentRequestId: wake.id } : {}), onBlocked: (reason, message) => { wait = { reason, message }; }, }); if (!admitted) { @@ -10462,7 +10468,7 @@ export function heartbeatService( await enqueueWakeup(run.agentId, { source: wake.source as WakeupOptions["source"], triggerDetail: (wake.triggerDetail ?? undefined) as WakeupOptions["triggerDetail"], reason, payload, contextSnapshot: context, requestedByActorType: "user", requestedByActorId, - ...(run.runtimeMode === "native" ? { queuedCommentRequestId: wake.id } : {}), + ...((run.runtimeMode === "native" || cancelledPreparation) ? { queuedCommentRequestId: wake.id } : {}), idempotencyKey: `remote-stop-comment:${run.id}:${wake.id}` }, wake.id); break; } From 215586d127e97c9301d86e769a39a15c13298ca2 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 3 Oct 2026 00:04:21 -0500 Subject: [PATCH 03/37] fix: settle interrupted preparation with a retained cancellation fence (#15022) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A cancelled preparation must preserve history while allowing a new user turn. > - Older preparation can retain its cancellation fence but omit the unwind marker. > - The controller from that older boot is gone, its lease expired, and no provider started. > - Use the same narrow preparation proof for this retained-fence state. > - The benefit is working Retry and saved-message recovery after interrupted startup. ## Linked Issues or Issue Description Refs #15020 and #15015. Related #13293 addresses post-launch recovery. **What happened?** A preparation interrupted before native selection retained `startupCancellation.beforeNativeSelection: true` but no preparation-settled marker. Its owner expired across restart and it had no environment leases. The missing-receipt compatibility rule did not recognize the retained fence, leaving Retry absent and saved messages deferred. **Expected behavior** Admit one new user turn when the preparation evidence agrees, the old controller expired, and cleanup is complete. Preserve previous results and do not replay the cancelled input. **Steps to reproduce** 1. Cancel Paperclip Runner preparation before runtime selection. 2. Retain the cancellation fence without an unwind marker or environment leases. 3. Restart after its old controller lease expires. 4. Send a new message, select Retry, or allow the saved-message worker to reconsider new input. 5. Confirm one successor receives only undelivered input. Repeat with live ownership, invocation evidence, or pending cleanup and confirm execution stays held. **Paperclip version or commit** Reproduced on `94f6f3eb4`. **Deployment mode** Authenticated private self-hosted server, built from source. ## What Changed - Accept a retained before-selection cancellation fence in the expired historical preparation proof. - Retain runtime, stage, adapter, ownership expiry, process, event, and cleanup requirements. - Extend Retry, fresh-message, partial-queue, concurrent-worker, and contradictory-evidence tests to both receipt states. - Document recovery when the preparation-settled marker was not retained. ## Verification - Red: five exact-state regressions fail on the parent commit. - Green: all 203 continuation tests pass, including the five new regressions and 13 additional negative evidence cases. Process recovery and queued-comment routes add 413 passing tests. - A read-only candidate service check against the retained live run returns Retry eligibility without changing any task state. - An unrelated containment assertion failed once in CI. All 85 tests in that route suite pass locally and the CI shard passes on rerun. - Workspace `pnpm -r typecheck` and `pnpm build` pass. - Final head `48240de3d`: all 54 CI checks pass, two optional Storybook checks skip, and Greptile is 5/5 with zero unresolved threads. The complete local monolithic suite is covered by sharded CI; the earlier local run was stopped after a workspace case failed, and that case passed in isolation. - After merge, deploy the exact merged commit and verify an actual agent reply through the task composer. ## Risks A cancellation receipt alone must not certify that provider execution stopped. This path still requires unresolved preparation, no native identity or coordinator, an expired controller from another boot, no launch or provider evidence, and completed environment cleanup. Current-boot preparation remains held until it settles. No schema or dependency change. ## Model Used OpenAI Codex, an agent based on GPT-6. The exact runtime model ID and context window are not exposed in this session. Used reasoning, repository tools, code execution, and browser inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- doc/execution-semantics.md | 6 ++-- .../src/services/cancelled-native-startup.ts | 5 ++-- .../explicit-native-continuation.test.ts | 29 +++++++++++-------- 3 files changed, 24 insertions(+), 16 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 326da73f0bd..72a0cb98324 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1156,10 +1156,12 @@ controller, lease, or result). It also checks for contradictory launch/process evidence and verifies local cleanup or exact remote termination receipts. The preparer must have finished or its startup lease must have expired. A missing PID alone does not establish this proof. -For older interrupted preparation rows without a cancellation receipt, the +For older interrupted preparation rows without a cancellation receipt, or with +a retained startup fence but no preparation-settled marker, the immutable Paperclip Runner adapter claim, unresolved runtime, and preparing stage must agree. The old controller must belong to another server boot and its lease -must have expired. No native identity, coordinator, result, adapter invocation, +must have expired. Only absent results or the retained before-selection +cancellation fence qualify. No native identity, coordinator, adapter invocation, provider event, or process-launch evidence may exist. Environment cleanup still requires the same receipts. This historical proof permits explicit Retry or a newer saved user message; it does not replay the cancelled input. diff --git a/server/src/services/cancelled-native-startup.ts b/server/src/services/cancelled-native-startup.ts index ef7fb126d5f..9bcadce31cf 100644 --- a/server/src/services/cancelled-native-startup.ts +++ b/server/src/services/cancelled-native-startup.ts @@ -28,14 +28,15 @@ export async function isCancelledNativeStartup(db: Db, run: Run, coordinator: Co if (run.status !== "cancelled" || !run.finishedAt || run.processPid || run.processGroupId || run.processStartedAt || run.sessionIdAfter) return false; const cancellation = run.resultJson?.startupCancellation as Record | undefined; - // Older builds did not retain the cancellation fence. Their immutable + // Older builds could omit the cancellation fence or its unwind marker. Their immutable // native-adapter claim and unresolved preparation stage still prove that // provider dispatch did not begin. Require an expired owner from another // server boot; neither a missing PID nor mutable agent settings is proof. const historicalBeforeSelection = run.runtimeMode === "legacy" && !run.runtimeModeResolvedAt && run.executionStage === "preparing" && !run.nativeIssueId && !run.nativeSessionId && !coordinator && claimedAdapterType(run) === "paperclip_runner" && run.errorCode === "operator_interrupted" && - run.resultJson === null && Boolean(run.controllerBootId && run.controllerBootId !== legacyControllerBootId && + (run.resultJson === null || cancellation?.beforeNativeSelection === true) && + Boolean(run.controllerBootId && run.controllerBootId !== legacyControllerBootId && run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt <= new Date()); const beforeSelection = historicalBeforeSelection || run.runtimeMode === "legacy" && !run.runtimeModeResolvedAt && !run.nativeSessionId && !coordinator && claimedAdapterType(run) === "paperclip_runner" && diff --git a/server/src/services/explicit-native-continuation.test.ts b/server/src/services/explicit-native-continuation.test.ts index 31b0cb0f4a8..9ddb4a898be 100644 --- a/server/src/services/explicit-native-continuation.test.ts +++ b/server/src/services/explicit-native-continuation.test.ts @@ -598,21 +598,24 @@ const support = await getEmbeddedPostgresTestSupport(); expect(await admit(f)).toMatchObject({ previousRunId: f.sourceRunId }); }); - async function seedHistoricalCancelledPreparation() { + async function seedHistoricalCancelledPreparation(retainedReceipt = false) { const f = await seedCancelledStartup(); await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.sourceRunId)); await db.delete(environmentLeases).where(eq(environmentLeases.heartbeatRunId, f.sourceRunId)); await db.update(heartbeatRuns).set({ runtimeMode: "legacy", runtimeModeResolvedAt: null, nativeIssueId: null, nativeSessionId: null, executionStage: "preparing", errorCode: "operator_interrupted", - runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } }, resultJson: null, + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } }, + resultJson: retainedReceipt ? { operatorInterrupted: true, stopReason: "cancelled", + startupCancellation: { beforeNativeSelection: true, requestedAt: "2026-09-11T10:00:00Z" } } : null, }).where(eq(heartbeatRuns.id, f.sourceRunId)); await db.update(issueRecoveryActions).set({ cause: "legacy_execution_requires_reconciliation" }) .where(eq(issueRecoveryActions.sourceIssueId, f.issueId)); return f; } - it.each(["message", "retry"])("recovers historical native preparation through an explicit %s", async kind => { - const f = await seedHistoricalCancelledPreparation(); + it.each([false, true].flatMap(receipt => ["message", "retry"].map(kind => ({ receipt, kind }))))( + "recovers historical native preparation through an explicit $kind (receipt: $receipt)", async ({ receipt, kind }) => { + const f = await seedHistoricalCancelledPreparation(receipt); expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ canRetry: true }); await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }); const successor = await heartbeatService(db).wakeup(f.agentId, { @@ -625,11 +628,13 @@ const support = await getEmbeddedPostgresTestSupport(); previousRunId: f.sourceRunId, explicitUserContinuation: { commentId: kind === "message" ? f.commentId : null } } }); expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull(); const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId)); - expect(source).toMatchObject({ status: "cancelled", runtimeMode: "legacy", resultJson: null }); + expect(source).toMatchObject({ status: "cancelled", runtimeMode: "legacy", + resultJson: receipt ? { startupCancellation: { beforeNativeSelection: true } } : null }); }); - it.each(["fresh", "earlier_delivered", "last_delivered"])("resumes saved input after historical native preparation expires across restart exactly once (%s)", async kind => { - const f = await seedHistoricalCancelledPreparation(); + it.each([false, true].flatMap(receipt => ["fresh", "earlier_delivered", "last_delivered"].map(kind => ({ receipt, kind }))))( + "resumes saved input after historical native preparation expires across restart exactly once ($kind, receipt: $receipt)", async ({ receipt, kind }) => { + const f = await seedHistoricalCancelledPreparation(receipt); await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }); await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60_000) }) .where(eq(heartbeatRuns.id, f.sourceRunId)); @@ -658,14 +663,14 @@ const support = await getEmbeddedPostgresTestSupport(); expect(successors).toHaveLength(1); expect(successors[0].contextSnapshot).toMatchObject({ wakeCommentIds: [f.commentId], explicitUserContinuation: { commentId: f.commentId }, forceFreshSession: true }); - const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, waiting.id)); - expect(receipt).toMatchObject({ status: "coalesced", runId: successors[0].id }); + const [queueReceipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, waiting.id)); + expect(queueReceipt).toMatchObject({ status: "coalesced", runId: successors[0].id }); expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull(); }); - it.each(["resolved", "dispatching", "adapter", "missing_boot", "current_boot", "missing_lease", "live_lease", "receipt", "launch", "invoked", "coordinator", "cleanup", "remote"])( - "holds historical native preparation with contradictory or incomplete %s evidence", async kind => { - const f = await seedHistoricalCancelledPreparation(); + it.each([false, true].flatMap(receipt => ["resolved", "dispatching", "adapter", "missing_boot", "current_boot", "missing_lease", "live_lease", "receipt", "launch", "invoked", "coordinator", "cleanup", "remote"].map(kind => ({ receipt, kind }))))( + "holds historical native preparation with contradictory or incomplete $kind evidence (receipt: $receipt)", async ({ receipt, kind }) => { + const f = await seedHistoricalCancelledPreparation(receipt); const patch = kind === "resolved" ? { runtimeModeResolvedAt: new Date() } : kind === "dispatching" ? { executionStage: "dispatching" } : kind === "adapter" ? { runnerProfileJson: { adapterDispatch: { adapterType: "process" } } } From 9ae3d8db3dc930f316f29e6e83fe6c325ac8bc06 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 3 Oct 2026 01:00:42 -0500 Subject: [PATCH 04/37] fix: keep pre-dispatch review waits out of execution recovery (#15024) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The queued-run gate can cancel a continuation that must wait for review. > - This cancellation happens before execution authority or a provider starts. > - Recovery currently treats the gate receipt as unknown provider execution. > - That mistake blocks a conversation after a successful reply and hides Retry. > - This pull request recognizes only the recorded, unclaimed review-wait state. > - Review waits keep their normal disposition path, and new user input can recover older false holds. ## Linked Issues or Issue Description Refs #15015, #15020, and #15022. Related #11614 narrows the review posture that causes cancellation; this change corrects recovery after a valid cancellation. **What happened?** After a successful agent reply, the queued-run gate cancelled an automatic continuation with `issue_continuation_waiting_on_review`. The gate retained `timeoutSource: stale_queued_run_gate` and a matching stop reason. No execution authority or provider started. Recovery still created an unknown-action hold, moved the task to Blocked, and hid Retry. **Expected behavior** Use normal review-wait disposition repair for this recorded state. Allow Retry, a new user message, or saved undelivered input to recover an older false hold after ordinary admission checks pass. Preserve the cancelled run and do not replay its input. **Steps to reproduce** 1. Finish an agent turn on an open task that has a real review target. 2. Let the automatic continuation reach the queued-run review gate. 3. Refresh after the cancelled run is checked by recovery. 4. Confirm a review wait is handled as a wait rather than unknown provider work. 5. Reproduce an older false hold for the same receipt, then request Retry or send a new message. 6. Confirm only one fresh turn starts, and contradictory execution or cleanup evidence retains the hold. **Paperclip version or commit** Reproduced on `215586d127`. **Deployment mode** Authenticated private self-hosted server, built from source. ## What Changed - Recognize the exact review-wait dispatch receipt only while all execution claims remain unset. - Exempt recovery only after checking retained launch and provider events, coordinators, and environment cleanup. Keep the synchronous classifier conservative without that database proof. - Apply the verified classification to automatic recovery, heartbeat retries, and stranded-queue release, so saved user input starts once. - Reuse guarded startup admission for Retry, new input, and saved input on older false holds. - Show a precise review-wait notice and keep continuation guidance consistent with Retry availability. - Verify provider events, launch events, coordinators, and cleanup before admission. - Add five initial red regressions, three additional red recovery evidence regressions, concurrent saved-input coverage, and negative evidence checks. - Document the review-wait contract. ## Verification - Red: five regressions fail on unchanged master. Existing review-wait and contradictory-evidence cases still pass. - Workspace `pnpm -r typecheck` passes. - All 752 affected recovery, continuation, queue, classification, and retry-scheduling tests pass. - Three additional review regressions failed before the database proof was added; all 12 focused review-wait cases then pass. - Workspace `pnpm build` passes. - Saved-input promotion and recovery notice regressions fail before their fixes and pass afterward. - Current-head CI has 54 passing checks and 2 skipped optional Storybook checks. Greptile reviewed `0bf1437110e1617ae4544afa41f4319eac763dba` at 5/5 with zero open findings. The complete suite runs in sharded CI. No complete local monolithic pass is claimed; an earlier long run was stopped, and its unrelated failing case passed in isolation. ## Risks The error code alone cannot establish that no provider started. This exception also requires the server gate receipt, matching stop reason, and null execution authority fields. User admission separately checks coordinator, launch and provider events, environment cleanup, pending decisions, ownership, task holds, budget, and active execution. No migration or dependency change. ## Model Used OpenAI Codex, an agent based on GPT-6. The exact runtime model ID and context window are not exposed in this session. Used reasoning, repository tools, code execution, and browser inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/execution-semantics.md | 2 + .../heartbeat-process-recovery.test.ts | 26 ++++++- .../wake-queue/adapters/postgres.test.ts | 40 +++++++++++ .../modules/wake-queue/adapters/postgres.ts | 4 +- .../src/services/cancelled-native-startup.ts | 6 +- server/src/services/execution-blocker.ts | 11 +-- .../explicit-native-continuation.test.ts | 67 +++++++++++++++++++ server/src/services/heartbeat.ts | 14 ++-- .../legacy-execution-recovery.test.ts | 21 ++++++ .../src/services/legacy-execution-recovery.ts | 7 ++ .../src/services/pre-dispatch-review-wait.ts | 51 ++++++++++++++ server/src/services/recovery/service.ts | 6 +- 12 files changed, 235 insertions(+), 20 deletions(-) create mode 100644 server/src/services/pre-dispatch-review-wait.ts diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 72a0cb98324..46b9e87e7f0 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -488,6 +488,8 @@ A new user message can continue a terminal native run whose process fields were The task thread exposes the guarded Retry action for failed or timed-out conversation runs and native preparation cancelled before provider startup when the never-started proof is verified. The server projects this eligibility on the recovery notice and rechecks it on Retry. Cleanup quarantine retains its inspection path, and non-conversation reconciliation gates remain enforced. Pending decisions, active execution, pause, budget, dependency, and ownership gates remain in force. A refused Retry reports its reason inline and remains available for another attempt. +A continuation cancelled by the queued-run gate while waiting for review did not start a provider. Its matching dispatch-gate receipt and unclaimed execution fields keep this deliberate wait out of unknown-action reconciliation. Pending review and decision gates still apply. If an older server already created a recovery hold for this exact state, Retry or a new user message can request one fresh turn after the server verifies there is no provider, coordinator, or unfinished cleanup. Saved user input is reconsidered through the same admission path; the cancelled turn is not replayed. + A valid recovery action must name: - the source issue and company diff --git a/server/src/__tests__/heartbeat-process-recovery.test.ts b/server/src/__tests__/heartbeat-process-recovery.test.ts index 07bef2d6d8c..d88dac7d329 100644 --- a/server/src/__tests__/heartbeat-process-recovery.test.ts +++ b/server/src/__tests__/heartbeat-process-recovery.test.ts @@ -6221,14 +6221,28 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]).toMatchObject({ status: "blocked", assigneeAgentId: agentId }); }); - it("converts a continuation parked for review into a dependency wait on its open sub-tasks", async () => { - const { companyId, agentId, issueId } = await seedStrandedIssueFixture({ + it.each([false, "receipt", "launch", "provider", "cleanup"])("converts a continuation parked for review into a dependency wait on its open sub-tasks (dispatch receipt: %s)", async dispatchReceipt => { + const { companyId, agentId, issueId, runId } = await seedStrandedIssueFixture({ status: "in_progress", runStatus: "cancelled", retryReason: "issue_continuation_needed", runErrorCode: "issue_continuation_waiting_on_review", runError: "Continuation parked: issue is waiting on review/approval", }); + if (dispatchReceipt) { + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId)); + await db.update(heartbeatRuns).set({ startedAt: null, + resultJson: { stopReason: "issue_continuation_waiting_on_review", timeoutSource: "stale_queued_run_gate" }, + }).where(eq(heartbeatRuns.id, run.id)); + } + if (dispatchReceipt === "launch" || dispatchReceipt === "provider") { + await db.insert(heartbeatRunEvents).values({ companyId, runId, agentId, seq: 1, + eventType: dispatchReceipt === "launch" ? "native.process_start_requested" : "provider.event", + stream: "system", level: "info", message: "Contradictory execution evidence" }); + } else if (dispatchReceipt === "cleanup") { + await db.insert(environmentLeases).values({ companyId, heartbeatRunId: runId, + provider: "local", status: "pending_cleanup", cleanupStatus: "failed" }); + } const issuePrefix = `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`; const openChildTodoId = randomUUID(); const openChildInProgressId = randomUUID(); @@ -6270,6 +6284,14 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { const heartbeat = heartbeatService(db); const result = await heartbeat.reconcileStrandedAssignedIssues(); + if (dispatchReceipt && dispatchReceipt !== "receipt") { + expect(result.waitingOnReviewResolved).toBe(0); + expect(result.escalated).toBe(1); + expect(await getExecutionBlocker(db, companyId, issueId)).toMatchObject({ + cause: "legacy_execution_requires_reconciliation", runId, canRetry: false, + }); + return; + } expect(result.waitingOnReviewResolved).toBe(1); expect(result.escalated).toBe(0); expect(result.issueIds).toEqual([issueId]); diff --git a/server/src/modules/wake-queue/adapters/postgres.test.ts b/server/src/modules/wake-queue/adapters/postgres.test.ts index 812cdec546b..9bc674d5fac 100644 --- a/server/src/modules/wake-queue/adapters/postgres.test.ts +++ b/server/src/modules/wake-queue/adapters/postgres.test.ts @@ -9,6 +9,8 @@ import { agents, companies, createDb, + environmentLeases, + heartbeatRunEvents, heartbeatRuns, issueComments, issueRecoveryActions, @@ -59,6 +61,8 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => { afterEach(async () => { await db.delete(activityLog); await db.delete(issueComments); + await db.delete(heartbeatRunEvents); + await db.delete(environmentLeases); // `heartbeat_runs.wakeup_request_id` references `agent_wakeup_requests.id`, // so the run row must go first. await db.delete(heartbeatRuns); @@ -416,6 +420,42 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => { expect(wake.status).toBe("deferred_issue_execution"); }); + it.each(["clean", "launch", "cleanup"])("promotes saved input only after a verified pre-dispatch review wait (%s)", async evidence => { + const companyId = await seedCompany(), agentId = await seedAgent({ companyId }); + const issueId = await seedIssue({ companyId, assigneeAgentId: agentId }); + const runId = await seedRun({ companyId, agentId, status: "cancelled", contextSnapshot: { issueId }, + errorCode: "issue_continuation_waiting_on_review" }); + await db.update(heartbeatRuns).set({ finishedAt: new Date(Date.now() - 1000), resultJson: { + stopReason: "issue_continuation_waiting_on_review", timeoutSource: "stale_queued_run_gate", + } }).where(eq(heartbeatRuns.id, runId)); + if (evidence === "launch") await db.insert(heartbeatRunEvents).values({ companyId, runId, agentId, seq: 1, + eventType: "native.process_start_requested", stream: "system", level: "info" }); + if (evidence === "cleanup") await db.insert(environmentLeases).values({ companyId, heartbeatRunId: runId, + provider: "local", status: "pending_cleanup", cleanupStatus: "failed" }); + const [comment] = await db.insert(issueComments).values({ companyId, issueId, authorType: "user", + authorUserId: "responsible-user", body: "Continue with this new message." }).returning(); + const wakeId = await seedDeferredWake({ companyId, agentId, issueId, requestedByActorId: "responsible-user", + payload: { commentId: comment.id, _paperclipWakeContext: { issueId, wakeReason: "issue_commented", wakeCommentIds: [comment.id] } } }); + const release = createReleaseIssueExecution({ issueLock: createPostgresWakeQueueAdapter(db, stubDeps), + recovery: { escalateStrandedAssignedIssue: async () => {}, escalateStrandedRecoveryIssueInPlace: async () => {} } }); + const result = await release({ companyId, runId, now: new Date() }); + const runs = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId)); + const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wakeId)); + if (evidence !== "clean") { + expect(result.outcome.kind).toBe("released"); + expect(runs).toHaveLength(1); + expect(wake).toMatchObject({ status: "deferred_issue_execution", runId: null }); + return; + } + expect(result.outcome.kind).toBe("promoted"); + expect(runs).toHaveLength(2); + const successor = runs.find(run => run.status === "queued")!; + expect(wake.runId).toBe(successor.id); + expect(successor.contextSnapshot).toMatchObject({ wakeCommentIds: [comment.id] }); + await release({ companyId, runId, now: new Date() }); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId))).toHaveLength(2); + }); + it("leaves deferred work untouched until the effective execution hold clears", async () => { const companyId = await seedCompany(); const agentId = await seedAgent({ companyId }); diff --git a/server/src/modules/wake-queue/adapters/postgres.ts b/server/src/modules/wake-queue/adapters/postgres.ts index 60b0e359b79..57d256e70be 100644 --- a/server/src/modules/wake-queue/adapters/postgres.ts +++ b/server/src/modules/wake-queue/adapters/postgres.ts @@ -19,7 +19,7 @@ import { nativeRunFinalizations, } from "@paperclipai/db"; import { hasConversationContinuationPolicy } from "../../../services/conversation-continuation.js"; -import { legacyExecutionNeedsReconciliation } from "../../../services/legacy-execution-recovery.js"; +import { legacyExecutionNeedsReconciliationWithEvidence } from "../../../services/legacy-execution-recovery.js"; import { authorizeFailedChatRunRetryWake, FailedChatRunRetryAuthorizationError, @@ -1104,7 +1104,7 @@ export function createPostgresWakeQueueAdapter(db: Db, deps: WakeQueuePostgresAd issueStatus: issueRow?.status ?? "", hasAssigneeUser: Boolean(issueRow?.assigneeUserId), assigneeAgentMatchesRunAgent: issueRow?.assigneeAgentId === run.agentId, - legacyExecutionNeedsReconciliation: legacyExecutionNeedsReconciliation(run), + legacyExecutionNeedsReconciliation: await legacyExecutionNeedsReconciliationWithEvidence(tx as unknown as Db, run), // An operator stop never promotes old queued work by itself. The // next explicit wake adopts those messages atomically when it // queues a run. diff --git a/server/src/services/cancelled-native-startup.ts b/server/src/services/cancelled-native-startup.ts index 9bcadce31cf..53a17527f48 100644 --- a/server/src/services/cancelled-native-startup.ts +++ b/server/src/services/cancelled-native-startup.ts @@ -1,3 +1,4 @@ +import { isPreDispatchReviewWait } from "./pre-dispatch-review-wait.js"; import { and, eq, inArray, isNotNull, or } from "drizzle-orm"; import { environmentLeases, heartbeatRunEvents, heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db"; import { claimedAdapterType } from "./conversation-continuation.js"; @@ -38,7 +39,8 @@ export async function isCancelledNativeStartup(db: Db, run: Run, coordinator: Co (run.resultJson === null || cancellation?.beforeNativeSelection === true) && Boolean(run.controllerBootId && run.controllerBootId !== legacyControllerBootId && run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt <= new Date()); - const beforeSelection = historicalBeforeSelection || run.runtimeMode === "legacy" && !run.runtimeModeResolvedAt && + const beforeReviewDispatch = isPreDispatchReviewWait(run) && !coordinator; + const beforeSelection = beforeReviewDispatch || historicalBeforeSelection || run.runtimeMode === "legacy" && !run.runtimeModeResolvedAt && !run.nativeSessionId && !coordinator && claimedAdapterType(run) === "paperclip_runner" && cancellation?.beforeNativeSelection === true; const neverClaimed = run.runtimeMode === "native" && coordinator && @@ -47,7 +49,7 @@ export async function isCancelledNativeStartup(db: Db, run: Run, coordinator: Co !coordinator.controllerPid && !coordinator.leaseOwner && !coordinator.leaseExpiresAt && !coordinator.resultId && !coordinator.failureDetail?.successorRunId; if (!beforeSelection && !neverClaimed) return false; - const settled = typeof run.resultJson?.startupPreparationSettledAt === "string"; + const settled = beforeReviewDispatch || typeof run.resultJson?.startupPreparationSettledAt === "string"; // The old preparer can still be unwinding even though the run is terminal. if (!settled && run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt > new Date()) return false; const leases = await db.select().from(environmentLeases).where(and( diff --git a/server/src/services/execution-blocker.ts b/server/src/services/execution-blocker.ts index 2d70f70add9..14c316b5a00 100644 --- a/server/src/services/execution-blocker.ts +++ b/server/src/services/execution-blocker.ts @@ -1,3 +1,4 @@ +import { isPreDispatchReviewWait } from "./pre-dispatch-review-wait.js"; import { and, desc, eq, gt, inArray, not, or, sql } from "drizzle-orm"; import { conversationRecoveryActionPredicate, getConversationOwnershipBlocker } from "./conversation-continuation.js"; import { z } from "zod"; @@ -79,13 +80,15 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st or consumed.context_snapshot->'wakeCommentIds' @> jsonb_build_array(${issueComments.id}::text)))`, )) : []; const cancellation = run ? readRunCancellation(run.resultJson) : null; - const runError = cancellation?.reason ?? (run?.status === "cancelled" - ? "Execution was cancelled; its source was not recorded." - : run?.error); const eligibleContinuation = Boolean(run && conversation?.assigneeAgentId === run.agentId && !["done", "cancelled"].includes(conversation.status) && canContinueCancelledRun(run)); const canRetry = Boolean(run && conversation?.assigneeAgentId === run.agentId && !["done", "cancelled"].includes(conversation.status) && await canRetryStoppedRun(db, run)); + const runError = run && canRetry && isPreDispatchReviewWait(run) + ? "Waiting for review; this continuation never started." + : cancellation?.reason ?? (run?.status === "cancelled" + ? "Execution was cancelled; its source was not recorded." + : run?.error); const [chatBinding] = eligibleContinuation || canRetry ? await db.select({ state: chatConversations.state, endpointStatus: chatEndpoints.status, connectionStatus: toolConnections.status, connectionEnabled: toolConnections.enabled, @@ -107,7 +110,7 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st } else { nextAction = "Send a new chat message to continue this conversation."; } - } else if (run?.status === "cancelled" && !eligibleContinuation && action.cause === "legacy_execution_requires_reconciliation") { + } else if (run?.status === "cancelled" && !eligibleContinuation && !canRetry && action.cause === "legacy_execution_requires_reconciliation") { nextAction += " Inspect the run before sending a new message to request continuation."; } diff --git a/server/src/services/explicit-native-continuation.test.ts b/server/src/services/explicit-native-continuation.test.ts index 9ddb4a898be..cd0d3ac143a 100644 --- a/server/src/services/explicit-native-continuation.test.ts +++ b/server/src/services/explicit-native-continuation.test.ts @@ -613,6 +613,73 @@ const support = await getEmbeddedPostgresTestSupport(); return f; } + async function seedCancelledReviewWait() { + const f = await seedHistoricalCancelledPreparation(); + await db.update(heartbeatRuns).set({ startedAt: null, executionStage: null, + controllerBootId: null, controllerLeaseExpiresAt: null, runnerProfileJson: null, + errorCode: "issue_continuation_waiting_on_review", + resultJson: { stopReason: "issue_continuation_waiting_on_review", timeoutSource: "stale_queued_run_gate" }, + }).where(eq(heartbeatRuns.id, f.sourceRunId)); + return f; + } + + it.each(["message", "retry"])("recovers a pre-dispatch review wait through an explicit %s", async kind => { + const f = await seedCancelledReviewWait(); + const notice = await getExecutionBlocker(db, f.companyId, f.issueId); + expect(notice).toMatchObject({ canRetry: true, runError: "Waiting for review; this continuation never started." }); + expect(notice?.nextAction).not.toContain("Inspect the run before sending a new message"); + await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }); + const successor = await heartbeatService(db).wakeup(f.agentId, { source: kind === "retry" ? "on_demand" : "automation", triggerDetail: "manual", + reason: kind === "retry" ? "retry_failed_run" : "issue_commented", + ...(kind === "retry" ? { failedRunId: f.sourceRunId } : {}), requestedByActorType: "user", requestedByActorId: "board", + payload: { issueId: f.issueId, ...(kind === "message" ? { commentId: f.commentId } : {}) }, + contextSnapshot: { issueId: f.issueId, ...(kind === "message" ? { wakeCommentId: f.commentId } : {}) }, + }); + expect(successor).toMatchObject({ status: "queued", contextSnapshot: { forceFreshSession: true, + previousRunId: f.sourceRunId, explicitUserContinuation: { commentId: kind === "message" ? f.commentId : null } } }); + expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull(); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId)))[0].status).toBe("cancelled"); + }); + + it("reconsiders saved input after a pre-dispatch review wait exactly once", async () => { + const f = await seedCancelledReviewWait(); + await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }); + const leaseId = randomUUID(); + await db.insert(environmentLeases).values({ id: leaseId, companyId: f.companyId, + heartbeatRunId: f.sourceRunId, provider: "local", status: "pending_cleanup", cleanupStatus: "failed" }); + await heartbeatService(db).wakeup(f.agentId, { source: "automation", reason: "issue_commented", + requestedByActorType: "user", requestedByActorId: "board", payload: { issueId: f.issueId, commentId: f.commentId }, + contextSnapshot: { issueId: f.issueId, wakeCommentId: f.commentId } }); + const [waiting] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId)); + expect(waiting.status).toBe("deferred_issue_execution"); + await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "succeeded" }) + .where(eq(environmentLeases.id, leaseId)); + await db.update(agentWakeupRequests).set({ updatedAt: new Date(0) }).where(eq(agentWakeupRequests.id, waiting.id)); + await Promise.all([heartbeatService(db).resumeExecutionWaitComments(), heartbeatService(db).resumeExecutionWaitComments()]); + const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued"))); + expect(successors).toHaveLength(1); + expect(successors[0].contextSnapshot).toMatchObject({ wakeCommentIds: [f.commentId], + explicitUserContinuation: { commentId: f.commentId }, forceFreshSession: true }); + expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, waiting.id)))[0]) + .toMatchObject({ status: "coalesced", runId: successors[0].id }); + expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull(); + }); + + it.each(["launch", "provider", "cleanup"])("retains a pre-dispatch review wait with contradictory %s evidence", async kind => { + const f = await seedCancelledReviewWait(); + if (kind === "cleanup") await db.insert(environmentLeases).values({ companyId: f.companyId, + heartbeatRunId: f.sourceRunId, provider: "local", status: "pending_cleanup", cleanupStatus: "failed" }); + else await db.insert(heartbeatRunEvents).values({ companyId: f.companyId, runId: f.sourceRunId, + agentId: f.agentId, seq: 1, eventType: kind === "launch" ? PROCESS_START_REQUESTED : "provider.event", + stream: "system", level: "info", message: "Contradictory execution evidence" }); + try { + expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ canRetry: false }); + expect(await admit(f)).toBeNull(); + } finally { + await db.delete(environmentLeases).where(eq(environmentLeases.heartbeatRunId, f.sourceRunId)); + } + }); + it.each([false, true].flatMap(receipt => ["message", "retry"].map(kind => ({ receipt, kind }))))( "recovers historical native preparation through an explicit $kind (receipt: $receipt)", async ({ receipt, kind }) => { const f = await seedHistoricalCancelledPreparation(receipt); diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 88f73ea4c6d..9ca39eb35cd 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -44,7 +44,7 @@ import { getNativeReviewAssignment, readNativeReviewAssignmentContext } from "./ import { claimQueuedNativeReviewRun } from "./native-runtime/native-review-dispatch.js"; import { buildNativeReviewRequest } from "./native-runtime/native-review-prompt.js"; import { - legacyExecutionNeedsReconciliation, + legacyExecutionNeedsReconciliationWithEvidence, settleInterruptedNativeBootstrap, terminalizeLegacyExecution, } from "./legacy-execution-recovery.js"; @@ -11771,7 +11771,7 @@ export function heartbeatService( "Native execution recovery owns this provider failure; a quota monitor cannot start a replacement.", ); } - if (legacyExecutionNeedsReconciliation(sourceRun)) { + if (await legacyExecutionNeedsReconciliationWithEvidence(db, sourceRun)) { await terminalizeLegacyExecution({ db, run: sourceRun, @@ -12912,7 +12912,7 @@ export function heartbeatService( } const updated = previousStatus && - legacyExecutionNeedsReconciliation({ + await legacyExecutionNeedsReconciliationWithEvidence(db, { ...previousStatus, status, ...patch, @@ -13002,7 +13002,7 @@ export function heartbeatService( } const updated = previousStatus && - legacyExecutionNeedsReconciliation({ + await legacyExecutionNeedsReconciliationWithEvidence(db, { ...previousStatus, status, ...patch, @@ -14519,7 +14519,7 @@ export function heartbeatService( // Native sessions have their own fenced same-run controller. Legacy // bootstrap recovery shares the durable delay and incident counter with // transient retries; process loss must not open a second retry budget. - if (run.runtimeMode === "native" || legacyExecutionNeedsReconciliation(run)) + if (run.runtimeMode === "native" || await legacyExecutionNeedsReconciliationWithEvidence(db, run)) return null; const scheduled = await scheduleBoundedRetryForRun(run, agent, { now }); return scheduled.outcome === "scheduled" ? scheduled.run : null; @@ -15494,7 +15494,7 @@ export function heartbeatService( }; } - if (legacyExecutionNeedsReconciliation(run)) { + if (await legacyExecutionNeedsReconciliationWithEvidence(db, run)) { return { outcome: "not_scheduled" as const, reason: @@ -25812,7 +25812,7 @@ export function heartbeatService( await scheduleBoundedRetryForRun(livenessRun, agent); } else if ( outcome === "failed" && - !legacyExecutionNeedsReconciliation(livenessRun) + !(await legacyExecutionNeedsReconciliationWithEvidence(db, livenessRun)) ) { await scheduleInteractionContinuationInfrastructureRetryIfEligible( livenessRun, diff --git a/server/src/services/legacy-execution-recovery.test.ts b/server/src/services/legacy-execution-recovery.test.ts index 52397bf506c..b1563e07d1d 100644 --- a/server/src/services/legacy-execution-recovery.test.ts +++ b/server/src/services/legacy-execution-recovery.test.ts @@ -1,3 +1,4 @@ +import { isPreDispatchReviewWait } from "./pre-dispatch-review-wait.js"; import { expect, it } from "vitest"; import { legacyExecutionNeedsReconciliation } from "./legacy-execution-recovery.js"; @@ -86,3 +87,23 @@ it("retains conversation retry eligibility for a transient restore lock timeout" workspaceRestoreFailure: "restore_lock_timeout", conversationContinuation: "continue_conversation_v1", } })).toBe(false); }); + + +it("recognizes only an unclaimed pre-dispatch review wait receipt", () => { + const run = { runtimeMode: "legacy", status: "cancelled", errorCode: "issue_continuation_waiting_on_review", + startedAt: null, runtimeModeResolvedAt: null, processPid: null, processGroupId: null, + processStartedAt: null, nativeIssueId: null, nativeSessionId: null, sessionIdAfter: null, + controllerBootId: null, controllerLeaseExpiresAt: null, executionStage: null, + resultJson: { stopReason: "issue_continuation_waiting_on_review", timeoutSource: "stale_queued_run_gate" } }; + expect(isPreDispatchReviewWait(run)).toBe(true); + expect(legacyExecutionNeedsReconciliation(run)).toBe(true); + for (const patch of [ + { startedAt: new Date() }, { processPid: 123 }, { processGroupId: 123 }, + { processStartedAt: new Date() }, { nativeSessionId: "provider-session" }, { nativeIssueId: "native-task" }, + { sessionIdAfter: "provider-session" }, { runtimeModeResolvedAt: new Date() }, + { controllerBootId: "live-controller" }, { controllerLeaseExpiresAt: new Date() }, { executionStage: "preparing" }, + { errorCode: "operator_interrupted" }, { resultJson: {} }, + { resultJson: { ...run.resultJson, stopReason: "cancelled" } }, + { resultJson: { ...run.resultJson, workspaceRestoreFailure: "restore_unsafe_archive" } }, + ]) expect(isPreDispatchReviewWait({ ...run, ...patch })).toBe(false); +}); diff --git a/server/src/services/legacy-execution-recovery.ts b/server/src/services/legacy-execution-recovery.ts index 98c4c03f7e5..6ab86f7ee43 100644 --- a/server/src/services/legacy-execution-recovery.ts +++ b/server/src/services/legacy-execution-recovery.ts @@ -1,3 +1,4 @@ +import { isPreDispatchReviewWaitVerified } from "./pre-dispatch-review-wait.js"; import { hasWorkspaceRestoreFailure } from "@paperclipai/shared"; import { normalizeMaxTurnStopReason } from "./heartbeat-stop-metadata.js"; import { claimedAdapterType, hasConversationContinuationPolicy } from "./conversation-continuation.js"; @@ -53,6 +54,12 @@ export function legacyExecutionNeedsReconciliation( ); } +/** Review-wait receipts are only exempt after retained execution evidence agrees. + * The synchronous classifier stays conservative for callers without a DB proof. */ +export async function legacyExecutionNeedsReconciliationWithEvidence(db: Db, run: Run): Promise { + return legacyExecutionNeedsReconciliation(run) && !(await isPreDispatchReviewWaitVerified(db, run)); +} + /** Persist the failed legacy run, owned lock release and operator decision together. */ export async function terminalizeLegacyExecution(input: { db: Db; diff --git a/server/src/services/pre-dispatch-review-wait.ts b/server/src/services/pre-dispatch-review-wait.ts new file mode 100644 index 00000000000..08ea49b498e --- /dev/null +++ b/server/src/services/pre-dispatch-review-wait.ts @@ -0,0 +1,51 @@ +import { and, eq, inArray, isNotNull, or } from "drizzle-orm"; +import { environmentLeases, heartbeatRunEvents, heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-local-process-stop.js"; +import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js"; + +type Run = typeof heartbeatRuns.$inferSelect; +type DispatchFields = "startedAt" | "runtimeModeResolvedAt" | "processPid" | "processGroupId" | + "processStartedAt" | "nativeIssueId" | "nativeSessionId" | "sessionIdAfter" | + "controllerBootId" | "controllerLeaseExpiresAt" | "executionStage"; + +/** The queued-run gate owns this receipt before execution authority is claimed. + * A deliberate review wait has no provider actions to reconcile. The error code + * alone is not proof; missing or conflicting dispatch evidence retains the hold. */ +export function isPreDispatchReviewWait( + run: Pick & Partial>, +): boolean { + return run.runtimeMode === "legacy" && run.status === "cancelled" && + run.errorCode === "issue_continuation_waiting_on_review" && + run.resultJson?.stopReason === run.errorCode && + run.resultJson?.timeoutSource === "stale_queued_run_gate" && + run.resultJson?.workspaceRestoreFailure !== "restore_unsafe_archive" && + run.startedAt === null && run.runtimeModeResolvedAt === null && + run.processPid === null && run.processGroupId === null && run.processStartedAt === null && + run.nativeIssueId === null && run.nativeSessionId === null && run.sessionIdAfter === null && + run.controllerBootId === null && run.controllerLeaseExpiresAt === null && run.executionStage === null; +} + +/** The row receipt is a candidate, not permission to discard conflicting + * retained execution or cleanup evidence. Historical false holds must be + * checked against the same evidence that gates explicit user admission. */ +export async function isPreDispatchReviewWaitVerified(db: Db, run: Run): Promise { + if (!isPreDispatchReviewWait(run)) return false; + const [coordinator] = await db.select({ id: nativeRunFinalizations.runId }).from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.companyId, run.companyId), eq(nativeRunFinalizations.runId, run.id), + )).limit(1); + if (coordinator) return false; + const leases = await db.select().from(environmentLeases).where(and( + eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id), + )); + if (leases.some(lease => lease.provider === "local" + ? !lease.releasedAt || lease.status === "pending_cleanup" || lease.cleanupStatus === "failed" + : !hasRemoteTerminationReceipt(lease))) return false; + const [execution] = await db.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents).where(and( + eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id), + or(isNotNull(heartbeatRunEvents.sourceEventId), + inArray(heartbeatRunEvents.eventType, ["adapter.invoke", PROCESS_START_REQUESTED, PROCESS_IDENTITY_RECORDED, + "harness.ready", "session.started", "session.resumed", "session.updated", "turn.started", + "provider.event", "provider.rpc_result", "tool.execution.started"])), + )).limit(1); + return !execution; +} diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index 113ea5ce1cd..99ed604dee8 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -89,7 +89,7 @@ import { budgetService } from "../budgets.js"; import { unadmittedChatWakeupCondition } from "../durable-chat-wakeup.js"; import { issueRecoveryActionService } from "../issue-recovery-actions.js"; import { - legacyExecutionNeedsReconciliation, + legacyExecutionNeedsReconciliationWithEvidence, terminalizeLegacyExecution, } from "../legacy-execution-recovery.js"; import { issueTreeControlService } from "../issue-tree-control.js"; @@ -1923,7 +1923,7 @@ export function recoveryService( // Failure recovery shares the durable incident budget and delay. It // cannot fall through into the productive-work continuation queue. if (predecessor.runtimeMode === "native") return null; - if (legacyExecutionNeedsReconciliation(predecessor)) { + if (await legacyExecutionNeedsReconciliationWithEvidence(db, predecessor)) { await terminalizeLegacyExecution({ db, run: predecessor, @@ -4603,7 +4603,7 @@ export function recoveryService( eq(heartbeatRuns.id, executionRecoverySource.id), ), ); - if (source && legacyExecutionNeedsReconciliation(source)) { + if (source && await legacyExecutionNeedsReconciliationWithEvidence(db, source)) { await terminalizeLegacyExecution({ db, run: source, From ffe5e9e2a8866767cbb48009040bfafd85b56576 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 3 Oct 2026 01:38:57 -0500 Subject: [PATCH 05/37] fix: retain execution evidence for Retry and saved input (#15033) ## Thinking Path > - Paperclip lets people steer and recover AI-agent conversations. > - Recovery eligibility depends on retained cancellation receipts. > - Run presentation intentionally omits result JSON on SQL_ASCII databases and reduces oversized output. > - Retry and the saved-input sweep mistakenly used that presentation read for admission. > - The banner could offer Retry while the endpoint rejected the same stopped run. > - Read the narrow execution-evidence fields for internal admission and keep normal presentation unchanged. ## Linked Issues or Issue Description Refs #15024 and #15015. Searched existing recovery and redaction PRs; no duplicate fix found. **What happened?** On a SQL_ASCII instance, a verified pre-dispatch review-wait cancellation offers Retry in the recovery notice. Clicking it returns an eligibility conflict, and a saved user message remains deferred. The notice reads the retained database receipt, but the endpoint and sweep read a presentation projection where `resultJson` is null. **Expected behavior** Retry and saved user input use the recorded execution evidence and ordinary admission gates, independent of presentation redaction. Public run reads retain their existing encoding and output-size protections. **Steps to reproduce** 1. Record a cancelled, unclaimed review-wait continuation and its recovery hold. 2. Use the SQL_ASCII presentation projection, where run result JSON is omitted. 3. Click Retry or save a new user message and let the recovery sweep inspect it. 4. Verify a fresh turn starts once, with no replay of consumed input. **Paperclip version or commit** Reproduced on `9ae3d8db3`. **Deployment mode** Authenticated private self-hosted server with a SQL_ASCII database. ## What Changed - Add an explicit internal read of cancellation, startup, review-wait, tool-inventory, and Stop evidence; omit provider diagnostics. - Use that read in the Retry route, wakeup validation, and saved-input continuation checks. - Preserve the distinction between absent result JSON and an unrecognized stored result. - Cover the reproduced SQL_ASCII Retry and saved-input failures, retained public redaction, native Stop behavior, and excluded provider output. - Document the presentation and admission distinction. ## Verification - Red: four selected assertions fail before the fix, including the SQL_ASCII eligibility conflict and saved input remaining deferred. - Targeted green regressions and existing native Stop cases pass. - Workspace `pnpm -r typecheck` and `pnpm build` pass. - All 626 affected recovery, continuation, and route tests pass, including 22 focused admission and native Stop cases. Current-head CI has 54 passing gates and 2 skipped optional Storybook checks. Greptile reviewed `9b94af91e295a4e8007dfc6bff6a8d532d945e36` at 5/5 with no findings or open review threads. No complete local monolithic pass is claimed; the complete suite runs in sharded CI. ## Risks Admission still checks recorded process and controller ownership, provider events, cleanup, company scope, user authority, pending decisions, and task holds. The evidence projection must retain every field used by these eligibility predicates; existing native Stop cases guard against dropping its acknowledgement receipt. No schema, dependency, UI, or public response change. ## Model Used OpenAI Codex, an agent based on GPT-6. The exact runtime model ID and context window are not exposed in this session. Used reasoning, repository tools, code execution, and browser inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- doc/execution-semantics.md | 2 + .../__tests__/agent-live-run-routes.test.ts | 2 +- server/src/routes/agents.ts | 2 +- .../explicit-native-continuation.test.ts | 40 ++++++++++++++--- server/src/services/heartbeat.ts | 44 ++++++++++++++----- 5 files changed, 71 insertions(+), 19 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 46b9e87e7f0..6d8ec4af309 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -668,6 +668,8 @@ The same bounded rule applies when the previous heartbeat reported waiting on a A continuation that the staleness gate cancelled with `issue_continuation_waiting_on_review` is a *deliberate park*, not a disappeared execution path. The latest run reported that the issue is waiting for review/approval (for example, an umbrella issue whose work was just decomposed into sub-tasks). Treating that park as a stranded run would retry it, then escalate it to `blocked` with a recovery action and an operator-facing failure notice — even though nothing failed and there is nothing for a human to do. +Execution admission reads a narrow server-owned cancellation-evidence projection. Ordinary run presentation can redact `resultJson` for database encoding or output size; that presentation projection must not decide Retry eligibility or saved-input recovery. The admission projection excludes provider diagnostics and preserves whether the stored result is absent. + Recovery rule for a parked-for-review continuation: - if the issue has a real waiting target — open (non-terminal) sub-tasks or existing unresolved blockers — Paperclip converts the deliberate wait into a first-class dependency wait: it sets the issue `blocked` by those issues, keeps the original assignee, and posts a plain-language comment explaining that the task will resume automatically when its dependencies finish. The issue then self-resumes through the normal `issue_blockers_resolved` path; no recovery action or escalation owner is involved diff --git a/server/src/__tests__/agent-live-run-routes.test.ts b/server/src/__tests__/agent-live-run-routes.test.ts index 28919226bde..55d978e0798 100644 --- a/server/src/__tests__/agent-live-run-routes.test.ts +++ b/server/src/__tests__/agent-live-run-routes.test.ts @@ -1163,7 +1163,7 @@ describe("agent live run routes", () => { expect(res.status, JSON.stringify(res.body)).toBe(202); expect(res.body).toEqual(receipt); expect(mockHeartbeatService.getRun).toHaveBeenCalledWith( - failedChatRunId, + failedChatRunId, { includeExecutionEvidence: true }, ); expect( mockChatRunRetries.prepareFailedChatRunRetry, diff --git a/server/src/routes/agents.ts b/server/src/routes/agents.ts index 1fc61ee9e11..094a02a656e 100644 --- a/server/src/routes/agents.ts +++ b/server/src/routes/agents.ts @@ -5998,7 +5998,7 @@ export function agentRoutes( "An exact failed-run retry cannot override its execution context.", ); } - const failedRun = await heartbeat.getRun(req.body.failedRunId); + const failedRun = await heartbeat.getRun(req.body.failedRunId, { includeExecutionEvidence: true }); if ( !failedRun || failedRun.companyId !== agent.companyId || diff --git a/server/src/services/explicit-native-continuation.test.ts b/server/src/services/explicit-native-continuation.test.ts index cd0d3ac143a..3ecd4596b41 100644 --- a/server/src/services/explicit-native-continuation.test.ts +++ b/server/src/services/explicit-native-continuation.test.ts @@ -623,13 +623,37 @@ const support = await getEmbeddedPostgresTestSupport(); return f; } - it.each(["message", "retry"])("recovers a pre-dispatch review wait through an explicit %s", async kind => { + it("keeps provider diagnostics out of admission evidence and preserves absent results", async () => { const f = await seedCancelledReviewWait(); + const heartbeat = heartbeatService(db); + await db.update(heartbeatRuns).set({ resultJson: { + stopReason: "issue_continuation_waiting_on_review", timeoutSource: "stale_queued_run_gate", + summary: "private provider output", providerPayload: "large provider output".repeat(10000), + } }).where(eq(heartbeatRuns.id, f.sourceRunId)); + const projected = await heartbeat.getRun(f.sourceRunId, { includeExecutionEvidence: true }); + expect(projected?.resultJson).toMatchObject({ stopReason: "issue_continuation_waiting_on_review", timeoutSource: "stale_queued_run_gate" }); + expect(projected?.resultJson).not.toHaveProperty("summary"); + expect(projected?.resultJson).not.toHaveProperty("providerPayload"); + await db.update(heartbeatRuns).set({ resultJson: null }).where(eq(heartbeatRuns.id, f.sourceRunId)); + expect((await heartbeat.getRun(f.sourceRunId, { includeExecutionEvidence: true }))?.resultJson).toBeNull(); + await db.update(heartbeatRuns).set({ resultJson: { unrecognizedReceipt: true } }).where(eq(heartbeatRuns.id, f.sourceRunId)); + expect((await heartbeat.getRun(f.sourceRunId, { includeExecutionEvidence: true }))?.resultJson).not.toBeNull(); + }); + + it.each([false, true].flatMap(ascii => ["message", "retry"].map(kind => ({ ascii, kind }))))( + "recovers a pre-dispatch review wait through an explicit $kind (SQL_ASCII: $ascii)", async ({ ascii, kind }) => { + const f = await seedCancelledReviewWait(); + const heartbeat = heartbeatService(db); + if (ascii) { + const encoding = vi.spyOn(db, "execute").mockResolvedValueOnce([{ server_encoding: "SQL_ASCII" }] as never); + try { expect((await heartbeat.getRun(f.sourceRunId))?.resultJson).toBeNull(); } + finally { encoding.mockRestore(); } + } const notice = await getExecutionBlocker(db, f.companyId, f.issueId); expect(notice).toMatchObject({ canRetry: true, runError: "Waiting for review; this continuation never started." }); expect(notice?.nextAction).not.toContain("Inspect the run before sending a new message"); await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }); - const successor = await heartbeatService(db).wakeup(f.agentId, { source: kind === "retry" ? "on_demand" : "automation", triggerDetail: "manual", + const successor = await heartbeat.wakeup(f.agentId, { source: kind === "retry" ? "on_demand" : "automation", triggerDetail: "manual", reason: kind === "retry" ? "retry_failed_run" : "issue_commented", ...(kind === "retry" ? { failedRunId: f.sourceRunId } : {}), requestedByActorType: "user", requestedByActorId: "board", payload: { issueId: f.issueId, ...(kind === "message" ? { commentId: f.commentId } : {}) }, @@ -641,13 +665,19 @@ const support = await getEmbeddedPostgresTestSupport(); expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId)))[0].status).toBe("cancelled"); }); - it("reconsiders saved input after a pre-dispatch review wait exactly once", async () => { + it.each([false, true])("reconsiders saved input after a pre-dispatch review wait exactly once (SQL_ASCII: %s)", async ascii => { const f = await seedCancelledReviewWait(); + const heartbeat = heartbeatService(db); + if (ascii) { + const encoding = vi.spyOn(db, "execute").mockResolvedValueOnce([{ server_encoding: "SQL_ASCII" }] as never); + try { expect((await heartbeat.getRun(f.sourceRunId))?.resultJson).toBeNull(); } + finally { encoding.mockRestore(); } + } await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }); const leaseId = randomUUID(); await db.insert(environmentLeases).values({ id: leaseId, companyId: f.companyId, heartbeatRunId: f.sourceRunId, provider: "local", status: "pending_cleanup", cleanupStatus: "failed" }); - await heartbeatService(db).wakeup(f.agentId, { source: "automation", reason: "issue_commented", + await heartbeat.wakeup(f.agentId, { source: "automation", reason: "issue_commented", requestedByActorType: "user", requestedByActorId: "board", payload: { issueId: f.issueId, commentId: f.commentId }, contextSnapshot: { issueId: f.issueId, wakeCommentId: f.commentId } }); const [waiting] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId)); @@ -655,7 +685,7 @@ const support = await getEmbeddedPostgresTestSupport(); await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "succeeded" }) .where(eq(environmentLeases.id, leaseId)); await db.update(agentWakeupRequests).set({ updatedAt: new Date(0) }).where(eq(agentWakeupRequests.id, waiting.id)); - await Promise.all([heartbeatService(db).resumeExecutionWaitComments(), heartbeatService(db).resumeExecutionWaitComments()]); + await Promise.all([heartbeat.resumeExecutionWaitComments(), heartbeat.resumeExecutionWaitComments()]); const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued"))); expect(successors).toHaveLength(1); expect(successors[0].contextSnapshot).toMatchObject({ wakeCommentIds: [f.commentId], diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 9ca39eb35cd..4c7f3a281c6 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -3558,6 +3558,27 @@ const heartbeatRunSafeResultJsonColumn = sql | null>` end `.as("resultJson"); +// Execution admission needs retained server receipts even when presentation +// projection omits resultJson (SQL_ASCII or oversized provider output). Select +// only the evidence used by eligibility; never retrieve provider diagnostics. +const heartbeatRunExecutionEvidenceColumn = sql | null>` + case when ${heartbeatRuns.resultJson} is null then null else jsonb_build_object( + 'startupCancellation', ${heartbeatRuns.resultJson} -> 'startupCancellation', + 'startupPreparationSettledAt', ${heartbeatRuns.resultJson} -> 'startupPreparationSettledAt', + 'stopReason', ${heartbeatRuns.resultJson} -> 'stopReason', + 'timeoutSource', ${heartbeatRuns.resultJson} -> 'timeoutSource', + 'workspaceRestoreFailure', ${heartbeatRuns.resultJson} -> 'workspaceRestoreFailure', + 'executionCancellation', ${heartbeatRuns.resultJson} -> 'executionCancellation', + 'nativeCancellation', ${heartbeatRuns.resultJson} -> 'nativeCancellation', + 'cancelledByActorType', ${heartbeatRuns.resultJson} -> 'cancelledByActorType', + 'cancelledByUserId', ${heartbeatRuns.resultJson} -> 'cancelledByUserId', + 'conversationContinuation', ${heartbeatRuns.resultJson} -> 'conversationContinuation', + 'cancellation', ${heartbeatRuns.resultJson} -> 'cancellation', + 'acpToolInventoryComplete', ${heartbeatRuns.resultJson} -> 'acpToolInventoryComplete', + 'acpPendingToolCount', ${heartbeatRuns.resultJson} -> 'acpPendingToolCount' + ) end +`.as("resultJson"); + const heartbeatRunSafeColumns = { ...getTableColumns(heartbeatRuns), processGroupId: heartbeatRunProcessGroupIdColumn, @@ -10384,7 +10405,7 @@ export function heartbeatService( !(await remoteExecutionHasStopped(db, run.companyId, run.id))) return; const issueId = run.nativeIssueId ?? (typeof run.contextSnapshot?.issueId === "string" ? run.contextSnapshot.issueId : null); if (!issueId) return; - const currentRun = run.runtimeMode === "native" ? await getRun(run.id) : null; + const currentRun = run.runtimeMode === "native" ? await getRun(run.id, { includeExecutionEvidence: true }) : null; const [coordinator] = currentRun ? await db.select({ phase: nativeRunFinalizations.phase, leaseOwner: nativeRunFinalizations.leaseOwner }).from(nativeRunFinalizations).where(and( eq(nativeRunFinalizations.companyId, run.companyId), eq(nativeRunFinalizations.runId, run.id), @@ -10399,7 +10420,7 @@ export function heartbeatService( await acknowledgedNativeStopExecutionHasStopped(db, currentRun) && !(await getExecutionBlocker(db, run.companyId, issueId)); const legacyContinuation = run.runtimeMode === "legacy" && - hasConversationContinuationPolicy((await getRun(run.id))?.resultJson) && + hasConversationContinuationPolicy((await getRun(run.id, { includeExecutionEvidence: true }))?.resultJson) && !(await getExecutionBlocker(db, run.companyId, issueId)); if (run.runtimeMode !== "native" && run.runtimeMode !== "legacy") return; const pending = await db.select().from(agentWakeupRequests).where(and( @@ -10630,7 +10651,7 @@ export function heartbeatService( const blocker = await getExecutionBlocker(db, wake.companyId, issueId); const sourceId = blocker?.runId; if (!sourceId || !isUuidLike(sourceId)) continue; - const run = await getRun(sourceId); + const run = await getRun(sourceId, { includeExecutionEvidence: true }); if (!run || run.companyId !== wake.companyId || run.agentId !== wake.agentId) continue; if (canContinueCancelledRun(run)) { await resumeSavedLegacyComments(wake.companyId, wake.id).catch(err => { @@ -10731,18 +10752,17 @@ export function heartbeatService( async function getRun( runId: string, - opts?: { unsafeFullResultJson?: boolean }, + opts?: { unsafeFullResultJson?: boolean; includeExecutionEvidence?: boolean }, ) { const safeForLegacyEncoding = !opts?.unsafeFullResultJson && (await hasUnsafeTextProjectionDatabase()); + const columns = opts?.unsafeFullResultJson + ? getTableColumns(heartbeatRuns) + : safeForLegacyEncoding ? heartbeatRunSqlAsciiSafeColumns : heartbeatRunSafeColumns; return db - .select( - opts?.unsafeFullResultJson - ? getTableColumns(heartbeatRuns) - : safeForLegacyEncoding - ? heartbeatRunSqlAsciiSafeColumns - : heartbeatRunSafeColumns, - ) + .select(opts?.includeExecutionEvidence + ? { ...columns, resultJson: heartbeatRunExecutionEvidenceColumn } + : columns) .from(heartbeatRuns) .where(eq(heartbeatRuns.id, runId)) .then((rows) => rows[0] ?? null); @@ -26905,7 +26925,7 @@ export function heartbeatService( } if (opts.failedRunId) { - const failed = await getRun(opts.failedRunId); + const failed = await getRun(opts.failedRunId, { includeExecutionEvidence: true }); if (opts.requestedByActorType !== "user" || !opts.requestedByActorId || reason !== "retry_failed_run" || source !== "on_demand" || triggerDetail !== "manual" || !failed || failed.companyId !== agent.companyId || failed.agentId !== agentId || From dd868ed125cd709506dd9b29fca640a44d580501 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 3 Oct 2026 06:20:38 -0500 Subject: [PATCH 06/37] fix(runner): share native completion tool guidance (#14961) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip manages AI agents and their work. > - Native Runner agents report completion through finish and block tools. > - The providers receive different descriptions for those tools. > - Completion guidance belongs with the tools that enforce the result. > - This pull request shares the descriptions and refreshes retained catalogs. > - A separate native suite checks completion and blocking on production defaults. > - Legacy agents retain their separate skill and API paths. ## Linked Issues or Issue Description Refs: #14920, #14948, #14985. **Current behavior** Native Codex and MCP bridges describe finish and block differently. Retained provider sessions can keep old descriptions. **Proposed behavior** Native providers receive the same finish and block descriptions. The descriptions cover report selection, validation feedback, returned outcomes, approval gates and final-answer timing. Retained native sessions refresh from v13 to v14. **Reason and benefit** Put the completion procedure next to its native tool. Preserve stock base instructions, schemas, permissions and terminal semantics. This PR now stands alone on master. It contains no reduced manual, shared prompt or operational-skill changes from #14948. ## What Changed - Add canonical native finish and block descriptions. Use them in direct Codex and both native MCP bridges. - Advance the native tool contract to v14. Cover old-v13 refresh without replacing task identity or prior history. - Check authenticated tool catalogs, provider start/resume frames and serialized daemon catalogs. - Add an independent, explicit-only native completion suite. Preserve the original assigned-skill durable-document journey. Pair it with a concrete whole-task blocker across Codex, ACPX Claude and OpenCode. - Verify the actual public production default bundle and budgets before execution. Require independent durable disposition, native result/terminal receipts and observable provider-final ordering. - Correct the blocker browser oracle to accept the requested explanation. Keep exact owner/action/scope checks. Calibrate positive, missing and contradictory replies. - Preserve only actual `tool_call` terminal names (`paperclip_finish` / `paperclip_block`) in the native compatibility run-log projection. Require the same named call ID through its finishing result; retain all other redaction boundaries. - Admit verified hosted shallow checkout/build hydration and bind the selected runnerd to exact source/archive/binary provenance. Hosted cells truthfully reuse the existing trusted build; local admission executes Rust calibration. Forward only public source/run identifiers through both launcher preflight subprocess paths. - Enforce single attempts in the launcher for opted-in fixtures. Keep ordinary retry policy unchanged. Run exact-source, credential-free admission before credential loading. ## Verification - Frozen candidate: `d6e59e4712a3158ab4cd7d58deff1389b4578c21`, based on master `59c07ede72dc08b8aba149a01cc11e0b7a204621`; historical descriptions: `e74ed61a69fbdd8b3a8f15dd6456bc3140246e33`. Exactly the five original native production files and six unit tests differ. Both carry identical corrected fixtures, strict named finishing-call grader, closed compatibility carrier and admission. Defaults, profiles/models/auth/permissions and manifest bytes match. - Actual launcher `prepareNativeCompletionPreflight` → `verifyNativeCompletionPreflight` admission passes on both exact refs with zero providers: candidate 132 / historical 127 selected TypeScript assertions, 128 Node calibrations and one Rust normalization calibration each; E2E typecheck, manifest checks, selected binary provenance and six-cell discovery pass. Each has 257 explicitly skipped unrelated assertions, not coverage. The credential-free environment calibration exercises both real prepare/verify subprocess options with public hosted identifiers and rejects credential/ambient overrides. Complete actual launcher prepare→verify also passes on both frozen refs with explicitly synthetic hosted metadata/verified archives, separately labeled as calibration rather than a trusted GitHub run. Exact framed provenance parsing and mock source identity are calibrated without relaxing the real verifier. - [Complete matched qualification report](https://github.com/paperclipai/paperclip/blob/532066620b88e8731a5211fbe1cbc48ce8c7dd1a/doc/plans/2026-10-02-native-completion-master-qualification.md), [immutable manifest](https://github.com/paperclipai/paperclip/blob/532066620b88e8731a5211fbe1cbc48ce8c7dd1a/doc/plans/2026-10-02-native-completion-calibrated-manifest.json) and [closed retained audit/hashes](https://github.com/paperclipai/paperclip/blob/532066620b88e8731a5211fbe1cbc48ce8c7dd1a/doc/plans/2026-10-02-native-completion-calibrated-results/comparison.json) are inspectable. All six candidate cells pass; historical descriptions pass five. Paired outcomes: **zero new failures, one new pass (Codex blocker), five unchanged passes, zero pending pairs**. [Candidate campaign](https://github.com/paperclipai/paperclip/actions/runs/37098728980) and [historical campaign](https://github.com/paperclipai/paperclip/actions/runs/37098815696) each execute six original attempt-1 native runs, with no campaign retry and successful cleanup. Their trusted workflow revision is `215586d127e97c9301d86e769a39a15c13298ca2`, separate from measured source. [Candidate public HTML](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37098728980-1/index.html) and [historical public HTML](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37098815696-1/index.html) retain declared screenshots. - Independent candidate evidence agrees with all original grades: 51 strict native checks, 12 served-default/budget checks and 21 original skill/document checks pass. The historical Codex blocker saves the correct whole-task blocker but omits the required marker from its actual provider final and identical saved reply. This is not semantic-summary fallback. Its original browser/matcher failure stays retained; the additional native snapshot/grade and workspace before/after digest were never written and are not fabricated by the separate API/PRP audit. Historical Codex completion has one failed finish followed by success within the same native run; the public receipt records no failure reason. All twelve runs and their usage remain counted. Reported model-cost subtotals are $0.00421482 historical/$0.00437391 candidate; Codex/Claude zero entries have unknown billing type, actual invoices are unverified and hosted execution cost is unmetered. One matched trial supports no extra failure within these six cases, not broad statistical or coding-quality equivalence. - Initial hosted `e18c2cf9` / `459455ac` and subsequent `0a9c5a7` / `00a761b` cohorts each stopped before providers in all twelve cells. The latter failed a mocked-receipt unit test under ambient hosted metadata; all source/build proofs passed. [All twelve later setup receipts](https://github.com/paperclipai/paperclip/blob/402ee94c52273ad58de355ae9a7d562dd22f8101/doc/plans/2026-10-02-native-completion-qualified-hosted-setup.json) are retained. [Exact failed setup receipts](https://github.com/paperclipai/paperclip/blob/27653eb1a8f8ce839776d760f4563f672e5a706c/doc/plans/2026-10-02-native-completion-master-hosted-setup.json) and the original manifest remain intact. Local sandbox-denied loopback and stale anchor-expectation attempts are retained separately; unchanged appropriate assertions were corrected/admitted before paid dispatch. Old anonymous OpenCode streams are not assigned inferred tool names or retroactively passed. - Full provider-free E2E support previously passed 927 tests in 67 files. Exact-head d6 normal CI run `37098409915`, attempt 1 passes full repository typecheck/build/tests, Runner Rust/static checks, all browser shards/aggregate and canary: 52 check-runs pass, four intentional skips, Snyk passes. Fresh Greptile check `111132956342` is 5/5 with zero unresolved threads. Source-specific deterministic tests do not substitute for the bounded live comparison. - Earlier native source `9138f570c341c251a5727c32d6615ce238bc8e03` is archived. Its [complete reduced-manual-context report](https://github.com/paperclipai/paperclip/blob/9138f570c341c251a5727c32d6615ce238bc8e03/doc/plans/2026-10-02-native-completion-live-comparison.md) remains intact, including original failures, grader limits and provider-free replay. It is not current-master-context qualification. ## Risks Changed tool text can change model behavior. The completed six-pair qualification shows no extra failing outcomes in this bounded trial; other tasks and repeated-run variance remain unmeasured. Observable final ordering does not prove provider feedback consumption. Public evidence can fail closed if a provider does not expose the required result sequence. This slice does not remove native fixed prompts or measure general coding quality. No database, schema, permission or legacy completion changes occur. ## Model Used OpenAI Codex, GPT-6 family, with code inspection, execution and tool use. The exact deployment ID and context-window size are not exposed in this session. They are unavailable rather than inferred from the model menu. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../crates/runner-core/src/provider_events.rs | 46 +++ .../src/contracts/completion-result.test.ts | 4 + .../src/contracts/completion-result.ts | 5 + .../codex-app-server-driver.lifecycle.test.ts | 10 + .../src/drivers/codex/codex-driver-values.ts | 10 +- .../src/drivers/opencode/mcp-bridge.test.ts | 10 +- .../src/drivers/opencode/mcp-bridge.ts | 6 +- .../src/drivers/runner-tool-bridge.test.ts | 10 +- .../src/drivers/runner-tool-bridge.ts | 6 +- .../src/live/runnerd-codex-transport.test.ts | 17 +- .../native-session-resume.test.ts | 6 + .../native-runtime/native-session-resume.ts | 2 +- tests/runner-e2e/NATIVE-COMPLETION.md | 70 +++++ tests/runner-e2e/README.md | 2 + tests/runner-e2e/automatic-retry.test.ts | 57 ++++ tests/runner-e2e/automatic-retry.ts | 29 ++ tests/runner-e2e/catalog.test.ts | 6 +- tests/runner-e2e/catalog.ts | 12 + .../terminal-tool-carrier.json | 55 ++++ tests/runner-e2e/launch.ts | 61 ++-- tests/runner-e2e/live-fixtures.ts | 3 +- .../runner-e2e/native-blocker-visible.spec.ts | 28 ++ tests/runner-e2e/native-blocker-visible.ts | 18 ++ .../native-completion-admission.test.ts | 108 +++++++ .../runner-e2e/native-completion-admission.ts | 60 ++++ tests/runner-e2e/native-completion-cases.ts | 47 +++ tests/runner-e2e/native-completion-checks.mjs | 297 ++++++++++++++++++ .../native-completion-checks.test.mjs | 182 +++++++++++ .../native-completion-defaults.test.ts | 74 +++++ .../runner-e2e/native-completion-defaults.ts | 70 +++++ .../native-completion-git-source.mjs | 166 ++++++++++ .../native-completion-git-source.test.mjs | 174 ++++++++++ .../native-completion-scoring.test.ts | 118 +++++++ tests/runner-e2e/native-completion-scoring.ts | 135 ++++++++ .../native-completion-source-contract.mjs | 155 +++++++++ ...native-completion-source-contract.test.mjs | 74 +++++ tests/runner-e2e/playwright-support.config.ts | 2 +- tests/runner-e2e/runner.spec.ts | 50 +++ tests/runner-e2e/types.ts | 2 + 39 files changed, 2135 insertions(+), 52 deletions(-) create mode 100644 tests/runner-e2e/NATIVE-COMPLETION.md create mode 100644 tests/runner-e2e/automatic-retry.test.ts create mode 100644 tests/runner-e2e/automatic-retry.ts create mode 100644 tests/runner-e2e/fixtures/native-completion/terminal-tool-carrier.json create mode 100644 tests/runner-e2e/native-blocker-visible.spec.ts create mode 100644 tests/runner-e2e/native-blocker-visible.ts create mode 100644 tests/runner-e2e/native-completion-admission.test.ts create mode 100644 tests/runner-e2e/native-completion-admission.ts create mode 100644 tests/runner-e2e/native-completion-cases.ts create mode 100644 tests/runner-e2e/native-completion-checks.mjs create mode 100644 tests/runner-e2e/native-completion-checks.test.mjs create mode 100644 tests/runner-e2e/native-completion-defaults.test.ts create mode 100644 tests/runner-e2e/native-completion-defaults.ts create mode 100644 tests/runner-e2e/native-completion-git-source.mjs create mode 100644 tests/runner-e2e/native-completion-git-source.test.mjs create mode 100644 tests/runner-e2e/native-completion-scoring.test.ts create mode 100644 tests/runner-e2e/native-completion-scoring.ts create mode 100644 tests/runner-e2e/native-completion-source-contract.mjs create mode 100644 tests/runner-e2e/native-completion-source-contract.test.mjs diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/provider_events.rs b/packages/paperclip-runner/runner/crates/runner-core/src/provider_events.rs index 759f5ef4c1b..be447aeb7ce 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/provider_events.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/provider_events.rs @@ -928,6 +928,18 @@ pub fn normalize_codex_notification(method: &str, params: &Value) -> Vec { ); expect(finish.inputSchema).toEqual(PRP_COMPLETION_RESULT_PROVIDER_INPUT_SCHEMA); expect(block.inputSchema).toEqual(PRP_BLOCK_RESULT_PROVIDER_INPUT_SCHEMA); + expect(finish.description).toBe(PRP_COMPLETION_TOOL_DESCRIPTION); + expect(block.description).toBe(PRP_BLOCK_TOOL_DESCRIPTION); }); it("allows only a response-wake continuation when completion explicitly yields", () => { diff --git a/packages/paperclip-runner/src/contracts/completion-result.ts b/packages/paperclip-runner/src/contracts/completion-result.ts index dec93a2f1de..b727a942965 100644 --- a/packages/paperclip-runner/src/contracts/completion-result.ts +++ b/packages/paperclip-runner/src/contracts/completion-result.ts @@ -1,6 +1,11 @@ /** Provider-neutral semantic completion tools and their strict model-facing schemas. */ export const PRP_COMPLETION_TOOL_NAME = "paperclip_finish" as const; export const PRP_BLOCK_TOOL_NAME = "paperclip_block" as const; +/** Native Runner guidance; legacy adapters use their own skill/API completion paths. */ +export const PRP_COMPLETION_TOOL_DESCRIPTION = + "Report completed work (done), work requiring review (needs_review), or an explicit wait for the next response (yielded with response_wake). Use the current completion contract and supporting evidence. If rejected, correct the report and retry. After acceptance, read the returned outcome; do not claim completion while gated. Explain any required approval with its supplied link and action, then write the final response and end the turn without further tool calls."; +export const PRP_BLOCK_TOOL_DESCRIPTION = + "Report work that cannot continue because of a concrete blocker. Identify the blocker, its owner, and the action needed to unblock it; use the current completion contract and supporting evidence. If rejected, correct the report and retry. After acceptance, read the returned outcome, explain the blocker and any required action in the final response, and end the turn without further tool calls."; export const PRP_SEMANTIC_TOOL_NAMES = [ PRP_COMPLETION_TOOL_NAME, PRP_BLOCK_TOOL_NAME, diff --git a/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.lifecycle.test.ts b/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.lifecycle.test.ts index 81e8a3def93..c5ade4c19c2 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.lifecycle.test.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.lifecycle.test.ts @@ -1,3 +1,9 @@ +import { + PRP_BLOCK_RESULT_PROVIDER_INPUT_SCHEMA, + PRP_BLOCK_TOOL_DESCRIPTION, + PRP_COMPLETION_RESULT_PROVIDER_INPUT_SCHEMA, + PRP_COMPLETION_TOOL_DESCRIPTION, +} from "../../contracts/completion-result.js"; import { CODEX_BLOCK_RESULT_OUTPUT_SCHEMA, CODEX_INVALID_REQUEST, @@ -540,6 +546,10 @@ describe("Codex app-server Codex driver", () => { expect(started.developerInstructions).toBe(conversationMode === "direct" ? undefined : baseInstructions); expect(resumed.developerInstructions).toBe(conversationMode === "direct" ? "" : baseInstructions); expect(resumed.dynamicTools).toEqual(started.dynamicTools); + expect(started.dynamicTools).toEqual(conversationMode === "direct" ? [] : [ + { name: "paperclip_finish", description: PRP_COMPLETION_TOOL_DESCRIPTION, inputSchema: PRP_COMPLETION_RESULT_PROVIDER_INPUT_SCHEMA }, + { name: "paperclip_block", description: PRP_BLOCK_TOOL_DESCRIPTION, inputSchema: PRP_BLOCK_RESULT_PROVIDER_INPUT_SCHEMA }, + ]); expect(resumed.config).toEqual(started.config); await recovered.session?.close({ reason: "verified additive recovery" }); }); diff --git a/packages/paperclip-runner/src/drivers/codex/codex-driver-values.ts b/packages/paperclip-runner/src/drivers/codex/codex-driver-values.ts index 68c32a3c9a6..5d4b9c9c7f2 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-driver-values.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-driver-values.ts @@ -1,5 +1,9 @@ import type { PersistedHarnessProviderIdentity } from "../../contracts/harness-driver.js"; import type { NativeUserMessage } from "../../contracts/types.js"; +import { + PRP_BLOCK_TOOL_DESCRIPTION, + PRP_COMPLETION_TOOL_DESCRIPTION, +} from "../../contracts/completion-result.js"; import { CODEX_BLOCK_RESULT_PROVIDER_INPUT_SCHEMA, CODEX_BLOCK_TOOL_NAME, @@ -232,8 +236,7 @@ export function differingJsonPaths( function finishToolSpec(): Record { return { name: CODEX_COMPLETION_TOOL_NAME, - description: - "Return the one semantic completion result for this task, including an explicit response_wake yield when waiting for the next response.", + description: PRP_COMPLETION_TOOL_DESCRIPTION, inputSchema: CODEX_RESULT_PROVIDER_INPUT_SCHEMA, }; } @@ -241,8 +244,7 @@ function finishToolSpec(): Record { function blockToolSpec(): Record { return { name: CODEX_BLOCK_TOOL_NAME, - description: - "Return the one semantic result when the task cannot continue.", + description: PRP_BLOCK_TOOL_DESCRIPTION, inputSchema: CODEX_BLOCK_RESULT_PROVIDER_INPUT_SCHEMA, }; } diff --git a/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.test.ts b/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.test.ts index 6539a14b52a..07681400ddd 100644 --- a/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.test.ts +++ b/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.test.ts @@ -1,4 +1,10 @@ import { afterEach, describe, expect, it, vi } from "vitest"; +import { + PRP_BLOCK_RESULT_PROVIDER_INPUT_SCHEMA, + PRP_BLOCK_TOOL_DESCRIPTION, + PRP_COMPLETION_RESULT_PROVIDER_INPUT_SCHEMA, + PRP_COMPLETION_TOOL_DESCRIPTION, +} from "../../contracts/completion-result.js"; import { canonicalOpenCodeMcpToolName, startOpenCodeMcpBridge, type OpenCodeMcpBridge } from "./mcp-bridge.js"; @@ -38,8 +44,8 @@ describe("OpenCode MCP bridge", () => { expect(await listed.json()).toMatchObject({ result: { tools: [ { name: "documents.read" }, - { name: "paperclip_finish" }, - { name: "paperclip_block" }, + { name: "paperclip_finish", description: PRP_COMPLETION_TOOL_DESCRIPTION, inputSchema: PRP_COMPLETION_RESULT_PROVIDER_INPUT_SCHEMA }, + { name: "paperclip_block", description: PRP_BLOCK_TOOL_DESCRIPTION, inputSchema: PRP_BLOCK_RESULT_PROVIDER_INPUT_SCHEMA }, ] }, }); }); diff --git a/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts b/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts index 79ce6770375..1bf1555dd16 100644 --- a/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts +++ b/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts @@ -4,8 +4,10 @@ import { Ajv2020, type ValidateFunction } from "ajv/dist/2020.js"; import { PRP_BLOCK_RESULT_PROVIDER_INPUT_SCHEMA, + PRP_BLOCK_TOOL_DESCRIPTION, PRP_BLOCK_TOOL_NAME, PRP_COMPLETION_RESULT_PROVIDER_INPUT_SCHEMA, + PRP_COMPLETION_TOOL_DESCRIPTION, PRP_COMPLETION_TOOL_NAME, } from "../../contracts/completion-result.js"; @@ -117,8 +119,8 @@ function normalizeTools( seen.add(name); } if (includeTerminalTools) for (const terminal of [ - { name: PRP_COMPLETION_TOOL_NAME, description: "Return the semantic completion result.", inputSchema: PRP_COMPLETION_RESULT_PROVIDER_INPUT_SCHEMA }, - { name: PRP_BLOCK_TOOL_NAME, description: "Return the semantic blocked result.", inputSchema: PRP_BLOCK_RESULT_PROVIDER_INPUT_SCHEMA }, + { name: PRP_COMPLETION_TOOL_NAME, description: PRP_COMPLETION_TOOL_DESCRIPTION, inputSchema: PRP_COMPLETION_RESULT_PROVIDER_INPUT_SCHEMA }, + { name: PRP_BLOCK_TOOL_NAME, description: PRP_BLOCK_TOOL_DESCRIPTION, inputSchema: PRP_BLOCK_RESULT_PROVIDER_INPUT_SCHEMA }, ]) { if (!seen.has(terminal.name)) tools.push(terminal); } diff --git a/packages/paperclip-runner/src/drivers/runner-tool-bridge.test.ts b/packages/paperclip-runner/src/drivers/runner-tool-bridge.test.ts index b552adcb28a..5bf87169d24 100644 --- a/packages/paperclip-runner/src/drivers/runner-tool-bridge.test.ts +++ b/packages/paperclip-runner/src/drivers/runner-tool-bridge.test.ts @@ -1,6 +1,12 @@ import { connect } from "node:net"; import { afterEach, describe, expect, it, vi } from "vitest"; +import { + PRP_BLOCK_RESULT_PROVIDER_INPUT_SCHEMA, + PRP_BLOCK_TOOL_DESCRIPTION, + PRP_COMPLETION_RESULT_PROVIDER_INPUT_SCHEMA, + PRP_COMPLETION_TOOL_DESCRIPTION, +} from "../contracts/completion-result.js"; import { canonicalRunnerToolName, @@ -48,8 +54,8 @@ describe("runner semantic MCP bridge", () => { result: { tools: [ { name: "documents.read" }, - { name: "paperclip_finish" }, - { name: "paperclip_block" }, + { name: "paperclip_finish", description: PRP_COMPLETION_TOOL_DESCRIPTION, inputSchema: PRP_COMPLETION_RESULT_PROVIDER_INPUT_SCHEMA }, + { name: "paperclip_block", description: PRP_BLOCK_TOOL_DESCRIPTION, inputSchema: PRP_BLOCK_RESULT_PROVIDER_INPUT_SCHEMA }, ], }, }); diff --git a/packages/paperclip-runner/src/drivers/runner-tool-bridge.ts b/packages/paperclip-runner/src/drivers/runner-tool-bridge.ts index e59535171be..c8207c71833 100644 --- a/packages/paperclip-runner/src/drivers/runner-tool-bridge.ts +++ b/packages/paperclip-runner/src/drivers/runner-tool-bridge.ts @@ -10,8 +10,10 @@ import { Ajv2020, type ValidateFunction } from "ajv/dist/2020.js"; import { PRP_BLOCK_RESULT_PROVIDER_INPUT_SCHEMA, + PRP_BLOCK_TOOL_DESCRIPTION, PRP_BLOCK_TOOL_NAME, PRP_COMPLETION_RESULT_PROVIDER_INPUT_SCHEMA, + PRP_COMPLETION_TOOL_DESCRIPTION, PRP_COMPLETION_TOOL_NAME, } from "../contracts/completion-result.js"; @@ -70,12 +72,12 @@ const MAX_RESULT_CHUNK_BYTES = 64 * 1024; const RESERVED_TOOLS: readonly RunnerToolDefinition[] = [ { name: PRP_COMPLETION_TOOL_NAME, - description: "Return the semantic completion result.", + description: PRP_COMPLETION_TOOL_DESCRIPTION, inputSchema: PRP_COMPLETION_RESULT_PROVIDER_INPUT_SCHEMA, }, { name: PRP_BLOCK_TOOL_NAME, - description: "Return the semantic blocked result.", + description: PRP_BLOCK_TOOL_DESCRIPTION, inputSchema: PRP_BLOCK_RESULT_PROVIDER_INPUT_SCHEMA, }, ]; diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts index 8ab547d2211..50544213ea3 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts @@ -54,6 +54,10 @@ import { CODEX_SKILLLESS_BASE_INSTRUCTIONS, createCodexTaskEnvelope, } from "../contracts/codex.js"; +import { + PRP_BLOCK_TOOL_DESCRIPTION, + PRP_COMPLETION_TOOL_DESCRIPTION, +} from "../contracts/completion-result.js"; import { CodexAppServerDriver, codexSemanticToolSpecs, @@ -1336,14 +1340,19 @@ it("includes ACPX terminal tools in the authenticated bridge catalog", () => { }); }); -it("preserves answer and internal wait descriptions in the serialized native tool catalog", () => { +it.each(["codex", "opencode", "claude_managed", "aws_agentcore", "acpx"] as const)("preserves answer and internal wait descriptions in the serialized native %s tool catalog", (provider) => { const catalog = JSON.parse( - JSON.stringify(authorizedToolSetForProvider("codex", codexSemanticToolSpecs())), + JSON.stringify(authorizedToolSetForProvider(provider, codexSemanticToolSpecs())), ); const finish = catalog.operations.find( (operation: { operationId: string }) => operation.operationId === "paperclip_finish", ); + const block = catalog.operations.find( + (operation: { operationId: string }) => operation.operationId === "paperclip_block", + ); + expect(finish.description).toBe(PRP_COMPLETION_TOOL_DESCRIPTION); + expect(block.description).toBe(PRP_BLOCK_TOOL_DESCRIPTION); expect(finish.inputSchema.properties.summary.description).toContain( "complete user-facing answer", ); @@ -4154,7 +4163,9 @@ it("captures exact provider frames and correlates Rust and TypeScript interpreta ); const tracePath = join(traceDirectory, "trace.ndjson"); const bundle = createCapabilityRunnerdCodexTransport({ - runnerBinary: defaultCapabilityRunnerdBinary(), + // Qualification builds a debug daemon for this exact source. Its selected + // binary must win over any separately staged product/runtime artifact. + runnerBinary: process.env.PAPERCLIP_STOCK_PREFLIGHT_RUNNERD ?? defaultCapabilityRunnerdBinary(), codexCommand: fakeCodex, codexArgs: fakeCodexArgs(traceDirectory, "--structured-activity"), stateDirectory: join(traceDirectory, "state"), diff --git a/server/src/services/native-runtime/native-session-resume.test.ts b/server/src/services/native-runtime/native-session-resume.test.ts index 59113de9b89..a8a893987cc 100644 --- a/server/src/services/native-runtime/native-session-resume.test.ts +++ b/server/src/services/native-runtime/native-session-resume.test.ts @@ -1875,6 +1875,12 @@ describe("rebindNativeSessionCheckpoint", () => { retainedFingerprint: "sha256:5b7b302db36f7ed6686f9ea1ba70bbf79ebd7fabf86953b548d966a2bc38b648", }, + { + contract: "native completion tool guidance", + // Deployed v13 local catalog before canonical finish/block descriptions. + retainedFingerprint: + "sha256:68a51d34e091c55ee5d0d2b563153454dd727d72db16e6a27c358d342ae489c9", + }, { contract: "task-bound human-input description", // Deployed v9 still advertises the generic mock-task question description. diff --git a/server/src/services/native-runtime/native-session-resume.ts b/server/src/services/native-runtime/native-session-resume.ts index d8a31f27460..bf49bea2b84 100644 --- a/server/src/services/native-runtime/native-session-resume.ts +++ b/server/src/services/native-runtime/native-session-resume.ts @@ -23,7 +23,7 @@ export function nativeToolContractFingerprintForTarget( return `sha256:${createHash("sha256") .update( JSON.stringify({ - schema: "paperclip.native-tool-contract.v13", + schema: "paperclip.native-tool-contract.v14", executionTargetKind, advertisementPolicy: { // Direct provider threads retain declarations from thread/start. diff --git a/tests/runner-e2e/NATIVE-COMPLETION.md b/tests/runner-e2e/NATIVE-COMPLETION.md new file mode 100644 index 00000000000..ca0ac463d25 --- /dev/null +++ b/tests/runner-e2e/NATIVE-COMPLETION.md @@ -0,0 +1,70 @@ +# Native completion guidance qualification + +This explicit-only suite independently qualifies PR #14961 on master +`59c07ede72dc08b8aba149a01cc11e0b7a204621`. It does not include PR #14948's +reduced manuals, shared prompts or operational-skill changes. The earlier +comparison on reduced defaults remains historical evidence, not admission for +this context. + +The six local cells pair the original `assigned-skill-explicit-invocation` +journey and `native-blocked-report` across `runner-codex`, +`runner-acpx-claude` and `runner-opencode`. Completion keeps the original task, +installed output skill, explicit invocation and independently saved issue +document checks. Blocking checks the durable whole-task blocker, exact external +owner/action, explanatory visible reply, unchanged workspace and absence of +extra tasks, agents, documents or interactions. + +Both use the actual production default hire: no custom QA instructions bundle. +Before execution, the public bundle receipt must hash to the unchanged master +AGENTS.md, and company/agent budgets must each be 1,000 cents. Profile model, +effort, credentials, skills, permissions and provider configuration are inherited. + +The native oracle requires exactly one succeeded native run attributed to this +issue, bound complete public events, one runner semantic proposal and authoritative +control-plane acceptance/terminal. A matched finishing tool start/result with the expected native terminal name +precedes the actual provider final; no new invocation follows the proposal. The final +must be persisted as the run's agent reply. Acceptance may precede or follow the +final. This proves observable ordering and durable disposition; it does not prove +the provider consumed feedback or establish general coding-quality equivalence. + +`single_attempt` is enforced by the launcher even when hosted invocation only +supplies `--id`. Every first failure, usage and cleanup receipt is retained; no +automatic infrastructure or behavioral retry occurs. The bounded comparison is +six cells per variant, twelve expected provider turns total, with the existing +per-cell deadlines (skill: original deadline; blocker: five minutes). + +Candidate and baseline use identical executable fixtures and public oracles. +Both include the same closed OpenCode compatibility projection: only actual +`tool_call` invocations named `paperclip_finish` or `paperclip_block` retain a +nested name, paired to the result by the same call ID. Arguments, arbitrary +tool names and results are not added. Old anonymous OpenCode streams cannot +retroactively satisfy this stricter identity check. +Only the five native description/fingerprint source files and their six +variant-specific unit tests may differ. The historical variant restores those +eleven files from master, including v13; candidate uses the archived native change +and v14. Source-contract hashes reject mixed revisions and changes to master +defaults, auth/runtime context or shared instructions. Derived capability metadata +is independently checked; it currently requires no regeneration in either variant. + +Credential-free admission runs before local environment/credential loading: + +```sh +node tests/runner-e2e/native-completion-checks.mjs --output-dir=/absolute/report/path +node tests/runner-e2e/native-completion-checks.mjs --verify=/absolute/report/path/preflight.json +pnpm test:e2e:runner -- --list --suite native-completion +``` + +Admission requires a committed clean exact source, variant-appropriate schema, +tool bridge/catalog and resume tests, independent oracle/default/retry calibrations, +original context-integrity calibrations, typecheck, manifest checks and exact +six-cell discovery. The Rust carrier calibration and built runner binary digest +are retained with source/build provenance. Generic stock-suite gates remain unchanged. Paid hosted runs +must dispatch the trusted default-branch workflow to distinct immutable target +branches using the exact six IDs, preserving authorization and secret boundaries. +The initial twelve hosted cells stopped at source admission with zero provider +calls. Their receipts and frozen refs remain separate from the recovery comparison. +Hosted depth-one checkouts use only the declared variant-specific immediate +parent proof when full ancestry is unavailable. The only admitted untracked +hydration is the exact build archive/checksum pair after checksum verification; +extra files, symlinks or source dirt remain failures. +Live results are pending until retained cell evidence is published. diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index a9aa8109009..68eb10ae516 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -422,6 +422,8 @@ pnpm test:e2e:runner -- --list --suite agent-chat-hardening pnpm test:e2e:runner -- --id agent-chat-hardening.runner-codex.local.stop-startup-new-resume ``` +The independent, explicit-only `native-completion` suite qualifies native finish/block descriptions on unchanged master defaults. It preserves the original assigned-skill document journey and pairs it with whole-task blocking across three native profiles, with enforced single attempts. See [NATIVE-COMPLETION.md](NATIVE-COMPLETION.md) for admission, provenance and limits. + `context-integrity` is an explicit-only local suite with two bounded cases across ten listed legacy/native profiles (20 cells). Six cells are pending-prerequisite profiles and are listed for discovery but rejected before provider credentials are diff --git a/tests/runner-e2e/automatic-retry.test.ts b/tests/runner-e2e/automatic-retry.test.ts new file mode 100644 index 00000000000..eca2fc7c1f4 --- /dev/null +++ b/tests/runner-e2e/automatic-retry.test.ts @@ -0,0 +1,57 @@ +import { describe, expect, it } from "vitest"; +import { effectiveAutomaticRetryLimit, executeWithAutomaticRetry } from "./automatic-retry.js"; +import type { FailureClass } from "./types.js"; + +const retryableClasses = ["transient_infrastructure", "provider_variance"] as const; +const allClasses: FailureClass[] = [...retryableClasses, "candidate_failure", "permanent_infrastructure", "secret_leak", "cleanup_failure"]; +const options = { maxAutomaticRetries: 1, ui: false, debug: false }; + +describe("explicit single-attempt fixture policy", () => { + for (const failureClass of allClasses) it(`retains the first ${failureClass} failure without a second attempt`, async () => { + const observedAttempts: number[] = [], retryNotifications: unknown[] = []; + const first = { status: "failed" as const, failureClass, attempt: 1, runIds: ["first-run"], cost: 3.25 }; + const result = await executeWithAutomaticRetry({ task: { automaticRetryPolicy: "single_attempt" }, options, + qualificationCandidate: false, cancelled: () => false, onRetry: result => retryNotifications.push(result), + runAttempt: async attempt => { observedAttempts.push(attempt); return first; } }); + expect(observedAttempts).toEqual([1]); + expect(retryNotifications).toEqual([]); + expect(result).toBe(first); + expect(result.runIds).toEqual(["first-run"]); + expect(result.cost).toBe(3.25); + }); + it("overrides any requested automatic retry count only for opted-in fixtures", () => { + expect(effectiveAutomaticRetryLimit({ automaticRetryPolicy: "single_attempt" }, 99)).toBe(0); + expect(effectiveAutomaticRetryLimit({}, 1)).toBe(1); + expect(effectiveAutomaticRetryLimit({}, 0)).toBe(0); + }); + for (const failureClass of retryableClasses) it(`preserves the ordinary ${failureClass} retry`, async () => { + const attempts: number[] = [], notifications: unknown[] = []; + const failed = { status: "failed" as const, failureClass, attempt: 1 }; + const passed = { status: "passed" as const, attempt: 2 }; + const result = await executeWithAutomaticRetry({ task: {}, options, qualificationCandidate: false, + cancelled: () => false, onRetry: result => notifications.push(result), + runAttempt: async attempt => { attempts.push(attempt); return attempt === 1 ? failed : passed; } }); + expect(attempts).toEqual([1, 2]); + expect(notifications).toEqual([failed]); + expect(result).toBe(passed); + }); + for (const restriction of ["qualificationCandidate", "ui", "debug", "zeroRetries"] as const) + it(`preserves the existing ${restriction} restriction`, async () => { + const attempts: number[] = []; + const first = { status: "failed" as const, failureClass: "provider_variance" as const }; + const result = await executeWithAutomaticRetry({ task: {}, options: { ...options, + ui: restriction === "ui", debug: restriction === "debug", maxAutomaticRetries: restriction === "zeroRetries" ? 0 : 1 }, + qualificationCandidate: restriction === "qualificationCandidate", cancelled: () => false, + onRetry: () => { throw new Error("Unexpected retry notification"); }, + runAttempt: async attempt => { attempts.push(attempt); return first; } }); + expect(attempts).toEqual([1]); expect(result).toBe(first); + }); + it("checks cancellation before an otherwise admitted second attempt", async () => { + const attempts: number[] = []; + await expect(executeWithAutomaticRetry({ task: {}, options, qualificationCandidate: false, + cancelled: () => true, onRetry: () => { throw new Error("Unexpected retry notification"); }, + runAttempt: async attempt => { attempts.push(attempt); return { status: "failed" as const, failureClass: "provider_variance" as const }; }, + })).rejects.toThrow("campaign cancelled"); + expect(attempts).toEqual([1]); + }); +}); diff --git a/tests/runner-e2e/automatic-retry.ts b/tests/runner-e2e/automatic-retry.ts new file mode 100644 index 00000000000..78fb35abab3 --- /dev/null +++ b/tests/runner-e2e/automatic-retry.ts @@ -0,0 +1,29 @@ +import { shouldRetryFailure } from "./failure-classifier.js"; +import type { FailureClass, RunnerTaskFixture } from "./types.js"; + +type RetryTask = Pick; +interface RetryableAttempt { status: "passed" | "failed"; failureClass?: FailureClass } +interface RetryOptions { maxAutomaticRetries: number; ui: boolean; debug: boolean } + +export function effectiveAutomaticRetryLimit(task: RetryTask, configuredLimit: number): number { + return task.automaticRetryPolicy === "single_attempt" ? 0 : configuredLimit; +} + +/** Keep the original failed result when the fixture admits exactly one attempt. */ +export async function executeWithAutomaticRetry(input: { + task: RetryTask; + options: RetryOptions; + qualificationCandidate: boolean; + runAttempt(attempt: 1 | 2): Promise; + cancelled(): boolean; + onRetry(result: Result): void; +}): Promise { + const firstResult = await input.runAttempt(1); + if (input.qualificationCandidate || input.options.ui || input.options.debug || + firstResult.status !== "failed" || !firstResult.failureClass || + !shouldRetryFailure(firstResult.failureClass, + effectiveAutomaticRetryLimit(input.task, input.options.maxAutomaticRetries))) return firstResult; + if (input.cancelled()) throw new Error("Runner E2E campaign cancelled"); + input.onRetry(firstResult); + return input.runAttempt(2); +} diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index 1e2a7a6561f..49748c8a6f6 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -136,10 +136,10 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 50, 20, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, + 6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 50, 6, 20, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, ]); - expect(validateRunnerCatalog()).toHaveLength(438); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(438); + expect(validateRunnerCatalog()).toHaveLength(444); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(444); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(48); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index 15038c55de9..89301b7f111 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1,3 +1,5 @@ +import { nativeCompletionTasks, nativeCompletionDefinitionDigest } from "./native-completion-cases.js"; +import { nativeCompletionProfile, NATIVE_COMPLETION_BUDGET_CENTS } from "./native-completion-defaults.js"; import { chatConfirmationTasks } from "./chat-cases.js"; import { hiringTemplateTasks, hiringTemplateProfile, hiringTemplateDefinitionDigest } from "./hiring-template-cases.js"; import { instructionPersistenceTask } from "./instruction-persistence.js"; @@ -1193,6 +1195,16 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ .map(task => `everyday-workflows.${profile.id}.daytona.${task.id}`))], definitionMetadata: { version: 4, instructions: "production", grading: "outcome-and-invariants", scheduling: "explicit-only" }, }, + { + id: "native-completion", label: "Native completion guidance", manualOnly: true, + description: "Production-default assigned-skill document completion and concrete whole-task blocking, with independent native result/final ordering.", + groups: ["native", "local"], + profiles: runnerProfiles.filter(profile => ["runner-codex", "runner-acpx-claude", "runner-opencode"].includes(profile.id)).map(nativeCompletionProfile), + environments: [localEnvironment], tasks: nativeCompletionTasks, expectedMatrixSize: 6, + definitionMetadata: { version: 1, fixtureDigest: nativeCompletionDefinitionDigest(), instructions: "unchanged-master-production-default", + providerTurns: 6, maximumAttemptsPerCell: 1, automaticRetryPolicy: "single_attempt", budgetMonthlyCents: NATIVE_COMPLETION_BUDGET_CENTS, + grading: "original-assigned-skill-durable-document-plus-native-disposition-and-observable-final-order", scheduling: "explicit-only" }, + }, { id: "context-integrity", label: "Context Integrity", diff --git a/tests/runner-e2e/fixtures/native-completion/terminal-tool-carrier.json b/tests/runner-e2e/fixtures/native-completion/terminal-tool-carrier.json new file mode 100644 index 00000000000..5f7e5144da3 --- /dev/null +++ b/tests/runner-e2e/fixtures/native-completion/terminal-tool-carrier.json @@ -0,0 +1,55 @@ +{ + "schema": "paperclip.native-completion-terminal-carrier.fixture.v1", + "cases": [ + { + "name": "paperclip_finish", + "providerInput": { + "item": { + "id": "terminal-call", + "type": "tool_call", + "name": "paperclip_finish", + "status": "inProgress", + "arguments": { + "secret": "not-for-the-log" + } + } + }, + "normalizedPayload": { + "provider": "codex", + "itemId": "terminal-call", + "kind": "tool_call", + "status": "running", + "channel": "detail", + "text": null, + "item": { + "name": "paperclip_finish" + } + } + }, + { + "name": "paperclip_block", + "providerInput": { + "item": { + "id": "terminal-call", + "type": "tool_call", + "name": "paperclip_block", + "status": "inProgress", + "arguments": { + "secret": "not-for-the-log" + } + } + }, + "normalizedPayload": { + "provider": "codex", + "itemId": "terminal-call", + "kind": "tool_call", + "status": "running", + "channel": "detail", + "text": null, + "item": { + "name": "paperclip_block" + } + } + } + ] +} diff --git a/tests/runner-e2e/launch.ts b/tests/runner-e2e/launch.ts index ceb5927ec79..7cafc2c8f09 100644 --- a/tests/runner-e2e/launch.ts +++ b/tests/runner-e2e/launch.ts @@ -21,7 +21,8 @@ import { import { isImmutableDaytonaImage, runnerMatrix } from "./catalog.js"; import { renderRunnerE2EDashboard } from "./dashboard.js"; import { packageEvidence } from "./evidence.js"; -import { classifyFailure, shouldRetryFailure } from "./failure-classifier.js"; +import { classifyFailure } from "./failure-classifier.js"; +import { effectiveAutomaticRetryLimit, executeWithAutomaticRetry } from "./automatic-retry.js"; import { buildRunnerCampaign } from "./history.js"; import { buildRunnerE2EProcessEnvironment, @@ -50,6 +51,7 @@ import { type RunnerE2EResult, } from "./types.js"; import { assertRunnerE2EPrerequisites } from "./prerequisites.js"; +import { assertNativeCompletionSelection, prepareNativeCompletionPreflight, NATIVE_COMPLETION_PREFLIGHT_ENV } from "./native-completion-admission.js"; import { reapNewDetachedDarwinSharedMemory, snapshotDarwinSharedMemory, @@ -1014,36 +1016,19 @@ async function runExecutionWithRetry(input: { options: ReturnType; }): Promise { const { execution, campaignId, options } = input; - const [firstResult] = await runAttempt({ - executions: [execution], - attempt: 1, - campaignId, - options, + return executeWithAutomaticRetry({ + task: execution.task, options, + qualificationCandidate: execution.profile.qualificationCandidate !== undefined, + cancelled: () => cancelled, + onRetry: result => console.warn( + `Retrying ${execution.id} in a fresh isolated harness after ${result.failureClass!.replaceAll("_", " ")}`, + ), + runAttempt: async attempt => { + const [result] = await runAttempt({ executions: [execution], attempt, campaignId, options }); + if (!result) throw new Error(`No result produced for ${execution.id} attempt ${attempt}`); + return result; + }, }); - if (!firstResult) throw new Error(`No result produced for ${execution.id}`); - if ( - execution.profile.qualificationCandidate !== undefined || - options.ui || - options.debug || - firstResult.status !== "failed" || - !firstResult.failureClass || - !shouldRetryFailure(firstResult.failureClass, options.maxAutomaticRetries) - ) { - return firstResult; - } - if (cancelled) throw new Error("Runner E2E campaign cancelled"); - console.warn( - `Retrying ${execution.id} in a fresh isolated harness after ${firstResult.failureClass.replaceAll("_", " ")}`, - ); - const [retryResult] = await runAttempt({ - executions: [execution], - attempt: 2, - campaignId, - options, - }); - if (!retryResult) - throw new Error(`No retry result produced for ${execution.id}`); - return retryResult; } async function runWithConcurrency( @@ -1098,6 +1083,15 @@ async function main() { // Keep admission before local-env loading and credential checks. Pending // profiles remain discoverable, but cannot reach a provider. assertRunnerE2EPrerequisites(executions); + assertNativeCompletionSelection(executions); + let nativeCampaignId: string | undefined; + if (executions.some(execution => execution.suite.id === "native-completion")) { + nativeCampaignId = cleanId(process.env.PAPERCLIP_E2E_CAMPAIGN_ID ?? + `local-${new Date().toISOString().replace(/[:.]/g, "-")}`); + const nativeCampaignDirectory = path.join(resultsRoot, nativeCampaignId); + await mkdir(nativeCampaignDirectory, { recursive: true }); + process.env[NATIVE_COMPLETION_PREFLIGHT_ENV] = prepareNativeCompletionPreflight(nativeCampaignDirectory); + } await loadLocalEnvironment(process.env); const missingCredentials = [ @@ -1119,7 +1113,7 @@ async function main() { ); } - const campaignId = cleanId( + const campaignId = nativeCampaignId ?? cleanId( process.env.PAPERCLIP_E2E_CAMPAIGN_ID ?? `local-${new Date().toISOString().replace(/[:.]/g, "-")}`, ); @@ -1132,6 +1126,11 @@ async function main() { version: 1, maxAutomaticRetries: options.maxAutomaticRetries, retryClasses: ["transient_infrastructure", "provider_variance"], + executions: executions.map(execution => ({ + executionId: execution.id, + automaticRetryPolicy: execution.task.automaticRetryPolicy ?? "default", + maxAutomaticRetries: effectiveAutomaticRetryLimit(execution.task, options.maxAutomaticRetries), + })), }, null, 2, diff --git a/tests/runner-e2e/live-fixtures.ts b/tests/runner-e2e/live-fixtures.ts index d8a4cbeccbe..da0aa8e19c9 100644 --- a/tests/runner-e2e/live-fixtures.ts +++ b/tests/runner-e2e/live-fixtures.ts @@ -1,3 +1,4 @@ +import { NATIVE_COMPLETION_BUDGET_CENTS } from "./native-completion-defaults.js"; import path from "node:path"; import { isManagedHiringCase } from "./chat-cases.js"; import { FixtureRegistry } from "./fixture-registry.js"; @@ -135,7 +136,7 @@ export async function setupLiveFixtures(input: { return api.post("/api/companies", { name: `Runner E2E ${execution.id} ${input.executionNonce}`, description: "Ephemeral paid full-stack runner acceptance fixture", - budgetMonthlyCents: execution.suite.id === "task-titles" ? TASK_TITLE_BUDGET_CENTS : 0, + budgetMonthlyCents: execution.suite.id === "native-completion" ? NATIVE_COMPLETION_BUDGET_CENTS : execution.suite.id === "task-titles" ? TASK_TITLE_BUDGET_CENTS : 0, }); }, async teardown() { diff --git a/tests/runner-e2e/native-blocker-visible.spec.ts b/tests/runner-e2e/native-blocker-visible.spec.ts new file mode 100644 index 00000000000..2a3fed5196c --- /dev/null +++ b/tests/runner-e2e/native-blocker-visible.spec.ts @@ -0,0 +1,28 @@ +import { expect, test } from "@playwright/test"; +import { assertNativeBlockerReply } from "./native-blocker-visible.js"; + +test("accepts a visible blocker explanation rather than requiring a marker-only reply", async ({ page }) => { + await page.setContent('
Deployment is blocked.
Release Owner must Grant deployment access. BLOCKED_probe
'); + await assertNativeBlockerReply(page.getByTestId("reply"), "BLOCKED_probe", 100); +}); + +test("accepts a future access condition while the task remains blocked", async ({ page }) => { + await page.setContent('
Deployment remains blocked until access is granted. Release Owner must Grant deployment access. BLOCKED_probe
'); + await assertNativeBlockerReply(page.getByTestId("reply"), "BLOCKED_probe", 100); +}); + +for (const [name, body] of [ + ["marker only", "BLOCKED_probe"], + ["missing owner", "Blocked. Someone must Grant deployment access. BLOCKED_probe"], + ["missing action", "Blocked. Release Owner must take action. BLOCKED_probe"], + ["missing reason", "Release Owner: Grant deployment access. BLOCKED_probe"], + ["contradictory disposition", "Deployment is not blocked. Release Owner completed Grant deployment access. BLOCKED_probe"], + ["resolved blocker", "Deployment is no longer blocked. Release Owner must Grant deployment access. BLOCKED_probe"], + ["granted access", "Blocked. Access already granted. Release Owner must Grant deployment access. BLOCKED_probe"], + ["duplicate replies", "Blocked. Release Owner must Grant deployment access. BLOCKED_probe
BLOCKED_probe"], +]) { + test(`rejects ${name}`, async ({ page }) => { + await page.setContent(`
${body}
`); + await expect(assertNativeBlockerReply(page.getByTestId("reply"), "BLOCKED_probe", 100)).rejects.toThrow(); + }); +} diff --git a/tests/runner-e2e/native-blocker-visible.ts b/tests/runner-e2e/native-blocker-visible.ts new file mode 100644 index 00000000000..b9deff3d502 --- /dev/null +++ b/tests/runner-e2e/native-blocker-visible.ts @@ -0,0 +1,18 @@ +import { expect, type Locator } from "@playwright/test"; + +/** Blocker replies include an explanation; ordinary marker-only replies do not. */ +export async function assertNativeBlockerReply(agentReplies: Locator, marker: string, timeout = 30_000): Promise { + const reply = agentReplies.filter({ hasText: marker }); + await expect(reply).toHaveCount(1, { timeout }); + await expect(reply).toBeVisible({ timeout }); + await expect(reply).toContainText("Release Owner", { timeout }); + await expect(reply).toContainText("Grant deployment access", { timeout }); + await expect(reply).not.toContainText( + /\b(?:not blocked|no longer blocked|access (?:is |has been |was )?already granted|completed Grant deployment access)\b/i, + { timeout }, + ); + await expect(reply).toContainText( + /\b(?:blocked|cannot proceed|can't proceed|missing|required access|not (?:yet )?granted|awaiting|waiting|unavailable)\b/i, + { timeout }, + ); +} diff --git a/tests/runner-e2e/native-completion-admission.test.ts b/tests/runner-e2e/native-completion-admission.test.ts new file mode 100644 index 00000000000..7a9b1ef8204 --- /dev/null +++ b/tests/runner-e2e/native-completion-admission.test.ts @@ -0,0 +1,108 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { spawnSync } from "node:child_process"; +import { assertNativeCompletionSelection, prepareNativeCompletionPreflight, nativeCompletionPreflightEnvironment, verifyNativeCompletionPreflight } from "./native-completion-admission.js"; +import { CREDENTIAL_NAMES, type MatrixExecution } from "./types.js"; + +vi.mock("node:child_process", () => ({ spawnSync: vi.fn() })); +const spawn = vi.mocked(spawnSync); +const campaignDirectory = "/fixture/results/gha-1-1-native-completion.fixture"; +beforeEach(() => { vi.resetAllMocks(); vi.unstubAllEnvs(); }); + +describe("native completion credential-free admission", () => { + it("allows toolchain paths and excludes every present or future credential", () => { + const source = { PATH: "/bin", HOME: "/home/fixture", CARGO_HOME: "/cargo", CI: "true", + GH_TOKEN: "secret", FUTURE_PROVIDER_API_KEY: "secret", NODE_OPTIONS: "ambient override", + ...Object.fromEntries(CREDENTIAL_NAMES.map(name => [name, "secret"])) }; + expect(nativeCompletionPreflightEnvironment(source)).toEqual({ PATH: "/bin", HOME: "/home/fixture", CARGO_HOME: "/cargo", CI: "true" }); + }); + it("carries only public hosted identity through actual prepare and verify subprocess options", () => { + const identity = { GITHUB_ACTIONS: "true", PAPERCLIP_RUNNER_E2E_SOURCE_SHA: "a".repeat(40), GITHUB_RUN_ID: "123", GITHUB_RUN_ATTEMPT: "1" }; + for (const [name, value] of Object.entries(identity)) vi.stubEnv(name, value); + for (const name of [...CREDENTIAL_NAMES, "GH_TOKEN", "FUTURE_PROVIDER_API_KEY", "NODE_OPTIONS"]) vi.stubEnv(name, "secret"); + spawn.mockReturnValueOnce({ status: 0 } as ReturnType); + spawn.mockReturnValueOnce({ status: 0, stdout: JSON.stringify({ passed: true, sourceSha: identity.PAPERCLIP_RUNNER_E2E_SOURCE_SHA }) } as ReturnType); + prepareNativeCompletionPreflight(campaignDirectory); + expect(spawn).toHaveBeenCalledTimes(2); + for (const call of spawn.mock.calls) { + expect(call[2]?.env).toMatchObject(identity); + for (const name of [...CREDENTIAL_NAMES, "GH_TOKEN", "FUTURE_PROVIDER_API_KEY", "NODE_OPTIONS"]) expect(call[2]?.env).not.toHaveProperty(name); + } + expect(spawn.mock.calls[0]?.[1]?.some(arg => arg.startsWith("--output-dir="))).toBe(true); + expect(spawn.mock.calls[1]?.[1]?.some(arg => arg.startsWith("--verify="))).toBe(true); + }); + it("rejects missing receipts without invoking a process", () => { + expect(() => verifyNativeCompletionPreflight(undefined)).toThrow("no prerequisite receipt"); + expect(spawn).not.toHaveBeenCalled(); + }); + it("fails before provider execution when the prerequisite subprocess fails", () => { + spawn.mockReturnValue({ status: 1 } as ReturnType); + expect(() => prepareNativeCompletionPreflight(campaignDirectory)).toThrow("before provider execution"); + expect(spawn).toHaveBeenCalledTimes(1); + }); + it("retains credential-free prerequisites inside the exact campaign root and verifies them", () => { + const sourceSha = "c".repeat(40); + vi.stubEnv("PAPERCLIP_RUNNER_E2E_SOURCE_SHA", sourceSha); + vi.stubEnv("OPENAI_API_KEY", "secret"); vi.stubEnv("UNLISTED_PROVIDER_CREDENTIAL", "secret"); + spawn.mockReturnValueOnce({ status: 0 } as ReturnType); + spawn.mockReturnValueOnce({ status: 0, stdout: JSON.stringify({ passed: true, sourceSha }) } as ReturnType); + const receipt = prepareNativeCompletionPreflight(campaignDirectory); + expect(receipt).toMatch(/^\/fixture\/results\/gha-1-1-native-completion\.fixture\/native-completion-prerequisites\/[^/]+\/preflight.json$/); + expect(spawn.mock.calls[1]?.[1]).toContain(`--verify=${receipt}`); + for (const call of spawn.mock.calls) { + expect(call[2]?.env).not.toHaveProperty("OPENAI_API_KEY"); + expect(call[2]?.env).not.toHaveProperty("UNLISTED_PROVIDER_CREDENTIAL"); + } + }); + it("rejects a workflow source override that differs from the exact receipt SHA", () => { + vi.stubEnv("PAPERCLIP_RUNNER_E2E_SOURCE_SHA", "a".repeat(40)); + spawn.mockReturnValue({ status: 0, stdout: JSON.stringify({ passed: true, sourceSha: "b".repeat(40) }) } as ReturnType); + expect(() => verifyNativeCompletionPreflight("/fixture/preflight.json")).toThrow("source SHA differs"); + spawn.mockReturnValue({ status: 0, stdout: JSON.stringify({ passed: true, sourceSha: "a".repeat(40) }) } as ReturnType); + expect(verifyNativeCompletionPreflight("/fixture/preflight.json").sourceSha).toBe("a".repeat(40)); + }); + it("refuses failed or unreadable receipt verification", () => { + spawn.mockReturnValue({ status: 1, stderr: "stale source" } as ReturnType); + expect(() => verifyNativeCompletionPreflight("/fixture/preflight.json")).toThrow("stale source"); + spawn.mockReturnValue({ status: 0, stdout: "malformed" } as ReturnType); + expect(() => verifyNativeCompletionPreflight("/fixture/preflight.json")).toThrow(); + }); +}); + +function selectedFixture() { + const tasks = ["assigned-skill-explicit-invocation", "native-blocked-report"].map(id => + ({ id, expectedRunCount: 1, automaticRetryPolicy: "single_attempt" })); + return { id: "native-completion.runner-codex.local.assigned-skill-explicit-invocation", + profile: { id: "runner-codex", generation: "native" }, environment: { id: "local" }, task: tasks[0], + suite: { id: "native-completion", manualOnly: true, expectedMatrixSize: 6, tasks } }; +} +describe("native completion selected-fixture policy", () => { + it("admits an exact subset of the six declared one-turn single-attempt cells", () => { + const base = selectedFixture(); + const executions = ["runner-codex", "runner-acpx-claude", "runner-opencode"].flatMap(id => base.suite.tasks.map(task => + ({ ...structuredClone(base), profile: { id, generation: "native" }, task, + id: `native-completion.${id}.local.${task.id}` }))); + expect(() => assertNativeCompletionSelection(executions as unknown as MatrixExecution[])).not.toThrow(); + expect(() => assertNativeCompletionSelection([executions[0]!] as unknown as MatrixExecution[])).not.toThrow(); + }); + const invalid: Array<[string, (fixture: ReturnType) => void]> = [ + ["a legacy generation", f => { f.profile.generation = "legacy"; }], + ["an unlisted profile", f => { f.profile.id = "runner-other"; }], + ["a remote environment", f => { f.environment.id = "daytona"; }], + ["an unexpected case", f => { f.task!.id = "other-task"; }], + ["an extra expected turn", f => { f.task!.expectedRunCount = 2; }], + ["a missing single-attempt policy", f => { f.task!.automaticRetryPolicy = "default"; }], + ["a forged execution ID", f => { f.id = "other-cell"; }], + ["an automatic suite", f => { f.suite.manualOnly = false; }], + ["an expanded matrix", f => { f.suite.expectedMatrixSize = 7; }], + ["a missing original completion case", f => { f.suite.tasks.shift(); }], + ["an extra task", f => { f.suite.tasks.push({ ...f.suite.tasks[1]!, id: "extra-task" }); }], + ]; + for (const [label, mutate] of invalid) it(`rejects ${label} before credentials`, () => { + const f = selectedFixture(); mutate(f); + expect(() => assertNativeCompletionSelection([f] as unknown as MatrixExecution[])).toThrow("before provider execution"); + }); + it("leaves ordinary suites and their retry policies unchanged", () => { + const f = selectedFixture(); f.suite.id = "ordinary-suite"; f.task!.automaticRetryPolicy = "default"; + expect(() => assertNativeCompletionSelection([f] as unknown as MatrixExecution[])).not.toThrow(); + }); +}); diff --git a/tests/runner-e2e/native-completion-admission.ts b/tests/runner-e2e/native-completion-admission.ts new file mode 100644 index 00000000000..ca429c405c5 --- /dev/null +++ b/tests/runner-e2e/native-completion-admission.ts @@ -0,0 +1,60 @@ +import { spawnSync } from "node:child_process"; +import { randomUUID } from "node:crypto"; +import path from "node:path"; +import type { MatrixExecution } from "./types.js"; + +const root = path.resolve(import.meta.dirname, "../.."); +export const NATIVE_COMPLETION_PREFLIGHT_ENV = "PAPERCLIP_RUNNER_E2E_NATIVE_COMPLETION_PREFLIGHT"; + +/** Explicit fixture policy is enforced even when the workflow supplies only --id. */ +export function assertNativeCompletionSelection(executions: readonly MatrixExecution[]): void { + const profiles = ["runner-codex", "runner-acpx-claude", "runner-opencode"]; + const tasks = ["assigned-skill-explicit-invocation", "native-blocked-report"]; + for (const execution of executions.filter(execution => execution.suite.id === "native-completion")) { + const suite = execution.suite; + if (!profiles.includes(execution.profile.id) || execution.profile.generation !== "native" || + execution.profile.qualificationCandidate !== undefined || execution.environment.id !== "local" || + !tasks.includes(execution.task.id) || execution.task.expectedRunCount !== 1 || + (execution.task.minimumExpectedRunCount !== undefined && execution.task.minimumExpectedRunCount !== 1) || + execution.task.automaticRetryPolicy !== "single_attempt" || + execution.id !== `native-completion.${execution.profile.id}.local.${execution.task.id}` || + suite.manualOnly !== true || suite.expectedMatrixSize !== 6 || suite.tasks.length !== 2 || + !tasks.every(id => suite.tasks.filter(task => task.id === id && task.expectedRunCount === 1 && + task.automaticRetryPolicy === "single_attempt").length === 1)) { + throw new Error("Native completion requires the declared local native fixtures, one turn and one attempt before provider execution."); + } + } +} + +/** Future credentials and ambient auth overrides stay outside prerequisite children. */ +export function nativeCompletionPreflightEnvironment(source: NodeJS.ProcessEnv): NodeJS.ProcessEnv { + return Object.fromEntries([ + "PATH", "HOME", "TMPDIR", "TMP", "TEMP", "SYSTEMROOT", "LANG", "LC_ALL", + "CARGO_HOME", "RUSTUP_HOME", "CI", "GITHUB_ACTIONS", + "PAPERCLIP_RUNNER_E2E_SOURCE_SHA", "GITHUB_RUN_ID", "GITHUB_RUN_ATTEMPT", + ].flatMap(name => source[name] === undefined ? [] : [[name, source[name]]])); +} + +export function verifyNativeCompletionPreflight(receiptPath: string | undefined) { + if (!receiptPath) throw new Error("Native completion has no prerequisite receipt; use the live launcher."); + const run = spawnSync(process.execPath, [path.join(root, "tests/runner-e2e/native-completion-checks.mjs"), `--verify=${receiptPath}`], { + cwd: root, env: nativeCompletionPreflightEnvironment(process.env), encoding: "utf8", timeout: 90_000, + }); + if (run.status !== 0) throw new Error(`Native completion prerequisite verification failed: ${run.stderr || run.error?.message || run.status}`); + const receipt = JSON.parse(run.stdout) as Record; + const expectedSourceSha = process.env.PAPERCLIP_RUNNER_E2E_SOURCE_SHA?.trim(); + if (expectedSourceSha && receipt.sourceSha !== expectedSourceSha) + throw new Error("Native completion prerequisite source SHA differs from the requested immutable source."); + return receipt; +} + +export function prepareNativeCompletionPreflight(campaignDirectory: string) { + const output = path.join(campaignDirectory, "native-completion-prerequisites", randomUUID()); + const receipt = path.join(output, "preflight.json"); + const run = spawnSync(process.execPath, [path.join(root, "tests/runner-e2e/native-completion-checks.mjs"), `--output-dir=${output}`], { + cwd: root, env: nativeCompletionPreflightEnvironment(process.env), stdio: "inherit", timeout: 35 * 60_000, + }); + if (run.status !== 0) throw new Error(`Native completion prerequisites failed before provider execution. Retained receipt: ${receipt}`); + verifyNativeCompletionPreflight(receipt); + return receipt; +} diff --git a/tests/runner-e2e/native-completion-cases.ts b/tests/runner-e2e/native-completion-cases.ts new file mode 100644 index 00000000000..8cea25b7c93 --- /dev/null +++ b/tests/runner-e2e/native-completion-cases.ts @@ -0,0 +1,47 @@ +import { createHash } from "node:crypto"; +import { readFileSync } from "node:fs"; +import type { RunnerTaskFixture } from "./types.js"; +import { contextIntegrityTasks } from "./context-integrity-cases.js"; + +export const nativeBlockedReportTask: RunnerTaskFixture = { + id: "native-blocked-report", label: "Native concrete blocker report", groups: ["native-completion"], + workMode: "standard", flow: "single_turn", expectedRunCount: 1, automaticRetryPolicy: "single_attempt", + attemptTimeoutMs: { local: 5 * 60_000, daytona: 5 * 60_000 }, + expectedTerminalState: { issue: "blocked", run: "succeeded" }, + buildTitle: nonce => `Native blocker report ${nonce}`, + buildVisibleMarker: nonce => `BLOCKED_${nonce.replace(/[^a-z0-9]/gi, "")}`, + buildPrompt: nonce => [ + "Deployment cannot proceed because required release access has not been granted.", + "Release Owner is the external owner of that grant. Do not attempt deployment, run shell/process commands, change files, invoke external services, create documents, agents, interactions or subtasks, or claim the access exists.", + 'Record the whole task as blocked and identify the owner as Release Owner. The exact unblock action is "Grant deployment access" (without quotation marks or trailing punctuation).', + `Include BLOCKED_${nonce.replace(/[^a-z0-9]/gi, "")} exactly once in your final explanation.`, + ].join("\n"), + buildMatchers: nonce => [ + { kind: "issue_status", expected: "blocked" }, { kind: "run_status", expected: "succeeded" }, + { kind: "runtime_mode", expected: "native" }, + { kind: "message_contains", expected: `BLOCKED_${nonce.replace(/[^a-z0-9]/gi, "")}` }, + { kind: "message_contains", expected: "Release Owner" }, + { kind: "message_contains", expected: "Grant deployment access" }, + { kind: "message_regex", pattern: "^(?![\\s\\S]*\\b(?:not blocked|no longer blocked|access (?:is |has been |was )?already granted|completed Grant deployment access)\\b)[\\s\\S]*\\b(?:blocked|cannot proceed|can't proceed|missing|required access|not (?:yet )?granted|awaiting|waiting|unavailable)\\b", flags: "i" }, + { kind: "json_path", path: "run.resultJson.nativeResult.reportedWorkDisposition", expected: "blocked" }, + { kind: "json_path", path: "run.resultJson.nativeResult.blocker.owner.name", expected: "Release Owner" }, + { kind: "json_path", path: "run.resultJson.nativeResult.blocker.unblockAction", expected: "Grant deployment access" }, + { kind: "json_path", path: "run.resultJson.nativeResult.blocker.scope", expected: "task_wide" }, + ], +}; + +const assignedSkill = contextIntegrityTasks.find(task => task.id === "assigned-skill-explicit-invocation"); +if (!assignedSkill) throw new Error("Missing original assigned-skill completion journey"); +// Preserve the original task prompt, skill, durable-document oracle and deadline. +export const nativeCompletionTasks: readonly RunnerTaskFixture[] = [ + { ...assignedSkill, automaticRetryPolicy: "single_attempt" }, nativeBlockedReportTask, +]; +export function nativeCompletionDefinitionDigest() { + const hash = createHash("sha256"); + for (const file of ["native-completion-cases.ts", "native-completion-scoring.ts", "native-completion-defaults.ts", + "native-blocker-visible.ts", "native-completion-admission.ts", "native-completion-checks.mjs", + "native-completion-source-contract.mjs", "automatic-retry.ts", "context-integrity-cases.ts", "context-integrity-flow.ts", "context-integrity-scoring.ts", "types.ts", "catalog.ts", "live-fixtures.ts", "runner.spec.ts", "launch.ts"]) { + hash.update(file); hash.update(readFileSync(new URL(file, import.meta.url))); + } + return hash.digest("hex"); +} diff --git a/tests/runner-e2e/native-completion-checks.mjs b/tests/runner-e2e/native-completion-checks.mjs new file mode 100644 index 00000000000..7d326e3bec5 --- /dev/null +++ b/tests/runner-e2e/native-completion-checks.mjs @@ -0,0 +1,297 @@ +// Independent, credential-free admission for the native completion comparison. +import { spawnSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import { existsSync, mkdirSync, readdirSync, readFileSync, writeFileSync } from "node:fs"; +import { join, resolve } from "node:path"; +import { + NATIVE_COMPLETION_SOURCE_CONTRACT, NATIVE_COMPLETION_SOURCE_FILES, + nativeCompletionSourceFingerprint, nativeSourceSha256, +} from "./native-completion-source-contract.mjs"; + +import { inspectNativeCompletionSourceMetadata, inspectNativeCompletionRunnerd, NATIVE_COMPLETION_RUNNERD_PATH } from "./native-completion-git-source.mjs"; + +const root = resolve(import.meta.dirname, "../.."); +export const NATIVE_COMPLETION_PREFLIGHT_SCHEMA = "paperclip.native-completion-preflight.v1"; +export const NATIVE_COMPLETION_CELL_IDS = ["runner-codex", "runner-acpx-claude", "runner-opencode"].flatMap(profile => + ["assigned-skill-explicit-invocation", "native-blocked-report"].map(task => `native-completion.${profile}.local.${task}`)); +export function nativeCompletionGates(variant) { + if (!["candidate", "historical"].includes(variant)) throw new Error("Unknown native completion source variant"); + return [ + { id: "NC-schemas", name: "Variant-native completion schemas", cwd: "packages/paperclip-runner", + files: ["src/contracts/completion-result.test.ts"], + required: ["distinguishes user-facing answer content", "propagates answer and wait descriptions", "requires a reason code for verification that was not run"] }, + { id: "NC-tools", name: "Native tool delivery and retained provider catalogs", cwd: "packages/paperclip-runner", files: [ + "src/drivers/codex/codex-app-server-driver.lifecycle.test.ts", "src/drivers/runner-tool-bridge.test.ts", + "src/drivers/opencode/mcp-bridge.test.ts", "src/live/runnerd-codex-transport.test.ts"], + testPattern: "binds to loopback|preserves stock Codex instructions on|includes ACPX terminal tools|preserves answer and internal wait descriptions", + required: ["runner semantic MCP bridge binds to loopback", "OpenCode MCP bridge binds to loopback", + "preserves stock Codex instructions on task recovery", "preserves stock Codex instructions on prepared recovery", + "preserves stock Codex instructions on direct recovery", "includes ACPX terminal tools in the authenticated bridge catalog", + ...(variant === "candidate" ? ["codex", "opencode", "claude_managed", "aws_agentcore", "acpx"].map(provider => + `preserves answer and internal wait descriptions in the serialized native ${provider} tool catalog`) : [])] }, + { id: "NC-resume", name: "Variant-native fingerprint and checkpoint refresh", cwd: ".", + files: ["server/src/services/native-runtime/native-session-resume.test.ts"], testPattern: "refreshes retained", + required: ["refreshes retained 'completion descriptions'", "refreshes retained 'task-bound human-input description'", + ...(variant === "candidate" ? ["native completion tool guidance"] : [])] }, + { id: "NC-eval", name: "Independent native oracle, defaults, admission and attempt policy", cwd: ".", + config: "tests/runner-e2e/vitest.config.ts", files: [ + "tests/runner-e2e/native-completion-scoring.test.ts", "tests/runner-e2e/native-completion-defaults.test.ts", + "tests/runner-e2e/native-completion-admission.test.ts", "tests/runner-e2e/automatic-retry.test.ts", + "tests/runner-e2e/context-integrity.test.ts", "tests/runner-e2e/context-integrity-evidence.test.ts", + "tests/runner-e2e/context-integrity-flow-logs.test.ts"], + required: ["allows toolchain paths and excludes every present or future credential", + "retains credential-free prerequisites inside the exact campaign root", + "retains the first transient_infrastructure failure without a second attempt", + "retains the first provider_variance failure without a second attempt"] }, + ]; +} +export const NATIVE_COMPLETION_COMMAND_GATE_IDS = ["NC-node", "NC-typecheck", "NC-manifest", "NC-discovery", "NC-rust-carrier"]; +export function nativeCompletionCommandGateIds(mode) { + return NATIVE_COMPLETION_COMMAND_GATE_IDS.map(id => mode === "trusted_hosted_archive" && id === "NC-rust-carrier" ? "NC-hosted-runnerd" : id); +} +export function gradeNativeCompletionGate(gate, report, exitCode) { + const assertions = (report?.testResults ?? []).flatMap(file => file.assertionResults ?? []); + const requirements = gate.required.map(name => ({ name, passed: assertions.some(assertion => + assertion.fullName?.includes(name) && assertion.status === "passed") })); + const files = gate.files.map(file => ({ file, passed: (report?.testResults ?? []).some(result => + result.name?.endsWith(file) && result.status === "passed" && result.assertionResults?.some(assertion => assertion.status === "passed") && + result.assertionResults.every(assertion => assertion.status === "passed" || (gate.testPattern && assertion.status === "skipped" && + !gate.required.some(name => assertion.fullName?.includes(name))))) })); + return { id: gate.id, name: gate.name, passed: exitCode === 0 && requirements.every(row => row.passed) && files.every(row => row.passed), + exitCode, files, requirements, total: report?.numTotalTests ?? 0, passedTests: report?.numPassedTests ?? 0, + failedTests: report?.numFailedTests ?? 0, pendingTests: report?.numPendingTests ?? 0 }; +} + +export function gradeNativeCompletionDiscovery(output, exitCode) { + const lines = output.trim().split(/\r?\n/).filter(Boolean); + const rows = lines.slice(1).map(line => line.split("\t")); + const ids = rows.map(row => row[0]); + return { passed: exitCode === 0 && lines[0] === "ID\tSUITE\tGENERATION\tPROVIDER\tMODEL\tCREDENTIALS" && + rows.length === 6 && new Set(ids).size === 6 && NATIVE_COMPLETION_CELL_IDS.every(id => ids.includes(id)) && + rows.every(row => row.length === 6 && row[1] === "native-completion" && row[2] === "native" && row[4]?.trim() && row[5]?.trim()), + executionIds: ids, expectedCells: 6, expectedTurns: 6, maximumAttemptsPerCell: 1 }; +} +/** capture() stores compact provenance JSON on its first line, then diagnostics. */ +export function gradeNativeCompletionHostedRunnerdEvidence(text, expected) { + try { return JSON.stringify(JSON.parse(text.split(/\r?\n/, 1)[0])) === JSON.stringify(expected); } + catch { return false; } +} +export function gradeNativeCompletionRustCarrier(output, exitCode) { + return exitCode === 0 && /^test provider_events::tests::preserves_closed_compatibility_terminal_tool_identity \.\.\. ok$/m.test(output) && + /test result: ok\. 1 passed; 0 failed;/.test(output); +} +export function nativeCompletionPrerequisiteEnvironment(source) { + return Object.fromEntries(["PATH", "HOME", "TMPDIR", "TMP", "TEMP", "SYSTEMROOT", "LANG", "LC_ALL", + "CARGO_HOME", "RUSTUP_HOME", "CI", "GITHUB_ACTIONS", "GITHUB_RUN_ID", "GITHUB_RUN_ATTEMPT", "PAPERCLIP_RUNNER_E2E_SOURCE_SHA"].flatMap(name => source[name] === undefined ? [] : [[name, source[name]]])); +} +function currentSource() { + const source = nativeCompletionSourceFingerprint(); + return { ...source, ...inspectNativeCompletionSourceMetadata({ repositoryRoot: root, + sourceFiles: NATIVE_COMPLETION_SOURCE_FILES, baseSha: source.baseSha, variant: source.variant, + shallowParentAnchors: NATIVE_COMPLETION_SOURCE_CONTRACT.shallowParentAnchors, + }) }; +} +function buildOutputFingerprint() { + const hash = createHash("sha256"); + function visit(relative) { + for (const entry of readdirSync(join(root, relative), { withFileTypes: true }).sort((a, b) => a.name.localeCompare(b.name))) { + const file = join(relative, entry.name); + if (entry.isDirectory()) visit(file); + else if (entry.isFile()) { const bytes = readFileSync(join(root, file)); hash.update(JSON.stringify([file, bytes.length])).update(bytes); } + else throw new Error(`Unexpected build output: ${file}`); + } + } + for (const directory of ["packages/shared/dist", "packages/plugins/sdk/dist", "packages/paperclip-runner/dist"]) { + if (!existsSync(join(root, directory, "index.js"))) throw new Error(`Missing prerequisite build output: ${directory}`); + visit(directory); + } + const daemon = runnerdBinary(); + const bytes = readFileSync(daemon); + hash.update(JSON.stringify(["debug/paperclip-runnerd", bytes.length])).update(bytes); + return hash.digest("hex"); +} +function runnerdBinary() { + return join(root, `${NATIVE_COMPLETION_RUNNERD_PATH}${process.platform === "win32" ? ".exe" : ""}`); +} +function runnerdProvenance(source, env) { + return inspectNativeCompletionRunnerd({ repositoryRoot: root, sourceSha: source.sha, + sourceFingerprint: source.fingerprint, environment: env }); +} +export function assertNativeCompletionPreflightReceipt(report, current) { + const expected = [...nativeCompletionGates(current.variant).map(gate => gate.id), ...nativeCompletionCommandGateIds(current.runnerdProvenance?.mode)]; + const hostedGate = report?.gates?.find(gate => gate.id === "NC-hosted-runnerd"); + const truthfulHosted = current.runnerdProvenance?.mode !== "trusted_hosted_archive" || + hostedGate?.executed === false && hostedGate?.calibration === "not_executed" && hostedGate?.total === 0 && + hostedGate?.passedTests === 0 && hostedGate?.reuse === "trusted_same_run_build"; + if (!truthfulHosted || report?.schema !== NATIVE_COMPLETION_PREFLIGHT_SCHEMA || report.passed !== true || report.providerCalls !== 0 || + report.live !== "not_run" || report.sourceSha !== current.sha || !/^[a-f0-9]{40}$/.test(current.sha ?? "") || + ![current.fingerprint, current.fixtureFingerprint, current.manifestFingerprint, current.buildOutputFingerprint] + .every(value => typeof value === "string" && /^[a-f0-9]{64}$/.test(value)) || + report.sourceFingerprint !== current.fingerprint || report.fixtureFingerprint !== current.fixtureFingerprint || + report.manifestFingerprint !== current.manifestFingerprint || report.variant !== current.variant || + report.baseSha !== NATIVE_COMPLETION_SOURCE_CONTRACT.baseSha || report.archiveSha !== NATIVE_COMPLETION_SOURCE_CONTRACT.archiveSha || + report.layering !== true || current.layering !== true || report.immutable !== true || current.immutable !== true || + !/^[a-f0-9]{64}$/.test(current.sourceMetadataFingerprint ?? "") || + report.sourceMetadataFingerprint !== current.sourceMetadataFingerprint || + !Array.isArray(report.sourceMetadataErrors) || report.sourceMetadataErrors.length !== 0 || current.sourceMetadataErrors?.length !== 0 || + !Array.isArray(report.sourceErrors) || report.sourceErrors.length !== 0 || current.sourceErrors?.length !== 0 || + report.setup?.passed !== true || report.setup.sdkExitCode !== 0 || report.setup.runnerTypeScriptExitCode !== 0 || + report.setup.runnerdExitCode !== (current.runnerdProvenance?.mode === "trusted_hosted_archive" ? null : 0) || + current.runnerdProvenance?.passed !== true || !["trusted_hosted_archive", "fresh_local_build"].includes(current.runnerdProvenance?.mode) || + JSON.stringify(report.setup.runnerdProvenance) !== JSON.stringify(current.runnerdProvenance) || + report.setup.runnerdSelectedPath !== current.runnerdProvenance.selectedPath || + report.setup.runnerdSha256 !== current.runnerdProvenance.binarySha256 || + !/^[a-f0-9]{64}$/.test(current.runnerdProvenance.binarySha256 ?? "") || + report.setup.runnerdSourceSha !== current.sha || report.setup.runnerdSourceFingerprint !== current.fingerprint || + report.setup.buildOutputFingerprint !== current.buildOutputFingerprint || + !Array.isArray(report.setup.evidence) || report.setup.evidence.length !== 3 || + ["setup-sdk.txt", "setup-runner-typescript.txt", "setup-runnerd.txt"].some(file => report.setup.evidence.filter(row => + row.file === file && /^[a-f0-9]{64}$/.test(row.sha256 ?? "")).length !== 1) || + !Array.isArray(report.gates) || report.gates.length !== expected.length || + expected.some(id => report.gates.filter(gate => gate.id === id && gate.passed === true && gate.exitCode === 0).length !== 1) || + report.expectedCells !== 6 || report.expectedTurns !== 6 || report.maximumAttemptsPerCell !== 1) { + throw new Error("Native completion requires passing prerequisites for this exact immutable source SHA, variant and fingerprint."); + } + return report; +} +function runCommand(command, args, env, timeout = 10 * 60_000, cwd = root) { + return spawnSync(command, args, { cwd, env, encoding: "utf8", timeout }); +} +function capture(output, name, run) { + const file = `${name}.txt`, bytes = `${run?.stdout ?? ""}\n${run?.stderr ?? ""}`; + writeFileSync(join(output, file), bytes); + return { file, sha256: nativeSourceSha256(bytes) }; +} +export function assertRetainedNativeCompletionEvidence(output, evidence) { + if (!evidence || typeof evidence.file !== "string" || !/^[A-Za-z0-9_.-]+$/.test(evidence.file) || + nativeSourceSha256(readFileSync(join(output, evidence.file))) !== evidence.sha256) + throw new Error("Native completion retained prerequisite evidence changed or is unavailable."); + return readFileSync(join(output, evidence.file), "utf8"); +} +function manifestCommands(env) { + return ["generate-capability-contract.mjs", "check-capability-inventory.mjs"].map(file => runCommand(process.execPath, + [join(root, "packages/paperclip-runner/scripts", file), ...(file.startsWith("generate-") ? ["--check"] : [])], env)); +} + +export function main(args = process.argv.slice(2)) { + if (args.includes("--list")) { + console.log(JSON.stringify({ variants: ["candidate", "historical"], gates: nativeCompletionGates("candidate"), + commands: { local: nativeCompletionCommandGateIds("fresh_local_build"), hosted: nativeCompletionCommandGateIds("trusted_hosted_archive") }, + cells: NATIVE_COMPLETION_CELL_IDS, providerCalls: 0 }, null, 2)); return; + } + if (args.some(arg => !arg.startsWith("--output-dir=") && !arg.startsWith("--verify="))) + throw new Error("Use --list, --output-dir= or --verify=; never paid providers."); + const source = currentSource(), env = nativeCompletionPrerequisiteEnvironment(process.env); + const verify = args.find(arg => arg.startsWith("--verify="))?.slice("--verify=".length); + if (verify) { + const report = assertNativeCompletionPreflightReceipt(JSON.parse(readFileSync(verify, "utf8")), { + ...source, buildOutputFingerprint: buildOutputFingerprint(), runnerdProvenance: runnerdProvenance(source, env), + }); + const output = resolve(verify, ".."); + for (const evidence of report.setup.evidence) assertRetainedNativeCompletionEvidence(output, evidence); + for (const gate of nativeCompletionGates(source.variant)) { + const retained = report.gates.find(row => row.id === gate.id); + const grade = gradeNativeCompletionGate(gate, JSON.parse(assertRetainedNativeCompletionEvidence(output, retained.evidence)), 0); + if (!grade.passed) throw new Error(`Missing or failed retained native prerequisite assertions: ${gate.id}`); + } + for (const id of nativeCompletionCommandGateIds(report.setup.runnerdProvenance.mode)) { + const retained = report.gates.find(row => row.id === id), text = assertRetainedNativeCompletionEvidence(output, retained.evidence); + if (id === "NC-node" && (!/# fail 0\b/.test(text) || !/# tests [1-9]\d*\b/.test(text))) + throw new Error("Missing passing native admission calibrations."); + if (id === "NC-discovery" && !gradeNativeCompletionDiscovery(text.split("\n\n")[0], 0).passed) + throw new Error("Native completion six-cell discovery changed."); + if (id === "NC-rust-carrier" && !gradeNativeCompletionRustCarrier(text, 0)) + throw new Error("Missing passing retained Rust terminal-tool carrier calibration."); + if (id === "NC-hosted-runnerd" && (retained.executed !== false || retained.calibration !== "not_executed" || + retained.total !== 0 || retained.passedTests !== 0 || + !gradeNativeCompletionHostedRunnerdEvidence(text, report.setup.runnerdProvenance))) + throw new Error("Hosted runnerd reuse must retain exact binary proof and report Rust calibration not executed."); + if (id === "NC-manifest" && manifestCommands(env).some(run => run.status !== 0)) + throw new Error("Generated native capability manifests are stale."); + } + console.log(JSON.stringify(report)); return report; + } + const output = resolve(args.find(arg => arg.startsWith("--output-dir="))?.slice("--output-dir=".length) ?? + join(root, "tests/runner-e2e/results", `native-completion-preflight-${new Date().toISOString().replaceAll(":", "-")}`)); + mkdirSync(output, { recursive: true }); + const report = { schema: NATIVE_COMPLETION_PREFLIGHT_SCHEMA, sourceSha: source.sha, sourceFingerprint: source.fingerprint, + fixtureFingerprint: source.fixtureFingerprint, manifestFingerprint: source.manifestFingerprint, + variant: source.variant, baseSha: source.baseSha, archiveSha: source.archiveSha, + sourceErrors: source.sourceErrors, immutable: source.immutable, layering: source.layering, + sourceMetadata: source.sourceMetadata, sourceMetadataErrors: source.sourceMetadataErrors, + sourceMetadataFingerprint: source.sourceMetadataFingerprint, + measuredAt: new Date().toISOString(), providerCalls: 0, live: "not_run", expectedCells: 6, expectedTurns: 6, maximumAttemptsPerCell: 1, + setup: { passed: false, sdkExitCode: null, runnerTypeScriptExitCode: null, runnerdExitCode: null, evidence: [] }, gates: [], passed: false }; + if (!source.sha || source.sourceErrors.length || !source.layering || !source.immutable) { + writeFileSync(join(output, "preflight.json"), JSON.stringify(report, null, 2) + "\n"); + console.error(`Native completion source admission failed before providers: ${[...source.sourceErrors, ...source.sourceMetadataErrors].join("; ")}`); + process.exitCode = 1; return report; + } + const sdk = runCommand(process.execPath, [join(root, "scripts/ensure-plugin-build-deps.mjs")], env, 5 * 60_000); + report.setup.sdkExitCode = sdk.status; report.setup.evidence.push(capture(output, "setup-sdk", sdk)); + const runner = sdk.status === 0 ? runCommand("pnpm", ["--filter", "@paperclipai/paperclip-runner", "build:typescript"], env) : null; + report.setup.runnerTypeScriptExitCode = runner?.status ?? null; + report.setup.evidence.push(capture(output, "setup-runner-typescript", runner)); + const hosted = env.GITHUB_ACTIONS === "true"; + const runnerd = sdk.status === 0 && runner?.status === 0 && !hosted ? runCommand("cargo", + ["build", "--locked", "--offline", "--workspace", "--bins"], env, + 10 * 60_000, join(root, "packages/paperclip-runner/runner")) : null; + report.setup.runnerdExitCode = runnerd?.status ?? null; + report.setup.runnerdProvenance = runnerdProvenance(source, env); + report.setup.evidence.push(capture(output, "setup-runnerd", hosted + ? { stdout: JSON.stringify(report.setup.runnerdProvenance), stderr: "Rust build and unit calibration not executed in this hosted cell; reusing trusted same-run build." } + : runnerd)); + report.setup.passed = sdk.status === 0 && runner?.status === 0 && (hosted || runnerd?.status === 0) && report.setup.runnerdProvenance.passed; + if (report.setup.passed) { + report.setup.buildOutputFingerprint = buildOutputFingerprint(); + report.setup.runnerdSha256 = report.setup.runnerdProvenance.binarySha256; + report.setup.runnerdSelectedPath = report.setup.runnerdProvenance.selectedPath; + report.setup.runnerdSourceSha = source.sha; + report.setup.runnerdSourceFingerprint = source.fingerprint; + for (const gate of nativeCompletionGates(source.variant)) { + console.log(`Checking ${gate.id}: ${gate.name}`); + const file = `${gate.id}.json`, run = runCommand(process.execPath, [join(root, "node_modules/vitest/vitest.mjs"), "run", ...gate.files, + ...(gate.config ? ["--config", gate.config] : []), ...(gate.testPattern ? ["--testNamePattern", gate.testPattern] : []), + "--reporter=default", "--reporter=json", `--outputFile.json=${join(output, file)}`], env, 10 * 60_000, resolve(root, gate.cwd)); + capture(output, gate.id, run); + let result; try { result = JSON.parse(readFileSync(join(output, file), "utf8")); } catch { /* Missing discovery fails closed. */ } + report.gates.push({ ...gradeNativeCompletionGate(gate, result, run.status), + evidence: existsSync(join(output, file)) ? { file, sha256: nativeSourceSha256(readFileSync(join(output, file))) } : null }); + } + const commands = [ + { id: "NC-node", command: process.execPath, args: ["--test", "--test-reporter=tap", "tests/runner-e2e/native-completion-checks.test.mjs", "tests/runner-e2e/native-completion-source-contract.test.mjs", "tests/runner-e2e/native-completion-git-source.test.mjs"] }, + { id: "NC-typecheck", command: process.execPath, args: ["node_modules/typescript/bin/tsc", "-p", "tests/runner-e2e/tsconfig.json"] }, + ...(!hosted ? [{ id: "NC-rust-carrier", command: "cargo", args: ["test", "--locked", "--offline", + "-p", "paperclip-runner-core", "--lib", "provider_events::tests::preserves_closed_compatibility_terminal_tool_identity", "--", "--exact"], + cwd: join(root, "packages/paperclip-runner/runner") }] : []), + { id: "NC-discovery", command: process.execPath, args: ["--import", "./server/node_modules/tsx/dist/loader.mjs", "tests/runner-e2e/launch.ts", "--list", "--suite", "native-completion"] }, + ]; + for (const command of commands) { + console.log(`Checking ${command.id}`); + const run = runCommand(command.command, command.args, env, 10 * 60_000, command.cwd ?? root); + report.gates.push({ id: command.id, passed: run.status === 0 && + (command.id !== "NC-discovery" || gradeNativeCompletionDiscovery(run.stdout, run.status).passed) && + (command.id !== "NC-rust-carrier" || gradeNativeCompletionRustCarrier(`${run.stdout ?? ""}\n${run.stderr ?? ""}`, run.status)) && + (command.id !== "NC-node" || /# fail 0\b/.test(run.stdout) && /# tests [1-9]\d*\b/.test(run.stdout)), + exitCode: run.status, evidence: capture(output, command.id, run) }); + } + if (hosted) report.gates.push({ id: "NC-hosted-runnerd", name: "Trusted hosted runnerd provenance; Rust calibration not executed", + passed: report.setup.runnerdProvenance.passed, exitCode: 0, executed: false, calibration: "not_executed", + reuse: "trusted_same_run_build", total: 0, passedTests: 0, evidence: capture(output, "NC-hosted-runnerd", { + stdout: JSON.stringify(report.setup.runnerdProvenance), stderr: "Rust unit calibration must be qualified by the separate exact-source local receipt and normal CI." }) }); + const manifest = manifestCommands(env), combined = { stdout: manifest.map(run => run.stdout ?? "").join("\n"), stderr: manifest.map(run => run.stderr ?? "").join("\n") }; + report.gates.push({ id: "NC-manifest", passed: manifest.every(run => run.status === 0), + exitCode: manifest.find(run => run.status !== 0)?.status ?? 0, evidence: capture(output, "NC-manifest", combined) }); + const after = currentSource(); + report.passed = report.gates.every(gate => gate.passed) && after.immutable && after.layering && + after.sha === source.sha && after.fingerprint === source.fingerprint && after.sourceErrors.length === 0 && + after.sourceMetadataFingerprint === source.sourceMetadataFingerprint && after.sourceMetadataErrors.length === 0 && + buildOutputFingerprint() === report.setup.buildOutputFingerprint && + JSON.stringify(runnerdProvenance(after, env)) === JSON.stringify(report.setup.runnerdProvenance); + } + writeFileSync(join(output, "preflight.json"), JSON.stringify(report, null, 2) + "\n"); + console.log(`Native completion prerequisite evidence: ${join(output, "preflight.json")}`); + if (!report.passed) process.exitCode = 1; + return report; +} +if (process.argv[1] && resolve(process.argv[1]) === resolve(import.meta.filename)) main(); diff --git a/tests/runner-e2e/native-completion-checks.test.mjs b/tests/runner-e2e/native-completion-checks.test.mjs new file mode 100644 index 00000000000..36ca65c0c71 --- /dev/null +++ b/tests/runner-e2e/native-completion-checks.test.mjs @@ -0,0 +1,182 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { mkdtempSync, writeFileSync, rmSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import { + nativeCompletionGates, gradeNativeCompletionGate, gradeNativeCompletionDiscovery, gradeNativeCompletionRustCarrier, gradeNativeCompletionHostedRunnerdEvidence, + assertNativeCompletionPreflightReceipt, assertRetainedNativeCompletionEvidence, nativeCompletionPrerequisiteEnvironment, + NATIVE_COMPLETION_PREFLIGHT_SCHEMA, NATIVE_COMPLETION_CELL_IDS, NATIVE_COMPLETION_COMMAND_GATE_IDS, nativeCompletionCommandGateIds, +} from "./native-completion-checks.mjs"; +import { nativeSourceSha256, NATIVE_COMPLETION_SOURCE_CONTRACT } from "./native-completion-source-contract.mjs"; + +const gate = { id: "NC-test", name: "Boundary", files: ["boundary.test.ts"], required: ["native assertion"] }; +const report = () => ({ testResults: [{ name: "/repo/boundary.test.ts", status: "passed", assertionResults: [ + { fullName: "Boundary native assertion", status: "passed" }] }], numTotalTests: 1, numPassedTests: 1 }); +test("native completion prerequisite gates exclude reduced manual and shared legacy assertions", () => { + for (const variant of ["candidate", "historical"]) { + const gates = nativeCompletionGates(variant); + assert.deepEqual(gates.map(gate => gate.id), ["NC-schemas", "NC-tools", "NC-resume", "NC-eval"]); + assert.ok(gates.every(gate => gate.files.length && gate.required.length)); + assert.ok(!gates.flatMap(gate => gate.files).some(file => /agent-skills-routes|onboarding-first-task-assets|server-utils\.test|prompt-sections/.test(file))); + } + assert.ok(nativeCompletionGates("candidate")[1].required.some(name => name.includes("serialized native codex"))); + assert.ok(!nativeCompletionGates("historical")[1].required.some(name => name.includes("serialized native codex"))); + assert.ok(!nativeCompletionGates("historical")[2].required.includes("native completion tool guidance")); + assert.throws(() => nativeCompletionGates("unknown")); +}); +test("native completion prerequisite admits an executed passing boundary", () => assert.equal(gradeNativeCompletionGate(gate, report(), 0).passed, true)); +for (const status of ["pending", "skipped", "failed", "todo"]) test(`native completion prerequisite rejects a ${status} required assertion`, () => { + const observation = report(); observation.testResults[0].assertionResults[0].status = status; + assert.equal(gradeNativeCompletionGate(gate, observation, 0).passed, false); +}); +test("native completion prerequisite rejects missing discovery, unrelated assertions and failed commands", () => { + assert.equal(gradeNativeCompletionGate(gate, undefined, 0).passed, false); + const different = report(); different.testResults[0].assertionResults[0].fullName = "Other"; + assert.equal(gradeNativeCompletionGate(gate, different, 0).passed, false); + assert.equal(gradeNativeCompletionGate(gate, report(), 1).passed, false); + assert.equal(gradeNativeCompletionGate({ ...gate, files: [...gate.files, "undiscovered.test.ts"] }, report(), 0).passed, false); +}); +test("native completion prerequisite allows filtering unrelated assertions but never the required assertion", () => { + const observation = report(); observation.testResults[0].assertionResults.push({ fullName: "Unrelated", status: "skipped" }); + assert.equal(gradeNativeCompletionGate({ ...gate, testPattern: "native assertion" }, observation, 0).passed, true); + assert.equal(gradeNativeCompletionGate(gate, observation, 0).passed, false); + observation.testResults[0].assertionResults[0].status = "skipped"; + assert.equal(gradeNativeCompletionGate({ ...gate, testPattern: "native assertion" }, observation, 0).passed, false); +}); +const discovery = () => ["ID\tSUITE\tGENERATION\tPROVIDER\tMODEL\tCREDENTIALS", ...NATIVE_COMPLETION_CELL_IDS.map(id => + `${id}\tnative-completion\tnative\tprovider\tmodel\tAPI_KEY`)].join("\n"); +test("native completion prerequisite discovers exactly six native single-turn cells", () => { + assert.deepEqual(gradeNativeCompletionDiscovery(discovery(), 0), { passed: true, executionIds: NATIVE_COMPLETION_CELL_IDS, + expectedCells: 6, expectedTurns: 6, maximumAttemptsPerCell: 1 }); +}); +for (const kind of ["missing", "duplicate", "legacy", "extra", "failed", "missing-model", "wrong-task"]) test(`native completion prerequisite rejects ${kind} discovery`, () => { + let output = discovery(); + if (kind === "missing") output = output.split("\n").slice(0, -1).join("\n"); + if (kind === "duplicate") output = output.replace(NATIVE_COMPLETION_CELL_IDS[1], NATIVE_COMPLETION_CELL_IDS[0]); + if (kind === "legacy") output = output.replace("\tnative\t", "\tlegacy\t"); + if (kind === "extra") output += `\n${output.split("\n")[1]}`; + if (kind === "missing-model") output = output.replace("\tmodel\t", "\t\t"); + if (kind === "wrong-task") output = output.replace("assigned-skill-explicit-invocation", "other-case"); + assert.equal(gradeNativeCompletionDiscovery(output, kind === "failed" ? 1 : 0).passed, false); +}); +test("native completion prerequisite excludes future credentials and auth overrides from every child", () => { + assert.deepEqual(nativeCompletionPrerequisiteEnvironment({ PATH: "/bin", HOME: "/fixture", CI: "true", + OPENAI_API_KEY: "secret", ANTHROPIC_API_KEY: "secret", FUTURE_TOKEN: "secret", NODE_OPTIONS: "secret", GH_TOKEN: "secret" }), + { PATH: "/bin", HOME: "/fixture", CI: "true" }); +}); +const runnerdProof = () => ({ schema: "paperclip.native-completion-runnerd-provenance.v1", mode: "fresh_local_build", passed: true, + selectedPath: "packages/paperclip-runner/runner/target/debug/paperclip-runnerd", binarySha256: "9".repeat(64), + sourceSha: "a".repeat(40), sourceFingerprint: "b".repeat(64), archiveSha256: null, workflowRunId: null, + workflowRunAttempt: null, artifactName: null, errors: [] }); +const current = variant => ({ runnerdProvenance: runnerdProof(), sha: "a".repeat(40), fingerprint: "b".repeat(64), fixtureFingerprint: "c".repeat(64), + manifestFingerprint: "d".repeat(64), buildOutputFingerprint: "e".repeat(64), runnerdSha256: "9".repeat(64), variant, immutable: true, layering: true, sourceErrors: [], sourceMetadataFingerprint: "f".repeat(64), sourceMetadataErrors: [] }); +const receipt = variant => ({ schema: NATIVE_COMPLETION_PREFLIGHT_SCHEMA, passed: true, providerCalls: 0, live: "not_run", + sourceSha: current(variant).sha, sourceFingerprint: current(variant).fingerprint, fixtureFingerprint: current(variant).fixtureFingerprint, + manifestFingerprint: current(variant).manifestFingerprint, variant, + baseSha: NATIVE_COMPLETION_SOURCE_CONTRACT.baseSha, archiveSha: NATIVE_COMPLETION_SOURCE_CONTRACT.archiveSha, immutable: true, layering: true, sourceErrors: [], + sourceMetadataFingerprint: current(variant).sourceMetadataFingerprint, sourceMetadataErrors: [], + expectedCells: 6, expectedTurns: 6, maximumAttemptsPerCell: 1, + setup: { passed: true, sdkExitCode: 0, runnerTypeScriptExitCode: 0, buildOutputFingerprint: current(variant).buildOutputFingerprint, + runnerdExitCode: 0, runnerdSha256: current(variant).runnerdSha256, runnerdProvenance: runnerdProof(), + runnerdSelectedPath: runnerdProof().selectedPath, + runnerdSourceSha: current(variant).sha, runnerdSourceFingerprint: current(variant).fingerprint, + evidence: ["setup-sdk.txt", "setup-runner-typescript.txt", "setup-runnerd.txt"].map(file => ({ file, sha256: "f".repeat(64) })) }, + gates: [...nativeCompletionGates(variant).map(gate => gate.id), ...NATIVE_COMPLETION_COMMAND_GATE_IDS].map(id => ({ id, passed: true, exitCode: 0 })) }); +for (const variant of ["candidate", "historical"]) test(`native completion prerequisite admits only the exact immutable ${variant} receipt`, () => { + assert.equal(assertNativeCompletionPreflightReceipt(receipt(variant), current(variant)).passed, true); +}); +const negatives = [ + ["old SHA", r => { r.sourceSha = "f".repeat(40); }], ["changed source", r => { r.sourceFingerprint = "f".repeat(64); }], + ["changed fixtures", r => { r.fixtureFingerprint = "f".repeat(64); }], ["stale manifests", r => { r.manifestFingerprint = "f".repeat(64); }], + ["wrong variant", r => { r.variant = "historical"; }], ["wrong base", r => { r.baseSha = "f".repeat(40); }], + ["wrong archive", r => { r.archiveSha = "f".repeat(40); }], ["dirty source", r => { r.immutable = false; }], + ["wrong master layering", r => { r.layering = false; }], ["source errors", r => { r.sourceErrors.push("missing.ts"); }], + ["paid provider calls", r => { r.providerCalls = 1; }], ["paid live run", r => { r.live = "run"; }], + ["missing gate", r => { r.gates.pop(); }], ["duplicate gate", r => { r.gates[1] = r.gates[0]; }], + ["failed gate", r => { r.gates[0].passed = false; }], ["nonzero exit", r => { r.gates[0].exitCode = 1; }], + ["failed SDK build", r => { r.setup.sdkExitCode = 1; }], ["missing Runner build", r => { delete r.setup.runnerTypeScriptExitCode; }], + ["changed build output", r => { r.setup.buildOutputFingerprint = "f".repeat(64); }], + ["extra cells", r => { r.expectedCells = 7; }], ["extra turns", r => { r.expectedTurns = 7; }], + ["additional attempts", r => { r.maximumAttemptsPerCell = 2; }], + ["missing runnerd provenance", r => { delete r.setup.runnerdProvenance; }], + ["wrong selected runnerd path", r => { r.setup.runnerdSelectedPath = "packages/paperclip-runner/dist/bin/paperclip-runnerd"; }], + ["claimed hosted reuse of local binary", r => { r.setup.runnerdProvenance.mode = "trusted_hosted_archive"; r.setup.runnerdExitCode = null; }], + ["failed runnerd build", r => { r.setup.runnerdExitCode = 1; }], + ["stale runnerd binary", r => { r.setup.runnerdSha256 = "8".repeat(64); }], + ["wrong runnerd source SHA", r => { r.setup.runnerdSourceSha = "b".repeat(40); }], + ["wrong runnerd source fingerprint", r => { r.setup.runnerdSourceFingerprint = "8".repeat(64); }], + ["changed hosted source metadata", r => { r.sourceMetadataFingerprint = "a".repeat(64); }], + ["source metadata errors", r => { r.sourceMetadataErrors.push("unknown lineage"); }], + ["missing setup evidence", r => { r.setup.evidence.pop(); }], + ["duplicate setup evidence", r => { r.setup.evidence[1] = r.setup.evidence[0]; }], +]; +for (const [name, mutate] of negatives) test(`native completion prerequisite rejects ${name} before providers`, () => { + const r = receipt("candidate"); mutate(r); + assert.throws(() => assertNativeCompletionPreflightReceipt(r, current("candidate")), /exact immutable source SHA, variant and fingerprint/); +}); +test("native completion prerequisite rejects a current dirty source and unavailable digests", () => { + assert.throws(() => assertNativeCompletionPreflightReceipt(receipt("candidate"), { ...current("candidate"), immutable: false })); + assert.throws(() => assertNativeCompletionPreflightReceipt(receipt("candidate"), { ...current("candidate"), fingerprint: null })); +}); + +test("native completion prerequisite rejects changed or missing retained evidence and unsafe paths", () => { + const output = mkdtempSync(join(tmpdir(), "native-completion-retained-")); + try { + const bytes = '{"passing":"native assertions"}', evidence = { file: "NC-tools.json", sha256: nativeSourceSha256(bytes) }; + writeFileSync(join(output, evidence.file), bytes); + assert.equal(assertRetainedNativeCompletionEvidence(output, evidence), bytes); + writeFileSync(join(output, evidence.file), "Changed retained assertions"); + assert.throws(() => assertRetainedNativeCompletionEvidence(output, evidence), /changed or is unavailable/); + assert.throws(() => assertRetainedNativeCompletionEvidence(output, { file: "missing.json", sha256: evidence.sha256 })); + assert.throws(() => assertRetainedNativeCompletionEvidence(output, { file: "../elsewhere.json", sha256: evidence.sha256 })); + assert.throws(() => assertRetainedNativeCompletionEvidence(output, null)); + } finally { rmSync(output, { recursive: true, force: true }); } +}); + +test("native completion prerequisite requires the executed exact Rust carrier calibration", () => { + const output = "test provider_events::tests::preserves_closed_compatibility_terminal_tool_identity ... ok\ntest result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out;"; + assert.equal(gradeNativeCompletionRustCarrier(output, 0), true); + assert.equal(gradeNativeCompletionRustCarrier(output, 1), false); + assert.equal(gradeNativeCompletionRustCarrier(output.replace(" ... ok", " ... FAILED"), 0), false); + assert.equal(gradeNativeCompletionRustCarrier(output.replace("1 passed", "0 passed"), 0), false); + assert.equal(gradeNativeCompletionRustCarrier(output.replace("preserves_closed_compatibility_terminal_tool_identity", "other_test"), 0), false); +}); + +function hostedFixture() { + const c = current("candidate"), r = receipt("candidate"); + c.runnerdProvenance = { ...runnerdProof(), mode: "trusted_hosted_archive", archiveSha256: "7".repeat(64), + workflowRunId: "12345", workflowRunAttempt: "2", artifactName: `runner-e2e-build-${c.sha}-12345-2` }; + r.setup.runnerdProvenance = structuredClone(c.runnerdProvenance); r.setup.runnerdExitCode = null; + r.gates = r.gates.map(gate => gate.id === "NC-rust-carrier" ? { ...gate, id: "NC-hosted-runnerd", executed: false, + calibration: "not_executed", total: 0, passedTests: 0, reuse: "trusted_same_run_build" } : gate); + return { c, r }; +} +test("native hosted receipt explicitly reuses verified runnerd bytes without claiming Rust calibration execution", () => { + const {c, r} = hostedFixture(); assert.equal(assertNativeCompletionPreflightReceipt(r, c).passed, true); + assert.equal(nativeCompletionCommandGateIds("trusted_hosted_archive").includes("NC-rust-carrier"), false); + assert.equal(nativeCompletionCommandGateIds("trusted_hosted_archive").includes("NC-hosted-runnerd"), true); +}); +for (const [name, mutate] of [ + ["executed Rust calibration", r => { r.gates.at(-1).executed = true; }], + ["positive Rust test count", r => { r.gates.at(-1).passedTests = 1; }], + ["missing explicit not-executed status", r => { delete r.gates.at(-1).calibration; }], + ["fresh Rust build claim", r => { r.setup.runnerdExitCode = 0; }], + ["different archive bytes", r => { r.setup.runnerdProvenance.archiveSha256 = "8".repeat(64); }], + ["different workflow run", r => { r.setup.runnerdProvenance.workflowRunId = "999"; }], +]) test(`native hosted receipt rejects ${name}`, () => { + const {c,r} = hostedFixture(); + // The hosted provenance gate is last in this deterministic fixture. + r.gates.sort((a,b) => a.id === "NC-hosted-runnerd" ? 1 : b.id === "NC-hosted-runnerd" ? -1 : 0); + mutate(r); assert.throws(() => assertNativeCompletionPreflightReceipt(r,c)); +}); + +test("hosted runnerd evidence verifies the exact compact JSON capture before its diagnostic line", () => { + const proof = hostedFixture().c.runnerdProvenance; + const capture = `${JSON.stringify(proof)}\nRust unit calibration must be qualified by the separate exact-source local receipt and normal CI.`; + assert.equal(gradeNativeCompletionHostedRunnerdEvidence(capture, proof), true); + assert.equal(gradeNativeCompletionHostedRunnerdEvidence(capture.replace("\n", "\r\n"), proof), true); + assert.equal(gradeNativeCompletionHostedRunnerdEvidence(capture, { ...proof, binarySha256: "f".repeat(64) }), false); + assert.equal(gradeNativeCompletionHostedRunnerdEvidence(`not-json\n${JSON.stringify(proof)}`, proof), false); + assert.equal(gradeNativeCompletionHostedRunnerdEvidence("", proof), false); +}); diff --git a/tests/runner-e2e/native-completion-defaults.test.ts b/tests/runner-e2e/native-completion-defaults.test.ts new file mode 100644 index 00000000000..21c5c6dc082 --- /dev/null +++ b/tests/runner-e2e/native-completion-defaults.test.ts @@ -0,0 +1,74 @@ +import { describe, expect, it } from "vitest"; +import { readFileSync } from "node:fs"; +import { captureNativeDefault, gradeNativeDefault, nativeCompletionProfile, nativeCompletionWorkspaceDigest, NATIVE_MASTER_DEFAULT_SHA256, type NativeDefaultReceipt } from "./native-completion-defaults.js"; +import { mkdtemp, mkdir, writeFile, symlink, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import type { RunnerProfileFixture } from "./types.js"; +import { runnerSuites } from "./catalog.js"; +import { contextIntegrityTasks } from "./context-integrity-cases.js"; + +const receipt = (): NativeDefaultReceipt => ({ schema: "paperclip.native-completion-default.v1", agentId: "agent", companyId: "company", + entryFile: "AGENTS.md", files: [{ path: "AGENTS.md", sha256: NATIVE_MASTER_DEFAULT_SHA256, bytes: 100 }], budgets: { company: 1000, agent: 1000 } }); +describe("native production defaults", () => { + it("hashes scoped file and symlink identity without reading an outside target", async () => { + const temporary = await mkdtemp(path.join(tmpdir(), "native-workspace-calibration-")); + try { + const root = path.join(temporary, "workspace"), outside = path.join(temporary, "outside"); + await mkdir(root); await mkdir(outside); + await writeFile(path.join(root, "file"), "initial"); await writeFile(path.join(outside, "private"), "first"); + await symlink(outside, path.join(root, "skill-link")); + const first = await nativeCompletionWorkspaceDigest(root); + await writeFile(path.join(outside, "private"), "changed outside"); + expect(await nativeCompletionWorkspaceDigest(root)).toBe(first); + await writeFile(path.join(root, "file"), "changed fixture"); + expect(await nativeCompletionWorkspaceDigest(root)).not.toBe(first); + } finally { await rm(temporary, { recursive: true, force: true }); } + }); + it("preserves the original assigned-skill prompt flow and oracle in exactly six explicit cells", () => { + const suite = runnerSuites.find(value => value.id === "native-completion")!; + const original = contextIntegrityTasks.find(value => value.id === "assigned-skill-explicit-invocation")!; + const completion = suite.tasks.find(value => value.id === original.id)!; + expect({ ...completion, automaticRetryPolicy: undefined }).toEqual({ ...original, automaticRetryPolicy: undefined }); + expect(suite.manualOnly).toBe(true); + expect(suite.expectedMatrixSize).toBe(6); + expect(suite.tasks).toHaveLength(2); + expect(suite.tasks.every(task => task.expectedRunCount === 1 && task.automaticRetryPolicy === "single_attempt")).toBe(true); + expect(suite.environments.map(value => value.id)).toEqual(["local"]); + expect(suite.profiles.map(value => value.id).sort()).toEqual(["runner-acpx-claude", "runner-codex", "runner-opencode"]); + }); + it("accepts only the unchanged master default and bounded budgets", () => expect(gradeNativeDefault(receipt()).passed).toBe(true)); + it.each(["qa", "reduced", "extra-file", "wrong-entry", "company-unbounded", "agent-unbounded"])("rejects %s override", name => { + const value = receipt(); + if (name === "qa" || name === "reduced") value.files[0]!.sha256 = name; + if (name === "extra-file") value.files.push({ path: "HEARTBEAT.md", sha256: "x", bytes: 1 }); + if (name === "wrong-entry") value.entryFile = "OTHER.md"; + if (name === "company-unbounded") value.budgets.company = 0; + if (name === "agent-unbounded") value.budgets.agent = 0; + expect(gradeNativeDefault(value).passed).toBe(false); + }); + it("strips only the fixture bundle and preserves auth skills permissions and model", () => { + const original = { instructionsBundle: { entryFile: "AGENTS.md", files: { "AGENTS.md": "QA" } }, + adapterConfig: { model: "same", env: { KEY: { type: "secret_ref", secretId: "id" } } }, + runtimeConfig: { permissionMode: "same" }, skills: ["same"], budgetMonthlyCents: 0 }; + const profile = { generation: "native", buildAgent: () => original } as unknown as RunnerProfileFixture; + const wrapped = nativeCompletionProfile(profile).buildAgent({} as never); + expect(wrapped).toEqual({ ...original, instructionsBundle: undefined, budgetMonthlyCents: 1000 }); + expect(Object.hasOwn(wrapped, "instructionsBundle")).toBe(false); + expect(original.instructionsBundle).toBeDefined(); + expect(() => nativeCompletionProfile({ ...profile, generation: "legacy" })).toThrow(); + }); + it("hashes actual public bundle bytes before execution without publishing the body", async () => { + const content = readFileSync(new URL("../../server/src/onboarding-assets/default/AGENTS.md", import.meta.url), "utf8"); + const called: string[] = []; + const api = { async get(path: string): Promise { + called.push(path); + return (path.endsWith("instructions-bundle") ? { entryFile: "AGENTS.md", files: [{ path: "AGENTS.md" }] } + : path.includes("instructions-bundle/file") ? { content } : { budgetMonthlyCents: 1000 }) as T; + } }; + const actual = await captureNativeDefault({ api, agentId: "agent", companyId: "company" }); + expect(gradeNativeDefault(actual).passed).toBe(true); + expect(called).toContain("/api/agents/agent/instructions-bundle/file?path=AGENTS.md"); + expect(JSON.stringify(actual)).not.toContain(content); + }); +}); diff --git a/tests/runner-e2e/native-completion-defaults.ts b/tests/runner-e2e/native-completion-defaults.ts new file mode 100644 index 00000000000..53efae177ba --- /dev/null +++ b/tests/runner-e2e/native-completion-defaults.ts @@ -0,0 +1,70 @@ +import { createHash } from "node:crypto"; +import { lstat, readFile, readdir, readlink } from "node:fs/promises"; +import path from "node:path"; +import type { RunnerProfileFixture } from "./types.js"; + +export const NATIVE_COMPLETION_BUDGET_CENTS = 1_000; +// Independent, immutable current-master default; not the reduced #14948 manual. +export const NATIVE_MASTER_DEFAULT_SHA256 = "e4d2375d722602cd744403292d99f6e6c9b7e8014d9cdfa6f9811ace03428e5f"; + +/** Scoped blocker proof: hash symlink identity without following any target. */ +export async function nativeCompletionWorkspaceDigest(root: string): Promise { + const hash = createHash("sha256"); + async function visit(relative: string) { + const entries = await readdir(path.join(root, relative), { withFileTypes: true }); + for (const entry of entries.sort((a, b) => a.name.localeCompare(b.name))) { + const file = path.join(relative, entry.name), absolute = path.join(root, file), stat = await lstat(absolute); + hash.update(JSON.stringify([file, stat.isSymbolicLink() ? "symlink" : stat.isDirectory() ? "directory" : stat.isFile() ? "file" : "other"])); + if (stat.isSymbolicLink()) hash.update(await readlink(absolute)); + else if (stat.isDirectory()) await visit(file); + else if (stat.isFile()) hash.update(await readFile(absolute)); + } + } + await visit(""); return hash.digest("hex"); +} + +export function nativeCompletionProfile(profile: RunnerProfileFixture): RunnerProfileFixture { + if (profile.generation !== "native") throw new Error("Native completion requires a native profile"); + return { ...profile, buildAgent(input) { + const { instructionsBundle: _fixtureBundle, ...agent } = profile.buildAgent(input); + return { ...agent, budgetMonthlyCents: NATIVE_COMPLETION_BUDGET_CENTS }; + } }; +} + +export interface NativeDefaultReceipt { + schema: "paperclip.native-completion-default.v1"; + agentId: string; + companyId: string; + entryFile: unknown; + files: Array<{ path: string; sha256: string; bytes: number }>; + budgets: { company: unknown; agent: unknown }; +} + +export function gradeNativeDefault(receipt: NativeDefaultReceipt) { + const checks = [ + { id: "production-default-bundle", passed: receipt.entryFile === "AGENTS.md" + && receipt.files.length === 1 && receipt.files[0]?.path === "AGENTS.md" + && receipt.files[0]?.sha256 === NATIVE_MASTER_DEFAULT_SHA256, + detail: "The served public hire bundle is the unchanged master default, with no QA or reduced-manual injection." }, + { id: "bounded-company-and-agent", passed: receipt.budgets.company === NATIVE_COMPLETION_BUDGET_CENTS + && receipt.budgets.agent === NATIVE_COMPLETION_BUDGET_CENTS, + detail: "Both public budgets enforce the declared 1,000-cent hard stop." }, + ]; + return { passed: checks.every(check => check.passed), checks }; +} + +export async function captureNativeDefault(input: { + api: { get(path: string): Promise }; agentId: string; companyId: string; +}): Promise { + const [bundle, company, agent] = await Promise.all([ + input.api.get<{ entryFile: unknown; files: Array<{ path: string }> }>(`/api/agents/${input.agentId}/instructions-bundle`), + input.api.get<{ budgetMonthlyCents: unknown }>(`/api/companies/${input.companyId}`), + input.api.get<{ budgetMonthlyCents: unknown }>(`/api/agents/${input.agentId}`), + ]); + const files = await Promise.all(bundle.files.map(async file => { + const detail = await input.api.get<{ content: string }>(`/api/agents/${input.agentId}/instructions-bundle/file?path=${encodeURIComponent(file.path)}`); + return { path: file.path, sha256: createHash("sha256").update(detail.content).digest("hex"), bytes: Buffer.byteLength(detail.content) }; + })); + return { schema: "paperclip.native-completion-default.v1", agentId: input.agentId, companyId: input.companyId, + entryFile: bundle.entryFile, files, budgets: { company: company.budgetMonthlyCents, agent: agent.budgetMonthlyCents } }; +} diff --git a/tests/runner-e2e/native-completion-git-source.mjs b/tests/runner-e2e/native-completion-git-source.mjs new file mode 100644 index 00000000000..0da48024d44 --- /dev/null +++ b/tests/runner-e2e/native-completion-git-source.mjs @@ -0,0 +1,166 @@ +import { spawnSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import { closeSync, constants, fstatSync, lstatSync, mkdtempSync, openSync, readFileSync, readSync, readdirSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, posix } from "node:path"; + +const hydrationDirectory = "runner-e2e-build"; +const archiveName = "runner-e2e-build-bundle.tar.gz"; +const checksumName = `${archiveName}.sha256`; +const hydrationPaths = [archiveName, checksumName].map(file => `${hydrationDirectory}/${file}`); +const sha256 = bytes => createHash("sha256").update(bytes).digest("hex"); + +/** A shallow commit still carries its hash-bound immediate parent in raw bytes. */ +export function verifyNativeCompletionParentAnchor({ sha, rawCommit, anchor }) { + if (!/^[a-f0-9]{40}$/.test(sha ?? "") || !/^[a-f0-9]{40}$/.test(anchor ?? "")) return false; + const bytes = Buffer.isBuffer(rawCommit) ? rawCommit : Buffer.from(rawCommit ?? ""); + const observed = createHash("sha1").update(`commit ${bytes.length}\0`).update(bytes).digest("hex"); + const parents = bytes.toString("utf8").split("\n\n", 1)[0].split("\n").filter(line => line.startsWith("parent ")); + return observed === sha && parents.length === 1 && parents[0] === `parent ${anchor}`; +} +function regularFile(path) { + const stat = lstatSync(path); + if (!stat.isFile() || stat.nlink !== 1) throw new Error("entry is not a regular, independently downloaded file"); + return stat; +} +function archiveDigest(path) { + const before = regularFile(path), fd = openSync(path, constants.O_RDONLY | (constants.O_NOFOLLOW ?? 0)); + try { + const opened = fstatSync(fd); + if (!opened.isFile() || opened.nlink !== 1 || opened.ino !== before.ino || opened.dev !== before.dev) + throw new Error("archive changed during verification"); + const hash = createHash("sha256"), buffer = Buffer.alloc(1024 * 1024); + let count; + while ((count = readSync(fd, buffer, 0, buffer.length, null)) > 0) hash.update(buffer.subarray(0, count)); + const after = fstatSync(fd); + if (after.size !== opened.size || after.mtimeMs !== opened.mtimeMs) throw new Error("archive changed during verification"); + return hash.digest("hex"); + } finally { closeSync(fd); } +} +export function inspectNativeCompletionBuildHydration(repositoryRoot) { + const directory = join(repositoryRoot, hydrationDirectory), errors = []; + let stat; + try { stat = lstatSync(directory); } + catch (error) { + if (error.code === "ENOENT") return { present: false, verified: false, archiveSha256: null, checksumSha256: null, errors }; + return { present: true, verified: false, archiveSha256: null, checksumSha256: null, errors: ["build hydration directory is unreadable"] }; + } + let archiveSha256 = null, checksumSha256 = null; + try { + if (!stat.isDirectory()) throw new Error("directory must be a real directory, not a symlink or file"); + const entries = readdirSync(directory).sort(); + if (JSON.stringify(entries) !== JSON.stringify([archiveName, checksumName].sort())) + throw new Error("directory must contain exactly the archive and its checksum, with no extra entries"); + const checksumPath = join(directory, checksumName), checksumStat = regularFile(checksumPath); + if (checksumStat.size > 128) throw new Error("checksum record is malformed"); + const checksum = readFileSync(checksumPath), match = /^([a-f0-9]{64}) {2}runner-e2e-build-bundle\.tar\.gz\n?$/.exec(checksum.toString("utf8")); + if (!match) throw new Error("checksum must be one exact SHA256 record for the downloaded archive"); + checksumSha256 = sha256(checksum); archiveSha256 = archiveDigest(join(directory, archiveName)); + if (archiveSha256 !== match[1]) throw new Error("archive checksum verification failed"); + } catch (error) { errors.push(`build hydration ${error.message}`); } + return { present: true, verified: errors.length === 0, archiveSha256, checksumSha256, errors }; +} + +export const NATIVE_COMPLETION_RUNNERD_PATH = "packages/paperclip-runner/runner/target/debug/paperclip-runnerd"; +/** Mirror the production default selector, and reject staged binaries that would take precedence. */ +export function inspectNativeCompletionRunnerd({ repositoryRoot, sourceSha, sourceFingerprint, environment }) { + const errors = [], hosted = environment.GITHUB_ACTIONS === "true"; + const selectedPath = `${NATIVE_COMPLETION_RUNNERD_PATH}${process.platform === "win32" ? ".exe" : ""}`; + const staged = join(repositoryRoot, `packages/paperclip-runner/dist/bin/paperclip-runnerd${process.platform === "win32" ? ".exe" : ""}`); + let binarySha256 = null, binaryBytes = null, archiveSha256 = null, archiveMemberBytes = null; + try { + try { lstatSync(staged); throw new Error("staged dist/bin runnerd would override the admitted debug executable"); } + catch (error) { if (error.code !== "ENOENT") throw error; } + let directory = repositoryRoot; + for (const part of selectedPath.split("/").slice(0, -1)) { + directory = join(directory, part); + if (!lstatSync(directory).isDirectory()) throw new Error("selected runnerd ancestor must be a real directory"); + } + const binary = join(repositoryRoot, selectedPath), stat = regularFile(binary); + if (process.platform !== "win32" && !(stat.mode & 0o111)) throw new Error("selected runnerd is not executable"); + binaryBytes = stat.size; + binarySha256 = archiveDigest(binary); + if (hosted) { + if (environment.PAPERCLIP_RUNNER_E2E_SOURCE_SHA !== sourceSha || !/^[a-f0-9]{40}$/.test(sourceSha ?? "")) + throw new Error("trusted hosted target SHA does not match admitted Git HEAD"); + if (![environment.GITHUB_RUN_ID, environment.GITHUB_RUN_ATTEMPT].every(value => typeof value === "string" && value === value.trim() && /^[1-9]\d*$/.test(value))) + throw new Error("trusted hosted workflow run and attempt are unavailable"); + const hydration = inspectNativeCompletionBuildHydration(repositoryRoot); + if (!hydration.verified) throw new Error(`hosted build archive is unverified: ${hydration.errors.join("; ")}`); + archiveSha256 = hydration.archiveSha256; + const archive = join(repositoryRoot, hydrationDirectory, archiveName); + const tar = args => spawnSync("tar", args, { encoding: "utf8", timeout: 120_000, maxBuffer: 16 * 1024 * 1024 }); + const names = tar(["-tzf", archive]); + if (names.status !== 0) throw new Error("hosted build archive cannot be listed"); + const matching = names.stdout.split(/\r?\n/).filter(name => name && posix.normalize(name.replace(/^\/+/, "")) === selectedPath); + if (matching.length !== 1 || matching[0] !== selectedPath) + throw new Error("hosted archive must contain exactly one canonical selected runnerd member"); + const details = tar(["-tvzf", archive, selectedPath]); + const rows = details.stdout.trim().split(/\r?\n/); + if (details.status !== 0 || rows.length !== 1 || !rows[0].startsWith("-") || !rows[0].endsWith(` ${selectedPath}`)) + throw new Error("hosted selected runnerd member must be a regular file"); + // Keep daemon bytes out of JavaScript memory; the selected executable supplies the exact expected size. + const temporary = mkdtempSync(join(tmpdir(), "native-completion-runnerd-member-")), member = join(temporary, "runnerd"); + let fd; + try { + fd = openSync(member, constants.O_WRONLY | constants.O_CREAT | constants.O_EXCL | (constants.O_NOFOLLOW ?? 0), 0o600); + const extracted = spawnSync("tar", ["-xOzf", archive, selectedPath], { timeout: 120_000, stdio: ["ignore", fd, "pipe"] }); + archiveMemberBytes = fstatSync(fd).size; + closeSync(fd); fd = undefined; + if (extracted.status !== 0 || archiveMemberBytes !== binaryBytes || archiveDigest(member) !== binarySha256) + throw new Error("selected runnerd byte count or hash differs from the verified hosted archive member"); + } finally { + if (fd !== undefined) closeSync(fd); + rmSync(temporary, { recursive: true, force: true }); + } + if (archiveDigest(archive) !== archiveSha256) throw new Error("hosted build archive changed during runnerd verification"); + } + } catch (error) { errors.push(`runnerd provenance ${error.message}`); } + return { schema: "paperclip.native-completion-runnerd-provenance.v1", mode: hosted ? "trusted_hosted_archive" : "fresh_local_build", + passed: errors.length === 0, selectedPath, binarySha256, binaryBytes, archiveMemberBytes, sourceSha, sourceFingerprint, archiveSha256, + workflowRunId: hosted ? environment.GITHUB_RUN_ID ?? null : null, + workflowRunAttempt: hosted ? environment.GITHUB_RUN_ATTEMPT ?? null : null, + artifactName: hosted ? `runner-e2e-build-${sourceSha}-${environment.GITHUB_RUN_ID}-${environment.GITHUB_RUN_ATTEMPT}` : null, + errors }; +} + +/** Preserve tracked-source cleanliness; allow only the verified workflow download. */ +export function inspectNativeCompletionSourceMetadata({ repositoryRoot, sourceFiles, baseSha, variant, shallowParentAnchors }) { + const git = (args, raw = false) => spawnSync("git", ["--no-replace-objects", ...args], { + cwd: repositoryRoot, encoding: raw ? undefined : "utf8", timeout: 30_000, + }); + const head = git(["rev-parse", "--verify", "HEAD"]), sha = head.status === 0 ? head.stdout.trim() : null; + const ancestor = git(["merge-base", "--is-ancestor", baseSha, "HEAD"]); + const shallow = git(["rev-parse", "--is-shallow-repository"]), isShallow = shallow.status === 0 && shallow.stdout.trim() === "true"; + const errors = []; + if (!/^[a-f0-9]{40}$/.test(sha ?? "")) errors.push("Git HEAD is unavailable or is not an immutable SHA1 commit"); + let layering = ancestor.status === 0, strategy = layering ? "full_ancestry" : null; + const anchor = shallowParentAnchors?.[variant] ?? null; + if (!layering && isShallow && anchor && sha) { + const commit = git(["cat-file", "commit", "HEAD"], true); + layering = commit.status === 0 && verifyNativeCompletionParentAnchor({ sha, rawCommit: commit.stdout, anchor }); + if (layering) strategy = "shallow_immediate_parent_anchor"; + } + if (!layering) errors.push(isShallow + ? "shallow Git HEAD does not have the sole hash-verified immediate parent admitted for this source variant" + : "full Git history does not prove the declared master base is an ancestor of HEAD"); + const tracked = git(["ls-files", "--error-unmatch", "--", ...sourceFiles]); + const clean = git(["diff", "--quiet", "HEAD", "--"]); + const status = git(["status", "--porcelain=v1", "-z", "--untracked-files=all"]); + const entries = status.status === 0 ? status.stdout.split("\0").filter(Boolean) : []; + const unexpected = entries.filter(entry => entry.slice(0, 3) !== "?? " || !hydrationPaths.includes(entry.slice(3))); + const hydration = inspectNativeCompletionBuildHydration(repositoryRoot); + if (tracked.status !== 0) errors.push("one or more admitted source paths are not tracked at HEAD"); + if (clean.status !== 0) errors.push("tracked source differs from HEAD"); + if (status.status !== 0) errors.push("Git worktree status is unavailable"); + if (unexpected.length) errors.push(`unexpected worktree entries: ${unexpected.join(", ")}`); + errors.push(...hydration.errors); + const immutable = tracked.status === 0 && clean.status === 0 && status.status === 0 && unexpected.length === 0 && + (!hydration.present || hydration.verified); + const metadata = { schema: "paperclip.native-completion-source-metadata.v1", + lineage: { strategy, shallow: isShallow, sha, baseSha, anchor: strategy === "shallow_immediate_parent_anchor" ? anchor : null }, + hydration: { present: hydration.present, verified: hydration.verified, archiveSha256: hydration.archiveSha256, checksumSha256: hydration.checksumSha256 }, + worktreeEntries: entries, errors }; + return { sha, layering, immutable, sourceMetadata: metadata, sourceMetadataErrors: errors, + sourceMetadataFingerprint: sha256(JSON.stringify(metadata)) }; +} diff --git a/tests/runner-e2e/native-completion-git-source.test.mjs b/tests/runner-e2e/native-completion-git-source.test.mjs new file mode 100644 index 00000000000..03681ee3454 --- /dev/null +++ b/tests/runner-e2e/native-completion-git-source.test.mjs @@ -0,0 +1,174 @@ +import assert from "node:assert/strict"; +import { execFileSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import { after, test } from "node:test"; +import { chmodSync, mkdtempSync, mkdirSync, readFileSync, rmSync, symlinkSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { pathToFileURL } from "node:url"; +import { inspectNativeCompletionSourceMetadata, inspectNativeCompletionBuildHydration, verifyNativeCompletionParentAnchor, + inspectNativeCompletionRunnerd, NATIVE_COMPLETION_RUNNERD_PATH } from "./native-completion-git-source.mjs"; + +const roots = []; +after(() => { for (const root of roots) rmSync(root, { recursive: true, force: true }); }); +const temporary = () => { const root = mkdtempSync(join(tmpdir(), "native-source-metadata-")); roots.push(root); return root; }; +const git = (root, args, input) => execFileSync("git", args, { cwd: root, input, encoding: "utf8", stdio: ["pipe", "pipe", "pipe"] }).trim(); +const digest = bytes => createHash("sha256").update(bytes).digest("hex"); +const archiveName = "runner-e2e-build-bundle.tar.gz", checksumName = `${archiveName}.sha256`; +function commit(root, tree, parents, message) { + const raw = `tree ${tree}\n${parents.map(parent => `parent ${parent}\n`).join("")}author Fixture 1 +0000\ncommitter Fixture 1 +0000\n\n${message}\n`; + return { raw, sha: git(root, ["hash-object", "-t", "commit", "-w", "--stdin"], raw) }; +} +function fixture(shallow = false, parentCount = 1) { + const source = temporary(); git(source, ["init", "--quiet"]); + writeFileSync(join(source, "source.txt"), "Immutable admitted source\n"); + const blob = git(source, ["hash-object", "-w", "source.txt"]); + const tree = git(source, ["mktree"], `100644 blob ${blob}\tsource.txt\n`); + const base = commit(source, tree, [], "Admitted base"), anchor = commit(source, tree, [base.sha], "Admitted native parent"); + const other = commit(source, tree, [base.sha], "Different parent"); + const head = commit(source, tree, parentCount === 2 ? [anchor.sha, other.sha] : [anchor.sha], "Native-only repair"); + git(source, ["update-ref", "refs/heads/fixture", head.sha]); git(source, ["checkout", "--quiet", "--force", "fixture"]); + let root = source; + if (shallow) { + root = join(temporary(), "checkout"); + git(source, ["clone", "--quiet", "--depth", "1", "--branch", "fixture", pathToFileURL(source).href, root]); + } + const input = { repositoryRoot: root, sourceFiles: ["source.txt"], baseSha: base.sha, variant: "candidate", + shallowParentAnchors: { candidate: anchor.sha, historical: other.sha } }; + return { root, source, base, anchor, other, head, input }; +} +function hydrate(root, bytes = Buffer.from("Exact downloaded build archive bytes")) { + const directory = join(root, "runner-e2e-build"); mkdirSync(directory); + writeFileSync(join(directory, archiveName), bytes); + writeFileSync(join(directory, checksumName), `${digest(bytes)} ${archiveName}\n`); + return directory; +} +test("native hosted-source metadata retains full ancestry and clean sources", () => { + const f = fixture(), result = inspectNativeCompletionSourceMetadata(f.input); + assert.equal(result.layering, true); assert.equal(result.immutable, true); assert.deepEqual(result.sourceMetadataErrors, []); + assert.equal(result.sourceMetadata.lineage.strategy, "full_ancestry"); +}); +test("native hosted-source metadata accepts an actual shallow clone with the exact raw parent anchor", () => { + const f = fixture(true); hydrate(f.root); + assert.equal(git(f.root, ["rev-parse", "--is-shallow-repository"]), "true"); + assert.throws(() => git(f.root, ["merge-base", "--is-ancestor", f.base.sha, "HEAD"])); + const result = inspectNativeCompletionSourceMetadata(f.input); + assert.equal(result.sha, f.head.sha); assert.equal(result.layering, true); assert.equal(result.immutable, true); + assert.deepEqual(result.sourceMetadataErrors, []); + assert.equal(result.sourceMetadata.lineage.strategy, "shallow_immediate_parent_anchor"); + assert.equal(result.sourceMetadata.lineage.anchor, f.anchor.sha); + assert.equal(result.sourceMetadata.hydration.verified, true); +}); +test("native hosted-source metadata requires the variant-specific shallow anchor", () => { + const f = fixture(true), result = inspectNativeCompletionSourceMetadata({ ...f.input, variant: "historical" }); + assert.equal(result.layering, false); assert.ok(result.sourceMetadataErrors.some(error => error.includes("sole hash-verified immediate parent"))); +}); +test("native hosted-source metadata never substitutes an anchor for missing full-clone ancestry", () => { + const f = fixture(), result = inspectNativeCompletionSourceMetadata({ ...f.input, baseSha: "f".repeat(40) }); + assert.equal(result.layering, false); assert.ok(result.sourceMetadataErrors.some(error => error.includes("full Git history"))); +}); +test("native hosted-source metadata rejects a shallow merge even with the admitted parent", () => { + const f = fixture(true, 2), result = inspectNativeCompletionSourceMetadata(f.input); + assert.equal(result.layering, false); assert.ok(result.sourceMetadataErrors.some(error => error.includes("sole hash-verified"))); +}); +test("native hosted-source metadata cryptographically binds all raw HEAD bytes and the sole parent", () => { + const f = fixture(); + assert.equal(verifyNativeCompletionParentAnchor({ sha: f.head.sha, rawCommit: f.head.raw, anchor: f.anchor.sha }), true); + for (const input of [ + { sha: "f".repeat(40), rawCommit: f.head.raw, anchor: f.anchor.sha }, + { sha: f.head.sha, rawCommit: `${f.head.raw}altered`, anchor: f.anchor.sha }, + { sha: f.head.sha, rawCommit: f.head.raw, anchor: f.other.sha }, + { sha: f.base.sha, rawCommit: f.base.raw, anchor: f.anchor.sha }, + { sha: null, rawCommit: f.head.raw, anchor: f.anchor.sha }, + ]) assert.equal(verifyNativeCompletionParentAnchor(input), false); +}); +test("native hosted-source metadata rejects dirty tracked and untracked admitted source", () => { + const f = fixture(); hydrate(f.root); writeFileSync(join(f.root, "source.txt"), "Changed admitted source\n"); + let result = inspectNativeCompletionSourceMetadata(f.input); + assert.equal(result.immutable, false); assert.ok(result.sourceMetadataErrors.includes("tracked source differs from HEAD")); + git(f.root, ["checkout", "HEAD", "--", "source.txt"]); writeFileSync(join(f.root, "extra-source.mjs"), "Unexpected code"); + result = inspectNativeCompletionSourceMetadata(f.input); + assert.equal(result.immutable, false); assert.ok(result.sourceMetadataErrors.some(error => error.includes("extra-source.mjs"))); + assert.equal(inspectNativeCompletionSourceMetadata({ ...f.input, sourceFiles: ["extra-source.mjs"] }).immutable, false); +}); +test("native hosted-source metadata verifies only the exact two downloaded regular files", () => { + const f = fixture(), directory = hydrate(f.root), first = inspectNativeCompletionSourceMetadata(f.input); + assert.equal(first.immutable, true); assert.equal(first.sourceMetadata.hydration.verified, true); + assert.equal(inspectNativeCompletionSourceMetadata(f.input).sourceMetadataFingerprint, first.sourceMetadataFingerprint); + writeFileSync(join(directory, archiveName), "Different verified archive"); + writeFileSync(join(directory, checksumName), `${digest("Different verified archive")} ${archiveName}\n`); + const changed = inspectNativeCompletionSourceMetadata(f.input); + assert.equal(changed.immutable, true); assert.notEqual(changed.sourceMetadataFingerprint, first.sourceMetadataFingerprint); +}); +const negatives = [ + ["missing checksum", (root, directory) => rmSync(join(directory, checksumName))], + ["checksum mismatch", (root, directory) => writeFileSync(join(directory, archiveName), "Unverified replacement")], + ["extra file", (root, directory) => writeFileSync(join(directory, "extra.mjs"), "Unexpected code")], + ["extra nested directory", (root, directory) => mkdirSync(join(directory, "nested"))], + ["malformed checksum", (root, directory) => writeFileSync(join(directory, checksumName), "malformed")], + ["wrong checksum filename", (root, directory) => writeFileSync(join(directory, checksumName), `${"a".repeat(64)} other.tar.gz\n`)], + ["multiple checksum records", (root, directory) => writeFileSync(join(directory, checksumName), readFileSync(join(directory, checksumName), "utf8").repeat(2))], + ["symlink archive", (root, directory) => { rmSync(join(directory, archiveName)); symlinkSync(join(root, "source.txt"), join(directory, archiveName)); }], + ["symlink checksum", (root, directory) => { rmSync(join(directory, checksumName)); symlinkSync(join(root, "source.txt"), join(directory, checksumName)); }], + ["symlink directory", (root, directory) => { rmSync(directory, { recursive: true }); symlinkSync(root, directory); }], +]; +for (const [name, mutate] of negatives) test(`native hosted-source metadata rejects ${name} hydration`, () => { + const f = fixture(), directory = hydrate(f.root); mutate(f.root, directory); + const hydration = inspectNativeCompletionBuildHydration(f.root), result = inspectNativeCompletionSourceMetadata(f.input); + assert.equal(hydration.verified, false); assert.ok(hydration.errors.length > 0); + assert.equal(result.immutable, false); assert.ok(result.sourceMetadataErrors.some(error => error.includes("build hydration"))); +}); +test("native hosted-source metadata rejects unrelated dirty trees despite verified hydration", () => { + const f = fixture(); hydrate(f.root); mkdirSync(join(f.root, "untrusted")); writeFileSync(join(f.root, "untrusted", "anything"), "extra"); + const result = inspectNativeCompletionSourceMetadata(f.input); + assert.equal(result.immutable, false); assert.ok(result.sourceMetadataErrors.some(error => error.includes("untrusted/anything"))); +}); + +function runnerdFixture() { + const root = temporary(), binary = join(root, NATIVE_COMPLETION_RUNNERD_PATH); + mkdirSync(join(binary, ".."), { recursive: true }); writeFileSync(binary, "Exact built runnerd fixture bytes"); chmodSync(binary, 0o755); + const directory = join(root, "runner-e2e-build"); mkdirSync(directory); + function pack(members = [NATIVE_COMPLETION_RUNNERD_PATH]) { + execFileSync("tar", ["-czf", join(directory, archiveName), ...members], { cwd: root }); + writeFileSync(join(directory, checksumName), `${digest(readFileSync(join(directory, archiveName)))} ${archiveName}\n`); + } + pack(); + const input = { repositoryRoot: root, sourceSha: "a".repeat(40), sourceFingerprint: "b".repeat(64), environment: { + GITHUB_ACTIONS: "true", PAPERCLIP_RUNNER_E2E_SOURCE_SHA: "a".repeat(40), GITHUB_RUN_ID: "12345", GITHUB_RUN_ATTEMPT: "2" } }; + return { root, binary, directory, pack, input }; +} +test("native hosted runnerd proof binds the actual selected debug executable to the trusted same-run archive", () => { + const f = runnerdFixture(), proof = inspectNativeCompletionRunnerd(f.input); + assert.equal(proof.passed, true); assert.equal(proof.mode, "trusted_hosted_archive"); + assert.equal(proof.selectedPath, NATIVE_COMPLETION_RUNNERD_PATH); + assert.equal(proof.binarySha256, digest(readFileSync(f.binary))); + assert.equal(proof.binaryBytes, readFileSync(f.binary).length); assert.equal(proof.archiveMemberBytes, proof.binaryBytes); + assert.equal(proof.artifactName, `runner-e2e-build-${f.input.sourceSha}-12345-2`); + assert.deepEqual(proof.errors, []); + const local = inspectNativeCompletionRunnerd({ ...f.input, environment: {} }); + assert.equal(local.passed, true); assert.equal(local.mode, "fresh_local_build"); + assert.equal(local.archiveSha256, null); assert.equal(local.workflowRunId, null); + assert.equal(local.binaryBytes, readFileSync(f.binary).length); assert.equal(local.archiveMemberBytes, null); +}); +for (const [name, mutate] of [ + ["wrong trusted source SHA", f => { f.input.environment.PAPERCLIP_RUNNER_E2E_SOURCE_SHA = "c".repeat(40); }], + ["missing workflow run", f => { delete f.input.environment.GITHUB_RUN_ID; }], + ["malformed workflow attempt", f => { f.input.environment.GITHUB_RUN_ATTEMPT = "../2"; }], + ["newline in workflow run", f => { f.input.environment.GITHUB_RUN_ID = "12345\n"; }], + ["unverified archive", f => { writeFileSync(join(f.directory, archiveName), "Replacement"); }], + ["different selected executable", f => { writeFileSync(f.binary, "Stale binary"); }], + ["missing selected executable", f => { rmSync(f.binary); }], + ["nonexecutable selected binary", f => { chmodSync(f.binary, 0o644); }], + ["same-size different binary hash", f => { writeFileSync(f.binary, Buffer.alloc(readFileSync(f.binary).length, 0x58)); }], + ["symlink selected executable", f => { const bytes = readFileSync(f.binary); rmSync(f.binary); writeFileSync(join(f.root, "other"), bytes); symlinkSync(join(f.root, "other"), f.binary); }], + ["symlink executable ancestor", f => { const bytes = readFileSync(f.binary), directory = join(f.binary, ".."); rmSync(directory, { recursive: true }); mkdirSync(join(f.root, "other")); writeFileSync(join(f.root, "other", "paperclip-runnerd"), bytes); chmodSync(join(f.root, "other", "paperclip-runnerd"), 0o755); symlinkSync(join(f.root, "other"), directory); }], + ["staged executable override", f => { const staged = join(f.root, "packages/paperclip-runner/dist/bin"); mkdirSync(staged, { recursive: true }); writeFileSync(join(staged, "paperclip-runnerd"), "Unexpected staged binary"); }], + ["duplicate canonical archive members", f => { f.pack([NATIVE_COMPLETION_RUNNERD_PATH, NATIVE_COMPLETION_RUNNERD_PATH]); }], + ["noncanonical archive member", f => { f.pack([`./${NATIVE_COMPLETION_RUNNERD_PATH}`]); }], + ["missing archive member", f => { writeFileSync(join(f.root, "other"), "Other"); f.pack(["other"]); }], + ["symlink archive member", f => { rmSync(f.binary); writeFileSync(join(f.root, "other"), "Other"); symlinkSync(join(f.root, "other"), f.binary); f.pack(); rmSync(f.binary); writeFileSync(f.binary, "Other"); chmodSync(f.binary, 0o755); }], +]) test(`native hosted runnerd proof rejects ${name}`, () => { + const f = runnerdFixture(); mutate(f); + const proof = inspectNativeCompletionRunnerd(f.input); + assert.equal(proof.passed, false); assert.ok(proof.errors.length > 0); +}); diff --git a/tests/runner-e2e/native-completion-scoring.test.ts b/tests/runner-e2e/native-completion-scoring.test.ts new file mode 100644 index 00000000000..8c476cbc32e --- /dev/null +++ b/tests/runner-e2e/native-completion-scoring.test.ts @@ -0,0 +1,118 @@ +import { readFileSync } from "node:fs"; +import { describe, expect, it } from "vitest"; +import { rehydrateRunnerdItemNotification } from "../../packages/paperclip-runner/src/live/runnerd-codex-transport.js"; +import { gradeNativeCompletion, type NativeCompletionObservation } from "./native-completion-scoring.js"; +type Row = Record; +function sample(blocked = true, compatibility = false): NativeCompletionObservation { + const body = blocked ? "Deployment remains blocked until access is granted. Release Owner must Grant deployment access. BLOCKED_probe" : "Saved the requested document."; + const result = { reportedWorkDisposition: blocked ? "blocked" : "done", ...(blocked ? { blocker: { owner: { name: "Release Owner" }, unblockAction: "Grant deployment access", scope: "task_wide" } } : {}) }; + const event = (seq: number, eventType: string, payload: Row, sourceKind = "runner"): Row => ({ seq, eventType, protocolSchemaVersion: 1, + sourceEventId: `event${seq}`, sourceInstanceId: sourceKind, sourceSeq: seq, + payload: { prpEvent: { schema: "paperclip.prp.event.v1", schemaVersion: 1, runId: "run", eventType, + sourceKind, sourceEventId: `event${seq}`, sourceInstanceId: sourceKind, sourceSeq: seq, payload } } }); + const tool = blocked ? "paperclip_block" : "paperclip_finish"; + return { caseId: blocked ? "native-blocked-report" : "assigned-skill-explicit-invocation", companyId: "company", agentId: "agent", marker: "BLOCKED_probe", + issue: { id: "issue", companyId: "company", assigneeAgentId: "agent", status: result.reportedWorkDisposition }, + runs: [{ id: "run", nativeIssueId: "issue", companyId: "company", agentId: "agent", status: "succeeded", runtimeMode: "native", resultJson: { nativeResult: result } }], + comments: [{ createdByRunId: "run", authorAgentId: "agent", body }], + initial: { issueIds: [], agentIds: ["agent"] }, state: { issueIds: ["issue"], agentIds: ["agent"], documentCount: blocked ? 0 : 1, interactionCount: 0 }, workspaceChanged: false, + events: [compatibility ? event(1, "item.started", { kind: "tool_call", item: { type: "tool_call", id: "tool", name: tool } }) : event(1, "tool.execution.started", { name: tool, executionId: "tool" }), event(2, "run.result.proposed", result), + compatibility ? event(3, "item.completed", { kind: "tool_result", item: { type: "tool_result", status: "completed", id: "tool" } }) : event(3, "tool.execution.completed", { name: tool, status: "completed", executionId: "tool" }), + event(4, "item.completed", { kind: "agentMessage", channel: "final", item: { type: "agentMessage", phase: "final_answer", text: body, channel: "final" } }), + event(5, "run.result.accepted", { result }, "control_plane"), event(6, "run.terminal", { runTerminalState: "succeeded", turnTerminalState: "completed" }, "control_plane")], + }; +} +function payload(input: NativeCompletionObservation, index: number): Row { return ((input.events[index]!.payload as Row).prpEvent as Row).payload as Row; } +describe("native completion independent oracle", () => { + it.each([[true, false], [true, true], [false, false], [false, true]])("accepts disposition %s compatibility %s with final before late control-plane acceptance", (blocked, compatibility) => expect(gradeNativeCompletion(sample(blocked, compatibility)).passed).toBe(true)); + it.each(["paperclip_finish", "paperclip_block"])("carries normalized %s identity through rehydration into the exact-call oracle", name => { + // The Rust normalization calibration asserts these exact fixture bytes. + const fixture = JSON.parse(readFileSync(new URL("./fixtures/native-completion/terminal-tool-carrier.json", import.meta.url), "utf8")) as { + cases: Array<{ name: string; normalizedPayload: Row }>; + }; + const normalized = fixture.cases.find(value => value.name === name)!.normalizedPayload; + const started = rehydrateRunnerdItemNotification(normalized, "thread", "turn"); + expect(started.item).toMatchObject({ id: "terminal-call", type: "tool_call", name }); + const completed = rehydrateRunnerdItemNotification({ provider: "codex", itemId: "terminal-call", kind: "tool_result", status: "completed", channel: "detail", text: null }, "thread", "turn"); + const value = sample(name === "paperclip_block", true); + payload(value, 0).item = started.item; + payload(value, 2).item = completed.item; + expect(gradeNativeCompletion(value).passed).toBe(true); + (payload(value, 2).item as Row).id = "different-call"; + expect(gradeNativeCompletion(value).checks.find(check => check.id === "final-after-tool-result")?.passed).toBe(false); + }); + it("accepts authoritative acceptance before the provider final", () => { + const value = sample(); + const events = [...value.events]; + value.events = [events[0]!, events[1]!, events[2]!, events[4]!, events[3]!, events[5]!].map((event, index) => ({ ...event, seq: index + 1 })); + expect(gradeNativeCompletion(value).passed).toBe(true); + }); + it("accepts complete public streams containing non-PRP lifecycle rows", () => { + const value = sample(); + value.events = [{ seq: 1, eventType: "lifecycle" }, ...value.events.map(event => ({ ...event, seq: Number(event.seq) + 1 }))]; + expect(gradeNativeCompletion(value).passed).toBe(true); + }); + it.each(["canonical", "compatibility"])("rejects an unmatched %s terminal result", kind => { + const value = sample(true, kind === "compatibility"); + if (kind === "canonical") payload(value, 2).executionId = "unmatched"; + else (payload(value, 2).item as Row).id = "unmatched"; + expect(gradeNativeCompletion(value).passed).toBe(false); + }); + it.each(["missing-name", "unrelated-named-tool", "contradictory-result-name"])("rejects compatibility finishing identity %s despite a matching ID", name => { + const value = sample(true, true); + if (name === "missing-name") delete (payload(value, 0).item as Row).name; + if (name === "unrelated-named-tool") (payload(value, 0).item as Row).name = "write_document"; + if (name === "contradictory-result-name") (payload(value, 2).item as Row).name = "write_document"; + expect(gradeNativeCompletion(value).checks.find(check => check.id === "final-after-tool-result")?.passed).toBe(false); + }); + it.each(["extra-run", "retry", "continuation", "wrong-account", "wrong-agent", "wrong-disposition", "punctuated-action", "wrong-scope", "missing-final", "summary-fallback", "pre-tool-final", "post-admission-call", "missing-acceptance", "runner-acceptance", "failed-terminal", "event-gap", "binding-mismatch", "extra-task", "extra-agent", "extra-document", "interaction", "changed-workspace", "process-call", "marker-only", "contradiction", "wrong-reply-run"])("rejects %s", name => { + const value = sample(); const run = value.runs[0]!; + if (name === "extra-run") value.runs = [run, { ...run, id: "extra" }]; + if (name === "retry") run.retryOfRunId = "prior"; + if (name === "continuation") run.continuationAttempt = 1; + if (name === "wrong-account") run.companyId = "other"; + if (name === "wrong-agent") run.agentId = "other"; + if (name === "wrong-disposition") value.issue.status = "done"; + if (name === "punctuated-action") ((run.resultJson as Row).nativeResult as {blocker: {unblockAction: string}}).blocker.unblockAction = "Grant deployment access."; + if (name === "wrong-scope") ((run.resultJson as Row).nativeResult as {blocker: {scope: string}}).blocker.scope = "step"; + if (name === "missing-final" || name === "summary-fallback") payload(value, 3).kind = "summary", (payload(value, 3).item as Row).type = "summary"; + if (name === "pre-tool-final") (payload(value, 2).name = "other"); + if (name === "post-admission-call") value.events[3]!.eventType = "tool.execution.started"; + if (name === "missing-acceptance") value.events[4]!.eventType = "other"; + if (name === "runner-acceptance") ((value.events[4]!.payload as Row).prpEvent as Row).sourceKind = "runner"; + if (name === "failed-terminal") payload(value, 5).runTerminalState = "failed"; + if (name === "event-gap") value.events[1]!.seq = 20; + if (name === "binding-mismatch") ((value.events[1]!.payload as Row).prpEvent as Row).runId = "other"; + if (name === "extra-task") value.state.issueIds.push("extra"); + if (name === "extra-agent") value.state.agentIds.push("extra"); + if (name === "extra-document") value.state.documentCount = 1; + if (name === "interaction") value.state.interactionCount = 1; + if (name === "changed-workspace") value.workspaceChanged = true; + if (name === "process-call") payload(value, 0).transport = "process"; + if (name === "marker-only" || name === "contradiction") { + const text = name === "marker-only" ? "BLOCKED_probe" : "Deployment is not blocked. Release Owner completed Grant deployment access. BLOCKED_probe"; + (payload(value, 3).item as Row).text = text; value.comments[0]!.body = text; + } + if (name === "wrong-reply-run") value.comments[0]!.createdByRunId = "other"; + expect(gradeNativeCompletion(value).passed).toBe(false); + }); + it.each([undefined, "other-issue"])("rejects missing or wrong native issue attribution %s", nativeIssueId => { + const value = sample(); value.runs[0]!.nativeIssueId = nativeIssueId; + expect(gradeNativeCompletion(value).passed).toBe(false); + }); + it.each(["missing-action-envelope", "wrong-action-source", "missing-source-identity", "extra-wrong-source-result"])("rejects %s", name => { + const value = sample(); + if (name === "missing-action-envelope") value.events[0]!.payload = {}; + if (name === "wrong-action-source") ((value.events[0]!.payload as Row).prpEvent as Row).sourceKind = "control_plane"; + if (name === "missing-source-identity") { + delete value.events[0]!.sourceEventId; + delete ((value.events[0]!.payload as Row).prpEvent as Row).sourceEventId; + } + if (name === "extra-wrong-source-result") { + const extra = structuredClone(value.events[4]!); + ((extra.payload as Row).prpEvent as Row).sourceKind = "runner"; + value.events = [...value.events, { ...extra, seq: 7 }]; + } + expect(gradeNativeCompletion(value).passed).toBe(false); + }); +}); diff --git a/tests/runner-e2e/native-completion-scoring.ts b/tests/runner-e2e/native-completion-scoring.ts new file mode 100644 index 00000000000..86740c45808 --- /dev/null +++ b/tests/runner-e2e/native-completion-scoring.ts @@ -0,0 +1,135 @@ +import { isValidNativePrpEnvelope } from "./native-event-envelope.js"; + +type Row = Record; +const row = (value: unknown): Row => value && typeof value === "object" && !Array.isArray(value) ? value as Row : {}; +const forbiddenKinds = new Set(["commandExecution", "process", "fileChange", "file_change", "apply_patch"]); +const toolKinds = new Set(["dynamicToolCall", "mcpToolCall", "commandExecution", "tool_call", "tool_use"]); +const terminalName = (value: unknown, expected: string) => typeof value === "string" && (value === expected || value.endsWith(`.${expected}`)); + +export interface NativeCompletionObservation { + caseId: "assigned-skill-explicit-invocation" | "native-blocked-report"; + companyId: string; agentId: string; issue: Row; runs: readonly Row[]; + comments: readonly Row[]; events: readonly Row[]; + state: { issueIds: string[]; agentIds: string[]; documentCount: number; interactionCount: number }; + initial: { issueIds: string[]; agentIds: string[] }; + workspaceChanged: boolean; + marker: string; +} + +/** Observable tool/result/final sequence, not a claim that the provider consumed feedback. */ +export function gradeNativeCompletion(input: NativeCompletionObservation) { + const checks: Array<{ id: string; passed: boolean; detail: string }> = []; + const check = (id: string, passed: boolean, detail: string) => checks.push({ id, passed, detail }); + const blocked = input.caseId === "native-blocked-report"; + const run = input.runs[0] ?? {}; + const expectedDisposition = blocked ? "blocked" : "done"; + const expectedTool = blocked ? "paperclip_block" : "paperclip_finish"; + const nativeResult = row(row(run.resultJson).nativeResult); + check("one-native-attempt", input.runs.length === 1 && run.status === "succeeded" && run.runtimeMode === "native" + && typeof run.id === "string" && run.id.length > 0 && typeof input.issue.id === "string" && input.issue.id.length > 0 + && run.nativeIssueId === input.issue.id + && run.companyId === input.companyId && run.agentId === input.agentId + && !run.retryOfRunId && !run.continuationAttempt && !input.issue.scheduledRetry, + "Exactly one succeeded native run, with no continuation/retry or uncounted extra work."); + check("durable-disposition", input.issue.companyId === input.companyId && input.issue.assigneeAgentId === input.agentId + && input.issue.status === expectedDisposition && nativeResult.reportedWorkDisposition === expectedDisposition, + "Public durable issue status and the native semantic result agree with the requested disposition."); + if (blocked) { + const blocker = row(nativeResult.blocker); + check("exact-blocker", row(blocker.owner).name === "Release Owner" && blocker.unblockAction === "Grant deployment access" + && blocker.scope === "task_wide", "The structured owner, exact action and whole-task scope are independently checked."); + } + + const events = input.events.map(event => ({ event, envelope: row(row(event.payload).prpEvent) })); + const bound = events.length > 0 && events.every(({ event, envelope }, index) => { + const action = /^(?:tool\.execution\.|item\.)/.test(String(event.eventType)); + const required = action || ["run.result.proposed", "run.result.accepted", "run.terminal"].includes(String(event.eventType)); + return event.seq === index + 1 + && (Object.keys(envelope).length === 0 ? !required : isValidNativePrpEnvelope(envelope, event.protocolSchemaVersion as number | undefined) + && (!action || envelope.sourceKind === "runner") + && typeof envelope.sourceEventId === "string" && envelope.sourceEventId.length > 0 + && typeof envelope.sourceInstanceId === "string" && envelope.sourceInstanceId.length > 0 + && Number.isSafeInteger(envelope.sourceSeq) && Number(envelope.sourceSeq) > 0 + && envelope.runId === run.id && envelope.eventType === event.eventType + && envelope.sourceEventId === event.sourceEventId && envelope.sourceInstanceId === event.sourceInstanceId + && envelope.sourceSeq === event.sourceSeq); + }); + check("complete-bound-events", bound, "A complete contiguous public PRP stream is bound to this run and persisted source identity."); + const selected = (type: string) => events.filter(({ event }) => event.eventType === type); + const proposed = selected("run.result.proposed"); + const accepted = selected("run.result.accepted"); + const terminal = selected("run.terminal"); + const acceptedResult = row(row(accepted[0]?.envelope.payload).result); + const terminalResult = row(terminal[0]?.envelope.payload); + check("authoritative-native-result", proposed.length === 1 && accepted.length === 1 && terminal.length === 1 + && proposed[0]?.envelope.sourceKind === "runner" && accepted[0]?.envelope.sourceKind === "control_plane" && terminal[0]?.envelope.sourceKind === "control_plane" + && row(proposed[0]?.envelope.payload).reportedWorkDisposition === expectedDisposition + && acceptedResult.reportedWorkDisposition === expectedDisposition + && terminalResult.runTerminalState === "succeeded" && terminalResult.turnTerminalState === "completed", + "One runner proposal is independently accepted and terminated by the control plane."); + const proposalSeq = Number(proposed[0]?.event.seq ?? -1); + const finishes = events.filter(({ event, envelope }) => { + const payload = row(envelope.payload), item = row(payload.item); + const completion = envelope.sourceKind === "runner" && Number(event.seq) > proposalSeq && ( + event.eventType === "tool.execution.completed" && terminalName(payload.name, expectedTool) && payload.status === "completed" + || event.eventType === "item.completed" && (payload.kind === "tool_result" || item.type === "tool_result") && item.status === "completed" + ); + if (!completion) return false; + return events.some(({ event: start, envelope: startEnvelope }) => { + const started = row(startEnvelope.payload), startedItem = row(started.item); + return startEnvelope.sourceKind === "runner" && Number(start.seq) < proposalSeq && ( + event.eventType === "tool.execution.completed" && start.eventType === "tool.execution.started" + && typeof payload.executionId === "string" && payload.executionId.length > 0 && started.executionId === payload.executionId && terminalName(started.name, expectedTool) + || event.eventType === "item.completed" && start.eventType === "item.started" + && (started.kind === "tool_call" || startedItem.type === "tool_call") + && typeof item.id === "string" && item.id.length > 0 && item.id === startedItem.id + && terminalName(startedItem.name, expectedTool) + && (item.name === undefined || item.name === null || terminalName(item.name, expectedTool)) + ); + }); + }); + const finals = events.filter(({ event, envelope }) => { + const payload = row(envelope.payload), item = row(payload.item); + return envelope.sourceKind === "runner" && event.eventType === "item.completed" + && (payload.kind === "agentMessage" || item.type === "agentMessage") + && (payload.channel === "final" || item.channel === "final") && item.phase === "final_answer" + && typeof item.text === "string" && item.text.trim().length > 0; + }); + const finishSeq = Number(finishes.at(-1)?.event.seq ?? -1); + const finalSeq = Number(finals[0]?.event.seq ?? -1); + const starts = events.filter(({ event, envelope }) => envelope.sourceKind === "runner" && ( + event.eventType === "tool.execution.started" || event.eventType === "item.started" + && toolKinds.has(String(row(envelope.payload).kind ?? row(row(envelope.payload).item).type)))); + check("final-after-tool-result", proposed.length === 1 && finishes.length > 0 && finals.length === 1 + && proposalSeq < finishSeq && finishSeq < finalSeq + && proposalSeq < Number(accepted[0]?.event.seq ?? -1) && Number(accepted[0]?.event.seq ?? -1) < Number(terminal[0]?.event.seq ?? -1) + && !starts.some(({ event }) => Number(event.seq) > proposalSeq), + "The provider final follows the observed finishing result; no new invocation follows admission. Control-plane acceptance precedes its terminal receipt independently."); + const finalText = String(row(row(finals[0]?.envelope.payload).item).text ?? "").trim(); + const replies = input.comments.filter(comment => comment.createdByRunId === run.id && comment.authorAgentId === input.agentId + && typeof comment.body === "string" && comment.body.trim() === finalText); + check("persisted-provider-final", finalText.length > 0 && replies.length === 1, + "A nonempty provider final is durably projected as this run's agent reply; semantic-summary fallback is insufficient."); + if (blocked) check("visible-blocker-content", finalText.split(input.marker).length === 2 + && finalText.includes("Release Owner") && finalText.includes("Grant deployment access") + && /\b(?:blocked|cannot proceed|can't proceed|missing|required access|not (?:yet )?granted|awaiting|waiting|unavailable)\b/i.test(finalText) + && !/\b(?:not blocked|no longer blocked|access (?:is |has been |was )?already granted|completed Grant deployment access)\b/i.test(finalText), + "The final explains the actual unresolved blocker and its owner/action, rather than supplying only a marker."); + const same = (a: string[], b: string[]) => a.length === b.length && a.every(id => b.includes(id)) && new Set(a).size === a.length; + check("bounded-durable-work", same(input.state.issueIds, [...input.initial.issueIds, String(input.issue.id)]) + && same(input.state.agentIds, input.initial.agentIds) && input.state.documentCount === (blocked ? 0 : 1) + && input.state.interactionCount === 0, + "No additional tasks, agents or interactions; completion permits its one required document and blocker permits none."); + if (blocked) { + const forbidden = events.some(({ event, envelope }) => { + const payload = row(envelope.payload), item = row(payload.item); + return envelope.sourceKind === "runner" && (forbiddenKinds.has(String(payload.kind ?? item.type)) + || String(event.eventType).startsWith("tool.execution.") && payload.transport === "process" + || /(?:write_document|create_task|hire_agent|register_deliverable|apply_patch|exec_command)/.test(String(payload.name ?? item.name ?? ""))); + }); + check("no-deployment-or-file-work", !input.workspaceChanged && !forbidden, + "The fixture workspace is unchanged and no process/file/deployment or extra-work tool is observed."); + } + return { schema: "paperclip.native-completion-observation.v2", passed: checks.every(value => value.passed), checks, + limitations: ["Exact provider feedback identity/consumption is not measured by the public sequence."] }; +} diff --git a/tests/runner-e2e/native-completion-source-contract.mjs b/tests/runner-e2e/native-completion-source-contract.mjs new file mode 100644 index 00000000000..6c6ea682ab1 --- /dev/null +++ b/tests/runner-e2e/native-completion-source-contract.mjs @@ -0,0 +1,155 @@ +// Exact native-only comparison boundary: executable fixtures stay byte-identical. +import { createHash } from "node:crypto"; +import { readFileSync } from "node:fs"; +import { join, resolve } from "node:path"; + +export const NATIVE_COMPLETION_SOURCE_CONTRACT = { + "schema": "paperclip.native-completion-source-contract.v1", + "baseSha": "59c07ede72dc08b8aba149a01cc11e0b7a204621", + "archiveSha": "9138f570c341c251a5727c32d6615ce238bc8e03", + "shallowParentAnchors": { + "candidate": "0a9c5a75164cd0b02115ff12273aadb6a86c9464", + "historical": "00a761b967f9f73b0f45069c0ba828fae277a76e" + }, + "variants": { + "candidate": { + "packages/paperclip-runner/src/contracts/completion-result.ts": "bd4bb79d2be5c4ee3765df67dfb4da98ef95bc532cac872be40cb41a6bff6cda", + "packages/paperclip-runner/src/drivers/codex/codex-driver-values.ts": "aff3b3d2d626fbaf504a7e46adb27cb8127d0afbc62a8ed58111938f1293de7d", + "packages/paperclip-runner/src/drivers/runner-tool-bridge.ts": "4cca0d7837e0f5a1f709abb0ba3982227288a01abd16a08107337416fde4a9e2", + "packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts": "ec1cd94c17be22d0556d4586fa6407b28e2309ed7c2437d95a32beeb0755cc4f", + "server/src/services/native-runtime/native-session-resume.ts": "5b27465e1a328412770fc27c4bc637e88b62022adc6a67709bb74187eadc465b", + "packages/paperclip-runner/src/contracts/completion-result.test.ts": "42bd18105505da7fd7963311bf862002df7158f8c0cf2a81b927dff4c47ac500", + "packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.lifecycle.test.ts": "2b15425514d8e3316baec48b8ce118394acf67d97b76cbab2a99916fc4e8e1d2", + "packages/paperclip-runner/src/drivers/runner-tool-bridge.test.ts": "097d4973ac4ca200e82af31aece0f9d5531e1756ffe739d4422bd8f65d479232", + "packages/paperclip-runner/src/drivers/opencode/mcp-bridge.test.ts": "533dd12ea00c61e012f5d596e03080fa1aaf63375ad7602c0de4e0a6c8370400", + "packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts": "8546181e5b3a63a194ee9ea0fc99c2bd97562f0c105fffde298fa6d74b22e83b", + "server/src/services/native-runtime/native-session-resume.test.ts": "57b498d76bc89801edc81d700cd87660a16074ae8bb8a43aa566205cc377d7b4" + }, + "historical": { + "packages/paperclip-runner/src/contracts/completion-result.ts": "00a684f9aea87f68eabf0974681c52461bc308631a1252a89c4ed087d7158b28", + "packages/paperclip-runner/src/drivers/codex/codex-driver-values.ts": "87929dd81b6fdc44fb46e5b14ac775ca7244354587028b3bd356a617dcf31907", + "packages/paperclip-runner/src/drivers/runner-tool-bridge.ts": "966696d39eb9554e5f40bd1d89d31efc426c3c01d2ce883b3b6e968db8480088", + "packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts": "7080ae2896fe7a81fdf1a1c4f7fcd1773611cbd3336896b5f9e861c5b0aa0ed4", + "server/src/services/native-runtime/native-session-resume.ts": "72b4f2e19c01a83ccd52e21102d2c5422265eec84b589ea75a7cffe40d1d0b8d", + "packages/paperclip-runner/src/contracts/completion-result.test.ts": "7c615dc36ad07b7659c3605f94fec89ae592727ce078a62fa45eb1bb549f26d5", + "packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.lifecycle.test.ts": "0b7e8b7a552d6a3d74d761e0e68b720ed02d621ded41a1c2076ca3fda0f96d0c", + "packages/paperclip-runner/src/drivers/runner-tool-bridge.test.ts": "1ce59108e0cfef78fa71f0da9b46af70fd7bf3887d4a662556afc072d2351653", + "packages/paperclip-runner/src/drivers/opencode/mcp-bridge.test.ts": "bb1b0447cca30f3a2984e9e832949108c43cefa35d765440d07ee8098be4ef0c", + "packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts": "624bbac680e1dfc87fff9fab71501e8208e0bdbc6aefcb45a008b8e6c9030a27", + "server/src/services/native-runtime/native-session-resume.test.ts": "ddd60db81cdb8d17a2b31ea934779687708b927f877f3fb6d0c249ceb805be62" + } + }, + "fixedContext": { + "server/src/onboarding-assets/default/AGENTS.md": "e4d2375d722602cd744403292d99f6e6c9b7e8014d9cdfa6f9811ace03428e5f", + "packages/adapter-utils/src/server-utils.ts": "916e879f70a90d31c89bc1486d5f84807c4ecafeab62961f67d84d1ebec7a173", + "skills/paperclip/SKILL.md": "19b4e17bc6588b087efce63673c3f2839ac09dba9c072f56b8e90924a66856de", + "server/src/services/onboarding-first-task-assets.ts": "c7fd1db7ed97850bd67e48a4d8c1470c8ff801f895f4f665b84690978f19b47e", + "server/src/services/default-agent-instructions.ts": "8bd3b59f900c132db5ed89b96524f35ba7c8dafb9d58db1dac888430f08c4d63", + "packages/paperclip-runner/src/contracts/runtime-context.ts": "a17e30a69f3cb38fba00c37e1a06753d723651a004e6050c9e77ae202a61264b", + "packages/paperclip-runner/src/backends/runtime-context.ts": "e7c46e81cc9c93f9a4f805b5091ef08c59ea70c62f208cfda70d6edeed363898", + "server/src/services/native-runtime/native-execution-input.ts": "e346bf3f163dfc5374c3c90fb515c1db6847bbab69fb59a49d73227478cef70d", + "server/src/services/native-runtime/completion-contracts.ts": "b1f4f7a9b04fea902c23313adb296e67f202ee5878424f85a7d0cebcb775ab69", + "server/src/services/native-runtime/runtime-context.ts": "13f71f749ad6a3a0612e3e268fdc2bc58ee26f70399dd120b1341abf00d1847b", + "packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts": "1a3b80e9b155a1e7e1f72313ad41bb0a397d6d42cb6aedf23704c1cf9121c0b8" + } +}; +export const NATIVE_COMPLETION_FIXTURE_FILES = [ + "tests/runner-e2e/native-completion-cases.ts", + "tests/runner-e2e/native-completion-scoring.ts", + "tests/runner-e2e/native-completion-scoring.test.ts", + "tests/runner-e2e/native-completion-defaults.ts", + "tests/runner-e2e/native-completion-defaults.test.ts", + "tests/runner-e2e/native-blocker-visible.ts", + "tests/runner-e2e/native-blocker-visible.spec.ts", + "tests/runner-e2e/native-completion-admission.ts", + "tests/runner-e2e/native-completion-admission.test.ts", + "tests/runner-e2e/native-completion-checks.mjs", + "tests/runner-e2e/native-completion-checks.test.mjs", + "tests/runner-e2e/native-completion-git-source.mjs", + "tests/runner-e2e/native-completion-git-source.test.mjs", + "tests/runner-e2e/fixtures/native-completion/terminal-tool-carrier.json", + "tests/runner-e2e/native-completion-source-contract.mjs", + "tests/runner-e2e/native-completion-source-contract.test.mjs", + "tests/runner-e2e/automatic-retry.ts", + "tests/runner-e2e/automatic-retry.test.ts", + "tests/runner-e2e/launch.ts", + "tests/runner-e2e/types.ts", + "tests/runner-e2e/catalog.ts", + "tests/runner-e2e/catalog.test.ts", + "tests/runner-e2e/runner.spec.ts", + "tests/runner-e2e/playwright-support.config.ts", + "tests/runner-e2e/context-integrity-cases.ts", + "tests/runner-e2e/context-integrity.test.ts", + "tests/runner-e2e/context-integrity-evidence.test.ts", + "tests/runner-e2e/context-integrity-flow-logs.test.ts", + "tests/runner-e2e/live-fixtures.ts", + "tests/runner-e2e/context-integrity-flow.ts", + "tests/runner-e2e/context-integrity-scoring.ts", + "tests/runner-e2e/selectors.ts", + "tests/runner-e2e/harness-env.ts", + "tests/runner-e2e/tsconfig.json", + "tests/runner-e2e/vitest.config.ts" +]; +export const NATIVE_COMPLETION_MANIFEST_FILES = [ + "packages/paperclip-runner/scripts/generate-capability-contract.mjs", + "packages/paperclip-runner/scripts/check-capability-inventory.mjs", + "packages/paperclip-runner/scripts/lib/capability-inventory.mjs", + "packages/paperclip-runner/spec/capability/source-contract.json", + "packages/paperclip-runner/spec/capability/capabilities.yaml", + "packages/paperclip-runner/spec/capability/eval-traceability.yaml", + "packages/paperclip-runner/spec/capability/mcp-tool-map.yaml", + "packages/paperclip-runner/spec/capability/inventory.schema.json", + "packages/paperclip-runner/src/generated/capability-contract.ts", + "packages/paperclip-runner/docs/capability-contract.md", + ...["capabilities.yaml", "mcp-tool-map.yaml", "eval-traceability.yaml", "capability-contract.md", "downstream-handoff.md"] + .map(file => `packages/paperclip-runner/generated/capability/${file}`), +]; +export const NATIVE_COMPLETION_SOURCE_FILES = [...new Set([ + ...Object.keys(NATIVE_COMPLETION_SOURCE_CONTRACT.variants.candidate), + ...Object.keys(NATIVE_COMPLETION_SOURCE_CONTRACT.fixedContext), + ...NATIVE_COMPLETION_FIXTURE_FILES, ...NATIVE_COMPLETION_MANIFEST_FILES, + "scripts/ensure-plugin-build-deps.mjs", + "packages/paperclip-runner/scripts/build-verified-provider-entrypoints.mjs", + "packages/paperclip-runner/src/live/runnerd-codex-transport.ts", + "packages/paperclip-runner/src/drivers/acpx/qualified-profiles.ts", + "packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts", + "packages/adapters/codex-local/src/index.ts", "packages/adapters/claude-local/src/index.ts", + "packages/paperclip-runner/package.json", "tsconfig.base.json", + "packages/paperclip-runner/runner/crates/runner-core/src/provider_events.rs", + "packages/paperclip-runner/runner/Cargo.toml", "packages/paperclip-runner/runner/Cargo.lock", + "packages/paperclip-runner/runner/crates/runner-core/Cargo.toml", +])].sort(); + +export const nativeSourceSha256 = bytes => createHash("sha256").update(bytes).digest("hex"); +function fingerprint(paths, read) { + const hash = createHash("sha256"); + for (const file of [...paths].sort()) { + const bytes = read(file); + hash.update(JSON.stringify([file, bytes.length])).update(bytes); + } + return hash.digest("hex"); +} + +/** Infer a variant only from all eleven matching production/assertion files. */ +export function nativeCompletionSourceFingerprint(read = file => readFileSync(join( + resolve(import.meta.dirname, "../.."), file)), contract = NATIVE_COMPLETION_SOURCE_CONTRACT) { + const observations = new Map(), sourceErrors = []; + for (const file of NATIVE_COMPLETION_SOURCE_FILES) { + try { observations.set(file, read(file)); } catch { sourceErrors.push(file); } + } + const matches = Object.entries(contract.variants).filter(([, expected]) => Object.entries(expected) + .every(([file, digest]) => observations.has(file) && nativeSourceSha256(observations.get(file)) === digest)); + const variant = matches.length === 1 ? matches[0][0] : null; + if (!variant) sourceErrors.push("native-completion: mixed or unknown production/assertion variant"); + for (const [file, digest] of Object.entries(contract.fixedContext)) + if (!observations.has(file) || nativeSourceSha256(observations.get(file)) !== digest) + sourceErrors.push(`native-completion: changed master context ${file}`); + const fromObservation = file => observations.get(file); + const complete = NATIVE_COMPLETION_SOURCE_FILES.every(file => observations.has(file)); + return { variant, sourceErrors, + fingerprint: complete ? fingerprint(NATIVE_COMPLETION_SOURCE_FILES, fromObservation) : null, + fixtureFingerprint: complete ? fingerprint(NATIVE_COMPLETION_FIXTURE_FILES, fromObservation) : null, + manifestFingerprint: complete ? fingerprint(NATIVE_COMPLETION_MANIFEST_FILES, fromObservation) : null, + baseSha: contract.baseSha, archiveSha: contract.archiveSha, + }; +} diff --git a/tests/runner-e2e/native-completion-source-contract.test.mjs b/tests/runner-e2e/native-completion-source-contract.test.mjs new file mode 100644 index 00000000000..d6044b2b51f --- /dev/null +++ b/tests/runner-e2e/native-completion-source-contract.test.mjs @@ -0,0 +1,74 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { + NATIVE_COMPLETION_SOURCE_CONTRACT as original, NATIVE_COMPLETION_SOURCE_FILES, + nativeCompletionSourceFingerprint, nativeSourceSha256, +} from "./native-completion-source-contract.mjs"; + +function fixture(variant = "candidate") { + const bytes = new Map(NATIVE_COMPLETION_SOURCE_FILES.map(file => [file, Buffer.from(`common:${file}`)])); + const variantFiles = Object.keys(original.variants.candidate), fixed = Object.keys(original.fixedContext); + const contract = { ...original, variants: Object.fromEntries(["candidate", "historical"].map(name => [name, + Object.fromEntries(variantFiles.map(file => [file, nativeSourceSha256(Buffer.from(`${name}:${file}`))]))])), + fixedContext: Object.fromEntries(fixed.map(file => [file, nativeSourceSha256(bytes.get(file))])) }; + for (const file of variantFiles) bytes.set(file, Buffer.from(`${variant}:${file}`)); + const read = file => { if (!bytes.has(file)) throw new Error("Missing source"); return bytes.get(file); }; + return { bytes, contract, read }; +} +for (const variant of ["candidate", "historical"]) test(`native source contract accepts the complete ${variant} source map`, () => { + const f = fixture(variant), result = nativeCompletionSourceFingerprint(f.read, f.contract); + assert.equal(result.variant, variant); assert.deepEqual(result.sourceErrors, []); + for (const key of ["fingerprint", "fixtureFingerprint", "manifestFingerprint"]) assert.match(result[key], /^[a-f0-9]{64}$/); +}); +test("native source contract keeps fixtures identical across candidate and historical variants", () => { + const a = fixture("candidate"), b = fixture("historical"); + const candidate = nativeCompletionSourceFingerprint(a.read, a.contract), historical = nativeCompletionSourceFingerprint(b.read, b.contract); + assert.equal(candidate.fixtureFingerprint, historical.fixtureFingerprint); + assert.equal(candidate.manifestFingerprint, historical.manifestFingerprint); + assert.notEqual(candidate.fingerprint, historical.fingerprint); +}); +for (const file of Object.keys(original.variants.candidate)) test(`native source contract rejects a mixed production/assertion source: ${file}`, () => { + const f = fixture(); f.bytes.set(file, Buffer.from(`historical:${file}`)); + const result = nativeCompletionSourceFingerprint(f.read, f.contract); + assert.equal(result.variant, null); assert.ok(result.sourceErrors.some(error => error.includes("mixed or unknown"))); +}); +for (const file of Object.keys(original.fixedContext)) test(`native source contract rejects changed master context: ${file}`, () => { + const f = fixture(); f.bytes.set(file, Buffer.from("Reduced or replaced context")); + const result = nativeCompletionSourceFingerprint(f.read, f.contract); + assert.ok(result.sourceErrors.includes(`native-completion: changed master context ${file}`)); +}); +test("native source contract rejects missing files and changes its digest when a fixture changes", () => { + const f = fixture(), before = nativeCompletionSourceFingerprint(f.read, f.contract); + f.bytes.set("tests/runner-e2e/native-completion-cases.ts", Buffer.from("Changed behavior")); + const changed = nativeCompletionSourceFingerprint(f.read, f.contract); + assert.notEqual(before.fixtureFingerprint, changed.fixtureFingerprint); + f.bytes.delete("tests/runner-e2e/native-completion-cases.ts"); + const missing = nativeCompletionSourceFingerprint(f.read, f.contract); + assert.equal(missing.fingerprint, null); assert.ok(missing.sourceErrors.includes("tests/runner-e2e/native-completion-cases.ts")); +}); +test("native source contract rejects an ambiguous identical variant map", () => { + const f = fixture(); f.contract.variants.historical = f.contract.variants.candidate; + assert.equal(nativeCompletionSourceFingerprint(f.read, f.contract).variant, null); +}); +test("native source contract binds common Rust carrier source and the shared behavioral projection fixture", () => { + const f = fixture(), before = nativeCompletionSourceFingerprint(f.read, f.contract); + f.bytes.set("packages/paperclip-runner/runner/crates/runner-core/src/provider_events.rs", Buffer.from("Changed carrier")); + const carrier = nativeCompletionSourceFingerprint(f.read, f.contract); + assert.equal(carrier.variant, "candidate"); assert.notEqual(carrier.fingerprint, before.fingerprint); + f.bytes.set("tests/runner-e2e/fixtures/native-completion/terminal-tool-carrier.json", Buffer.from("Changed projection")); + const projection = nativeCompletionSourceFingerprint(f.read, f.contract); + assert.notEqual(projection.fingerprint, carrier.fingerprint); assert.notEqual(projection.fixtureFingerprint, carrier.fixtureFingerprint); +}); +test("native source contract binds exactly five production and six variant assertion files", () => { + const files = Object.keys(original.variants.candidate); + assert.equal(files.length, 11); assert.equal(files.filter(file => file.endsWith(".test.ts")).length, 6); + assert.deepEqual(files, Object.keys(original.variants.historical)); + assert.ok(files.every(file => original.variants.candidate[file] !== original.variants.historical[file])); + assert.equal(original.baseSha, "59c07ede72dc08b8aba149a01cc11e0b7a204621"); + assert.equal(original.archiveSha, "9138f570c341c251a5727c32d6615ce238bc8e03"); + assert.deepEqual(original.shallowParentAnchors, { + candidate: "0a9c5a75164cd0b02115ff12273aadb6a86c9464", + historical: "00a761b967f9f73b0f45069c0ba828fae277a76e", + }); + assert.ok(!NATIVE_COMPLETION_SOURCE_FILES.some(file => file.includes("stock-harness") || file.endsWith("issue-documents.md"))); +}); diff --git a/tests/runner-e2e/playwright-support.config.ts b/tests/runner-e2e/playwright-support.config.ts index 11a00cb0529..6d2d2976a5e 100644 --- a/tests/runner-e2e/playwright-support.config.ts +++ b/tests/runner-e2e/playwright-support.config.ts @@ -3,7 +3,7 @@ import { defineConfig } from "@playwright/test"; /** Browser-only harness regressions: no Paperclip instance or provider credentials. */ export default defineConfig({ testDir: ".", - testMatch: ["screenshot-readiness.spec.ts", "service-worker-reload.spec.ts", "lost-send.spec.ts", "chat-restart.spec.ts", "settings-toggle.spec.ts", "browser-bootstrap-diagnostics.spec.ts", "service-worker-module-loading.spec.ts", "browser-bootstrap-recovery.spec.ts", "blocker-input.spec.ts"], + testMatch: ["screenshot-readiness.spec.ts", "service-worker-reload.spec.ts", "lost-send.spec.ts", "chat-restart.spec.ts", "settings-toggle.spec.ts", "browser-bootstrap-diagnostics.spec.ts", "service-worker-module-loading.spec.ts", "browser-bootstrap-recovery.spec.ts", "blocker-input.spec.ts", "native-blocker-visible.spec.ts"], workers: 1, retries: 0, timeout: 10_000, diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index 3833d3cef02..46cc8e96aaa 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -1,3 +1,7 @@ +import { assertNativeCompletionSelection, NATIVE_COMPLETION_PREFLIGHT_ENV, verifyNativeCompletionPreflight } from "./native-completion-admission.js"; +import { captureNativeDefault, gradeNativeDefault, nativeCompletionWorkspaceDigest } from "./native-completion-defaults.js"; +import { gradeNativeCompletion } from "./native-completion-scoring.js"; +import { assertNativeBlockerReply } from "./native-blocker-visible.js"; import { warmManagedFileEvidence } from "./warm-managed-files.js"; import { gitFinalizationEvidence, gitStreamingEvidence, setupGitStreamingWorkspace } from "./daytona-git-streaming.js"; import { runsCompletionUpdateProbe, completionQualityControls, completionQualityStatus, judgeCompletionQuality, reserveCompletionQuality, type CompletionQualityRecord } from "./completion-quality.js"; @@ -552,6 +556,10 @@ for (const execution of executions) { let apiResponseSourceId: string | undefined; let titleCreation: { issue: unknown; submitted: unknown } | undefined; let lifecycleBlockerId: string | null = null; + if (execution.suite.id === "native-completion") { + assertNativeCompletionSelection([execution]); + verifyNativeCompletionPreflight(process.env[NATIVE_COMPLETION_PREFLIGHT_ENV]); + } const credentials = credentialValues(); const secrets = normalizedSecrets(Object.values(credentials)); const api = new RunnerApi(request); @@ -560,6 +568,9 @@ for (const execution of executions) { const networkDiagnostics: Array> = []; const pageLifecycleDiagnostics: Array> = []; const browserBootstrap = observeBrowserBootstrap(page); + let nativeInitial: { issueIds: string[]; agentIds: string[]; workspaceDigest: string | null } | undefined; + const nativeWorkspaceDigest = () => execution.task.id === "native-blocked-report" + ? nativeCompletionWorkspaceDigest(workspacePath!) : Promise.resolve(null); let fixtures: LiveFixtureValues | undefined; let reviewProvider: Awaited> | undefined; let issue: IssueRecord | undefined; @@ -842,6 +853,18 @@ for (const execution of executions) { daytonaImage: process.env.PAPERCLIP_E2E_DAYTONA_IMAGE, }); + if (execution.suite.id === "native-completion") { + const receipt = await captureNativeDefault({ api, agentId: fixtures.agent.id, companyId: fixtures.company.id }); + const grade = gradeNativeDefault(receipt); + await writeSanitizedJson(snapshotsDir, "native-default-before-execution.json", { receipt, grade }, secrets); + if (!grade.passed) throw new Error("Native production-default hire admission failed before execution"); + const [issues, agents, workspaceDigest] = await Promise.all([ + api.get>(`/api/companies/${fixtures.company.id}/issues?limit=100`), + api.get>(`/api/companies/${fixtures.company.id}/agents`), nativeWorkspaceDigest(), + ]); + nativeInitial = { issueIds: issues.map(value => value.id), agentIds: agents.map(value => value.id), workspaceDigest }; + } + if (execution.suite.id === "api-response-reading") { const source = await api.post<{ id: string }>(`/api/companies/${fixtures.company.id}/issues`, { title: `Synthetic diagnostic evidence ${nonce}`, @@ -2574,6 +2597,8 @@ for (const execution of executions) { await expect(proofLink).toHaveAttribute("href", `/api/attachments/${downloadedResponseProof!.attachmentId}/content`); } else if (execution.suite.id === "task-titles") { await expect(visibleAgentReplies.filter({ hasText: marker }).last()).toBeVisible({ timeout: 30_000 }); + } else if (execution.suite.id === "native-completion" && execution.task.id === "native-blocked-report") { + await assertNativeBlockerReply(visibleAgentReplies, marker); } else if (execution.task.flow === "warm_three_turn") { // Prove the persisted user-facing response is visible, independently // of the byte-for-byte workspace checks and lease continuity checks. @@ -2633,6 +2658,31 @@ for (const execution of executions) { ); } } + if (execution.suite.id === "native-completion") { + if (!issue || !nativeInitial) throw new Error("Missing native qualification issue or pre-execution receipt"); + const [currentIssue, companyRuns, issues, agents, comments, documents, interactions, workspaceDigest] = await Promise.all([ + api.get(`/api/issues/${issue.id}`), + api.get(`/api/companies/${fixtures.company.id}/heartbeat-runs?limit=100`), + api.get>(`/api/companies/${fixtures.company.id}/issues?limit=100`), + api.get>(`/api/companies/${fixtures.company.id}/agents`), + api.get(`/api/issues/${issue.id}/comments`), + api.get(`/api/issues/${issue.id}/documents`), + api.get(`/api/issues/${issue.id}/interactions`), nativeWorkspaceDigest(), + ]); + const detailedRuns = await Promise.all(companyRuns.map(candidate => api.get(`/api/heartbeat-runs/${candidate.id}`))); + selectedRuns = detailedRuns; issue = currentIssue; + const events = detailedRuns.length === 1 ? await collectRunEvents((afterSeq, limit) => api.get(`/api/heartbeat-runs/${detailedRuns[0]!.id}/events?afterSeq=${afterSeq}&limit=${limit}`)) : []; + const observation = { caseId: execution.task.id as "assigned-skill-explicit-invocation" | "native-blocked-report", + companyId: fixtures.company.id, agentId: fixtures.agent.id, issue: currentIssue as unknown as Record, + runs: detailedRuns as unknown as Record[], comments: comments as unknown as Record[], events: events as unknown as Record[], + initial: nativeInitial, state: { issueIds: issues.map(value => value.id), agentIds: agents.map(value => value.id), documentCount: documents.length, interactionCount: interactions.length }, + workspaceChanged: workspaceDigest !== nativeInitial.workspaceDigest, marker }; + const grade = gradeNativeCompletion(observation); + const integrity = detailedRuns.flatMap(candidate => nativeRunEventIntegrityFailures(candidate, events)); + await writeSanitizedJson(snapshotsDir, "native-completion.json", { observation, grade, integrity }, secrets); + matcherResults.push(...grade.checks.map(check => ({ matcher: { kind: "json_path" as const, path: `nativeCompletion.${check.id}`, expected: true }, passed: check.passed, detail: check.detail }))); + if (!grade.passed || integrity.length) throw new Error(`Native completion qualification failed: ${[...grade.checks.filter(check => !check.passed).map(check => check.id), ...integrity].join("; ")}`); + } if (runsCompletionUpdateProbe(execution) && credentials.OPENAI_API_KEY) { const qualification = completionQualityStatus(completionQuality); if (qualification === "unqualified") { diff --git a/tests/runner-e2e/types.ts b/tests/runner-e2e/types.ts index 0d230d87437..b0e3c1608a9 100644 --- a/tests/runner-e2e/types.ts +++ b/tests/runner-e2e/types.ts @@ -138,6 +138,8 @@ export interface RunnerTaskFixture { expectedRunCount: number; /** Optional lower bound; expectedRunCount remains the maximum/cost estimate. */ minimumExpectedRunCount?: number; + /** Admit only the first attempt, including provider or infrastructure failures. */ + automaticRetryPolicy?: "single_attempt"; attemptTimeoutMs: Readonly>; expectedTerminalState: { issue: "done" | "in_review" | "blocked" | "in_progress"; From 78e003449827540175e2441c05bac9eeec8dae98 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 3 Oct 2026 06:42:51 -0500 Subject: [PATCH 07/37] fix(evals): account for hiring completion notifications (#15007) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E evals check real hiring and delegated task completion. > - The hiring fixture requires three requested CEO turns and two coder executions. > - The server can also wake the CEO when each delegated task completes. > - Two exact-five-run guards rejected these valid completion turns in all four retained cells. > - This pull request validates bounded completion turns in both guards. > - The benefit is accurate workflow grading while all actual runs and coverage failures remain visible. ## Linked Issues or Issue Description Refs: #14985, #14948, #14961. **What happened?** The original hiring comparison reports Codex Fail → Fail and Claude Fail → Fail. Each cell has seven successful runs. The five requested work turns are accompanied by two server task-completion notifications. All six other delivery checks pass. **Expected behavior** Require exactly three distinct user-requested CEO turns and one coder execution for each of two known tasks. Admit at most two strictly attributed server completion turns, including one turn that batches both tasks. Reject unknown, duplicate, failed, retried or extra-work runs. **Steps to reproduce** Inspect the retained four-cell report linked below. Each original result fails `five-successful-turns`. The same exact count was also enforced by the final chat-flow guard. ## What Changed - Add one typed lifecycle helper shared by the hiring scorer and the hiring-only final chat guard. - Validate public run ledgers, company/user/account identity, request attribution, task origins, completion deliveries, timing and replies. - Keep exactly five required work turns; declare seven maximum total turns for cost and timeout planning. - Count all actual runs, including notification runs and unexpected resets. Keep other chat count guards unchanged. - Version the hiring grader as v3 (turn accounting v2) and include the helper and chat guard in its definition digest. - Keep source-read, exact coder-body and all six other delivery checks unchanged. - Add 144 focused helper/scorer/settlement calibrations and separately versioned exact retained-input replay reports. - Retry complete bracketed observations, await both owed callbacks and attributed replies, and refresh the final guard consistently. - Reject unrelated completion writes and failed mutation attempts using exact canonical/native action IDs. Missing identity mapping is uncomparable action coverage. ## Verification - All 977 credential-free E2E support tests pass across 64 files, including 144 focused lifecycle/action/scorer/settlement calibrations. - E2E typecheck, ordinary plugin SDK and Runner TypeScript dependency builds, capability contract/inventory checks and the existing two-cell hiring discovery pass. - [Executable replay report](https://github.com/paperclipai/paperclip/blob/fed1729018cc100f5f4bbfb692777e49009c423b/doc/plans/2026-10-02-hiring-executable-accounting-replay.md) pins current code revision `e4077ade1818d98b9862ae79ee1d49a007dcf9c1`, v3 definition digest, exact source/input hashes and each original/new check. - The stricter replay verifies both Codex variants through both executable guards. ACPX Claude action attribution remains unresolved/uncomparable because provider execution IDs cannot be exactly joined to native request IDs; guards fail closed. No notification writes are observed. All six other outcomes and every original source/template coverage check stay unchanged. Original files and Fail → Fail machine verdicts remain preserved; zero providers are called. - Full attempts remain uncomparable in both profiles. Historical Claude also keeps its six-backtick exact-template mismatch. This grading repair does not prove model-performance equivalence. - The limited sidecar-v1 and initial executable-v2 passes checked notification-created tasks but could miss unrelated document writes. Those assessments remain preserved and do not prove harmless notifications. The stricter v3 replay is separate. - [Original measurement and separate sidecar](https://github.com/paperclipai/paperclip/blob/8eb517ca1497687237163bdef4dfc4d3332ea916/doc/plans/2026-10-02-hiring-template-live-comparison.md) retain 28 actual runs, eight automatic notifications, four successful cleanups and unknown actual model charges. No models are rerun. - The branch is replayed on master `59c07ede7`. Intervening master changes are UI-only; eval source bytes and replay verdicts match. The four-cell provider-free replay was repeated against the reachable code revision. - Initial-head normal CI retained browser failures in agent-run denial feedback and touch-picker scroll position. Those browser paths and imports were unchanged, but their cause was not established. The necessary review-fix head passes both browser checks; no blind rerun was requested. - Local full repository typecheck/test/build were not repeated. Exact-head normal CI passes the required repository gates, including typecheck, tests, build and browser shards. Fresh Greptile review completed on `fed1729018cc100f5f4bbfb692777e49009c423b` with 5/5 and zero unresolved threads. An independent rerun of the 144 focused helper/scorer/settlement tests passes on the unchanged head. **Merge readiness:** This PR repairs the evaluator. Its positive and negative calibrations pass, both guards reject missing action attribution, current-head CI and review pass, and there are no merge conflicts. The retained ACPX cells remain uncomparable because their action IDs cannot be joined. That coverage limit remains a separate follow-up; it does not require relaxing this grader or changing the old results. No model calls, production instructions, carrier changes, or historical regrades are part of this readiness update. ## Risks - Missing or inconsistent public lifecycle evidence fails the bounded helper. The focused calibrations reject plausible false positives and malformed observations. Unmatched action IDs fail closed and are reported as uncomparable rather than a model task regression. - Source-read evidence remains incomplete. This PR does not change provider event carriers or relax the coverage oracle. - The versioned count check differs from original v1 results. Reports retain both versions and exact input hashes. ## Model Used OpenAI Codex, GPT-6 family as identified by this session. The exact deployment ID and context-window size are not exposed. The assistant used reasoning, repository tools, code execution and delegated calibration work. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip normal CI gates are green (exact head `fed1729018cc100f5f4bbfb692777e49009c423b`; fresh review tracked separately below) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (completed exact-head review; zero unresolved threads) - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/evals.md | 6 +- ...2-hiring-executable-accounting-replay.json | 740 ++++++++++++++++ ...-02-hiring-executable-accounting-replay.md | 59 ++ ...iring-executable-accounting-replay.v2.json | 815 ++++++++++++++++++ tests/runner-e2e/README.md | 26 +- tests/runner-e2e/chat-flow.ts | 35 +- tests/runner-e2e/hiring-template-cases.ts | 7 +- tests/runner-e2e/hiring-template-flow.ts | 79 +- tests/runner-e2e/hiring-template-scoring.ts | 17 +- .../hiring-template-turn-accounting.test.ts | 274 ++++++ .../hiring-template-turn-accounting.ts | 317 +++++++ .../hiring-template-turn-fixture.ts | 110 +++ tests/runner-e2e/hiring-template.test.ts | 190 +++- 13 files changed, 2631 insertions(+), 44 deletions(-) create mode 100644 doc/plans/2026-10-02-hiring-executable-accounting-replay.json create mode 100644 doc/plans/2026-10-02-hiring-executable-accounting-replay.md create mode 100644 doc/plans/2026-10-02-hiring-executable-accounting-replay.v2.json create mode 100644 tests/runner-e2e/hiring-template-turn-accounting.test.ts create mode 100644 tests/runner-e2e/hiring-template-turn-accounting.ts create mode 100644 tests/runner-e2e/hiring-template-turn-fixture.ts diff --git a/doc/evals.md b/doc/evals.md index 9c0474e9f16..f936899203b 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -81,7 +81,9 @@ The explicit-only [production hiring templates suite](../tests/runner-e2e/README adds two local native Codex/Claude cells. It exercises API-created production CEO defaults, an explicitly requested hiring skill/reference read, a permanent coder hire, independently computed saved JSON fixtures and worker reuse. -Each cell expects five turns. Source/read coverage and workflow outcome are +Each cell requires five work turns and admits at most two strictly attributed +server task-completion turns. Every actual run remains counted; unknown or +extra-work turns fail. Source/read coverage and workflow outcome are separate: missing read provenance leaves the candidate/baseline pair uncomparable even if work succeeds. Baseline bundles and coder examples derive from their own source revision, without requiring candidate wording or length. @@ -386,3 +388,5 @@ The explicit-only Product E2E `confirmation-replies` suite tests conversational approval and rejection, persisted message provenance, approval before execution, ambiguous proposals, and the existing card-click path with native Claude/Codex. See the [suite contract](../tests/runner-e2e/README.md#conversational-confirmation-replies-explicit-only). + +Hiring notification accounting now also requires exact completed action attribution. Missing native/provider ID mapping is uncomparable evidence; it must not be reported as a model task regression or waived through name/order matching. The fixture waits for both known completion callbacks and settled bracketed observations, including the gap before pending outbox work becomes a wake. Strict action replay and original machine verdicts are retained separately. diff --git a/doc/plans/2026-10-02-hiring-executable-accounting-replay.json b/doc/plans/2026-10-02-hiring-executable-accounting-replay.json new file mode 100644 index 00000000000..7a3c6caaf07 --- /dev/null +++ b/doc/plans/2026-10-02-hiring-executable-accounting-replay.json @@ -0,0 +1,740 @@ +{ + "schema": "paperclip.hiring-executable-accounting-replay.v1", + "assessmentType": "Provider-free replay through both new executable guards; original model measurements and verdicts unchanged", + "evaluatedCodeRevision": "eef64009dc144b91b245a731ea9a1c3c07406a19", + "evaluatedCodeHashes": { + "tests/runner-e2e/hiring-template-cases.ts": "06915e33cc9a5241cd28a7184fa33d098c30aed4114accea4f93838aaba3738f", + "tests/runner-e2e/hiring-template-scoring.ts": "a9e1efa7e11bad3aa8dc873a28325b3153799251740ae87c4623a08f7d45a5cb", + "tests/runner-e2e/hiring-template-flow.ts": "f3c6d29dc5019955dda4372ead7a9fdfbde447b7b92ef103527b9393ec60bd30", + "tests/runner-e2e/hiring-template-turn-accounting.ts": "8f2ad8b09e419796b372929b59dc7f86d41dcbf3a3144c45e329a1aa988b383b", + "tests/runner-e2e/chat-flow.ts": "09dd87ef7aef299ddbd1fad65b72817c042595f9e4a4e699c430497ac053a2d4" + }, + "graderVersion": "paperclip.hiring-templates.v2", + "definitionDigest": "4d527906a967ca71c4c0e62cfbdc3405419931bcd6741b0630c4223bed2d4b6a", + "providerCalls": 0, + "newProviderRuns": 0, + "automaticRetries": 0, + "summary": { + "originalPairs": "Codex Fail→Fail; Claude Fail→Fail", + "correctedWorkflowOutcomePairs": "Codex Pass→Pass; Claude Pass→Pass", + "fullAttemptComparison": "Uncomparable in both profiles and variants; source-read coverage unchanged", + "originalMachineFailuresPreserved": 4, + "correctedScorerGuardsPassed": 4, + "correctedFinalChatGuardsPassed": 4, + "unchangedCoverageAssessments": 4, + "unchangedOtherOutcomeAssessments": 4, + "actualMeasuredRuns": 28, + "actualModelChargesKnown": false, + "broadEquivalenceEstablished": false + }, + "assessments": [ + { + "variant": "candidate", + "profile": "runner-codex", + "original": { + "sourceRevision": "9f5404ad3aacbe76777952759414d34fd381e674", + "graderVersion": "paperclip.hiring-templates.v1", + "definitionDigest": "4d7f18420889325af89aa48eaaf021e8178aa6d26d504b92041635a15741f004", + "machineStatus": "failed", + "outcomePassed": false, + "countCheckPassed": false, + "comparisonStatus": "uncomparable" + }, + "corrected": { + "graderVersion": "paperclip.hiring-templates.v2", + "definitionDigest": "4d527906a967ca71c4c0e62cfbdc3405419931bcd6741b0630c4223bed2d4b6a", + "outcomePassed": true, + "comparisonStatus": "uncomparable", + "scorerGuardPassed": true, + "finalChatGuardPassed": true, + "turnAccounting": { + "version": "paperclip.hiring-template-turn-accounting.v1", + "counts": { + "requiredWorkTurns": 5, + "maximumCompletionTurns": 2, + "maximumTotalTurns": 7, + "requestedLeadTurns": 3, + "coderTurns": 2, + "completionTurns": 2, + "unclassifiedTurns": 0, + "snapshotRunCount": 7, + "actualRunCount": 7, + "costAccountingRunCount": 7 + }, + "predicates": [ + { + "id": "known-fixture-context", + "passed": true + }, + { + "id": "complete-public-run-ledger", + "passed": true + }, + { + "id": "exact-five-required-work-turns", + "passed": true + }, + { + "id": "successful-native-without-retries", + "passed": true + }, + { + "id": "resolved-company-account-and-identity", + "passed": true + }, + { + "id": "three-distinct-requested-chat-turns", + "passed": true + }, + { + "id": "one-coder-execution-per-known-task", + "passed": true + }, + { + "id": "task-origins-are-first-two-requested-turns", + "passed": true + }, + { + "id": "bounded-server-completion-receipts", + "passed": true + }, + { + "id": "completion-runs-have-attributed-chat-replies", + "passed": true + }, + { + "id": "no-notification-created-extra-tasks", + "passed": true + } + ], + "passed": true + }, + "checks": [ + { + "id": "one-coder-hire", + "dimension": "outcome", + "passed": true, + "detail": "Exactly one permanent coding teammate reports to the lead." + }, + { + "id": "execution-account", + "dimension": "outcome", + "passed": true, + "detail": "Hire keeps the lead model and managed account binding." + }, + { + "id": "two-worker-tasks", + "dimension": "outcome", + "passed": true, + "detail": "Both independent tasks are completed by the same coder in the chosen project." + }, + { + "id": "bounded-work-and-completion-turns", + "dimension": "outcome", + "passed": true, + "detail": "Exactly three requested lead turns and two coder executions, plus at most two strictly attributed completion notifications; every actual run remains counted." + }, + { + "id": "initial-json-artifact", + "dimension": "outcome", + "passed": true, + "detail": "Independent computation checks each original input/value and worker authorship." + }, + { + "id": "reused-json-artifact", + "dimension": "outcome", + "passed": true, + "detail": "The reused coder applies the changed separator to every input." + }, + { + "id": "original-preserved", + "dimension": "outcome", + "passed": true, + "detail": "Reuse preserves the original document, revision and task identity." + }, + { + "id": "source-fingerprints", + "dimension": "coverage", + "passed": true, + "detail": "Served instruction and hiring source bytes match the evaluated revision." + }, + { + "id": "production-ceo-bundle", + "dimension": "coverage", + "passed": true, + "detail": "The API-created lead receives this revision's default CEO bundle, without a fixture override." + }, + { + "id": "assigned-hiring-skill", + "dimension": "coverage", + "passed": true, + "detail": "Production CEO defaults assign the hiring skill." + }, + { + "id": "production-source-reads", + "dimension": "coverage", + "passed": false, + "detail": "Completed lead read receipts before the hire prove the explicitly requested skill, guide, checklist and coder example paths. Unrecognized or missing reads leave coverage uncomparable." + }, + { + "id": "supplied-coder-instructions", + "dimension": "coverage", + "passed": true, + "detail": "Saved hired instructions use the source revision's coder example with its company/name placeholders filled." + }, + { + "id": "hired-instructions-durable", + "dimension": "coverage", + "passed": true, + "detail": "The same saved instruction bundle survives the reused worker execution." + }, + { + "id": "hired-skills-durable", + "dimension": "coverage", + "passed": true, + "detail": "The saved skill selections survive the reused worker execution." + } + ] + }, + "coverageUnchanged": true, + "otherOutcomesUnchanged": true, + "originalBytesUnchanged": true, + "inputSha256": { + "result": "44cbf64cfabd0257f31fb4d6909018d346f5667b0a7b3d7e855a2ece7e29c292", + "hiringTemplate": "9609ddcb81c6bee3c5db97312138b4ea3e294da13ea540e41f363c826a514067", + "apiState": "ef02af29bcf61ff3eace0549ca164a28c7950d1a3e14fc58c89bf9cb0f944b4d" + } + }, + { + "variant": "candidate", + "profile": "runner-acpx-claude", + "original": { + "sourceRevision": "9f5404ad3aacbe76777952759414d34fd381e674", + "graderVersion": "paperclip.hiring-templates.v1", + "definitionDigest": "4d7f18420889325af89aa48eaaf021e8178aa6d26d504b92041635a15741f004", + "machineStatus": "failed", + "outcomePassed": false, + "countCheckPassed": false, + "comparisonStatus": "uncomparable" + }, + "corrected": { + "graderVersion": "paperclip.hiring-templates.v2", + "definitionDigest": "4d527906a967ca71c4c0e62cfbdc3405419931bcd6741b0630c4223bed2d4b6a", + "outcomePassed": true, + "comparisonStatus": "uncomparable", + "scorerGuardPassed": true, + "finalChatGuardPassed": true, + "turnAccounting": { + "version": "paperclip.hiring-template-turn-accounting.v1", + "counts": { + "requiredWorkTurns": 5, + "maximumCompletionTurns": 2, + "maximumTotalTurns": 7, + "requestedLeadTurns": 3, + "coderTurns": 2, + "completionTurns": 2, + "unclassifiedTurns": 0, + "snapshotRunCount": 7, + "actualRunCount": 7, + "costAccountingRunCount": 7 + }, + "predicates": [ + { + "id": "known-fixture-context", + "passed": true + }, + { + "id": "complete-public-run-ledger", + "passed": true + }, + { + "id": "exact-five-required-work-turns", + "passed": true + }, + { + "id": "successful-native-without-retries", + "passed": true + }, + { + "id": "resolved-company-account-and-identity", + "passed": true + }, + { + "id": "three-distinct-requested-chat-turns", + "passed": true + }, + { + "id": "one-coder-execution-per-known-task", + "passed": true + }, + { + "id": "task-origins-are-first-two-requested-turns", + "passed": true + }, + { + "id": "bounded-server-completion-receipts", + "passed": true + }, + { + "id": "completion-runs-have-attributed-chat-replies", + "passed": true + }, + { + "id": "no-notification-created-extra-tasks", + "passed": true + } + ], + "passed": true + }, + "checks": [ + { + "id": "one-coder-hire", + "dimension": "outcome", + "passed": true, + "detail": "Exactly one permanent coding teammate reports to the lead." + }, + { + "id": "execution-account", + "dimension": "outcome", + "passed": true, + "detail": "Hire keeps the lead model and managed account binding." + }, + { + "id": "two-worker-tasks", + "dimension": "outcome", + "passed": true, + "detail": "Both independent tasks are completed by the same coder in the chosen project." + }, + { + "id": "bounded-work-and-completion-turns", + "dimension": "outcome", + "passed": true, + "detail": "Exactly three requested lead turns and two coder executions, plus at most two strictly attributed completion notifications; every actual run remains counted." + }, + { + "id": "initial-json-artifact", + "dimension": "outcome", + "passed": true, + "detail": "Independent computation checks each original input/value and worker authorship." + }, + { + "id": "reused-json-artifact", + "dimension": "outcome", + "passed": true, + "detail": "The reused coder applies the changed separator to every input." + }, + { + "id": "original-preserved", + "dimension": "outcome", + "passed": true, + "detail": "Reuse preserves the original document, revision and task identity." + }, + { + "id": "source-fingerprints", + "dimension": "coverage", + "passed": true, + "detail": "Served instruction and hiring source bytes match the evaluated revision." + }, + { + "id": "production-ceo-bundle", + "dimension": "coverage", + "passed": true, + "detail": "The API-created lead receives this revision's default CEO bundle, without a fixture override." + }, + { + "id": "assigned-hiring-skill", + "dimension": "coverage", + "passed": true, + "detail": "Production CEO defaults assign the hiring skill." + }, + { + "id": "production-source-reads", + "dimension": "coverage", + "passed": false, + "detail": "Completed lead read receipts before the hire prove the explicitly requested skill, guide, checklist and coder example paths. Unrecognized or missing reads leave coverage uncomparable." + }, + { + "id": "supplied-coder-instructions", + "dimension": "coverage", + "passed": true, + "detail": "Saved hired instructions use the source revision's coder example with its company/name placeholders filled." + }, + { + "id": "hired-instructions-durable", + "dimension": "coverage", + "passed": true, + "detail": "The same saved instruction bundle survives the reused worker execution." + }, + { + "id": "hired-skills-durable", + "dimension": "coverage", + "passed": true, + "detail": "The saved skill selections survive the reused worker execution." + } + ] + }, + "coverageUnchanged": true, + "otherOutcomesUnchanged": true, + "originalBytesUnchanged": true, + "inputSha256": { + "result": "a83aecc26ef0af2e231cb364299cf0cb2f2ccd83b6998913dcd33be72d4c0824", + "hiringTemplate": "4f83f523b955120c5baac18572dce752b278dbc9ff765b940c0e17c37ab28ae0", + "apiState": "a208a3e3568fe9aacbec8d683a502a0d304a47571a1b97c229d2936e5d1bb087" + } + }, + { + "variant": "baseline", + "profile": "runner-codex", + "original": { + "sourceRevision": "296a4df85e8bcc97a160fc78c291b17adb828196", + "graderVersion": "paperclip.hiring-templates.v1", + "definitionDigest": "4d7f18420889325af89aa48eaaf021e8178aa6d26d504b92041635a15741f004", + "machineStatus": "failed", + "outcomePassed": false, + "countCheckPassed": false, + "comparisonStatus": "uncomparable" + }, + "corrected": { + "graderVersion": "paperclip.hiring-templates.v2", + "definitionDigest": "4d527906a967ca71c4c0e62cfbdc3405419931bcd6741b0630c4223bed2d4b6a", + "outcomePassed": true, + "comparisonStatus": "uncomparable", + "scorerGuardPassed": true, + "finalChatGuardPassed": true, + "turnAccounting": { + "version": "paperclip.hiring-template-turn-accounting.v1", + "counts": { + "requiredWorkTurns": 5, + "maximumCompletionTurns": 2, + "maximumTotalTurns": 7, + "requestedLeadTurns": 3, + "coderTurns": 2, + "completionTurns": 2, + "unclassifiedTurns": 0, + "snapshotRunCount": 7, + "actualRunCount": 7, + "costAccountingRunCount": 7 + }, + "predicates": [ + { + "id": "known-fixture-context", + "passed": true + }, + { + "id": "complete-public-run-ledger", + "passed": true + }, + { + "id": "exact-five-required-work-turns", + "passed": true + }, + { + "id": "successful-native-without-retries", + "passed": true + }, + { + "id": "resolved-company-account-and-identity", + "passed": true + }, + { + "id": "three-distinct-requested-chat-turns", + "passed": true + }, + { + "id": "one-coder-execution-per-known-task", + "passed": true + }, + { + "id": "task-origins-are-first-two-requested-turns", + "passed": true + }, + { + "id": "bounded-server-completion-receipts", + "passed": true + }, + { + "id": "completion-runs-have-attributed-chat-replies", + "passed": true + }, + { + "id": "no-notification-created-extra-tasks", + "passed": true + } + ], + "passed": true + }, + "checks": [ + { + "id": "one-coder-hire", + "dimension": "outcome", + "passed": true, + "detail": "Exactly one permanent coding teammate reports to the lead." + }, + { + "id": "execution-account", + "dimension": "outcome", + "passed": true, + "detail": "Hire keeps the lead model and managed account binding." + }, + { + "id": "two-worker-tasks", + "dimension": "outcome", + "passed": true, + "detail": "Both independent tasks are completed by the same coder in the chosen project." + }, + { + "id": "bounded-work-and-completion-turns", + "dimension": "outcome", + "passed": true, + "detail": "Exactly three requested lead turns and two coder executions, plus at most two strictly attributed completion notifications; every actual run remains counted." + }, + { + "id": "initial-json-artifact", + "dimension": "outcome", + "passed": true, + "detail": "Independent computation checks each original input/value and worker authorship." + }, + { + "id": "reused-json-artifact", + "dimension": "outcome", + "passed": true, + "detail": "The reused coder applies the changed separator to every input." + }, + { + "id": "original-preserved", + "dimension": "outcome", + "passed": true, + "detail": "Reuse preserves the original document, revision and task identity." + }, + { + "id": "source-fingerprints", + "dimension": "coverage", + "passed": true, + "detail": "Served instruction and hiring source bytes match the evaluated revision." + }, + { + "id": "production-ceo-bundle", + "dimension": "coverage", + "passed": true, + "detail": "The API-created lead receives this revision's default CEO bundle, without a fixture override." + }, + { + "id": "assigned-hiring-skill", + "dimension": "coverage", + "passed": true, + "detail": "Production CEO defaults assign the hiring skill." + }, + { + "id": "production-source-reads", + "dimension": "coverage", + "passed": false, + "detail": "Completed lead read receipts before the hire prove the explicitly requested skill, guide, checklist and coder example paths. Unrecognized or missing reads leave coverage uncomparable." + }, + { + "id": "supplied-coder-instructions", + "dimension": "coverage", + "passed": true, + "detail": "Saved hired instructions use the source revision's coder example with its company/name placeholders filled." + }, + { + "id": "hired-instructions-durable", + "dimension": "coverage", + "passed": true, + "detail": "The same saved instruction bundle survives the reused worker execution." + }, + { + "id": "hired-skills-durable", + "dimension": "coverage", + "passed": true, + "detail": "The saved skill selections survive the reused worker execution." + } + ] + }, + "coverageUnchanged": true, + "otherOutcomesUnchanged": true, + "originalBytesUnchanged": true, + "inputSha256": { + "result": "d912d8400bfff4f7f922aa6b264f9ce5fff1ba2d893abf03790c4cfea352261f", + "hiringTemplate": "88b23e4d2cc8831a5d88b8479cc69c0d1c2e17a6a5243707b4bd19716a244aca", + "apiState": "dff6bb13bb54d45024966d9cfbeb6dbe6f2241e674b0b1350c1384863202a97f" + } + }, + { + "variant": "baseline", + "profile": "runner-acpx-claude", + "original": { + "sourceRevision": "296a4df85e8bcc97a160fc78c291b17adb828196", + "graderVersion": "paperclip.hiring-templates.v1", + "definitionDigest": "4d7f18420889325af89aa48eaaf021e8178aa6d26d504b92041635a15741f004", + "machineStatus": "failed", + "outcomePassed": false, + "countCheckPassed": false, + "comparisonStatus": "uncomparable" + }, + "corrected": { + "graderVersion": "paperclip.hiring-templates.v2", + "definitionDigest": "4d527906a967ca71c4c0e62cfbdc3405419931bcd6741b0630c4223bed2d4b6a", + "outcomePassed": true, + "comparisonStatus": "uncomparable", + "scorerGuardPassed": true, + "finalChatGuardPassed": true, + "turnAccounting": { + "version": "paperclip.hiring-template-turn-accounting.v1", + "counts": { + "requiredWorkTurns": 5, + "maximumCompletionTurns": 2, + "maximumTotalTurns": 7, + "requestedLeadTurns": 3, + "coderTurns": 2, + "completionTurns": 2, + "unclassifiedTurns": 0, + "snapshotRunCount": 7, + "actualRunCount": 7, + "costAccountingRunCount": 7 + }, + "predicates": [ + { + "id": "known-fixture-context", + "passed": true + }, + { + "id": "complete-public-run-ledger", + "passed": true + }, + { + "id": "exact-five-required-work-turns", + "passed": true + }, + { + "id": "successful-native-without-retries", + "passed": true + }, + { + "id": "resolved-company-account-and-identity", + "passed": true + }, + { + "id": "three-distinct-requested-chat-turns", + "passed": true + }, + { + "id": "one-coder-execution-per-known-task", + "passed": true + }, + { + "id": "task-origins-are-first-two-requested-turns", + "passed": true + }, + { + "id": "bounded-server-completion-receipts", + "passed": true + }, + { + "id": "completion-runs-have-attributed-chat-replies", + "passed": true + }, + { + "id": "no-notification-created-extra-tasks", + "passed": true + } + ], + "passed": true + }, + "checks": [ + { + "id": "one-coder-hire", + "dimension": "outcome", + "passed": true, + "detail": "Exactly one permanent coding teammate reports to the lead." + }, + { + "id": "execution-account", + "dimension": "outcome", + "passed": true, + "detail": "Hire keeps the lead model and managed account binding." + }, + { + "id": "two-worker-tasks", + "dimension": "outcome", + "passed": true, + "detail": "Both independent tasks are completed by the same coder in the chosen project." + }, + { + "id": "bounded-work-and-completion-turns", + "dimension": "outcome", + "passed": true, + "detail": "Exactly three requested lead turns and two coder executions, plus at most two strictly attributed completion notifications; every actual run remains counted." + }, + { + "id": "initial-json-artifact", + "dimension": "outcome", + "passed": true, + "detail": "Independent computation checks each original input/value and worker authorship." + }, + { + "id": "reused-json-artifact", + "dimension": "outcome", + "passed": true, + "detail": "The reused coder applies the changed separator to every input." + }, + { + "id": "original-preserved", + "dimension": "outcome", + "passed": true, + "detail": "Reuse preserves the original document, revision and task identity." + }, + { + "id": "source-fingerprints", + "dimension": "coverage", + "passed": true, + "detail": "Served instruction and hiring source bytes match the evaluated revision." + }, + { + "id": "production-ceo-bundle", + "dimension": "coverage", + "passed": true, + "detail": "The API-created lead receives this revision's default CEO bundle, without a fixture override." + }, + { + "id": "assigned-hiring-skill", + "dimension": "coverage", + "passed": true, + "detail": "Production CEO defaults assign the hiring skill." + }, + { + "id": "production-source-reads", + "dimension": "coverage", + "passed": false, + "detail": "Completed lead read receipts before the hire prove the explicitly requested skill, guide, checklist and coder example paths. Unrecognized or missing reads leave coverage uncomparable." + }, + { + "id": "supplied-coder-instructions", + "dimension": "coverage", + "passed": false, + "detail": "Saved hired instructions use the source revision's coder example with its company/name placeholders filled." + }, + { + "id": "hired-instructions-durable", + "dimension": "coverage", + "passed": true, + "detail": "The same saved instruction bundle survives the reused worker execution." + }, + { + "id": "hired-skills-durable", + "dimension": "coverage", + "passed": true, + "detail": "The saved skill selections survive the reused worker execution." + } + ] + }, + "coverageUnchanged": true, + "otherOutcomesUnchanged": true, + "originalBytesUnchanged": true, + "inputSha256": { + "result": "844763ddc83461b2bea620f153f4bfe0bfee20a27bf33a60017a45c774ce2f6a", + "hiringTemplate": "27b153798472e206970c46358febdda363cb75f892734757e76994117f0d2c9b", + "apiState": "441cf7272ea28c5e1182adf48dba09a401e88c6cad3a99e72c8cf3f46b1c5b41" + } + } + ] +} diff --git a/doc/plans/2026-10-02-hiring-executable-accounting-replay.md b/doc/plans/2026-10-02-hiring-executable-accounting-replay.md new file mode 100644 index 00000000000..21314b61930 --- /dev/null +++ b/doc/plans/2026-10-02-hiring-executable-accounting-replay.md @@ -0,0 +1,59 @@ +# Executable hiring lifecycle accounting repair + +**Current TL;DR:** No new model calls were made. The original Codex and Claude pairs remain Fail → Fail. The stricter v3 executable replay verifies Codex accounting in both variants; Claude notification action attribution remains unresolved/uncomparable because provider and native request IDs cannot be exactly joined. No notification writes are observed in the retained native API receipts. Source-read coverage remains uncomparable for both profiles, independently of action attribution. Earlier sidecar-v1 and initial executable-v2 passes remain below with their limited task-only proof identified. + +## Current stricter executable replay + +Code revision: `e4077ade1818d98b9862ae79ee1d49a007dcf9c1`. Hiring grader: `paperclip.hiring-templates.v3`; turn accounting: `paperclip.hiring-template-turn-accounting.v2`. The separately versioned [v2 JSON receipt](2026-10-02-hiring-executable-accounting-replay.v2.json) pins exact source hashes and unchanged original inputs. + +| Profile | Original machine grades | Limited sidecar v1 / initial executable v2 | Stricter executable accounting | Source-read coverage | +| --- | --- | --- | --- | --- | +| Codex | Fail → Fail | Pass → Pass | Verified / both new guards pass in each variant | Uncomparable both | +| ACPX Claude | Fail → Fail | Pass → Pass | Unresolved/uncomparable action attribution in each variant; guards fail closed | Uncomparable both; historical six-backtick exact-template mismatch remains | + +The limited sidecar and initial executable check rejected notification-created tasks but could miss an unrelated document write during a notification. Their original results and hashes are preserved; they do not prove every notification was harmless. The stricter helper adds a twelfth predicate and an independent action-coverage check. It requires complete contiguous event streams, an accepted control-plane result and succeeded/completed terminal, exact canonical/native action IDs, successful known GET API receipts or verified reads/discovery, and attributed native chat finish. Document/task/agent mutations, failed mutation attempts, incomplete streams and unknown or unmatched actions cannot pass. A readonly hint alone cannot qualify a generic API call. + +Codex verifies 10 candidate and 11 baseline notification tool executions, including five and seven exactly joined successful GET calls, with zero unresolved actions. ACPX's native request IDs and provider execution IDs use separate namespaces; names, ordering and counts cannot safely join them. All observed ACPX native API calls are successful GETs, but the stronger cross-ledger proof is missing (five candidate/four baseline unresolved observations). This is an evidence limitation, not a newly observed task failure or a prompt regression. No production carriers or source-read grading are changed. + +The observation repair retries entire bracketed snapshots, waits both known Done task callbacks and attributed chat replies (including batching), checks untruncated pending-wake diagnostics, and requires two identical complete observations. It covers the quiet gap before pending outbox work enqueues a wake. Accounted coalesced wakes are admitted only with a linked terminal run and known callbacks. The final guard performs that same complete refresh rather than mixing a stale evidence snapshot with new runs. The generic five-work-turn helper still admits zero notifications when none are owed; this production delegated fixture owes two task completions. + +All six non-count outcome checks and every original source/template coverage check remain byte-identical in all four replay projections. All 28 actual model runs remain counted; original input hashes and failed verdicts are unchanged. The stricter action check is separately classified as coverage when attribution is missing. No broad equivalence is established. + +- 977 credential-free support tests pass across 64 files, including 144 focused helper/scorer/settlement integrations. +- E2E typecheck, exact two-cell discovery, canonical capability contract/inventory checks and diff checks pass. +- Two current-head review findings are fixed: racing observations and unchecked completion-turn writes. Fresh review/CI is required on this source. +- Initial-head CI retained two unrelated browser failures: exact agent-run denial feedback and touch-picker scroll position. Neither browser path imports the changed runner-e2e harness. They are preserved without a blind rerun; the necessary review-fix head runs normal CI. +- No providers, old campaign retries or extra paid scope are launched. Actual prior model charges remain unknown. + +## Initial executable v2 replay (limited notification proof) + +TL;DR: This repairs a grading defect; it does not change model instructions or rerun models. The original Codex and Claude pairs remain Fail → Fail. Provider-free replay through both corrected executable paths passes all four retained cells; source-read coverage remains uncomparable. This is a grading correction, not a new model-performance result. + +The original fixture required exactly five total runs. Production adds legitimate server completion turns after delegated work. The corrected v2 contract requires exactly three distinct user-requested CEO turns and two coder executions. At most two additional completion turns must pass strict public identity/account/task/delivery/timing/reply attribution. One turn may batch both completed tasks. Unknown, duplicate, failed, retried runs, notification-created tasks and missing public observations fail. This initial version did not inspect unrelated document writes during notifications; see the stricter current assessment above. + +Both executable count guards use the shared helper: the hiring scorer and the hiring-only final chat guard. The catalog declares five required / seven maximum total runs so timeout/cost planning includes notifications. Non-hiring count guards remain unchanged. Source-read and exact coder-body grading remain unchanged, including the historical Claude six-backtick mismatch. The grader version and full helper/chat/source digest change. + +## Preserved measurement + +The immutable [original and bounded sidecar report](https://github.com/paperclipai/paperclip/blob/8eb517ca1497687237163bdef4dfc4d3332ea916/doc/plans/2026-10-02-hiring-template-live-comparison.md) retains candidate `9f5404ad3aacbe76777952759414d34fd381e674` and historical `296a4df85e8bcc97a160fc78c291b17adb828196`, the identical original fixture digest, 28 actual successful runs, eight automatic completion turns, four successful cleanups and unknown actual charges. It calibrates sidecar v1 with 69 passing tests and records exact original input/grader hashes. The executable repair is a new source/grader revision; it cannot rewrite those measurements or prove performance equivalence. + +## Provider-free verification + +The executable code revision is `eef64009dc144b91b245a731ea9a1c3c07406a19`; report-only commits do not change its grader bytes. The [JSON receipt](2026-10-02-hiring-executable-accounting-replay.json) pins all five grader/flow/helper source hashes, v2 definition digest and original result/hiring/API input hashes. + +| Profile | Original executable result | Corrected retained workflow outcome | Both new count guards | Source coverage | +| --- | --- | --- | --- | --- | +| Codex | Fail → Fail | Pass → Pass | Pass in both variants | Uncomparable both | +| ACPX Claude | Fail → Fail | Pass → Pass | Pass in both variants | Uncomparable both; historical exact coder body still fails | + +All six other outcome checks and every coverage check are byte-identical in the replay's check projection. Original input files and failed grades remain unchanged. All 28 actual model runs remain counted. The new helper has 11 lifecycle predicates, factoring the sidecar's lifecycle rule without requiring an already-computed original result. + +- All 946 credential-free E2E support tests pass across 64 files, including 99 helper calibrations and the four added scorer/final-guard integrations. +- Fifteen focused hiring/run-count tests pass. The integrations admit five core turns, six with batching and seven with distinct notifications; reject an arbitrary wake despite the same total count; require public observations; retain source/template coverage failures; and preserve non-hiring count guards. +- E2E typecheck, normal plugin SDK/Runner TypeScript dependency builds, canonical capability contract/inventory checks and existing two-cell discovery pass. +- Retained replay executes the actual new scorer and final chat guard in each of the four original cells, using the unchanged saved API observations. Both guards pass 4/4; coverage and the six non-count outcomes stay unchanged 4/4. +- Full repository CI and fresh review are pending on the separate draft PR. Local general repository typecheck/test/build were not repeated; exact-head CI must supply those gates before handoff. + +Initial development calibration caught mismatched synthetic account data and a deliberately changed template not preserved across reuse. The fixture data was corrected; assertions were kept. Initial cold typecheck lacked built Runner declarations; normal dependency builds resolved it. A temporary replay script's CommonJS extension rejected top-level await; the same script executed as ESM. These provider-free setup/development attempts remain private and are not live model results. No provider runs, old campaign retries or paid scope expansion are authorized for this repair. Public replay receipts will include only grader/input hashes, counts, predicate/check results and original verdicts; credentials, provider session IDs and hidden reasoning remain private. + +The branch was safely replayed on master `59c07ede7` before PR creation. The intervening master changes touch only two agent-provider UI files; all eval source bytes are unchanged. The four-cell provider-free replay was repeated against the final reachable code revision, with the same input hashes, checks and zero providers. diff --git a/doc/plans/2026-10-02-hiring-executable-accounting-replay.v2.json b/doc/plans/2026-10-02-hiring-executable-accounting-replay.v2.json new file mode 100644 index 00000000000..9c34e86b199 --- /dev/null +++ b/doc/plans/2026-10-02-hiring-executable-accounting-replay.v2.json @@ -0,0 +1,815 @@ +{ + "schema": "paperclip.hiring-executable-accounting-replay.v2", + "assessmentType": "Provider-free replay through both new executable guards; original model measurements and verdicts unchanged", + "evaluatedCodeRevision": "e4077ade1818d98b9862ae79ee1d49a007dcf9c1", + "evaluatedCodeHashes": { + "tests/runner-e2e/hiring-template-cases.ts": "a440ebf5a932525d388f507f8b47490a535d377c9f9c9beba6144f1fc2214b7c", + "tests/runner-e2e/hiring-template-scoring.ts": "5f600b8329c981f1fc94ecc1b70a94b78d4a32a5b8859600208cbe1f91be2dac", + "tests/runner-e2e/hiring-template-flow.ts": "10b2415ad6ee293286db0a8cc8cb7a6a363d993a70560ab0a82f668018b5e3f5", + "tests/runner-e2e/hiring-template-turn-accounting.ts": "13792c69669b99748087d6a78af8ab35c6293d1ad3e3396efcbe39620169ad35", + "tests/runner-e2e/chat-flow.ts": "85d8c3da832871f38cc4ed940046d28ed7b3edf17486ce43b0f3313c06d17d7a" + }, + "graderVersion": "paperclip.hiring-templates.v3", + "definitionDigest": "cddbcb6b2f3050e065599011ace2faee048047fae6ea9bc9183db8c5744e2f38", + "providerCalls": 0, + "newProviderRuns": 0, + "automaticRetries": 0, + "summary": { + "originalPairs": "Codex Fail→Fail; Claude Fail→Fail", + "correctedWorkflowOutcomePairs": "Codex Pass→Pass; Claude action attribution unresolved/uncomparable in both variants (not a measured task failure)", + "fullAttemptComparison": "Uncomparable in both profiles and variants; source-read coverage unchanged", + "originalMachineFailuresPreserved": 4, + "correctedScorerGuardsPassed": 2, + "correctedFinalChatGuardsPassed": 2, + "strictActionVerifiedAssessments": 2, + "strictActionUncomparableAssessments": 2, + "unchangedOriginalCoverageAssessments": 4, + "unchangedOtherOutcomeAssessments": 4, + "actualMeasuredRuns": 28, + "actualModelChargesKnown": false, + "broadEquivalenceEstablished": false, + "priorSidecarLimitation": "Sidecar v1 and initial executable v2 checked notification-created tasks, not unrelated document writes. Their original passes remain preserved." + }, + "assessments": [ + { + "variant": "candidate", + "profile": "runner-codex", + "original": { + "sourceRevision": "9f5404ad3aacbe76777952759414d34fd381e674", + "graderVersion": "paperclip.hiring-templates.v1", + "definitionDigest": "4d7f18420889325af89aa48eaaf021e8178aa6d26d504b92041635a15741f004", + "machineStatus": "failed", + "outcomePassed": false, + "countCheckPassed": false, + "comparisonStatus": "uncomparable" + }, + "corrected": { + "graderVersion": "paperclip.hiring-templates.v3", + "definitionDigest": "cddbcb6b2f3050e065599011ace2faee048047fae6ea9bc9183db8c5744e2f38", + "outcomePassed": true, + "comparisonStatus": "uncomparable", + "scorerGuardPassed": true, + "finalChatGuardPassed": true, + "assessmentStatus": "verified", + "turnAccounting": { + "version": "paperclip.hiring-template-turn-accounting.v2", + "counts": { + "requiredWorkTurns": 5, + "maximumCompletionTurns": 2, + "maximumTotalTurns": 7, + "requestedLeadTurns": 3, + "coderTurns": 2, + "completionTurns": 2, + "unclassifiedTurns": 0, + "snapshotRunCount": 7, + "actualRunCount": 7, + "costAccountingRunCount": 7 + }, + "predicates": [ + { + "id": "known-fixture-context", + "passed": true + }, + { + "id": "complete-public-run-ledger", + "passed": true + }, + { + "id": "exact-five-required-work-turns", + "passed": true + }, + { + "id": "successful-native-without-retries", + "passed": true + }, + { + "id": "resolved-company-account-and-identity", + "passed": true + }, + { + "id": "three-distinct-requested-chat-turns", + "passed": true + }, + { + "id": "one-coder-execution-per-known-task", + "passed": true + }, + { + "id": "task-origins-are-first-two-requested-turns", + "passed": true + }, + { + "id": "bounded-server-completion-receipts", + "passed": true + }, + { + "id": "completion-runs-have-attributed-chat-replies", + "passed": true + }, + { + "id": "no-notification-created-extra-tasks", + "passed": true + }, + { + "id": "completion-turns-only-report-actions", + "passed": true + } + ], + "actionEvidence": { + "status": "verified", + "notificationRuns": 2, + "canonicalExecutions": 10, + "matchedNativeApiCalls": 5, + "unknownExecutions": 0 + }, + "passed": true + }, + "checks": [ + { + "id": "one-coder-hire", + "dimension": "outcome", + "passed": true, + "detail": "Exactly one permanent coding teammate reports to the lead." + }, + { + "id": "execution-account", + "dimension": "outcome", + "passed": true, + "detail": "Hire keeps the lead model and managed account binding." + }, + { + "id": "two-worker-tasks", + "dimension": "outcome", + "passed": true, + "detail": "Both independent tasks are completed by the same coder in the chosen project." + }, + { + "id": "bounded-work-and-completion-turns", + "dimension": "outcome", + "passed": true, + "detail": "Exactly three requested lead turns and two coder executions, plus at most two strictly attributed completion notifications; every actual run remains counted." + }, + { + "id": "initial-json-artifact", + "dimension": "outcome", + "passed": true, + "detail": "Independent computation checks each original input/value and worker authorship." + }, + { + "id": "reused-json-artifact", + "dimension": "outcome", + "passed": true, + "detail": "The reused coder applies the changed separator to every input." + }, + { + "id": "original-preserved", + "dimension": "outcome", + "passed": true, + "detail": "Reuse preserves the original document, revision and task identity." + }, + { + "id": "source-fingerprints", + "dimension": "coverage", + "passed": true, + "detail": "Served instruction and hiring source bytes match the evaluated revision." + }, + { + "id": "production-ceo-bundle", + "dimension": "coverage", + "passed": true, + "detail": "The API-created lead receives this revision's default CEO bundle, without a fixture override." + }, + { + "id": "assigned-hiring-skill", + "dimension": "coverage", + "passed": true, + "detail": "Production CEO defaults assign the hiring skill." + }, + { + "id": "production-source-reads", + "dimension": "coverage", + "passed": false, + "detail": "Completed lead read receipts before the hire prove the explicitly requested skill, guide, checklist and coder example paths. Unrecognized or missing reads leave coverage uncomparable." + }, + { + "id": "supplied-coder-instructions", + "dimension": "coverage", + "passed": true, + "detail": "Saved hired instructions use the source revision's coder example with its company/name placeholders filled." + }, + { + "id": "hired-instructions-durable", + "dimension": "coverage", + "passed": true, + "detail": "The same saved instruction bundle survives the reused worker execution." + }, + { + "id": "hired-skills-durable", + "dimension": "coverage", + "passed": true, + "detail": "The saved skill selections survive the reused worker execution." + }, + { + "id": "completion-action-attribution", + "dimension": "coverage", + "passed": true, + "detail": "Notification actions require exact canonical/native identities; missing cross-namespace mapping is uncomparable, not proof of extra work." + } + ] + }, + "coverageUnchanged": true, + "otherOutcomesUnchanged": true, + "originalBytesUnchanged": true, + "inputSha256": { + "result": "44cbf64cfabd0257f31fb4d6909018d346f5667b0a7b3d7e855a2ece7e29c292", + "hiringTemplate": "9609ddcb81c6bee3c5db97312138b4ea3e294da13ea540e41f363c826a514067", + "apiState": "ef02af29bcf61ff3eace0549ca164a28c7950d1a3e14fc58c89bf9cb0f944b4d" + } + }, + { + "variant": "candidate", + "profile": "runner-acpx-claude", + "original": { + "sourceRevision": "9f5404ad3aacbe76777952759414d34fd381e674", + "graderVersion": "paperclip.hiring-templates.v1", + "definitionDigest": "4d7f18420889325af89aa48eaaf021e8178aa6d26d504b92041635a15741f004", + "machineStatus": "failed", + "outcomePassed": false, + "countCheckPassed": false, + "comparisonStatus": "uncomparable" + }, + "corrected": { + "graderVersion": "paperclip.hiring-templates.v3", + "definitionDigest": "cddbcb6b2f3050e065599011ace2faee048047fae6ea9bc9183db8c5744e2f38", + "outcomePassed": false, + "comparisonStatus": "uncomparable", + "scorerGuardPassed": false, + "finalChatGuardPassed": false, + "assessmentStatus": "uncomparable", + "turnAccounting": { + "version": "paperclip.hiring-template-turn-accounting.v2", + "counts": { + "requiredWorkTurns": 5, + "maximumCompletionTurns": 2, + "maximumTotalTurns": 7, + "requestedLeadTurns": 3, + "coderTurns": 2, + "completionTurns": 2, + "unclassifiedTurns": 0, + "snapshotRunCount": 7, + "actualRunCount": 7, + "costAccountingRunCount": 7 + }, + "predicates": [ + { + "id": "known-fixture-context", + "passed": true + }, + { + "id": "complete-public-run-ledger", + "passed": true + }, + { + "id": "exact-five-required-work-turns", + "passed": true + }, + { + "id": "successful-native-without-retries", + "passed": true + }, + { + "id": "resolved-company-account-and-identity", + "passed": true + }, + { + "id": "three-distinct-requested-chat-turns", + "passed": true + }, + { + "id": "one-coder-execution-per-known-task", + "passed": true + }, + { + "id": "task-origins-are-first-two-requested-turns", + "passed": true + }, + { + "id": "bounded-server-completion-receipts", + "passed": true + }, + { + "id": "completion-runs-have-attributed-chat-replies", + "passed": true + }, + { + "id": "no-notification-created-extra-tasks", + "passed": true + }, + { + "id": "completion-turns-only-report-actions", + "passed": false + } + ], + "actionEvidence": { + "status": "uncomparable", + "notificationRuns": 2, + "canonicalExecutions": 3, + "matchedNativeApiCalls": 0, + "unknownExecutions": 5 + }, + "passed": false + }, + "checks": [ + { + "id": "one-coder-hire", + "dimension": "outcome", + "passed": true, + "detail": "Exactly one permanent coding teammate reports to the lead." + }, + { + "id": "execution-account", + "dimension": "outcome", + "passed": true, + "detail": "Hire keeps the lead model and managed account binding." + }, + { + "id": "two-worker-tasks", + "dimension": "outcome", + "passed": true, + "detail": "Both independent tasks are completed by the same coder in the chosen project." + }, + { + "id": "bounded-work-and-completion-turns", + "dimension": "outcome", + "passed": false, + "detail": "Exactly three requested lead turns and two coder executions, plus at most two strictly attributed completion notifications; every actual run remains counted." + }, + { + "id": "initial-json-artifact", + "dimension": "outcome", + "passed": true, + "detail": "Independent computation checks each original input/value and worker authorship." + }, + { + "id": "reused-json-artifact", + "dimension": "outcome", + "passed": true, + "detail": "The reused coder applies the changed separator to every input." + }, + { + "id": "original-preserved", + "dimension": "outcome", + "passed": true, + "detail": "Reuse preserves the original document, revision and task identity." + }, + { + "id": "source-fingerprints", + "dimension": "coverage", + "passed": true, + "detail": "Served instruction and hiring source bytes match the evaluated revision." + }, + { + "id": "production-ceo-bundle", + "dimension": "coverage", + "passed": true, + "detail": "The API-created lead receives this revision's default CEO bundle, without a fixture override." + }, + { + "id": "assigned-hiring-skill", + "dimension": "coverage", + "passed": true, + "detail": "Production CEO defaults assign the hiring skill." + }, + { + "id": "production-source-reads", + "dimension": "coverage", + "passed": false, + "detail": "Completed lead read receipts before the hire prove the explicitly requested skill, guide, checklist and coder example paths. Unrecognized or missing reads leave coverage uncomparable." + }, + { + "id": "supplied-coder-instructions", + "dimension": "coverage", + "passed": true, + "detail": "Saved hired instructions use the source revision's coder example with its company/name placeholders filled." + }, + { + "id": "hired-instructions-durable", + "dimension": "coverage", + "passed": true, + "detail": "The same saved instruction bundle survives the reused worker execution." + }, + { + "id": "hired-skills-durable", + "dimension": "coverage", + "passed": true, + "detail": "The saved skill selections survive the reused worker execution." + }, + { + "id": "completion-action-attribution", + "dimension": "coverage", + "passed": false, + "detail": "Notification actions require exact canonical/native identities; missing cross-namespace mapping is uncomparable, not proof of extra work." + } + ] + }, + "coverageUnchanged": true, + "otherOutcomesUnchanged": true, + "originalBytesUnchanged": true, + "inputSha256": { + "result": "a83aecc26ef0af2e231cb364299cf0cb2f2ccd83b6998913dcd33be72d4c0824", + "hiringTemplate": "4f83f523b955120c5baac18572dce752b278dbc9ff765b940c0e17c37ab28ae0", + "apiState": "a208a3e3568fe9aacbec8d683a502a0d304a47571a1b97c229d2936e5d1bb087" + } + }, + { + "variant": "baseline", + "profile": "runner-codex", + "original": { + "sourceRevision": "296a4df85e8bcc97a160fc78c291b17adb828196", + "graderVersion": "paperclip.hiring-templates.v1", + "definitionDigest": "4d7f18420889325af89aa48eaaf021e8178aa6d26d504b92041635a15741f004", + "machineStatus": "failed", + "outcomePassed": false, + "countCheckPassed": false, + "comparisonStatus": "uncomparable" + }, + "corrected": { + "graderVersion": "paperclip.hiring-templates.v3", + "definitionDigest": "cddbcb6b2f3050e065599011ace2faee048047fae6ea9bc9183db8c5744e2f38", + "outcomePassed": true, + "comparisonStatus": "uncomparable", + "scorerGuardPassed": true, + "finalChatGuardPassed": true, + "assessmentStatus": "verified", + "turnAccounting": { + "version": "paperclip.hiring-template-turn-accounting.v2", + "counts": { + "requiredWorkTurns": 5, + "maximumCompletionTurns": 2, + "maximumTotalTurns": 7, + "requestedLeadTurns": 3, + "coderTurns": 2, + "completionTurns": 2, + "unclassifiedTurns": 0, + "snapshotRunCount": 7, + "actualRunCount": 7, + "costAccountingRunCount": 7 + }, + "predicates": [ + { + "id": "known-fixture-context", + "passed": true + }, + { + "id": "complete-public-run-ledger", + "passed": true + }, + { + "id": "exact-five-required-work-turns", + "passed": true + }, + { + "id": "successful-native-without-retries", + "passed": true + }, + { + "id": "resolved-company-account-and-identity", + "passed": true + }, + { + "id": "three-distinct-requested-chat-turns", + "passed": true + }, + { + "id": "one-coder-execution-per-known-task", + "passed": true + }, + { + "id": "task-origins-are-first-two-requested-turns", + "passed": true + }, + { + "id": "bounded-server-completion-receipts", + "passed": true + }, + { + "id": "completion-runs-have-attributed-chat-replies", + "passed": true + }, + { + "id": "no-notification-created-extra-tasks", + "passed": true + }, + { + "id": "completion-turns-only-report-actions", + "passed": true + } + ], + "actionEvidence": { + "status": "verified", + "notificationRuns": 2, + "canonicalExecutions": 11, + "matchedNativeApiCalls": 7, + "unknownExecutions": 0 + }, + "passed": true + }, + "checks": [ + { + "id": "one-coder-hire", + "dimension": "outcome", + "passed": true, + "detail": "Exactly one permanent coding teammate reports to the lead." + }, + { + "id": "execution-account", + "dimension": "outcome", + "passed": true, + "detail": "Hire keeps the lead model and managed account binding." + }, + { + "id": "two-worker-tasks", + "dimension": "outcome", + "passed": true, + "detail": "Both independent tasks are completed by the same coder in the chosen project." + }, + { + "id": "bounded-work-and-completion-turns", + "dimension": "outcome", + "passed": true, + "detail": "Exactly three requested lead turns and two coder executions, plus at most two strictly attributed completion notifications; every actual run remains counted." + }, + { + "id": "initial-json-artifact", + "dimension": "outcome", + "passed": true, + "detail": "Independent computation checks each original input/value and worker authorship." + }, + { + "id": "reused-json-artifact", + "dimension": "outcome", + "passed": true, + "detail": "The reused coder applies the changed separator to every input." + }, + { + "id": "original-preserved", + "dimension": "outcome", + "passed": true, + "detail": "Reuse preserves the original document, revision and task identity." + }, + { + "id": "source-fingerprints", + "dimension": "coverage", + "passed": true, + "detail": "Served instruction and hiring source bytes match the evaluated revision." + }, + { + "id": "production-ceo-bundle", + "dimension": "coverage", + "passed": true, + "detail": "The API-created lead receives this revision's default CEO bundle, without a fixture override." + }, + { + "id": "assigned-hiring-skill", + "dimension": "coverage", + "passed": true, + "detail": "Production CEO defaults assign the hiring skill." + }, + { + "id": "production-source-reads", + "dimension": "coverage", + "passed": false, + "detail": "Completed lead read receipts before the hire prove the explicitly requested skill, guide, checklist and coder example paths. Unrecognized or missing reads leave coverage uncomparable." + }, + { + "id": "supplied-coder-instructions", + "dimension": "coverage", + "passed": true, + "detail": "Saved hired instructions use the source revision's coder example with its company/name placeholders filled." + }, + { + "id": "hired-instructions-durable", + "dimension": "coverage", + "passed": true, + "detail": "The same saved instruction bundle survives the reused worker execution." + }, + { + "id": "hired-skills-durable", + "dimension": "coverage", + "passed": true, + "detail": "The saved skill selections survive the reused worker execution." + }, + { + "id": "completion-action-attribution", + "dimension": "coverage", + "passed": true, + "detail": "Notification actions require exact canonical/native identities; missing cross-namespace mapping is uncomparable, not proof of extra work." + } + ] + }, + "coverageUnchanged": true, + "otherOutcomesUnchanged": true, + "originalBytesUnchanged": true, + "inputSha256": { + "result": "d912d8400bfff4f7f922aa6b264f9ce5fff1ba2d893abf03790c4cfea352261f", + "hiringTemplate": "88b23e4d2cc8831a5d88b8479cc69c0d1c2e17a6a5243707b4bd19716a244aca", + "apiState": "dff6bb13bb54d45024966d9cfbeb6dbe6f2241e674b0b1350c1384863202a97f" + } + }, + { + "variant": "baseline", + "profile": "runner-acpx-claude", + "original": { + "sourceRevision": "296a4df85e8bcc97a160fc78c291b17adb828196", + "graderVersion": "paperclip.hiring-templates.v1", + "definitionDigest": "4d7f18420889325af89aa48eaaf021e8178aa6d26d504b92041635a15741f004", + "machineStatus": "failed", + "outcomePassed": false, + "countCheckPassed": false, + "comparisonStatus": "uncomparable" + }, + "corrected": { + "graderVersion": "paperclip.hiring-templates.v3", + "definitionDigest": "cddbcb6b2f3050e065599011ace2faee048047fae6ea9bc9183db8c5744e2f38", + "outcomePassed": false, + "comparisonStatus": "uncomparable", + "scorerGuardPassed": false, + "finalChatGuardPassed": false, + "assessmentStatus": "uncomparable", + "turnAccounting": { + "version": "paperclip.hiring-template-turn-accounting.v2", + "counts": { + "requiredWorkTurns": 5, + "maximumCompletionTurns": 2, + "maximumTotalTurns": 7, + "requestedLeadTurns": 3, + "coderTurns": 2, + "completionTurns": 2, + "unclassifiedTurns": 0, + "snapshotRunCount": 7, + "actualRunCount": 7, + "costAccountingRunCount": 7 + }, + "predicates": [ + { + "id": "known-fixture-context", + "passed": true + }, + { + "id": "complete-public-run-ledger", + "passed": true + }, + { + "id": "exact-five-required-work-turns", + "passed": true + }, + { + "id": "successful-native-without-retries", + "passed": true + }, + { + "id": "resolved-company-account-and-identity", + "passed": true + }, + { + "id": "three-distinct-requested-chat-turns", + "passed": true + }, + { + "id": "one-coder-execution-per-known-task", + "passed": true + }, + { + "id": "task-origins-are-first-two-requested-turns", + "passed": true + }, + { + "id": "bounded-server-completion-receipts", + "passed": true + }, + { + "id": "completion-runs-have-attributed-chat-replies", + "passed": true + }, + { + "id": "no-notification-created-extra-tasks", + "passed": true + }, + { + "id": "completion-turns-only-report-actions", + "passed": false + } + ], + "actionEvidence": { + "status": "uncomparable", + "notificationRuns": 2, + "canonicalExecutions": 3, + "matchedNativeApiCalls": 0, + "unknownExecutions": 4 + }, + "passed": false + }, + "checks": [ + { + "id": "one-coder-hire", + "dimension": "outcome", + "passed": true, + "detail": "Exactly one permanent coding teammate reports to the lead." + }, + { + "id": "execution-account", + "dimension": "outcome", + "passed": true, + "detail": "Hire keeps the lead model and managed account binding." + }, + { + "id": "two-worker-tasks", + "dimension": "outcome", + "passed": true, + "detail": "Both independent tasks are completed by the same coder in the chosen project." + }, + { + "id": "bounded-work-and-completion-turns", + "dimension": "outcome", + "passed": false, + "detail": "Exactly three requested lead turns and two coder executions, plus at most two strictly attributed completion notifications; every actual run remains counted." + }, + { + "id": "initial-json-artifact", + "dimension": "outcome", + "passed": true, + "detail": "Independent computation checks each original input/value and worker authorship." + }, + { + "id": "reused-json-artifact", + "dimension": "outcome", + "passed": true, + "detail": "The reused coder applies the changed separator to every input." + }, + { + "id": "original-preserved", + "dimension": "outcome", + "passed": true, + "detail": "Reuse preserves the original document, revision and task identity." + }, + { + "id": "source-fingerprints", + "dimension": "coverage", + "passed": true, + "detail": "Served instruction and hiring source bytes match the evaluated revision." + }, + { + "id": "production-ceo-bundle", + "dimension": "coverage", + "passed": true, + "detail": "The API-created lead receives this revision's default CEO bundle, without a fixture override." + }, + { + "id": "assigned-hiring-skill", + "dimension": "coverage", + "passed": true, + "detail": "Production CEO defaults assign the hiring skill." + }, + { + "id": "production-source-reads", + "dimension": "coverage", + "passed": false, + "detail": "Completed lead read receipts before the hire prove the explicitly requested skill, guide, checklist and coder example paths. Unrecognized or missing reads leave coverage uncomparable." + }, + { + "id": "supplied-coder-instructions", + "dimension": "coverage", + "passed": false, + "detail": "Saved hired instructions use the source revision's coder example with its company/name placeholders filled." + }, + { + "id": "hired-instructions-durable", + "dimension": "coverage", + "passed": true, + "detail": "The same saved instruction bundle survives the reused worker execution." + }, + { + "id": "hired-skills-durable", + "dimension": "coverage", + "passed": true, + "detail": "The saved skill selections survive the reused worker execution." + }, + { + "id": "completion-action-attribution", + "dimension": "coverage", + "passed": false, + "detail": "Notification actions require exact canonical/native identities; missing cross-namespace mapping is uncomparable, not proof of extra work." + } + ] + }, + "coverageUnchanged": true, + "otherOutcomesUnchanged": true, + "originalBytesUnchanged": true, + "inputSha256": { + "result": "844763ddc83461b2bea620f153f4bfe0bfee20a27bf33a60017a45c774ce2f6a", + "hiringTemplate": "27b153798472e206970c46358febdda363cb75f892734757e76994117f0d2c9b", + "apiState": "441cf7272ea28c5e1182adf48dba09a401e88c6cad3a99e72c8cf3f46b1c5b41" + } + } + ] +} diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 68eb10ae516..0da021312dc 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -1508,15 +1508,26 @@ explicit fixture user request. It is not an additional production requirement. A follow-up delegates a second fixture to the same coder with underscore separators while preserving the original. A final read-only chat turn requests recorded task status. Three CEO turns and two actual worker executions make -**five expected turns per cell**, with a **15-minute deadline** and 1,000-cent +**five required work turns per cell**, plus at most **two strictly attributed +automatic task-completion turns** (seven total maximum), with a **15-minute +deadline** and 1,000-cent company/CEO budget hard stops. Normal managed-account fixture cleanup and company-wide cancellation apply. Both cells opt into the existing native API tools. No model-authored code is executed by the grading host. The independent oracle checks every JSON input and computed value, authorship, -two distinct completed tasks, project/reporting identity, exact five-run count, +two distinct completed tasks, project/reporting identity, exactly three user-requested +CEO turns and one coder execution per task, managed execution-account attribution, original document preservation, and -worker reuse. It separately checks the production CEO bundle, assigned hiring +worker reuse. Bounded completion turns must have the same company, managed +account, responsible user, chat generation and known completed tasks; unique +server delivery/update receipts; valid completion timing; and a run-attributed +chat reply. One completion turn may batch both tasks. Unknown, duplicate, +failed, retried or extra work runs, and notification-created tasks fail. Every +actual run remains in usage/cost accounting. The hiring scorer and final chat +count guard use the same rule. All other chat count guards stay unchanged. + +The versioned `paperclip.hiring-templates.v3` oracle separately checks the production CEO bundle, assigned hiring skill, source hashes, completed pre-hire read receipts, the saved source-derived coder example, and durable instruction/skill selections. @@ -1524,7 +1535,8 @@ coder example, and durable instruction/skill selections. bytes on each evaluated revision. A historical four-file CEO bundle and long coder example are admissible; the candidate is not imposed on the baseline. Instruction bytes and word counts are measurements, without a size pass/fail -threshold. The definition digest fingerprints the cases, flow and grader; +threshold. The definition digest fingerprints the cases, flow, grader, shared +turn-accounting helper and final chat guard; source evidence also fingerprints the loader, generic execution contract, selected CEO files and production hiring references. Use the same fixture revision, scenario nonce, profile/model, managed account method and local @@ -1566,3 +1578,9 @@ The existing `first-task` suite uses the actual onboarding wizard and captures the changed chief-of-staff persona and skill selections; it needs no fixture change for that default selection. Hiring from that wizard-created chief of staff remains a separate follow-up qualification. + +### Hiring completion accounting evidence + +The v3 hiring grader uses turn-accounting v2 in both executable guards. It requires complete per-run public event streams, exact native tool-use/result pairing and canonical execution IDs for completion actions. Only successful known GET issue/document/comment operations, verified reads/discovery, and attributed native chat finish are admitted. Writes, failed mutation attempts, incomplete streams and unknown actions cannot pass. Separate ACPX host request IDs and provider execution IDs are not joined by name/order/count; missing mapping is uncomparable action coverage, not a measured task failure. The original source-read and exact template checks remain unchanged. + +The live fixture retries entire bracketed observations, waits for both known task callbacks and attributed replies (including batching), checks untruncated pending-wake diagnostics, and requires two equal settled observations. Silence before outbox enqueue is not delivery. Five-turn generic accounting remains calibrated for no owed notifications; this delegated fixture owes two completions. All actual runs remain counted for usage and cost. Retained original, limited sidecar-v1, initial executable, and stricter v3 assessments remain separately versioned; no models are rerun by the repair. diff --git a/tests/runner-e2e/chat-flow.ts b/tests/runner-e2e/chat-flow.ts index cd48e0bd23f..4d48931feca 100644 --- a/tests/runner-e2e/chat-flow.ts +++ b/tests/runner-e2e/chat-flow.ts @@ -1,4 +1,6 @@ import { runHiringTemplateFlow } from "./hiring-template-flow.js"; +import { gradeHiringTemplateTurns } from "./hiring-template-turn-accounting.js"; +import type { HiringTemplateEvidence } from "./hiring-template-scoring.js"; import { runAmbiguousConfirmationReply, runUnansweredQuestionReturn } from "./confirmation-replies.js"; import { expect, type Page } from "@playwright/test"; import { pollUntil, type RunnerApi } from "./api.js"; @@ -17,6 +19,21 @@ import { runActiveReassignment, runWorkerCrash, runAnswerQuality } from "./chat- import { matchesRunCount, minimumRunCount } from "./run-count.js"; import { runChatCompletionUpdate } from "./completion-update-flow.js"; +/** Hiring alone admits bounded, verified lifecycle notifications. Other suites keep their count contract. */ +export function assertChatFlowRunCount(input: { + suiteId: string; task: Parameters[0]; runs: ChatRun[]; + hiringEvidence?: HiringTemplateEvidence; hiringApiState?: unknown; +}) { + expect(matchesRunCount(input.task, input.runs.length), "Declared total provider-run bounds").toBe(true); + if (input.suiteId !== "hiring-templates") return; + const accounting = gradeHiringTemplateTurns({ + evidence: input.hiringEvidence ? { ...input.hiringEvidence, runs: input.runs } : undefined, + apiState: input.hiringApiState, + }); + expect(accounting.passed, `Hiring lifecycle: ${accounting.predicates.filter(p => !p.passed).map(p => p.id).join(", ")}`).toBe(true); + return accounting; +} + // Public API observations only: this driver never fabricates provider results or writes DB state. export interface ChatIssue { id: string; @@ -419,6 +436,8 @@ export async function runChatFlow(input: ChatFlowInput) { await idle(count); }; const noTasks = async () => expect(await tasks()).toHaveLength(0); + let hiringEvidence: HiringTemplateEvidence | undefined; + let refreshHiringEvidence: (() => Promise) | undefined; try { if (caseId === "enable-disable-resume") await enableChatThroughSettings(input); else await api.patch("/api/instance/settings/experimental", { @@ -441,7 +460,9 @@ export async function runChatFlow(input: ChatFlowInput) { await runChatCompletionUpdate({ input, marker, allRuns, issue: () => issue!, refreshIssue: async () => { issue = await api.get(chatPath); if (issue) input.observe(issue, await allRuns()); } }); } else if (execution.suite.id === "hiring-templates") { - await runHiringTemplateFlow({ input, issue: () => issue!, turn, tasks, allRuns }); + const hiring = await runHiringTemplateFlow({ input, issue: () => issue!, turn, tasks, allRuns }); + hiringEvidence = hiring.evidence; + refreshHiringEvidence = hiring.refresh; } else if (execution.suite.id === "agent-chat-qualification") { const context = { input, marker, issue: () => issue!, idle, allRuns, comments, expectedStops, refreshIssue: async () => { issue = await api.get(chatPath); input.observe(issue, await allRuns()); } }; @@ -982,7 +1003,17 @@ export async function runChatFlow(input: ChatFlowInput) { }); } await idle(minimumRunCount(execution.task)); - expect(matchesRunCount(execution.task, runs.filter((run) => !isResetRun(run)).length)).toBe(true); + if (execution.suite.id === "hiring-templates") { + // Refresh the whole consistent evidence generation for the final guard. + hiringEvidence = await refreshHiringEvidence!(); + runs = hiringEvidence.runs; + } + assertChatFlowRunCount({ suiteId: execution.suite.id, task: execution.task, + // Hiring must account for every actual company run, including unexpected resets. + runs: execution.suite.id === "hiring-templates" ? runs : runs.filter((run) => !isResetRun(run)), + hiringEvidence, hiringApiState: execution.suite.id === "hiring-templates" + ? hiringEvidence!.turnApiState : undefined, + }); for (const run of runs.filter((run) => !isResetRun(run))) { expect(run.runtimeMode).toBe(execution.profile.expectedRuntimeMode); expect(run.status).toBe( diff --git a/tests/runner-e2e/hiring-template-cases.ts b/tests/runner-e2e/hiring-template-cases.ts index 6291ce4a72a..7850dd355bf 100644 --- a/tests/runner-e2e/hiring-template-cases.ts +++ b/tests/runner-e2e/hiring-template-cases.ts @@ -2,7 +2,7 @@ import { createHash } from "node:crypto"; import { readFileSync } from "node:fs"; import type { RunnerProfileFixture, RunnerTaskFixture } from "./types.js"; -export const HIRING_TEMPLATE_GRADER_VERSION = "paperclip.hiring-templates.v1"; +export const HIRING_TEMPLATE_GRADER_VERSION = "paperclip.hiring-templates.v3"; export const HIRING_TEMPLATE_SKILL_KEY = "paperclipai/paperclip/paperclip-create-agent"; export const HIRING_TEMPLATE_READ_FILES = [ "SKILL.md", @@ -17,7 +17,8 @@ export const HIRING_TEMPLATE_SOURCE_FILES = [ "skills/paperclip-create-agent/references/baseline-role-guide.md", ] as const; export const hiringTemplateDefinitionDigest = createHash("sha256").update( - ["cases", "scoring", "flow"].map(part => readFileSync(new URL(`./hiring-template-${part}.ts`, import.meta.url), "utf8")).join("\n"), + ["hiring-template-cases.ts", "hiring-template-scoring.ts", "hiring-template-flow.ts", "hiring-template-turn-accounting.ts", "chat-flow.ts"] + .map(file => readFileSync(new URL(`./${file}`, import.meta.url), "utf8")).join("\n"), ).digest("hex"); export function hiringTemplateProfile(profile: RunnerProfileFixture): RunnerProfileFixture { @@ -49,7 +50,7 @@ export function hiringTemplateScenario(nonce: string, projectName: string) { export const hiringTemplateTasks: readonly RunnerTaskFixture[] = [{ id: "hire-coder-template-reuse", label: "Hire with production templates and reuse the coder", - groups: ["chat"], flow: "agent_chat", workMode: "standard", expectedRunCount: 5, + groups: ["chat"], flow: "agent_chat", workMode: "standard", expectedRunCount: 7, minimumExpectedRunCount: 5, attemptTimeoutMs: { local: 15 * 60_000, daytona: 15 * 60_000 }, expectedTerminalState: { issue: "in_review", run: "succeeded" }, buildTitle: nonce => `Production hiring templates ${nonce}`, diff --git a/tests/runner-e2e/hiring-template-flow.ts b/tests/runner-e2e/hiring-template-flow.ts index 6976788f917..203b4a1a496 100644 --- a/tests/runner-e2e/hiring-template-flow.ts +++ b/tests/runner-e2e/hiring-template-flow.ts @@ -6,7 +6,7 @@ import { HIRING_TEMPLATE_GRADER_VERSION, HIRING_TEMPLATE_READ_FILES, HIRING_TEMP HIRING_TEMPLATE_SOURCE_FILES, hiringTemplateDefinitionDigest, hiringTemplateScenario } from "./hiring-template-cases.js"; import { gradeHiringTemplate, hiringTemplateHash, hiringTemplateSize, type HiringAgent, type HiringDocument, type HiringInstructionSnapshot, type HiringTemplateEvidence } from "./hiring-template-scoring.js"; -import type { RunnerApi } from "./api.js"; +import { pollUntil, type RunnerApi } from "./api.js"; export async function readHiringInstructions(api: Pick, agentId: string): Promise { const bundle = await api.get<{ mode: string | null; entryFile: string; files: Array<{ path: string; binary?: boolean }> }>(`/api/agents/${agentId}/instructions-bundle`); @@ -62,6 +62,61 @@ export function renderHiringCoderExample(reference: string, agentName: string, c return example.replaceAll("{{agentName}}", agentName).replaceAll("{{companyName}}", companyName).replaceAll("{{managerTitle}}", managerTitle).replaceAll("{{issuePrefix}}", issuePrefix); } +/** Reads only existing public surfaces; this does not fabricate lifecycle evidence. */ +export async function readHiringTurnApiState(api: Pick, chatIssueId: string, allRuns: () => Promise) { + const [issue, comments, runs, wakes] = await Promise.all([ + api.get(`/api/issues/${chatIssueId}`), + api.get(`/api/issues/${chatIssueId}/comments`), + allRuns(), + api.get<{ events: Array<{ kind: string; status?: string; finishedAt?: string | null; runId?: string | null }>; truncated: boolean }>(`/api/issues/${chatIssueId}/diagnostics/wakes`), + ]); + return { issue, comments, runs, wakes }; +} + +type HiringTurnApiState = Awaited>; +interface HiringObservation { + agents: HiringAgent[]; tasks: ChatIssue[]; runs: ChatRun[]; readRuns: HiringTemplateEvidence["readRuns"]; + apiState?: HiringTurnApiState; finalApiState?: HiringTurnApiState; +} +/** Retry whole observations, never splice different ledger generations together. */ +export async function waitForSettledHiringObservation( + load: () => Promise, options: { deadlineAt?: number; intervalMs?: number } = {}, +) { + let previous: string | undefined; + return pollUntil({ label: "hiring work and completion deliveries settle consistently", + deadlineAt: options.deadlineAt ?? Date.now() + 90_000, intervalMs: options.intervalMs ?? 1_000, + load, accept: observation => { + const before = observation.apiState, after = observation.finalApiState; + const stable = before && after && JSON.stringify(before) === JSON.stringify(after) + && JSON.stringify(observation.runs) === JSON.stringify(after.runs); + const completionRuns = observation.runs.filter(run => run.contextSnapshot?.wakeReason === "chat_task_completed"); + const completionsObserved = observation.tasks.filter(task => task.status === "done").every(task => { + const callbacks = completionRuns.filter(run => Array.isArray(run.contextSnapshot?.chatCompletionUpdates) + && run.contextSnapshot.chatCompletionUpdates.filter((update: { id?: string }) => update.id === task.id).length === 1); + return callbacks.length === 1 && after?.comments.some(value => { + const comment = value as Record; + return comment.createdByRunId === callbacks[0]!.id && comment.issueId === after.issue.id + && comment.authorAgentId === callbacks[0]!.agentId && !comment.authorUserId + && typeof comment.body === "string" && comment.body.trim().length > 0; + }); + }); + const pending = !after || after.wakes.truncated || after.wakes.events.some(wake => { + if (wake.kind !== "wake_request") return false; + if (wake.status === "coalesced") return !completionsObserved || !observation.runs.some(run => run.id === wake.runId + && ["succeeded", "failed", "cancelled"].includes(run.status)); + return !["completed", "skipped", "failed", "cancelled"].includes(wake.status ?? ""); + }) || observation.runs.some(run => ["queued", "running", "scheduled_retry"].includes(run.status)); + // Await both known callbacks, even before the pending outbox enqueues a + // wake. Reply attribution proves the server acknowledged each delivery. + const signature = stable && !pending && completionsObserved && after.issue.conversationState === "waiting" + ? JSON.stringify(observation) : undefined; + const settled = Boolean(signature && signature === previous); + previous = signature; + return settled; + }, + }); +} + export async function runHiringTemplateFlow(context: { input: ChatFlowInput; issue(): ChatIssue; turn(message: string, count: number): Promise; tasks(): Promise; allRuns(): Promise; @@ -80,16 +135,27 @@ export async function runHiringTemplateFlow(context: { }; let source: Awaited> | undefined; let scenario: ReturnType | undefined; - async function refresh() { + async function loadObservation() { const observed = await Promise.all([api.get(`${company}/agents`), tasks(), allRuns()]); - [evidence.agents, evidence.tasks, evidence.runs] = observed; - evidence.readRuns = await Promise.all(evidence.runs.map(async run => { + const [agents, taskRows, runs] = observed; + const apiState = evidence.chatIssueId ? await readHiringTurnApiState(api, evidence.chatIssueId, allRuns) : undefined; + const readRuns = await Promise.all(runs.map(async run => { const events = await collectRunEvents<{ seq?: number; eventType?: string; payload?: unknown; createdAt?: string }>( (afterSeq, limit) => api.get(`/api/heartbeat-runs/${run.id}/events?afterSeq=${afterSeq}&limit=${limit}`), ); return { runId: run.id, agentId: run.agentId, events }; })); + // Bracket event collection with a fresh ledger/wake/comment observation. + const finalApiState = evidence.chatIssueId ? await readHiringTurnApiState(api, evidence.chatIssueId, allRuns) : undefined; + return { agents, tasks: taskRows, runs, readRuns, apiState, finalApiState }; } + async function refresh(settle = false) { + const observed = settle && evidence.chatIssueId + ? await waitForSettledHiringObservation(loadObservation) : await loadObservation(); + Object.assign(evidence, { agents: observed.agents, tasks: observed.tasks, runs: observed.runs, + readRuns: observed.readRuns, turnApiState: observed.finalApiState }); + } + try { await api.patch(`${company}/budgets`, { budgetMonthlyCents: 1_000 }); await api.patch(`/api/agents/${f.agent.id}/budgets`, { budgetMonthlyCents: 1_000 }); @@ -125,7 +191,7 @@ export async function runHiringTemplateFlow(context: { await turn(scenario.statusPrompt(initial[0]!.identifier ?? initial[0]!.id, second.identifier ?? second.id), 5); evidence.hiredInstructionsAfterReuse = await readHiringInstructions(api, hired.id); evidence.hiredSkillsAfterReuse = await readHiringSkillSelections(api, f.company.id, hired.id); - await refresh(); + await refresh(true); const result = gradeHiringTemplate(evidence); for (const check of result.checks) input.check?.(`hiringTemplates.${check.dimension}.${check.id}`, check.passed, check.detail); await input.evidence("hiring-template.json", { schema: HIRING_TEMPLATE_GRADER_VERSION, definitionDigest: hiringTemplateDefinitionDigest, @@ -133,9 +199,10 @@ export async function runHiringTemplateFlow(context: { const failed = result.checks.filter(check => !check.passed); if (!result.outcomePassed) throw new Error(`Hiring-template workflow outcome failed: ${failed.filter(check => check.dimension === "outcome").map(check => check.id).join(", ")}`); if (result.comparisonStatus === "uncomparable") throw new Error(`Hiring-template source coverage is uncomparable: ${failed.filter(check => check.dimension === "coverage").map(check => check.id).join(", ")}`); + return { evidence, refresh: async () => { await refresh(true); return evidence; } }; } finally { let observationError: string | undefined; - try { await refresh(); } catch (error) { observationError = String(error); } + try { await refresh(Boolean(scenario)); } catch (error) { observationError = String(error); } await input.evidence("hiring-template.json", { schema: HIRING_TEMPLATE_GRADER_VERSION, definitionDigest: hiringTemplateDefinitionDigest, budgetGuard: { companyMonthlyCents: 1_000, leadMonthlyCents: 1_000 }, scenario, source, evidence, result: gradeHiringTemplate(evidence), observationError }); diff --git a/tests/runner-e2e/hiring-template-scoring.ts b/tests/runner-e2e/hiring-template-scoring.ts index 5cfe2f9320b..39480ea2e46 100644 --- a/tests/runner-e2e/hiring-template-scoring.ts +++ b/tests/runner-e2e/hiring-template-scoring.ts @@ -1,6 +1,7 @@ import { createHash } from "node:crypto"; import { HIRING_TEMPLATE_READ_FILES, HIRING_TEMPLATE_SKILL_KEY } from "./hiring-template-cases.js"; import type { ChatIssue, ChatRun } from "./chat-flow.js"; +import { gradeHiringTemplateTurns } from "./hiring-template-turn-accounting.js"; const record = (value: unknown): Record => value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; @@ -22,6 +23,8 @@ export interface HiringTemplateEvidence { connectionId: string; binding: unknown; tasks: ChatIssue[]; runs: ChatRun[]; first?: HiringDocument; firstAfterReuse?: HiringDocument; second?: HiringDocument; readRuns: HiringReadRun[]; + /** Independent public chat/comment/run observations for strict lifecycle accounting. */ + turnApiState?: unknown; } function expectedFixture(inputs: readonly string[], marker: string, separator: "-" | "_") { @@ -130,13 +133,9 @@ export function gradeHiringTemplate(e: HiringTemplateEvidence) { check("two-worker-tasks", "outcome", ids.every(Boolean) && new Set(ids).size === 2 && e.tasks.length === 2 && ids.every(id => e.tasks.some(t => t.id === id && t.status === "done" && t.assigneeAgentId === hired?.id && t.projectId === e.projectId && !t.parentId)), "Both independent tasks are completed by the same coder in the chosen project."); - check("five-successful-turns", "outcome", e.runs.length === 5 && e.runs.every(r => r.status === "succeeded" && r.runtimeMode === "native") - && e.runs.filter(r => r.agentId === e.leadId && r.contextSnapshot?.issueId === e.chatIssueId).length === 3 - && ids.every(id => { - const runs = e.runs.filter(r => r.contextSnapshot?.issueId === id); - return runs.length === 1 && runs[0]?.agentId === hired?.id - && record(runs[0]?.contextSnapshot?.aiConnection).connectionId === e.connectionId; - }), "Five turns include one actual coder execution for each task, attributed to the managed account."); + const turnAccounting = gradeHiringTemplateTurns({ evidence: e, apiState: e.turnApiState }); + check("bounded-work-and-completion-turns", "outcome", turnAccounting.passed, + "Exactly three requested lead turns and two coder executions, plus at most two strictly attributed completion notifications; every actual run remains counted."); check("initial-json-artifact", "outcome", documentMatches(e.first, expectedFixture(e.inputs, e.marker, "-")) && e.first?.createdByAgentId === hired?.id, "Independent computation checks each original input/value and worker authorship."); check("reused-json-artifact", "outcome", documentMatches(e.second, expectedFixture(e.inputs, `REUSE${e.marker}`, "_")) @@ -161,8 +160,10 @@ export function gradeHiringTemplate(e: HiringTemplateEvidence) { && e.hiredInstructions.entryFile === "AGENTS.md" && instruction.trim() === expectedCoder, "Saved hired instructions use the source revision's coder example with its company/name placeholders filled."); check("hired-instructions-durable", "coverage", Boolean(e.hiredInstructions) && sameJson(e.hiredInstructions, e.hiredInstructionsAfterReuse), "The same saved instruction bundle survives the reused worker execution."); check("hired-skills-durable", "coverage", Boolean(e.hiredSkills) && sameJson(e.hiredSkills, e.hiredSkillsAfterReuse), "The saved skill selections survive the reused worker execution."); + check("completion-action-attribution", "coverage", turnAccounting.actionEvidence.status !== "uncomparable", + "Notification actions require exact canonical/native identities; missing cross-namespace mapping is uncomparable, not proof of extra work."); return { - checks, readReceipts: receipts, + checks, readReceipts: receipts, turnAccounting, outcomePassed: checks.filter(c => c.dimension === "outcome").every(c => c.passed), comparisonStatus: checks.filter(c => c.dimension === "coverage").every(c => c.passed) ? "comparable" : "uncomparable", instructionSizes: { ceo: Object.fromEntries(Object.entries(e.leadInstructions?.files ?? {}).map(([file, content]) => [file, hiringTemplateSize(content)])), coder: hiringTemplateSize(instruction) }, diff --git a/tests/runner-e2e/hiring-template-turn-accounting.test.ts b/tests/runner-e2e/hiring-template-turn-accounting.test.ts new file mode 100644 index 00000000000..2955d091343 --- /dev/null +++ b/tests/runner-e2e/hiring-template-turn-accounting.test.ts @@ -0,0 +1,274 @@ +import assert from "node:assert/strict"; +import { it } from "vitest"; +import { gradeHiringTemplateTurns, HIRING_TEMPLATE_TURN_ACCOUNTING_VERSION, + type HiringTemplateTurnPredicateId } from "./hiring-template-turn-accounting.js"; +import { createHiringTemplateTurnFixture as fixture, hiringTurnFixtureTimestamp as timestamp } from "./hiring-template-turn-fixture.js"; + +type Fixture = ReturnType; +type FixtureContext = Fixture["evidence"]["runs"][number]["contextSnapshot"]; +const e = (f: Fixture) => f.evidence; +const notification = (f: Fixture) => e(f).runs.find(r => r.id === "notify-1")!; +const worker = (f: Fixture) => e(f).runs.find(r => r.id === "worker-first")!; +const request = (f: Fixture) => e(f).runs.find(r => r.id === "lead-first")!; +const sync = (f: Fixture) => { f.apiState.runs = structuredClone(e(f).runs); return f; }; +const fails = (f: { evidence: unknown; apiState: unknown }, predicate?: HiringTemplateTurnPredicateId) => { + const result = gradeHiringTemplateTurns(f); + assert.equal(result.passed, false); + assert.equal(result.predicates.length, 12); + if (predicate) assert.equal(result.predicates.find(p => p.id === predicate)?.passed, false, predicate); +}; + +// Public positive and negative lifecycle cases ported from the published v1 +// retained-evidence sidecar. Its original-grade/provenance wrapper stays separate. +for (const [count, label] of [[0, "five exact work turns"], [1, "six runs with one batched completion"], [2, "seven runs with distinct completions"]] as const) { + it(`accepts ${label} while counting every run toward costs`, () => { + const result = gradeHiringTemplateTurns(fixture(count)); + assert.equal(result.passed, true); + assert.equal(result.version, HIRING_TEMPLATE_TURN_ACCOUNTING_VERSION); + assert.equal(result.predicates.length, 12); + assert.ok(result.predicates.every(predicate => predicate.passed)); + assert.deepEqual(result.counts, { requiredWorkTurns: 5, maximumCompletionTurns: 2, maximumTotalTurns: 7, + requestedLeadTurns: 3, coderTurns: 2, completionTurns: count, unclassifiedTurns: 0, + snapshotRunCount: 5 + count, actualRunCount: 5 + count, costAccountingRunCount: 5 + count }); + assert.deepEqual(Object.keys(result).sort(), ["actionEvidence", "counts", "passed", "predicates", "version"]); + }); +} + +const negatives: Array<[string, (f: Fixture) => void, HiringTemplateTurnPredicateId]> = [ + ["an eighth run", f => { e(f).runs.push({ ...structuredClone(notification(f)), id: "extra" }); }, "exact-five-required-work-turns"], + ["a duplicate run ID", f => { e(f).runs.push(structuredClone(notification(f))); }, "complete-public-run-ledger"], + ["an arbitrary chat wake", f => { notification(f).contextSnapshot.wakeReason = "heartbeat_timer"; }, "exact-five-required-work-turns"], + ["a completion label on a coder run", f => { notification(f).agentId = "coder"; }, "exact-five-required-work-turns"], + ["an extra requested lead turn", f => { e(f).runs.push({ ...structuredClone(request(f)), id: "lead-extra" }); }, "exact-five-required-work-turns"], + ["an extra coder execution", f => { e(f).runs.push({ ...structuredClone(worker(f)), id: "worker-extra" }); }, "exact-five-required-work-turns"], + ["a missing requested lead turn", f => { e(f).runs = e(f).runs.filter(r => r.id !== "lead-status"); }, "exact-five-required-work-turns"], + ["a missing coder execution", f => { e(f).runs = e(f).runs.filter(r => r.id !== "worker-first"); }, "exact-five-required-work-turns"], + ["a missing production lead identity", f => { e(f).agents[0].id = "other"; }, "known-fixture-context"], + ["a failed notification", f => { notification(f).status = "failed"; }, "successful-native-without-retries"], + ["a legacy notification runtime", f => { notification(f).runtimeMode = "legacy"; }, "successful-native-without-retries"], + ["a retried notification", f => { notification(f).retryOfRunId = "prior"; }, "successful-native-without-retries"], + ["hidden process retries", f => { notification(f).processLossRetryCount = 1; }, "successful-native-without-retries"], + ["a scheduled retry", f => { notification(f).scheduledRetryAttempt = 1; }, "successful-native-without-retries"], + ["a continuation run", f => { notification(f).continuationAttempt = 1; }, "successful-native-without-retries"], + ["a run in another company", f => { notification(f).companyId = "other"; }, "resolved-company-account-and-identity"], + ["a wrong managed account", f => { notification(f).contextSnapshot.aiConnection.connectionId = "other"; }, "resolved-company-account-and-identity"], + ["a wrong provider", f => { worker(f).contextSnapshot.aiConnection.provider = "other"; }, "resolved-company-account-and-identity"], + ["a wrong auth method", f => { worker(f).contextSnapshot.aiConnection.method = "host"; }, "resolved-company-account-and-identity"], + ["a wrong responsible-user binding", f => { notification(f).contextSnapshot.aiConnection.responsibleUserId = "other"; }, "resolved-company-account-and-identity"], + ["a wrong run responsible user", f => { request(f).responsibleUserId = "other"; }, "resolved-company-account-and-identity"], + ["an accepted foreign identity", f => { notification(f).identityHistory[0].responsibleUserId = "other"; }, "resolved-company-account-and-identity"], + ["an identity receipt for another run", f => { notification(f).identityHistory[0].runId = "other"; }, "resolved-company-account-and-identity"], + ["a missing instruction receipt", f => { request(f).identityHistory.pop(); }, "three-distinct-requested-chat-turns"], + ["a rejected instruction receipt", f => { request(f).identityHistory[1].status = "rejected"; }, "three-distinct-requested-chat-turns"], + ["a wrong instruction comment", f => { request(f).identityHistory[1].messageId = "other"; }, "three-distinct-requested-chat-turns"], + ["a forged request author", f => { f.apiState.comments[0].authorUserId = "other"; }, "three-distinct-requested-chat-turns"], + ["an additional user request", f => { f.apiState.comments.push({ ...f.apiState.comments[0], id: "request-extra" }); }, "three-distinct-requested-chat-turns"], + ["two runs for one request comment", f => { e(f).runs.find(r => r.id === "lead-reuse")!.contextSnapshot.wakeCommentId = "request-1"; }, "three-distinct-requested-chat-turns"], + ["a different requested chat generation", f => { request(f).contextSnapshot.conversationSessionGeneration = 1; }, "three-distinct-requested-chat-turns"], + ["a worker assigned to another agent", f => { e(f).tasks[0].assigneeAgentId = "other"; }, "one-coder-execution-per-known-task"], + ["a task outside the selected project", f => { e(f).tasks[0].projectId = "other"; }, "one-coder-execution-per-known-task"], + ["a notification-created task", f => { e(f).tasks[0].originRunId = notification(f).id; }, "no-notification-created-extra-tasks"], + ["a task created by the status-only turn", f => { e(f).tasks[0].originRunId = "lead-status"; }, "task-origins-are-first-two-requested-turns"], + ["an extra task without an execution", f => { e(f).tasks.push({ ...e(f).tasks[0], id: "extra-task" }); }, "no-notification-created-extra-tasks"], + ["a wake without delivery IDs", f => { delete notification(f).contextSnapshot.chatCompletionDeliveryIds; }, "bounded-server-completion-receipts"], + ["a wake without task updates", f => { delete notification(f).contextSnapshot.chatCompletionUpdates; }, "bounded-server-completion-receipts"], + ["mismatched receipt cardinality", f => { notification(f).contextSnapshot.chatCompletionDeliveryIds!.push("extra-delivery"); }, "bounded-server-completion-receipts"], + ["a duplicate delivery across runs", f => { e(f).runs.find(r => r.id === "notify-2")!.contextSnapshot.chatCompletionDeliveryIds = notification(f).contextSnapshot.chatCompletionDeliveryIds; }, "bounded-server-completion-receipts"], + ["a duplicate task notification with a fresh delivery ID", f => { e(f).runs.find(r => r.id === "notify-2")!.contextSnapshot.chatCompletionUpdates = structuredClone(notification(f).contextSnapshot.chatCompletionUpdates); }, "bounded-server-completion-receipts"], + ["an unknown notified task", f => { notification(f).contextSnapshot.chatCompletionUpdates![0].id = "unknown"; }, "bounded-server-completion-receipts"], + ["a pre-completion wake", f => { notification(f).startedAt = timestamp(10); }, "bounded-server-completion-receipts"], + ["a false completedAt receipt", f => { notification(f).contextSnapshot.chatCompletionUpdates![0].completedAt = timestamp(10); }, "bounded-server-completion-receipts"], + ["a false task identifier", f => { notification(f).contextSnapshot.chatCompletionUpdates![0].identifier = "OTHER-1"; }, "bounded-server-completion-receipts"], + ["a wrong task link", f => { notification(f).contextSnapshot.chatCompletionUpdates![0].url = "/issues/OTHER-1"; }, "bounded-server-completion-receipts"], + ["a false task status", f => { notification(f).contextSnapshot.chatCompletionUpdates![0].status = "in_progress"; }, "bounded-server-completion-receipts"], + ["a task without a saved result", f => { notification(f).contextSnapshot.chatCompletionUpdates![0].hasSavedDocuments = false; }, "bounded-server-completion-receipts"], + ["a stale completion generation", f => { notification(f).contextSnapshot.conversationSessionGeneration = 1; }, "bounded-server-completion-receipts"], + ["a manually invoked completion label", f => { notification(f).invocationSource = "on_demand"; }, "bounded-server-completion-receipts"], + ["a mixed request/completion wake", f => { notification(f).contextSnapshot.wakeCommentId = "request-1"; }, "bounded-server-completion-receipts"], + ["a missing completion dispatch identity", f => { notification(f).identityHistory = []; }, "bounded-server-completion-receipts"], + ["a missing completion reply", f => { f.apiState.comments = f.apiState.comments.filter(c => c.id !== "reply-1"); }, "completion-runs-have-attributed-chat-replies"], + ["a reply attributed to a different run", f => { f.apiState.comments.find(c => c.id === "reply-1")!.createdByRunId = "lead-first"; }, "completion-runs-have-attributed-chat-replies"], + ["a reply by another agent", f => { f.apiState.comments.find(c => c.id === "reply-1")!.authorAgentId = "coder"; }, "completion-runs-have-attributed-chat-replies"], + ["a reply in another chat", f => { f.apiState.comments.find(c => c.id === "reply-1")!.issueId = "other"; }, "completion-runs-have-attributed-chat-replies"], + ["a reply before task completion", f => { f.apiState.comments.find(c => c.id === "reply-1")!.createdAt = timestamp(10); }, "completion-runs-have-attributed-chat-replies"], +]; +for (const [label, mutate, predicate] of negatives) it(`rejects ${label}`, () => { + const f = fixture(); mutate(f); fails(sync(f), predicate); +}); +it("rejects extra runs present only in the final public ledger and counts their cost", () => { + const f = fixture(); f.apiState.runs.push({ ...structuredClone(notification(f)), id: "extra" }); + fails(f, "complete-public-run-ledger"); + assert.equal(gradeHiringTemplateTurns(f).counts.costAccountingRunCount, 8); +}); +it("rejects an account discrepancy between retained ledgers", () => { + const f = fixture(); f.apiState.runs[0]!.contextSnapshot.aiConnection.method = "host"; + fails(f, "complete-public-run-ledger"); +}); +it("does not require a prior grade, artifact outcome, source coverage or report provenance", () => { + const f = fixture(); + assert.equal(gradeHiringTemplateTurns({ ...f, result: { outcomePassed: false, comparisonStatus: "uncomparable" } } as typeof f).passed, true); + assert.deepEqual(Object.keys(f).sort(), ["apiState", "evidence"]); +}); +it("does not modify observations or expose opaque run content", () => { + const f = fixture(); notification(f).nativeSessionId = "PRIVATE_SESSION"; + notification(f).hiddenReasoning = "PRIVATE_REASONING"; notification(f).credential = "PRIVATE_CREDENTIAL"; + const before = JSON.stringify(f), result = gradeHiringTemplateTurns(f); + assert.equal(JSON.stringify(f), before); + assert.equal(result.version, HIRING_TEMPLATE_TURN_ACCOUNTING_VERSION); + assert.doesNotMatch(JSON.stringify(result), /PRIVATE_|"company"|"board"|"account"|"notify-1"|"lead-first"/); +}); + +const missingObservations = [ + "evidence.agents", "evidence.tasks", "evidence.runs", "evidence.first", "evidence.second", "evidence.binding", "evidence.connectionId", + "apiState.issue", "apiState.runs", "apiState.comments", "apiState.issue.companyId", "apiState.issue.conversationUserId", + "apiState.issue.conversationAgentId", "apiState.issue.conversationSessionGeneration", + "evidence.runs.0.identityHistory", "evidence.runs.0.contextSnapshot.aiConnection", "evidence.runs.0.startedAt", "evidence.runs.0.finishedAt", + "evidence.tasks.0.completedAt", "evidence.tasks.0.originRunId", "apiState.comments.0.createdAt", "apiState.comments.3.createdByRunId", +]; +for (const path of missingObservations) it(`fails closed without ${path}`, () => { + const f = fixture(), keys = path.split("."); + let target = f as unknown as Record; + for (const key of keys.slice(0, -1)) target = target[key] as Record; + delete target[keys.at(-1)!]; + fails(f); +}); +for (const [label, value] of [["null", null], ["undefined", undefined], ["string", "bad"], ["number", 42], ["boolean", true], ["array", []]] as const) { + it(`rejects ${label} evidence without throwing`, () => fails({ evidence: value, apiState: fixture().apiState })); + it(`rejects ${label} API state without throwing`, () => fails({ evidence: fixture().evidence, apiState: value })); +} +it("fails closed on unstringifiable accounting fields", () => { + for (const value of [1n, Object.assign({}, { nested: {} })]) { + const f = fixture(); + if (typeof value === "object") value.nested = value; + notification(f).contextSnapshot = { ...notification(f).contextSnapshot, chatCompletionUpdates: value } as unknown as FixtureContext; + fails(sync(f), "complete-public-run-ledger"); + } +}); +it("retains the observed public cost count when evidence getters throw", () => { + const f = fixture(); + const evidence = new Proxy(f.evidence, { get() { throw new Error("Malformed observation"); } }); + const result = gradeHiringTemplateTurns({ evidence, apiState: f.apiState }); + assert.equal(result.passed, false); + assert.equal(result.counts.costAccountingRunCount, 7); +}); + +it("rejects a notification write_document despite valid tasks and replies", () => { + const f = fixture(); + for (const event of f.evidence.readRuns[0]!.events) { + const payload = event.payload.prpEvent.payload as Record; + if (payload.name) payload.name = "write_document"; + if (payload.item?.name) payload.item.name = "write_document"; + } + fails(f, "completion-turns-only-report-actions"); + assert.equal(gradeHiringTemplateTurns(f).actionEvidence.status, "violated"); +}); +it("rejects a generic call_api document PUT even though it completes successfully", () => { + const f = fixture(); + for (const event of f.evidence.readRuns[0]!.events) { + const item = (event.payload.prpEvent.payload as Record).item; + if (item?.input) item.input.operationId = "PUT /api/issues/{id}/documents/{key}"; + if (item?.result) item.result.apiOperationId = "PUT /api/issues/{id}/documents/{key}"; + } + fails(f, "completion-turns-only-report-actions"); + assert.equal(gradeHiringTemplateTurns(f).actionEvidence.status, "violated"); +}); +for (const defect of ["missing-stream", "missing-terminal", "missing-start", "unmatched-result", "separate-namespace", + "api-error", "different-operation", "unknown-tool", "duplicate-result", "sequence-gap"] as const) { + it(`does not certify notification action evidence with ${defect}`, () => { + const f = fixture(), ledger = f.evidence.readRuns[0]!, events = ledger.events; + const start = events[1]!.payload.prpEvent.payload as Record; + const result = events[2]!.payload.prpEvent.payload as Record; + if (defect === "missing-stream") f.evidence.readRuns = []; + if (defect === "missing-terminal") ledger.events = events.slice(0, -1); + if (defect === "missing-start") ledger.events = events.filter(e => e.eventType !== "item.started"); + if (defect === "unmatched-result") result.item.tool_use_id = "other"; + if (defect === "separate-namespace") { start.item.id = result.item.id = result.item.tool_use_id = "host-request"; } + if (defect === "api-error") result.item.result.ok = false; + if (defect === "different-operation") result.item.result.apiOperationId = "GET /api/issues/{id}"; + if (defect === "unknown-tool") (events[0]!.payload.prpEvent.payload as Record).name = "unknown"; + if (defect === "duplicate-result") { ledger.events.splice(3, 0, structuredClone(events[2]!)); ledger.events.forEach((e, i) => e.seq = i + 1); } + if (defect === "sequence-gap") events[2]!.seq++; + fails(f, "completion-turns-only-report-actions"); + }); +} +it("allows successful native readonly reads without claiming source identity", () => { + const f = fixture(); + for (const ledger of f.evidence.readRuns) ledger.events = ledger.events.filter(event => !event.eventType.startsWith("item.")) + .map((event, i) => { const payload = event.payload.prpEvent.payload as Record; + if (payload.name) Object.assign(payload, { name: "Read File", operation: "read", readOnly: true, transport: "builtin" }); + return { ...event, seq: i + 1 }; }); + assert.equal(gradeHiringTemplateTurns(f).passed, true); +}); + +for (const [label, value] of [["non-2xx API result", { status: 500 }], ["explicit API error", { error: "failed" }], + ["API is_error flag", { is_error: true }], ["API isError flag", { isError: true }]] as const) { + it(`rejects ${label}`, () => { + const f = fixture(); + Object.assign((f.evidence.readRuns[0]!.events[2]!.payload.prpEvent.payload as Record).item.result, value); + fails(f, "completion-turns-only-report-actions"); + }); +} +it("does not trust a readonly hint on an unattributed call_api", () => { + const f = fixture(), ledger = f.evidence.readRuns[0]!; + ledger.events = ledger.events.filter(event => !event.eventType.startsWith("item.")); + ledger.events.forEach((event, i) => { + event.seq = i + 1; + const payload = event.payload.prpEvent.payload as Record; + if (payload.name) Object.assign(payload, { operation: "read", readOnly: true }); + }); + fails(f, "completion-turns-only-report-actions"); +}); +for (const [transport, operation, readOnly] of [["dynamic", "unknown", null], ["mcp", "execute", false]] as const) { + it(`rejects ${transport} write_document with its actual production ${operation} shape`, () => { + const f = fixture(); + for (const event of f.evidence.readRuns[0]!.events) { + const payload = event.payload.prpEvent.payload as Record; + if (payload.name) Object.assign(payload, { name: "write_document", transport, operation, readOnly }); + if (payload.item?.name) payload.item.name = "write_document"; + } + fails(f, "completion-turns-only-report-actions"); + assert.equal(gradeHiringTemplateTurns(f).actionEvidence.status, "violated"); + }); +} +it("rejects an attempted mutation even if it fails and a benign read follows", () => { + const f = fixture(), ledger = f.evidence.readRuns[0]!; + const mutation = structuredClone(ledger.events.slice(0, 4)); + for (const event of mutation) { + const payload = event.payload.prpEvent.payload as Record; + if (payload.executionId) payload.executionId = "mutation"; + if (payload.item) { payload.item.id = "mutation"; if (payload.item.tool_use_id) payload.item.tool_use_id = "mutation"; } + if (payload.item?.input) payload.item.input.operationId = "POST /api/companies/{companyId}/issues"; + if (payload.item?.result) Object.assign(payload.item.result, { apiOperationId: "POST /api/companies/{companyId}/issues", ok: false, status: 403 }); + } + ledger.events.unshift(...mutation); + ledger.events.forEach((event, i) => event.seq = i + 1); + fails(f, "completion-turns-only-report-actions"); + assert.equal(gradeHiringTemplateTurns(f).actionEvidence.status, "violated"); +}); +it("requires an accepted disposition and succeeded terminal in the action stream", () => { + for (const defect of ["accepted", "terminal"] as const) { + const f = fixture(), events = f.evidence.readRuns[0]!.events; + const payload = events[defect === "accepted" ? 4 : 5]!.payload.prpEvent.payload as Record; + if (defect === "accepted") payload.result.schema = "wrong"; + else payload.runTerminalState = "failed"; + fails(f, "completion-turns-only-report-actions"); + } +}); + +it("admits a correlated native finish without forcing Done versus yielded chat disposition", () => { + for (const disposition of ["done", "yielded"]) { + const f = fixture(), ledger = f.evidence.readRuns[0]!; + const start = structuredClone(ledger.events[0]!), finish = structuredClone(ledger.events[3]!); + for (const event of [start, finish]) Object.assign(event.payload.prpEvent.payload, + { executionId: "finish", name: "paperclip_finish", transport: "dynamic" }); + const proposal = { seq: 0, eventType: "run.result.proposed", payload: { prpEvent: { + schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", itemId: "finish", + payload: { schema: "paperclip.run_result.v1", reportedWorkDisposition: disposition }, + } } }; + ledger.events.splice(4, 0, start, proposal as unknown as typeof start, finish); + ledger.events.forEach((event, i) => event.seq = i + 1); + assert.equal(gradeHiringTemplateTurns(f).passed, true); + (proposal.payload.prpEvent as Record).itemId = "unrelated"; + fails(f, "completion-turns-only-report-actions"); + } +}); diff --git a/tests/runner-e2e/hiring-template-turn-accounting.ts b/tests/runner-e2e/hiring-template-turn-accounting.ts new file mode 100644 index 00000000000..f3bea9a5695 --- /dev/null +++ b/tests/runner-e2e/hiring-template-turn-accounting.ts @@ -0,0 +1,317 @@ +/** Strict public lifecycle accounting for the bounded hiring-template journey. + * Three requested CEO turns and two coder executions remain exact. Up to two + * server completion turns are separately admitted and still count toward costs. + * Outcome artifacts and source/read coverage are graded by their own checks. + */ +export const HIRING_TEMPLATE_TURN_ACCOUNTING_VERSION = "paperclip.hiring-template-turn-accounting.v2"; +const predicateIds = [ + "known-fixture-context", + "complete-public-run-ledger", + "exact-five-required-work-turns", + "successful-native-without-retries", + "resolved-company-account-and-identity", + "three-distinct-requested-chat-turns", + "one-coder-execution-per-known-task", + "task-origins-are-first-two-requested-turns", + "bounded-server-completion-receipts", + "completion-runs-have-attributed-chat-replies", + "no-notification-created-extra-tasks", + "completion-turns-only-report-actions", +] as const; +export type HiringTemplateTurnPredicateId = typeof predicateIds[number]; +export interface HiringTemplateTurnPredicate { id: HiringTemplateTurnPredicateId; passed: boolean } +export interface HiringTemplateTurnCounts { + requiredWorkTurns: 5; maximumCompletionTurns: 2; maximumTotalTurns: 7; + requestedLeadTurns: number; coderTurns: number; completionTurns: number; unclassifiedTurns: number; + snapshotRunCount: number; actualRunCount: number; costAccountingRunCount: number; +} +export interface HiringTemplateTurnAccountingResult { + version: typeof HIRING_TEMPLATE_TURN_ACCOUNTING_VERSION; + counts: HiringTemplateTurnCounts; + predicates: HiringTemplateTurnPredicate[]; + passed: boolean; + actionEvidence: HiringCompletionActionEvidence; +} +export interface HiringCompletionActionEvidence { + status: "verified" | "violated" | "uncomparable"; notificationRuns: number; + canonicalExecutions: number; matchedNativeApiCalls: number; unknownExecutions: number; +} +type JsonRecord = Record; +const object = (value: unknown): JsonRecord => + value && typeof value === "object" && !Array.isArray(value) ? value as JsonRecord : {}; +const rows = (value: unknown): JsonRecord[] => Array.isArray(value) ? value.map(object) : []; +const present = (value: unknown): value is string => typeof value === "string" && value.trim().length > 0; +const unique = (values: readonly unknown[]) => values.every(present) && new Set(values).size === values.length; +const sameSet = (a: readonly unknown[], b: readonly unknown[]) => + a.length === b.length && unique(a) && unique(b) && a.every(v => b.includes(v)); +const date = (value: unknown) => typeof value === "string" ? Date.parse(value) : NaN; +const zero = (value: unknown) => value == null || value === 0; +const noList = (value: unknown) => value == null || Array.isArray(value) && value.length === 0; +const context = (run: JsonRecord) => object(run.contextSnapshot); +const accepted = (run: JsonRecord) => rows(run.identityHistory).filter(i => i.status === "accepted"); +const dispatchIdentity = (run: JsonRecord, cause: string, user: unknown) => accepted(run).some(i => i.cause === cause + && i.responsibleUserId === user && !i.messageId && Number.isFinite(date(i.acceptedAt))); + +// Compare only public accounting fields; malformed projections fail closed. +function accountingProjection(run: JsonRecord): string | null { + try { + return JSON.stringify({ id: run.id, companyId: run.companyId, + agentId: run.agentId, status: run.status, runtimeMode: run.runtimeMode, + responsibleUserId: run.responsibleUserId, invocationSource: run.invocationSource, + triggerDetail: run.triggerDetail, startedAt: run.startedAt, finishedAt: run.finishedAt, + retryOfRunId: run.retryOfRunId, processLossRetryCount: run.processLossRetryCount, + scheduledRetryAttempt: run.scheduledRetryAttempt, continuationAttempt: run.continuationAttempt, + context: Object.fromEntries(["source", "issueId", "wakeReason", "wakeSource", "wakeTriggerDetail", "wakeCommentId", + "wakeCommentIds", "conversationSessionGeneration", "chatCompletionDeliveryIds", "chatCompletionUpdates"] + .map(key => [key, context(run)[key]])), + account: Object.fromEntries(["connectionId", "responsibleUserId", "provider", "method", "mode"] + .map(key => [key, object(context(run).aiConnection)[key]])), + identities: rows(run.identityHistory).map(i => Object.fromEntries(["runId", "companyId", "responsibleUserId", + "status", "cause", "messageId", "acceptedAt"].map(key => [key, i[key]]))) }); + } catch { return null; } +} + +const notificationReadApiOperations = new Set([ + "GET /api/issues/{id}", "GET /api/issues/{id}/comments", "GET /api/issues/{id}/documents", "GET /api/issues/{id}/documents/{key}", +]); +const discoveryNames = new Set(["search_api", "search_tasks"]); +const forbiddenWorkTools = new Set(["write_document", "create_task", "hire_agent", "register_deliverable", "create_skill", + "update_agent_instructions", "restore_agent_instructions", "create_project", "reassign_task", "set_dependencies", "set_task_title"]); +const errored = (value: JsonRecord) => Boolean(value.error || value.is_error || value.isError); +function eventEnvelope(event: JsonRecord) { return object(object(event.payload).prpEvent); } +/** Only exact per-run IDs join provider executions to successful native actions. + * ACPX host request IDs and provider stream IDs cannot be joined by order/name. + */ +export function gradeHiringCompletionActions(notifications: JsonRecord[], readRuns: unknown): HiringCompletionActionEvidence { + const ledgers = rows(readRuns); + const result: HiringCompletionActionEvidence = { status: "verified", notificationRuns: notifications.length, + canonicalExecutions: 0, matchedNativeApiCalls: 0, unknownExecutions: 0 }; + const unresolved = () => { result.unknownExecutions++; if (result.status !== "violated") result.status = "uncomparable"; }; + const violation = () => { result.status = "violated"; }; + for (const run of notifications) { + const matching = ledgers.filter(ledger => ledger.runId === run.id && ledger.agentId === run.agentId); + if (matching.length !== 1) { unresolved(); continue; } + const events = rows(matching[0]!.events), terminals = events.filter(event => event.eventType === "run.terminal"); + const acceptedResults = events.filter(event => event.eventType === "run.result.accepted"); + const terminalEnvelope = eventEnvelope(terminals[0] ?? {}), terminal = object(terminalEnvelope.payload); + const acceptedEnvelope = eventEnvelope(acceptedResults[0] ?? {}); + if (!events.length || terminals.length !== 1 || acceptedResults.length !== 1 + || terminalEnvelope.sourceKind !== "control_plane" || terminal.schema !== "paperclip.prp.terminal.v1" + || terminal.runTerminalState !== "succeeded" || terminal.turnTerminalState !== "completed" + || acceptedEnvelope.sourceKind !== "control_plane" || object(object(acceptedEnvelope.payload).result).schema !== "paperclip.run_result.v1" + || !events.every((event, i) => Number.isSafeInteger(event.seq) && event.seq === i + 1)) { unresolved(); continue; } + const starts = new Map(), completed = new Map(); + const nativeStarts = new Map(), nativeResults = new Map(); + for (const event of events) { + const envelope = eventEnvelope(event), payload = object(envelope.payload), item = object(payload.item); + if (event.eventType?.toString().startsWith("tool.execution.")) { + if (payload.schema !== "paperclip.tool.execution.v1" || !present(payload.executionId)) { unresolved(); continue; } + if (event.eventType === "tool.execution.started") { + if (starts.has(payload.executionId)) violation(); + starts.set(payload.executionId, payload); + } else if (event.eventType === "tool.execution.completed") { + if (completed.has(payload.executionId) || payload.status !== "completed" || errored(payload) + || payload.exitCode != null && payload.exitCode !== 0) violation(); + completed.set(payload.executionId, payload); + } else if (["tool.execution.failed", "tool.execution.cancelled", "tool.execution.interrupted"].includes(String(event.eventType))) violation(); + } + if (item.type !== "tool_use" && item.type !== "tool_result") continue; + if (envelope.schema !== "paperclip.prp.event.v1" || envelope.schemaVersion !== 1 || envelope.sourceKind !== "runner" + || !present(item.id) || errored(item)) { unresolved(); continue; } + if (event.eventType === "item.started" && item.type === "tool_use") { + if (nativeStarts.has(item.id)) violation(); + nativeStarts.set(item.id, item); + } else if (event.eventType === "item.completed" && item.type === "tool_result") { + if (item.tool_use_id !== item.id || nativeResults.has(item.id) || errored(object(item.result))) violation(); + nativeResults.set(item.id, item); + } else unresolved(); + } + if (!sameSet([...starts.keys()], [...completed.keys()]) || !sameSet([...nativeStarts.keys()], [...nativeResults.keys()])) unresolved(); + for (const [id, start] of nativeStarts) { + const receipt = object(nativeResults.get(id)?.result), input = object(start.input); + if (start.name === "call_api") { + if (present(input.operationId) && !notificationReadApiOperations.has(input.operationId)) violation(); + if (!present(input.operationId) || input.operationId !== receipt.apiOperationId || receipt.ok !== true + || !Number.isInteger(receipt.status) || Number(receipt.status) < 200 || Number(receipt.status) >= 300 || errored(receipt)) { unresolved(); continue; } + if (!notificationReadApiOperations.has(input.operationId)) violation(); + } else if (start.name === "search_api") { + if (!Array.isArray(receipt.results) || !Number.isInteger(receipt.total) || Number(receipt.total) < 0 || typeof receipt.guidance !== "string" + || receipt.nextCursor != null && typeof receipt.nextCursor !== "string") unresolved(); + } else if (start.name === "search_tasks") { + if (!Array.isArray(receipt.tasks)) unresolved(); + } else if (forbiddenWorkTools.has(String(start.name))) violation(); + else unresolved(); + } + for (const [id, start] of starts) { + result.canonicalExecutions++; + const finish = completed.get(id) ?? {}, native = nativeStarts.get(id); + if (start.name !== finish.name) { unresolved(); continue; } + if (forbiddenWorkTools.has(String(start.name))) { violation(); continue; } + if (start.name === "paperclip_finish") { + const proposals = events.filter(event => event.eventType === "run.result.proposed"); + const proposal = eventEnvelope(proposals[0] ?? {}); + if (start.transport !== "dynamic" || finish.transport !== "dynamic" || proposals.length !== 1 + || proposal.sourceKind !== "runner" || proposal.itemId !== id + || object(proposal.payload).schema !== "paperclip.run_result.v1") unresolved(); + continue; + } + if (start.name === "ToolSearch" && start.transport === "builtin" && finish.transport === "builtin") continue; + if (start.name === "call_api" || discoveryNames.has(String(start.name))) { + if (!native || native.name !== start.name) { unresolved(); continue; } + if (start.name === "call_api") result.matchedNativeApiCalls++; + } else if (start.readOnly === true && finish.readOnly === true && ["read", "search", "list"].includes(String(start.operation)) + && start.operation === finish.operation) continue; + else unresolved(); + } + // Every native action must also have an exact canonical provider identity. + if ([...nativeStarts.keys()].some(id => !starts.has(id))) unresolved(); + } + return result; +} + +function evaluate(evidence: unknown, apiState: unknown): HiringTemplateTurnAccountingResult { + const e = object(evidence), api = object(apiState); + const chat = object(api.issue), user = chat.conversationUserId, company = chat.companyId; + const runs = rows(e.runs), publicRuns = rows(api.runs), comments = rows(api.comments); + const agents = rows(e.agents), tasks = rows(e.tasks), lead = e.leadId; + const coder = agents.find(a => a.id !== lead && a.name === e.hireName); + const taskIds = [object(e.first).issueId, object(e.second).issueId]; + const taskById = new Map(tasks.map(t => [t.id, t])); + const requests = comments.filter(c => c.authorUserId).sort((a, b) => date(a.createdAt) - date(b.createdAt)); + const requested: JsonRecord[] = [], workers: JsonRecord[] = [], notifications: JsonRecord[] = [], unknown: JsonRecord[] = []; + for (const run of runs) { + const c = context(run); + if (run.agentId === lead && c.issueId === e.chatIssueId && c.wakeReason === "issue_commented") requested.push(run); + else if (run.agentId === coder?.id && taskIds.includes(c.issueId) && c.wakeReason === "issue_assigned") workers.push(run); + else if (run.agentId === lead && c.issueId === e.chatIssueId && c.wakeReason === "chat_task_completed") notifications.push(run); + else unknown.push(run); + } + const predicates: HiringTemplateTurnPredicate[] = []; + const check = (id: HiringTemplateTurnPredicateId, passed: unknown) => predicates.push({ id, passed: Boolean(passed) }); + const binding = object(e.binding); + check("known-fixture-context", present(company) && present(user) && present(lead) && present(coder?.id) + && chat.id === e.chatIssueId && chat.conversationAgentId === lead + && typeof chat.conversationSessionGeneration === "number" && Number.isInteger(chat.conversationSessionGeneration) && chat.conversationSessionGeneration >= 0 + && agents.length === 2 && agents.some(a => a.id === lead) + && unique(agents.map(a => a.id)) && agents.every(a => a.companyId === company) + && present(e.connectionId) && present(binding.provider) && present(binding.method) && binding.mode === "responsible_user" + && unique(taskIds) && unique(tasks.map(t => t.id)) && sameSet(taskIds, tasks.map(t => t.id))); + check("complete-public-run-ledger", sameSet(runs.map(r => r.id), publicRuns.map(r => r.id)) + && runs.every(r => { + const projection = accountingProjection(r); + return projection !== null && projection === accountingProjection(publicRuns.find(p => p.id === r.id) ?? {}); + })); + check("exact-five-required-work-turns", requested.length === 3 && workers.length === 2 && unknown.length === 0 + && notifications.length <= 2 && runs.length === 5 + notifications.length && runs.length <= 7); + check("successful-native-without-retries", runs.length > 0 && runs.every(r => r.status === "succeeded" + && r.runtimeMode === "native" && Number.isFinite(date(r.startedAt)) && Number.isFinite(date(r.finishedAt)) + && date(r.finishedAt) >= date(r.startedAt) && !r.retryOfRunId && zero(r.processLossRetryCount) + && zero(r.scheduledRetryAttempt) && zero(r.continuationAttempt))); + check("resolved-company-account-and-identity", runs.length > 0 && runs.every(r => { + const account = object(context(r).aiConnection), identities = accepted(r); + return r.companyId === company && r.responsibleUserId === user && account.responsibleUserId === user + && account.connectionId === e.connectionId && ["provider", "method", "mode"].every(k => account[k] === binding[k]) + && identities.length > 0 && identities.every(i => i.responsibleUserId === user + && i.runId === r.id && i.companyId === company); + })); + check("three-distinct-requested-chat-turns", requests.length === 3 && unique(requests.map(c => c.id)) + && sameSet(requested.map(r => context(r).wakeCommentId), requests.map(c => c.id)) + && requests.every(c => c.companyId === company && c.issueId === chat.id && c.authorUserId === user + && !c.authorAgentId && present(c.body) && Number.isFinite(date(c.createdAt))) + && requested.every(r => { + const c = context(r), comment = requests.find(p => p.id === c.wakeCommentId); + return r.invocationSource === "on_demand" && r.triggerDetail === "manual" && c.source === "issue.comment" + && c.wakeSource === "on_demand" && c.wakeTriggerDetail === "manual" + && c.conversationSessionGeneration === chat.conversationSessionGeneration + && noList(c.chatCompletionDeliveryIds) && noList(c.chatCompletionUpdates) + && sameSet(Array.isArray(c.wakeCommentIds) ? c.wakeCommentIds : [], [c.wakeCommentId]) + && comment && date(r.startedAt) >= date(comment.createdAt) + && dispatchIdentity(r, "issue_commented", user) + && accepted(r).some(i => i.cause === "instruction" && i.messageId === c.wakeCommentId + && i.responsibleUserId === user && Number.isFinite(date(i.acceptedAt))); + })); + check("one-coder-execution-per-known-task", tasks.length === 2 && taskIds.every(id => { + const task = taskById.get(id), matching = workers.filter(r => context(r).issueId === id), run = matching[0] ?? {}; + const c = context(run ?? {}); + return task && matching.length === 1 && task.companyId === company && task.assigneeAgentId === coder?.id + && task.projectId === e.projectId && !task.parentId && task.status === "done" && task.responsibleUserId === user + && Number.isFinite(date(task.completedAt)) && date(task.completedAt) >= date(run.startedAt) + && run.invocationSource === "assignment" && run.triggerDetail === "system" + && c.source === "paperclip_runner.create_task" && c.wakeSource === "assignment" && c.wakeTriggerDetail === "system" + && noList(c.chatCompletionDeliveryIds) && noList(c.chatCompletionUpdates) + && dispatchIdentity(run, "issue_assigned", user); + })); + check("task-origins-are-first-two-requested-turns", taskIds.every((id, index) => { + const task = taskById.get(id), comment = requests[index]; + const run = requested.find(r => context(r).wakeCommentId === comment?.id); + return task && run && task.createdByAgentId === lead && !task.createdByUserId && task.originRunId === run.id + && date(task.createdAt) >= date(run.startedAt); + })); + const deliveryIds = new Set(), notifiedTasks = new Set(); + check("bounded-server-completion-receipts", notifications.every(r => { + const c = context(r), ids = Array.isArray(c.chatCompletionDeliveryIds) ? c.chatCompletionDeliveryIds : []; + const updates = rows(c.chatCompletionUpdates); + let valid = r.invocationSource === "automation" && r.triggerDetail === "system" + && c.wakeSource === "automation" && c.wakeTriggerDetail === "system" && !c.source + && c.conversationSessionGeneration === chat.conversationSessionGeneration + && !c.wakeCommentId && noList(c.wakeCommentIds) + && ids.length > 0 && ids.length <= 2 && ids.length === updates.length && unique(ids) + && ids.every(id => !deliveryIds.has(id)) && dispatchIdentity(r, "chat_task_completed", user); + for (const update of updates) { + const task = taskById.get(update.id); + valid = Boolean(valid && task && !notifiedTasks.has(update.id) && update.status === "done" + && task.status === "done" && update.identifier === task.identifier && update.completedAt === task.completedAt + && present(task.identifier) && update.url === `/issues/${task.identifier}` && update.hasSavedDocuments === true + && date(r.startedAt) >= date(task.completedAt)); + notifiedTasks.add(update.id); + } + ids.forEach(id => deliveryIds.add(id)); + return valid; + })); + check("completion-runs-have-attributed-chat-replies", notifications.every(r => { + const updates = rows(context(r).chatCompletionUpdates); + const completedAt = Math.max(...updates.map(u => date(u.completedAt))); + return updates.length > 0 && comments.some(c => c.companyId === company && c.issueId === chat.id + && c.authorAgentId === lead && !c.authorUserId && c.createdByRunId === r.id && present(c.body) + && (c.conversationSessionGeneration == null || c.conversationSessionGeneration === chat.conversationSessionGeneration) + && date(c.createdAt) >= completedAt && date(c.createdAt) >= date(r.startedAt)); + })); + check("no-notification-created-extra-tasks", tasks.length === 2 && tasks.every(t => + !notifications.some(r => r.id === t.originRunId))); + const actionEvidence = gradeHiringCompletionActions(notifications, e.readRuns); + check("completion-turns-only-report-actions", actionEvidence.status === "verified"); + return { + version: HIRING_TEMPLATE_TURN_ACCOUNTING_VERSION, + counts: { requiredWorkTurns: 5, maximumCompletionTurns: 2, maximumTotalTurns: 7, + requestedLeadTurns: requested.length, coderTurns: workers.length, completionTurns: notifications.length, + unclassifiedTurns: unknown.length, snapshotRunCount: runs.length, actualRunCount: publicRuns.length, + costAccountingRunCount: publicRuns.length }, + predicates, actionEvidence, passed: predicates.every(predicate => predicate.passed), + }; +} + +/** Accept untrusted evidence observations without needing an existing result. */ +export function gradeHiringTemplateTurns(input: { evidence: unknown; apiState: unknown }): HiringTemplateTurnAccountingResult { + let snapshotRunCount = 0, actualRunCount = 0; + try { + const runs = object(object(input).apiState).runs; + actualRunCount = Array.isArray(runs) ? runs.length : 0; + } catch { /* An inaccessible public ledger is a failed observation. */ } + try { + const runs = object(object(input).evidence).runs; + snapshotRunCount = Array.isArray(runs) ? runs.length : 0; + } catch { /* Keep public cost accounting when the evidence is malformed. */ } + try { + const value = object(input); + return evaluate(value.evidence, value.apiState); + } catch { + return { + version: HIRING_TEMPLATE_TURN_ACCOUNTING_VERSION, + counts: { requiredWorkTurns: 5, maximumCompletionTurns: 2, maximumTotalTurns: 7, + requestedLeadTurns: 0, coderTurns: 0, completionTurns: 0, unclassifiedTurns: snapshotRunCount, + snapshotRunCount, actualRunCount, costAccountingRunCount: actualRunCount }, + predicates: predicateIds.map(id => ({ id, passed: false })), passed: false, + actionEvidence: { status: "uncomparable", notificationRuns: 0, canonicalExecutions: 0, matchedNativeApiCalls: 0, unknownExecutions: 0 }, + }; + } +} diff --git a/tests/runner-e2e/hiring-template-turn-fixture.ts b/tests/runner-e2e/hiring-template-turn-fixture.ts new file mode 100644 index 00000000000..81aebc8307c --- /dev/null +++ b/tests/runner-e2e/hiring-template-turn-fixture.ts @@ -0,0 +1,110 @@ +// Test-only public lifecycle facts shared by turn and outcome calibrations. +interface FixtureIdentity { + runId: string; companyId: string; responsibleUserId: string; status: string; + cause: string; messageId: string | null; acceptedAt: string; +} +interface FixtureUpdate { + id: string; identifier: string; status: string; completedAt: string; + url: string; hasSavedDocuments: boolean; +} +interface FixtureRun { + id: string; companyId: string; agentId: string; responsibleUserId: string; + invocationSource: string; triggerDetail: string; status: string; runtimeMode: string; + startedAt: string; finishedAt: string; retryOfRunId: string | null; + processLossRetryCount: number; scheduledRetryAttempt: number; continuationAttempt: number; + identityHistory: FixtureIdentity[]; + contextSnapshot: { + source?: string; issueId: string; wakeReason: string; wakeSource: string; + wakeTriggerDetail: string; conversationSessionGeneration: number; + wakeCommentId?: string; wakeCommentIds: string[]; + aiConnection: { provider: string; method: string; mode: string; connectionId: string; responsibleUserId: string }; + chatCompletionDeliveryIds?: string[]; chatCompletionUpdates?: FixtureUpdate[]; + }; + [key: string]: unknown; +} +interface FixtureComment { + id: string; companyId: string; issueId: string; authorUserId: string | null; + authorAgentId: string | null; body: string; createdAt: string; + createdByRunId?: string; conversationSessionGeneration?: number | null; +} +export const hiringTurnFixtureTimestamp = (seconds: number) => new Date(Date.UTC(2026, 9, 2, 0, 0, seconds)).toISOString(); + +/** Complete canonical/native read-action stream with one exact execution ID. */ +export function hiringNotificationActionEvents() { + const id = "read-action"; + const payload = { schema: "paperclip.tool.execution.v1", executionId: id, transport: "dynamic", + operation: "unknown", name: "call_api", readOnly: null, status: "running", exitCode: null }; + const items: Array<{ eventType: string; payload: Record }> = [ + { eventType: "tool.execution.started", payload }, + { eventType: "item.started", payload: { kind: "tool", item: { id, type: "tool_use", name: "call_api", + input: { operationId: "GET /api/issues/{id}/documents", pathParams: {} } } } }, + { eventType: "item.completed", payload: { kind: "tool", item: { id, tool_use_id: id, type: "tool_result", + result: { apiOperationId: "GET /api/issues/{id}/documents", ok: true, status: 200 } } } }, + { eventType: "tool.execution.completed", payload: { ...payload, status: "completed" } }, + { eventType: "run.result.accepted", payload: { result: { schema: "paperclip.run_result.v1" } } }, + { eventType: "run.terminal", payload: { schema: "paperclip.prp.terminal.v1", runTerminalState: "succeeded", turnTerminalState: "completed" } }, + ]; + return items.map((event, i) => ({ seq: i + 1, eventType: event.eventType, payload: { prpEvent: { + schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: event.eventType.startsWith("run.") ? "control_plane" : "runner", payload: event.payload, + } } })); +} + +export function createHiringTemplateTurnFixture(notificationCount = 2) { + if (!Number.isInteger(notificationCount) || notificationCount < 0 || notificationCount > 2) + throw new RangeError("The positive lifecycle fixture accepts zero, one or two notification turns."); + const timestamp = hiringTurnFixtureTimestamp; + const company = "company", user = "board", lead = "lead", coder = "coder", chat = "chat"; + const binding = { provider: "anthropic", method: "api_key", mode: "responsible_user" }; + function run(id: string, agentId: string, issueId: string, reason: string, start: number, finish: number, source?: string): FixtureRun { + const invocationSource = reason === "issue_commented" ? "on_demand" + : reason === "issue_assigned" ? "assignment" : "automation"; + const triggerDetail = reason === "issue_commented" ? "manual" : "system"; + return { id, companyId: company, agentId, responsibleUserId: user, invocationSource, triggerDetail, + status: "succeeded", runtimeMode: "native", startedAt: timestamp(start), finishedAt: timestamp(finish), + retryOfRunId: null, processLossRetryCount: 0, scheduledRetryAttempt: 0, continuationAttempt: 0, + identityHistory: [{ runId: id, companyId: company, responsibleUserId: user, status: "accepted", + cause: reason, messageId: null, acceptedAt: timestamp(start) }], + contextSnapshot: { source, issueId, wakeReason: reason, wakeSource: invocationSource, + wakeTriggerDetail: triggerDetail, conversationSessionGeneration: 0, wakeCommentIds: [], + aiConnection: { ...binding, connectionId: "account", responsibleUserId: user } } }; + } + const requested = [run("lead-first", lead, chat, "issue_commented", 1, 10, "issue.comment"), + run("lead-reuse", lead, chat, "issue_commented", 21, 30, "issue.comment"), + run("lead-status", lead, chat, "issue_commented", 51, 55, "issue.comment")]; + const comments: FixtureComment[] = requested.map((r, i) => { + const id = `request-${i + 1}`; + r.contextSnapshot.wakeCommentId = id; + r.contextSnapshot.wakeCommentIds = [id]; + r.identityHistory.push({ runId: r.id, companyId: company, responsibleUserId: user, status: "accepted", + cause: "instruction", messageId: id, acceptedAt: r.startedAt }); + return { id, companyId: company, issueId: chat, authorUserId: user, authorAgentId: null, + body: `Fixture request ${i + 1}`, createdAt: timestamp([0, 20, 50][i]!) }; + }); + const workers = [run("worker-first", coder, "first-task", "issue_assigned", 4, 12, "paperclip_runner.create_task"), + run("worker-reuse", coder, "second-task", "issue_assigned", 25, 40, "paperclip_runner.create_task")]; + const tasks = workers.map((r, i) => ({ id: r.contextSnapshot.issueId, companyId: company, projectId: "project", + title: r.contextSnapshot.issueId, parentId: null as string | null, status: "done", identifier: `RUN-${i + 2}`, responsibleUserId: user, + assigneeAgentId: coder, createdByAgentId: lead, createdByUserId: null as string | null, originRunId: requested[i]!.id, + createdAt: timestamp([2, 22][i]!), completedAt: timestamp([11, 39][i]!) })); + const notifications: FixtureRun[] = []; + for (let i = 0; i < notificationCount; i++) { + const r = run(`notify-${i + 1}`, lead, chat, "chat_task_completed", notificationCount === 1 ? 41 : [13, 41][i]!, + notificationCount === 1 ? 45 : [18, 45][i]!); + const covered = notificationCount === 1 ? tasks : [tasks[i]!]; + r.contextSnapshot.chatCompletionDeliveryIds = covered.map(t => `delivery-${t.id}`); + r.contextSnapshot.chatCompletionUpdates = covered.map(t => ({ id: t.id, identifier: t.identifier, + status: "done", completedAt: t.completedAt, url: `/issues/${t.identifier}`, hasSavedDocuments: true })); + comments.push({ id: `reply-${i + 1}`, companyId: company, issueId: chat, authorAgentId: lead, + authorUserId: null, createdByRunId: r.id, body: "The saved fixture is complete.", + conversationSessionGeneration: null, createdAt: r.finishedAt }); + notifications.push(r); + } + const runs = [...requested, ...workers, ...notifications]; + const evidence = { leadId: lead, chatIssueId: chat, hireName: "Fixture Coder", projectId: "project", connectionId: "account", binding, + agents: [{ id: lead, companyId: company, name: "Fixture CEO" }, { id: coder, companyId: company, name: "Fixture Coder" }], + first: { issueId: tasks[0]!.id }, second: { issueId: tasks[1]!.id }, runs, tasks, + readRuns: notifications.map(run => ({ runId: run.id, agentId: run.agentId, events: hiringNotificationActionEvents() })) }; + const apiState = { issue: { id: chat, companyId: company, conversationUserId: user, + conversationAgentId: lead, conversationSessionGeneration: 0 }, comments, runs: structuredClone(runs) }; + return { evidence, apiState }; +} diff --git a/tests/runner-e2e/hiring-template.test.ts b/tests/runner-e2e/hiring-template.test.ts index 0038dc26f58..f047815f857 100644 --- a/tests/runner-e2e/hiring-template.test.ts +++ b/tests/runner-e2e/hiring-template.test.ts @@ -5,12 +5,14 @@ import { buildRunnerE2EProcessEnvironment } from "./harness-env.js"; import { canonicalProviderEventsFromAcpxRuntimeEvent, canonicalProviderEventsFromCodex } from "../../packages/paperclip-runner/src/provider-events.js"; import { loadDefaultAgentInstructionsBundle } from "../../server/src/services/default-agent-instructions.js"; import { HIRING_TEMPLATE_READ_FILES, HIRING_TEMPLATE_SKILL_KEY, hiringTemplateInputs, hiringTemplateScenario } from "./hiring-template-cases.js"; -import { readHiringInstructions, readHiringTemplateSources, renderHiringCoderExample } from "./hiring-template-flow.js"; +import { readHiringInstructions, readHiringTemplateSources, renderHiringCoderExample, waitForSettledHiringObservation } from "./hiring-template-flow.js"; import { gradeHiringTemplate, hiringTemplateHash, hiringTemplateReadReceipts, type HiringTemplateEvidence } from "./hiring-template-scoring.js"; import type { RunnerApi } from "./api.js"; +import { createHiringTemplateTurnFixture } from "./hiring-template-turn-fixture.js"; +import { assertChatFlowRunCount } from "./chat-flow.js"; const beforeHire = "2026-10-01T12:00:00.000Z", hiredAt = "2026-10-01T12:01:00.000Z"; -const binding = { provider: "openai", method: "api_key", mode: "responsible_user" }; +const binding = { provider: "anthropic", method: "api_key", mode: "responsible_user" }; const ceo = { "AGENTS.md": "You are the CEO. Lead the company." }; const coder = "You are Casey, a software engineer at Fixture Company. Own software implementation and maintenance."; // Expected values are stated independently of the implementation under test. @@ -22,27 +24,28 @@ function commandEvents(file: string) { command: `cat /workspace/.agents/skills/paperclip-create-agent/${file}`, aggregatedOutput: "source bytes", } }).map(event => ({ eventType: event.eventType, createdAt: beforeHire, payload: { prpEvent: event } })); } -function validEvidence(): HiringTemplateEvidence { +function validEvidence(notificationCount = 0): HiringTemplateEvidence { + const turnFixture = createHiringTemplateTurnFixture(notificationCount); const sourceFiles = [...HIRING_TEMPLATE_READ_FILES.map(file => `skills/paperclip-create-agent/${file}`), "skills/paperclip-create-agent/references/baseline-role-guide.md", "server/src/onboarding-assets/ceo/AGENTS.md"]; const hashes = Object.fromEntries(sourceFiles.map(file => [file, hiringTemplateHash(file)])); - const tasks = ["first", "second"].map(id => ({ id, companyId: "company", title: id, status: "done", assigneeAgentId: "coder", projectId: "project", parentId: null })); - const runs = ["lead-1", "lead-2", "lead-3", "first", "second"].map(id => ({ id, companyId: "company", agentId: id.startsWith("lead") ? "lead" : "coder", - status: "succeeded", runtimeMode: "native", contextSnapshot: { issueId: id.startsWith("lead") ? "chat" : id, aiConnection: { connectionId: "account" } } })); + const tasks = turnFixture.evidence.tasks; + const runs = turnFixture.evidence.runs; const document = (issueId: string, reference: string, outputs: string[]) => ({ issueId, key: "fixture", latestRevisionId: `${issueId}-revision`, createdByAgentId: "coder", body: JSON.stringify({ reference, entries: hiringTemplateInputs.map((input, index) => ({ input, value: outputs[index] })) }) }); - const first = document("first", "HIREfixture", values); + const first = document("first-task", "HIREfixture", values); return { leadId: "lead", chatIssueId: "chat", hireName: "Casey", marker: "HIREfixture", projectId: "project", inputs: hiringTemplateInputs, expectedCeoFiles: ceo, leadInstructions: { mode: "managed", entryFile: "AGENTS.md", files: ceo }, expectedSourceHashes: hashes, servedSourceHashes: { ...hashes }, assignedSkills: [HIRING_TEMPLATE_SKILL_KEY], coderExample: coder, hiredInstructions: { mode: "managed", entryFile: "AGENTS.md", files: { "AGENTS.md": coder } }, hiredInstructionsAfterReuse: { mode: "managed", entryFile: "AGENTS.md", files: { "AGENTS.md": coder } }, hiredSkills: [], hiredSkillsAfterReuse: [], - agents: [{ id: "lead", name: "CEO", adapterConfig: { model: "model" } }, - { id: "coder", name: "Casey", role: "engineer", reportsTo: "lead", adapterType: "paperclip_runner", createdAt: hiredAt, + agents: [{ ...turnFixture.evidence.agents[0], id: "lead", name: "CEO", adapterConfig: { model: "model" } }, + { ...turnFixture.evidence.agents[1], id: "coder", name: "Casey", role: "engineer", reportsTo: "lead", adapterType: "paperclip_runner", createdAt: hiredAt, adapterConfig: { model: "model" }, runtimeConfig: { aiConnection: binding } }], - connectionId: "account", binding, tasks, runs, first, firstAfterReuse: { ...first }, second: document("second", "REUSEHIREfixture", reuseValues), - readRuns: [{ runId: "lead-1", agentId: "lead", events: HIRING_TEMPLATE_READ_FILES.flatMap(commandEvents) }] }; + connectionId: "account", binding, tasks, runs, first, firstAfterReuse: { ...first }, second: document("second-task", "REUSEHIREfixture", reuseValues), + turnApiState: turnFixture.apiState, + readRuns: [...turnFixture.evidence.readRuns, { runId: "lead-1", agentId: "lead", events: HIRING_TEMPLATE_READ_FILES.flatMap(commandEvents) }] }; } function fails(evidence: HiringTemplateEvidence, id: string) { expect(gradeHiringTemplate(evidence).checks.find(check => check.id === id)?.passed, id).toBe(false); @@ -63,7 +66,7 @@ describe("production hiring template oracle", () => { fails({ ...e, first: { ...e.first!, createdByAgentId: "lead" } }, "initial-json-artifact"); }); - it("requires the real hired identity, account, two distinct tasks and exactly five successful turns", () => { + it("requires the real hired identity, account, two distinct tasks and exactly five required successful work turns", () => { const e = validEvidence(); fails({ ...e, agents: [...e.agents, { ...e.agents[1]!, id: "replacement" }] }, "one-coder-hire"); for (const wrong of [{ role: "qa" }, { reportsTo: "somebody" }, { adapterType: "codex_local" }, { name: "Another coder" }]) { @@ -74,10 +77,10 @@ describe("production hiring template oracle", () => { for (const patch of [{ assigneeAgentId: "lead" }, { parentId: "chat" }, { projectId: "other" }, { status: "backlog" }]) { fails({ ...e, tasks: [e.tasks[0]!, { ...e.tasks[1]!, ...patch }] }, "two-worker-tasks"); } - fails({ ...e, runs: e.runs.slice(1) }, "five-successful-turns"); - fails({ ...e, runs: [...e.runs, { ...e.runs[0]!, id: "extra" }] }, "five-successful-turns"); - fails({ ...e, runs: e.runs.map(r => r.id === "second" ? { ...r, contextSnapshot: { issueId: "second", aiConnection: { connectionId: "other" } } } : r) }, "five-successful-turns"); - fails({ ...e, runs: e.runs.map(r => r.id === "lead-3" ? { ...r, agentId: "coder" } : r) }, "five-successful-turns"); + fails({ ...e, runs: e.runs.slice(1) }, "bounded-work-and-completion-turns"); + fails({ ...e, runs: [...e.runs, { ...e.runs[0]!, id: "extra" }] }, "bounded-work-and-completion-turns"); + fails({ ...e, runs: e.runs.map(r => r.id === "worker-reuse" ? { ...r, contextSnapshot: { issueId: "second-task", aiConnection: { connectionId: "other" } } } : r) }, "bounded-work-and-completion-turns"); + fails({ ...e, runs: e.runs.map(r => r.id === "lead-status" ? { ...r, agentId: "coder" } : r) }, "bounded-work-and-completion-turns"); fails({ ...e, firstAfterReuse: { ...e.first!, latestRevisionId: "modified" } }, "original-preserved"); }); @@ -110,7 +113,7 @@ describe("production hiring template oracle", () => { it("separates successful workflow outcomes from missing or mismatched source coverage", () => { const e = validEvidence(); - for (const patch of [{ readRuns: [] }, { assignedSkills: [] }, { expectedSourceHashes: {} }, { servedSourceHashes: {} }, + for (const patch of [{ readRuns: e.readRuns.filter(run => run.runId !== "lead-1") }, { assignedSkills: [] }, { expectedSourceHashes: {} }, { servedSourceHashes: {} }, { servedSourceHashes: { ...e.servedSourceHashes, "skills/paperclip-create-agent/SKILL.md": hiringTemplateHash("other checkout") } }]) { expect(gradeHiringTemplate({ ...e, ...patch })).toMatchObject({ outcomePassed: true, comparisonStatus: "uncomparable" }); } @@ -120,7 +123,7 @@ describe("production hiring template oracle", () => { }); it("requires a completed pre-hire lead read rather than an echoed path, failed read or listing", () => { - const e = validEvidence(), good = e.readRuns[0]!; + const e = validEvidence(), good = e.readRuns.find(run => run.runId === "lead-1")!; for (const command of ["echo /workspace/.agents/skills/paperclip-create-agent/SKILL.md", "ls /workspace/.agents/skills/paperclip-create-agent/SKILL.md", "cat /workspace/.agents/skills/wrong-skill/SKILL.md", "cat $SKILL/SKILL.md", "cat /workspace/.agents/skills/paperclip-create-agent/SKILL.md > /dev/null", ...["cat --help", "cat --version", "head --help", "tail --version", "head -n 0", "tail -c 0", "sed -n ''", "sed -n '1q'", "sed -n 'q'", "sed --help", "sed -n '1,200w /tmp/other'", "cat -n"].map(prefix => `${prefix} /workspace/.agents/skills/paperclip-create-agent/SKILL.md`), @@ -167,13 +170,71 @@ describe("production hiring template oracle", () => { }); }); +describe("executable hiring lifecycle count guards", () => { + const task = { expectedRunCount: 7, minimumExpectedRunCount: 5 }; + it("grades and admits five required work turns with zero, batched or distinct completion turns", () => { + for (const count of [0, 1, 2]) { + const evidence = validEvidence(count); + const result = gradeHiringTemplate(evidence); + expect(result).toMatchObject({ outcomePassed: true, comparisonStatus: "comparable", turnAccounting: { + passed: true, counts: { requestedLeadTurns: 3, coderTurns: 2, completionTurns: count, + actualRunCount: 5 + count, costAccountingRunCount: 5 + count, unclassifiedTurns: 0 }, + } }); + expect(assertChatFlowRunCount({ suiteId: "hiring-templates", task, runs: evidence.runs, + hiringEvidence: evidence, hiringApiState: evidence.turnApiState })?.passed).toBe(true); + } + }); + it("rejects an arbitrary same-count wake in both executable paths", () => { + const evidence = validEvidence(2); + const notification = evidence.runs.find(run => run.contextSnapshot?.wakeReason === "chat_task_completed")!; + notification.contextSnapshot!.wakeReason = "heartbeat_timer"; + (evidence.turnApiState as { runs: unknown[] }).runs = structuredClone(evidence.runs); + fails(evidence, "bounded-work-and-completion-turns"); + expect(() => assertChatFlowRunCount({ suiteId: "hiring-templates", task, runs: evidence.runs, + hiringEvidence: evidence, hiringApiState: evidence.turnApiState })).toThrow(); + }); + it("rejects unrelated notification document writes in both executable paths", () => { + const evidence = validEvidence(2); + const ledger = evidence.readRuns.find(run => run.runId === "notify-1")!; + for (const event of ledger.events) { + const payload = (event.payload as Record).prpEvent.payload; + if (payload.name) payload.name = "write_document"; + if (payload.item?.name) payload.item.name = "write_document"; + } + fails(evidence, "bounded-work-and-completion-turns"); + expect(() => assertChatFlowRunCount({ suiteId: "hiring-templates", task, runs: evidence.runs, + hiringEvidence: evidence, hiringApiState: evidence.turnApiState })).toThrow(); + expect(gradeHiringTemplate(evidence).turnAccounting.actionEvidence.status).toBe("violated"); + }); + it("requires public lifecycle observations and preserves source/template coverage failures", () => { + const evidence = validEvidence(2); + fails({ ...evidence, turnApiState: undefined }, "bounded-work-and-completion-turns"); + expect(() => assertChatFlowRunCount({ suiteId: "hiring-templates", task, runs: evidence.runs, + hiringEvidence: evidence })).toThrow(); + const changedInstructions = { ...evidence.hiredInstructions!, files: { "AGENTS.md": `${coder} Changed punctuation.` } }; + const result = gradeHiringTemplate({ ...evidence, readRuns: evidence.readRuns.filter(run => run.runId !== "lead-1"), hiredInstructions: changedInstructions, + hiredInstructionsAfterReuse: structuredClone(changedInstructions) }); + expect(result).toMatchObject({ outcomePassed: true, comparisonStatus: "uncomparable", turnAccounting: { passed: true } }); + expect(result.checks.filter(check => check.dimension === "coverage" && !check.passed).map(check => check.id)) + .toEqual(["production-source-reads", "supplied-coder-instructions"]); + }); + it("keeps every non-hiring chat count guard unchanged", () => { + const runs = validEvidence(2).runs; + for (const suiteId of ["agent-chat", "agent-chat-hardening", "agent-chat-qualification", "context-integrity"]) { + expect(() => assertChatFlowRunCount({ suiteId, task: { expectedRunCount: 2 }, runs: runs.slice(0, 2) })).not.toThrow(); + expect(() => assertChatFlowRunCount({ suiteId, task: { expectedRunCount: 2 }, runs: runs.slice(0, 3) })).toThrow(); + } + }); +}); + describe("production hiring fixture wiring and source observations", () => { - it("keeps two explicit local native cells, production permissions and five-turn scope", () => { + it("keeps two explicit local native cells, production permissions and bounded five-work-turn scope", () => { const cells = runnerMatrix.filter(cell => cell.suite.id === "hiring-templates"); expect(cells.map(cell => cell.id)).toEqual(["hiring-templates.runner-codex.local.hire-coder-template-reuse", "hiring-templates.runner-acpx-claude.local.hire-coder-template-reuse"]); expect(runnerSuites.find(suite => suite.id === "hiring-templates")?.manualOnly).toBe(true); for (const cell of cells) { - expect(cell.task.expectedRunCount).toBe(5); + expect(cell.task.expectedRunCount).toBe(7); + expect(cell.task.minimumExpectedRunCount).toBe(5); expect(cell.task.attemptTimeoutMs?.local).toBe(15 * 60_000); expect(buildRunnerE2EProcessEnvironment({}, [cell]).PAPERCLIP_RUNNER_API_TOOLS_ENABLED).toBe("true"); const payload = cell.profile.buildAgent({ executionId: "fixture", workspacePath: "/workspace", environmentId: "env", environmentFixtureId: "local", @@ -222,3 +283,92 @@ describe("production hiring fixture wiring and source observations", () => { expect(() => renderHiringCoderExample("No example", "Casey", "Company", "CEO", "FIX")).toThrow(); }); }); + +describe("settled hiring observation", () => { + function observation(count = 2) { + const f = createHiringTemplateTurnFixture(count); + const apiState = { ...f.apiState, issue: { ...f.apiState.issue, title: "Hiring fixture", status: "in_review", conversationState: "waiting" }, + wakes: { events: [], truncated: false } }; + return { agents: f.evidence.agents, tasks: f.evidence.tasks, runs: f.evidence.runs, + readRuns: [], apiState, finalApiState: structuredClone(apiState) }; + } + it("retries the whole snapshot when a completion appears between ledger reads", async () => { + const racing = observation(), settled = observation(); + racing.runs = racing.runs.filter(run => run.id !== "notify-2"); + racing.apiState.runs = structuredClone(racing.runs); + let reads = 0; + const value = await waitForSettledHiringObservation(async () => ++reads === 1 ? racing : settled, + { deadlineAt: Date.now() + 1000, intervalMs: 0 }); + expect(reads).toBe(3); + expect(value.runs).toHaveLength(7); + }); + it("awaits pending completion wakes and then two stable observations", async () => { + const pending = observation(), settled = observation(); + const wakes = { events: [{ kind: "wake_request", status: "queued", finishedAt: null }], truncated: false }; + pending.apiState.wakes = pending.finalApiState.wakes = wakes as typeof pending.apiState.wakes; + let reads = 0; + await waitForSettledHiringObservation(async () => ++reads === 1 ? pending : settled, + { deadlineAt: Date.now() + 1000, intervalMs: 0 }); + expect(reads).toBe(3); + }); + it("awaits an outbox completion not yet represented by a wake or run", async () => { + const awaitingCallback = observation(0), settled = observation(1); + let reads = 0; + const value = await waitForSettledHiringObservation(async () => ++reads === 1 ? awaitingCallback : settled, + { deadlineAt: Date.now() + 1000, intervalMs: 0 }); + expect(reads).toBe(3); + expect(value.runs).toHaveLength(6); + }); + it("never admits truncated diagnostics, active runs or inconsistent observations", async () => { + for (const defect of ["truncated", "active", "inconsistent"] as const) { + const bad = observation(); + if (defect === "truncated") bad.apiState.wakes.truncated = bad.finalApiState.wakes.truncated = true; + if (defect === "active") bad.runs[0]!.status = "running"; + if (defect === "inconsistent") bad.finalApiState.comments.pop(); + await expect(waitForSettledHiringObservation(async () => bad, + { deadlineAt: Date.now() + 10, intervalMs: 0 })).rejects.toThrow(/Timed out/); + } + }); +}); + +it("requires attributable callback replies and does not ignore unresolved coalesced or unknown wakes", async () => { + const f = createHiringTemplateTurnFixture(2); + const state = { ...f.apiState, issue: { ...f.apiState.issue, title: "Fixture", status: "in_review", conversationState: "waiting" }, + wakes: { events: [] as Array<{ kind: string; status: string; runId?: string }>, truncated: false } }; + const valid = { agents: f.evidence.agents, tasks: f.evidence.tasks, runs: f.evidence.runs, + readRuns: f.evidence.readRuns, apiState: state, finalApiState: structuredClone(state) }; + for (const defect of ["reply", "coalesced", "unknown", "scheduled-retry", "duplicate-callback"] as const) { + const bad = structuredClone(valid); + if (defect === "reply") bad.apiState.comments = bad.finalApiState.comments = []; + if (defect === "coalesced") bad.apiState.wakes.events = bad.finalApiState.wakes.events = [{ kind: "wake_request", status: "coalesced", runId: "missing" }]; + if (defect === "unknown") bad.apiState.wakes.events = bad.finalApiState.wakes.events = [{ kind: "wake_request", status: "other" }]; + if (defect === "scheduled-retry") bad.runs[0]!.status = "scheduled_retry"; + if (defect === "duplicate-callback") bad.runs.push(structuredClone(bad.runs.find(run => run.id === "notify-1")!)); + await expect(waitForSettledHiringObservation(async () => bad, + { deadlineAt: Date.now() + 10, intervalMs: 0 })).rejects.toThrow(/Timed out/); + } +}); + +it("admits completed and accounted coalesced wake bookkeeping after callbacks", async () => { + const f = createHiringTemplateTurnFixture(2); + const state = { ...f.apiState, issue: { ...f.apiState.issue, title: "Fixture", status: "in_review", conversationState: "waiting" }, + wakes: { events: [{ kind: "wake_request", status: "completed", runId: "notify-1" }, + { kind: "wake_request", status: "coalesced", runId: "notify-2" }], truncated: false } }; + const observation = { agents: f.evidence.agents, tasks: f.evidence.tasks, runs: f.evidence.runs, + readRuns: f.evidence.readRuns, apiState: state, finalApiState: structuredClone(state) }; + let reads = 0; + await waitForSettledHiringObservation(async () => { reads++; return observation; }, + { deadlineAt: Date.now() + 1000, intervalMs: 0 }); + expect(reads).toBe(2); +}); + +it("classifies absent notification action identity as uncomparable separately from unchanged source coverage", () => { + const evidence = validEvidence(2); + evidence.readRuns = evidence.readRuns.filter(run => run.runId === "lead-1"); + const result = gradeHiringTemplate(evidence); + expect(result.comparisonStatus).toBe("uncomparable"); + expect(result.turnAccounting.actionEvidence.status).toBe("uncomparable"); + expect(result.checks.filter(check => check.dimension === "coverage" && !check.passed).map(check => check.id)) + .toEqual(["completion-action-attribution"]); + expect(result.checks.find(check => check.id === "production-source-reads")?.passed).toBe(true); +}); From 569c7203aa24b95440682983ce7940ba1d4247bd Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 3 Oct 2026 06:52:49 -0500 Subject: [PATCH 08/37] fix(ui): use latest issue runtime callbacks (#14863) ## Thinking Path > - Paperclip lets people manage AI agents and their work. > - The issue page uses an external-store adapter for comments. > - The adapter must keep one identity during an unrelated render. > - The adapter must also call the newest send and cancel functions. > - Passive effects update those functions too late for a synchronous runtime call. > - This pull request updates the function refs during render and adds a regression test. > - The benefit is a stable comment thread with no stale comment action. ## Linked Issues or Issue Description Refs #3678 **What happened?** The issue comment runtime can call the prior send function after a render. The callback refs do not update until passive effects run. **Expected behavior** The stable runtime adapter must call the newest function as soon as the render supplies it. **Steps to reproduce** 1. Render the issue runtime with one send function. 2. Render it again with a new send function and the same thread data. 3. Call the stable adapter before passive effects run. 4. Observe that the prior send function runs. **Paperclip version or commit** `6395cae072` **Deployment mode** Local development from source. **Agent adapter(s) involved** This is a core UI bug. It is not adapter-specific. ## What Changed - Update the latest send and cancel refs during render. - Keep the external-store adapter stable across callback-only renders. - Add a test for a runtime callback before passive effects run. ## Verification - `pnpm --filter @paperclipai/ui exec vitest run src/hooks/usePaperclipIssueRuntime.test.tsx` - `pnpm --filter @paperclipai/ui typecheck` - `pnpm check:token-gates` - `pnpm -r typecheck` - `pnpm test:run` - `pnpm build` ## Risks - Low risk. The change only updates two refs earlier in the same render. - The adapter identity and its data dependencies do not change. > This bug fix does not add or overlap with a roadmap feature. ## Model Used - OpenAI Codex with GPT-5. The exact deployed model ID and context window are not exposed. Reasoning, tool use, and local code execution were enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../hooks/usePaperclipIssueRuntime.test.tsx | 86 ++++++++++++++++++- ui/src/hooks/usePaperclipIssueRuntime.ts | 12 +-- 2 files changed, 85 insertions(+), 13 deletions(-) diff --git a/ui/src/hooks/usePaperclipIssueRuntime.test.tsx b/ui/src/hooks/usePaperclipIssueRuntime.test.tsx index 4d51c8ce248..81a38deb264 100644 --- a/ui/src/hooks/usePaperclipIssueRuntime.test.tsx +++ b/ui/src/hooks/usePaperclipIssueRuntime.test.tsx @@ -1,6 +1,6 @@ // @vitest-environment jsdom -import { act } from "react"; +import { act, useLayoutEffect } from "react"; import { createRoot } from "react-dom/client"; import { afterEach, describe, expect, it, vi } from "vitest"; import type { @@ -10,9 +10,22 @@ import type { } from "@assistant-ui/react"; import { usePaperclipIssueRuntime } from "./usePaperclipIssueRuntime"; -const { useExternalStoreRuntimeMock } = vi.hoisted(() => ({ - useExternalStoreRuntimeMock: vi.fn(() => ({ kind: "runtime" })), -})); +const { externalStoreRuntimeState, useExternalStoreRuntimeMock } = vi.hoisted( + () => { + const state: { + adapter: ExternalStoreAdapter | undefined; + } = { adapter: undefined }; + return { + externalStoreRuntimeState: state, + useExternalStoreRuntimeMock: vi.fn( + (adapter: ExternalStoreAdapter) => { + state.adapter = adapter; + return { kind: "runtime" }; + }, + ), + }; + }, +); vi.mock("@assistant-ui/react", () => ({ useExternalStoreRuntime: useExternalStoreRuntimeMock, @@ -26,6 +39,7 @@ function HookHarness({ isRunning, onSend, onCancel, + onLayout, }: { messages: readonly ThreadMessage[]; isRunning: boolean; @@ -38,6 +52,7 @@ function HookHarness({ }; }) => Promise; onCancel?: (() => Promise) | undefined; + onLayout?: ((adapter: ExternalStoreAdapter) => void) | undefined; }) { usePaperclipIssueRuntime({ messages, @@ -45,6 +60,11 @@ function HookHarness({ onSend, onCancel, }); + useLayoutEffect(() => { + if (externalStoreRuntimeState.adapter && onLayout) { + onLayout(externalStoreRuntimeState.adapter); + } + }, [onLayout]); return null; } @@ -86,6 +106,7 @@ function createAssistantMessage(id: string, text: string): ThreadMessage { describe("usePaperclipIssueRuntime", () => { afterEach(() => { useExternalStoreRuntimeMock.mockReset(); + externalStoreRuntimeState.adapter = undefined; }); it("forwards explicit upload selection separately from Markdown content", async () => { @@ -177,6 +198,63 @@ describe("usePaperclipIssueRuntime", () => { container.remove(); }); + it("uses the latest callbacks before passive effects run", async () => { + const container = document.createElement("div"); + document.body.appendChild(container); + const root = createRoot(container); + const messages: ThreadMessage[] = [createUserMessage("message-1", "hello")]; + const firstOnSend = vi.fn(async () => {}); + const secondOnSend = vi.fn(async () => {}); + const firstOnCancel = vi.fn(async () => {}); + const secondOnCancel = vi.fn(async () => {}); + + act(() => { + root.render( + , + ); + }); + + let immediateSend: Promise | undefined; + let immediateCancel: Promise | undefined; + + act(() => { + root.render( + { + immediateSend = adapter.onNew?.( + createAppendMessage("layout runtime callback"), + ); + immediateCancel = adapter.onCancel?.(); + }} + />, + ); + }); + await Promise.all([immediateSend, immediateCancel]); + + expect(firstOnSend).not.toHaveBeenCalled(); + expect(firstOnCancel).not.toHaveBeenCalled(); + expect(secondOnSend).toHaveBeenCalledWith({ + body: "layout runtime callback", + reopen: undefined, + reassignment: undefined, + }); + expect(secondOnCancel).toHaveBeenCalledOnce(); + + act(() => { + root.unmount(); + }); + container.remove(); + }); + it("rebuilds the adapter when thread data changes", () => { const container = document.createElement("div"); document.body.appendChild(container); diff --git a/ui/src/hooks/usePaperclipIssueRuntime.ts b/ui/src/hooks/usePaperclipIssueRuntime.ts index 494e3a444b1..19d6fdf66f8 100644 --- a/ui/src/hooks/usePaperclipIssueRuntime.ts +++ b/ui/src/hooks/usePaperclipIssueRuntime.ts @@ -1,4 +1,4 @@ -import { useEffect, useMemo, useRef } from "react"; +import { useMemo, useRef } from "react"; import { useExternalStoreRuntime, type ThreadMessage, @@ -52,14 +52,8 @@ export function usePaperclipIssueRuntime({ }: UsePaperclipIssueRuntimeOptions) { const onSendRef = useRef(onSend); const onCancelRef = useRef(onCancel); - - useEffect(() => { - onSendRef.current = onSend; - }, [onSend]); - - useEffect(() => { - onCancelRef.current = onCancel; - }, [onCancel]); + onSendRef.current = onSend; + onCancelRef.current = onCancel; const adapter = useMemo>( () => ({ From b17019e14d04273ebff538a85385f2659f6d9ef1 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 3 Oct 2026 12:32:42 -0500 Subject: [PATCH 09/37] fix(agents): reduce default instructions and qualify stock harnesses (#14948) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its adapters supply task context and access to Paperclip skills and tools. > - The default hire manual and shared prompts also repeat general work procedures. > - Those procedures overlap with stock provider instructions and the Paperclip skill. > - Existing E2E fixtures supply a QA manual, so they do not qualify the production default. > - This pull request reduces the generic instructions and adds real default-hire coverage. > - The benefit is less competing guidance, with inspectable evidence for preserved skills and task context. ## Linked Issues or Issue Description Refs: #14920. That merged change preserves native Codex base instructions. This PR covers the default manual, shared legacy prompts, operational skill guidance, and the narrowly approved ACP skill-discovery/session-environment repair for measured delivery and credential-persistence failures. **What existing behavior does this improve?** New non-CEO hires without a custom bundle and legacy task/chat startup and continuation prompts. **Current behavior** The shipped default manual contains 602 words. Generic task/chat prompts and ordinary resume deltas repeat work procedures already available through the harness and Paperclip skill. **Proposed behavior** The default manual contains only the eight-word company identity. Shared startup prompts retain identity and connection guidance. Ordinary resume deltas retain current work context without the generic execution contract. **Reason and benefit** Let the stock harness guide general work. Keep Paperclip-specific capabilities and independently test default hires, skills, ordered comments, and chat restart. **Breaking changes** New default hires receive less guidance. Existing saved manuals, explicit custom bundles, CEO templates, and specialized wake contracts retain their behavior. The obsolete includeExecutionContract option remains accepted for source compatibility. ## What Changed - Reduce the default hire manual to one sentence. - Reduce shared task/chat defaults and remove the generic ordinary-resume contract. - Keep connection guidance, auth, skills, custom prompts, and specialized wake context. - Add credential-free instruction-boundary gates and 26 explicit Product E2E cells across eight legacy/native profiles, including two focused Paperclip-storage cases. - Capture public hire receipts before providers run, then grade delivered prompts and independent task/chat outcomes. - Add an early legacy skill API recipe for saving a task document, checking the saved revision receipt and linking the document. Improve stock task/heartbeat skill-selection metadata and show a clickable Markdown UI-link example. Keep native tool completion separate. - Advertise bounded routing descriptions and exact successfully staged SKILL.md paths in legacy ACP Claude; keep full bodies on demand and preserve remote path rebasing. - Remove only the provider environment from copied persisted ACP session records, while loading current run credentials and preserving all other options/conversation state. - Regenerate both capability metadata inventories and reject stale manifests/inventories before provider admission. - Publish the original reduction and focused skill-repair comparisons, preserving all failures, automatic recovery, cost coverage and limitations. ## Verification **Behavioral qualification remains pending.** Original legacy ACP Claude loses the issue document only in the reduced cohort beneath an unchanged credential failure. A source-backed diagnosis finds that neither ordinary assignment reads the staged operational skill, while the runtime persists provider environment in session state. The new common repairs expose skill metadata/path and omit persisted env; strict document and credential guards stay intact. [Inspectable diagnosis and retained hashes](https://github.com/paperclipai/paperclip/blob/9f654db4541d3d002769c988f6e51fc0b08dadbd/doc/plans/2026-10-03-legacy-acp-claude-readiness.md). Current repair head `de0965984ff3edf611ae6d0e7ca5c7d5ae3947bb` incorporates master `569c7203aa24b95440682983ce7940ba1d4247bd` (merged #14961/#15007). All 222 affected adapter tests, adapter-utils/E2E typechecks, and final 96 variant/grader/retry calibrations pass. The frozen historical comparator is `c25697f4260b6f3adfea143c3ae9932e2f42986d`: 8,280 of 8,291 paths identical, exactly two production instruction paths plus nine declared unit expectations differ. The operational skill/discovery/environment repairs, selected model/profile/task/core grader/auth/permissions/retry policy are identical. Both actual launcher prepare→verify admissions pass with zero providers. [Immutable manifest and exact receipts](https://github.com/paperclipai/paperclip/blob/9f654db4541d3d002769c988f6e51fc0b08dadbd/doc/plans/2026-10-03-legacy-acp-claude-evidence/manifest.json). One original legacy ACP Claude cell per variant is authorized, with enforced single campaign attempts, 12-minute deadlines and company/agent 1,000-cent hard stops; every product recovery run/cost is counted. Actual live outcomes are pending. Current normal CI has one failed server shard and failed aggregate verify under diagnosis; other normal gates including typecheck/build/Rust/all eight browser shards pass. Fresh review completed successfully; the valid historical startup/resume masking finding was fixed with per-invocation task/chat checks and strict complete-snapshot capture, calibrated and resolved. Prior heads, failures and campaigns below remain historical evidence, not checks on this repair head. - Prior head `36aa4d81c49a1a8f6f04b1a068fae19aa901955f` is replayed on merged hiring master `862a5758ba0e88a33232c1f1fa645e85c38a3113`. All 52 current-head checks pass with two intentional Storybook skips, including repository typecheck/test/build and the browser shard. Fresh Greptile is 5/5 with zero unresolved review threads. Exact-head stock prerequisites pass 599 assertions (598 TypeScript + 1 Rust), all six gates and retained receipt verification, zero providers/source errors. Fingerprint `a7f5a22d860a88fe20cce213c6d5e0004788f32c8363930729aea4fd740ad16d`. Combined catalog/hiring calibrations pass 67 assertions, E2E typecheck and 26-cell stock discovery pass. Canonical contract/inventory checks and the later issue-derived reference calibration are retained; that reference-only follow-up is not live-qualified by earlier frozen runs. - Prior full repository typecheck/build passed. The complete local Vitest run executed 14,956 tests: 14,870 passed, 83 skipped, three timing failures. All three affected files passed unchanged narrow reruns; original failures remain retained. Current-head CI now passes the full general checks; the original local failures remain retained. - The original 24-pair default-manual/shared-prompt comparison has two new overall classic Claude/OpenCode document-delivery failures plus an additional legacy ACP Claude document loss beneath an unchanged credential-guard failure (not closed by later runs), two newly passing OpenCode ordered cases, seven unchanged failures and 13 unchanged passes. Equal 15/24 totals do not establish behavioral equivalence. [Complete original report](https://github.com/paperclipai/paperclip/blob/875f4c397d9e8c3f12f39dedd59abaf1eaf5236e/doc/plans/2026-10-02-stock-harness-live-comparison.md). - The skill-only repair holds the eight-word manual/shared prompts and merged #14920 fixed. All four matched profile configurations and 203 fixture/behavior files match. Candidate `abd0b628ca642c09a54a4edc56a5227402f6686e` varies only the two skill sources against baseline `bc83fe030234439ac51279502a28803958963e2e`. [Candidate workflow](https://github.com/paperclipai/paperclip/actions/runs/37060885547) and [baseline workflow](https://github.com/paperclipai/paperclip/actions/runs/37060888047) each pass 571 exact-source prerequisites before providers; all eight cells clean up successfully. Failed campaigns publish successfully and remain failed. - Repair pairs: Claude original Fail → Pass; Claude explicit Pass → Pass; both OpenCode cases Fail → Fail. Explicit OpenCode's handoff worsens beneath the unchanged failing UI-link grade: baseline gives a clickable API URL, candidate gives a code-formatted path without an anchor. The request's usable-link wording is narrower in the UI-only oracle. [Complete repair report and safe projection](https://github.com/paperclipai/paperclip/blob/875f4c397d9e8c3f12f39dedd59abaf1eaf5236e/doc/plans/2026-10-02-legacy-document-skill-repair.md). - The subsequent narrow stock metadata/link correction has two matched Pass → Pass cases, zero new machine failures/passes and no pending pairs. Both original-case handoff links remain deficient: candidate uses a wrong PAP prefix, baseline supplies a bare prefix-less slug path; the preserved original oracle only requires a durable document. Both explicit clickable UI-link cases pass revision/content/link grading. All four exact-source 587-check gates, single assignment runs and cleanup pass. This does not establish fix causality because baseline also succeeds. [Candidate workflow](https://github.com/paperclipai/paperclip/actions/runs/37069547401) freezes `fe9dc1e3c518825242ed889ab9c8352986f8c2ed`; [matched baseline](https://github.com/paperclipai/paperclip/actions/runs/37069552374) freezes `0d7ecfa96d72fba79b7f0a25052b42c0686c0488`. This is a skill-only comparison with reduced manuals/shared prompts held constant, not a repeat of the historical-manual comparison. Only original and clarified explicit classic OpenCode cases are selected, two per variant/four expected turns. 8,242 other tracked files and both profile hashes match; protected workflows admit each exact source before credentials. [Complete qualification report](https://github.com/paperclipai/paperclip/blob/74d0d3d945f4c52d0814b5a845ab5bd09f33cd6b/doc/plans/2026-10-02-opencode-skill-routing-link-qualification.md). Candidate original loads Paperclip/reference before saving publicly; baseline original loads it after writing locally, then saves publicly within the same assignment. Reported cost totals are $0.0107824490 candidate / $0.0107909015 baseline, with unmetered runtime. The later reference-only issue-derived link correction is provider-free calibrated and **not live-qualified** by these frozen runs; no further paid runs. - Retained tool calls show the repaired original OpenCode assignment loads only its assigned output skill before writing locally. Operational Paperclip is first loaded during automatic disposition recovery; its early recipe is visible then, but it never saves the missing document. Explicit candidate loads Paperclip and reads the new reference before saving successfully. All nine actual runs are counted. Reported LLM totals are $0.3802537209 baseline and $0.4918990161 candidate; local runtime is unmetered. - Initial setup, packaging, cancelled/missing-cell recovery, callback test and relative-output attempts remain retained. No completed provider failure was rerun. Frozen measurement branches are unchanged by later canonical metadata maintenance. - Run `pnpm test:e2e:runner:stock-harness`, `pnpm test:e2e:runner:unit`, and `pnpm test:e2e:runner:typecheck`. Select `stock-harness` explicitly for paid execution; it is excluded from `--all`. Prior-head integration: `36aa4d81c49a1a8f6f04b1a068fae19aa901955f` replays this PR on merged hiring #14985 (`862a5758ba0e88a33232c1f1fa645e85c38a3113`), preserving the four explicit custom-CEO-bundle checks, minimal generic manual boundary, and both suites. The combined fixture catalog and hiring calibrations pass 67 assertions; exact-head stock prerequisites pass 599 assertions (598 TypeScript + 1 Rust), all six gates and retained-receipt verification, zero providers/source errors, fingerprint `a7f5a22d860a88fe20cce213c6d5e0004788f32c8363930729aea4fd740ad16d`. E2E typecheck and 26-cell stock discovery pass. Fresh current-head CI passes all 52 checks with two intentional skips, and fresh Greptile is 5/5 with zero unresolved review threads. The prior source-plan browser failure is retained: a deterministic process fixture replayed its last `fixture:plan` command on `chat_task_completed`, writing revision 2 with identical body after the approval handoff. This was not paid provider execution. Rebased current-head CI passes the same assertion without an old-head retry or a change to that browser fixture. The merged hiring change was measured separately on immutable matched unions, with this reduced/shared/operational context and native completion guidance held constant. [Complete original two-profile report](https://github.com/paperclipai/paperclip/blob/f0512647656be78e48abd8c22a3078db8bf6bcd2/doc/plans/2026-10-02-hiring-template-live-comparison.md): [candidate](https://github.com/paperclipai/paperclip/actions/runs/37075466208) / [historical baseline](https://github.com/paperclipai/paperclip/actions/runs/37075469463), 705 provider-free prerequisites each. Both pairs are unchanged Fail → Fail on the exact-five count, with six core delivery checks passing all four cells; 28 actual successful runs include eight automatic completion wakes, zero retries, four successful cleanups. Source-read coverage is uncomparable, actual model charges unknown. Separately versioned provider-free accounting remains analytical work; original verdicts are preserved. This does not rerun or qualify the completed default-manual or native campaigns. ## Risks - Legacy ACP Claude's additional delivery loss is not closed by any later matched run and blocks the no-extra-failing-behavior merge criterion. Legacy document delivery may have relied on the prior manual/shared prompts. The early skill repair improves Claude in one trial; the later OpenCode pairs pass in both variants and cannot establish causality or robust recovery. Both original-case links remain deficient beneath the storage-only grade. The later issue-derived reference correction has only provider-free validation. Native finish/block descriptions must not be supplied to legacy agents. - The comparison holds merged native Codex fix #14920 constant; it cannot measure that fix's before/after task performance. - These bounded skill/context/chat workflows do not measure general coding quality. Unrepresented providers remain unqualified. - Saved manuals and old Codex sessions are not automatically migrated. Codex through ACP still has a separate base-instruction follow-up. ## Model Used OpenAI Codex, GPT-6 family as identified by this session. The exact deployment ID and context-window size are not exposed. The assistant used reasoning, repository tools, code execution, and delegated PR/eval work. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (relevant suites and all three unchanged narrow reruns pass; complete-run timing failures retained in Verification) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green on the new repair head (prior-head checks retained above) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups on the new repair head - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/evals.md | 8 + ...26-10-02-legacy-document-skill-repair.json | 1187 ++++++++++ ...2026-10-02-legacy-document-skill-repair.md | 51 + ...code-skill-routing-link-qualification.json | 1418 ++++++++++++ ...encode-skill-routing-link-qualification.md | 75 + ...6-10-02-stock-harness-live-comparison.json | 357 ++++ ...026-10-02-stock-harness-live-comparison.md | 110 + ...10-02-stock-harness-paperclip-checklist.md | 308 ++- package.json | 1 + .../ephemeral-session-environment.test.ts | 165 ++ .../ephemeral-session-environment.ts | 32 + .../src/acpx-engine/execute.test.ts | 104 +- .../adapter-utils/src/acpx-engine/execute.ts | 39 +- .../adapter-utils/src/prompt-sections.test.ts | 13 +- .../adapter-utils/src/server-utils.test.ts | 134 +- packages/adapter-utils/src/server-utils.ts | 57 +- .../cursor-cloud/src/server/execute.test.ts | 5 +- .../src/server/prompt-rendering.test.ts | 6 +- .../opencode-local/src/server/execute.test.ts | 5 +- .../src/server/execute.remote.test.ts | 12 +- .../docs/capability-contract.md | 177 +- .../generated/capability/capabilities.yaml | 143 +- .../capability/capability-contract.md | 4 +- .../check-capability-inventory.test.mjs | 2 +- .../generate-capability-contract.d.mts | 1 + .../scripts/generate-capability-contract.mjs | 2 +- .../scripts/lib/capability-inventory.mjs | 12 +- .../spec/capability/capabilities.yaml | 1904 +++++++++-------- .../spec/capability/source-contract.json | 1 + .../src/generated/capability-contract.ts | 4 +- .../test/capability-contract.test.mjs | 15 +- .../src/__tests__/agent-skills-routes.test.ts | 32 +- .../src/__tests__/codex-local-execute.test.ts | 3 +- .../src/__tests__/workspace-runtime.test.ts | 35 +- .../src/onboarding-assets/default/AGENTS.md | 25 +- server/src/routes/agents.ts | 4 +- .../services/onboarding-first-task-assets.ts | 4 +- skills/paperclip/SKILL.md | 10 +- .../paperclip/references/issue-documents.md | 58 + tests/runner-e2e/FIXTURES.md | 11 + tests/runner-e2e/README.md | 8 + tests/runner-e2e/STOCK-HARNESS.md | 248 +++ tests/runner-e2e/catalog.test.ts | 6 +- tests/runner-e2e/catalog.ts | 28 +- tests/runner-e2e/context-integrity-cases.ts | 24 +- tests/runner-e2e/context-integrity-flow.ts | 16 +- tests/runner-e2e/context-integrity-scoring.ts | 31 +- .../historical-default-agents.md | 24 + tests/runner-e2e/launch.ts | 25 +- tests/runner-e2e/live-fixtures.test.ts | 24 + tests/runner-e2e/live-fixtures.ts | 5 +- .../native-completion-defaults.test.ts | 10 +- tests/runner-e2e/paperclip-document.test.ts | 89 + tests/runner-e2e/runner.spec.ts | 36 + .../runner-e2e/select-rerun-artifacts.test.ts | 13 + .../stock-harness-admission.test.ts | 44 + tests/runner-e2e/stock-harness-admission.ts | 38 + tests/runner-e2e/stock-harness-checks.mjs | 246 +++ .../runner-e2e/stock-harness-checks.test.mjs | 79 + tests/runner-e2e/stock-harness-digest.test.ts | 27 + .../stock-harness-instruction-variant.d.mts | 7 + .../stock-harness-instruction-variant.mjs | 24 + ...stock-harness-instruction-variant.test.mjs | 48 + .../runner-e2e/stock-harness-manifest.test.ts | 21 + tests/runner-e2e/stock-harness-manifest.ts | 19 + tests/runner-e2e/stock-harness.test.ts | 189 ++ tests/runner-e2e/stock-harness.ts | 184 ++ tests/runner-e2e/vitest.config.ts | 2 +- 68 files changed, 6635 insertions(+), 1414 deletions(-) create mode 100644 doc/plans/2026-10-02-legacy-document-skill-repair.json create mode 100644 doc/plans/2026-10-02-legacy-document-skill-repair.md create mode 100644 doc/plans/2026-10-02-opencode-skill-routing-link-qualification.json create mode 100644 doc/plans/2026-10-02-opencode-skill-routing-link-qualification.md create mode 100644 doc/plans/2026-10-02-stock-harness-live-comparison.json create mode 100644 doc/plans/2026-10-02-stock-harness-live-comparison.md create mode 100644 packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.test.ts create mode 100644 packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.ts create mode 100644 packages/paperclip-runner/scripts/generate-capability-contract.d.mts create mode 100644 skills/paperclip/references/issue-documents.md create mode 100644 tests/runner-e2e/STOCK-HARNESS.md create mode 100644 tests/runner-e2e/fixtures/stock-harness/historical-default-agents.md create mode 100644 tests/runner-e2e/paperclip-document.test.ts create mode 100644 tests/runner-e2e/stock-harness-admission.test.ts create mode 100644 tests/runner-e2e/stock-harness-admission.ts create mode 100644 tests/runner-e2e/stock-harness-checks.mjs create mode 100644 tests/runner-e2e/stock-harness-checks.test.mjs create mode 100644 tests/runner-e2e/stock-harness-digest.test.ts create mode 100644 tests/runner-e2e/stock-harness-instruction-variant.d.mts create mode 100644 tests/runner-e2e/stock-harness-instruction-variant.mjs create mode 100644 tests/runner-e2e/stock-harness-instruction-variant.test.mjs create mode 100644 tests/runner-e2e/stock-harness-manifest.test.ts create mode 100644 tests/runner-e2e/stock-harness-manifest.ts create mode 100644 tests/runner-e2e/stock-harness.test.ts create mode 100644 tests/runner-e2e/stock-harness.ts diff --git a/doc/evals.md b/doc/evals.md index f936899203b..c529972e76d 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -95,6 +95,14 @@ execution ID because `--all` excludes explicit-only suites. Each cell applies a 1,000-cent company and agent budget hard stop before task creation and records both limits in its evidence. +The explicit-only [stock-harness suite](../tests/runner-e2e/STOCK-HARNESS.md) +reuses skill, ordered-continuation, and chat-restart journeys across eight local +legacy/native profiles with production-default hires. It closes the custom QA +manual coverage gap. Its required credential-free prerequisite maps vendor +instruction layering, the tiny hire bundle, and shared startup/resume reductions +to executable checks. The 24 live cells are configured; no live qualification is +claimed from their setup or unit calibration. + The explicit-only `agent-chat-stories` suite covers the experimental settings lifecycle for a configured native agent and follow-ups during active work. Its fixture-driven file wait and persisted-plan oracle are documented in the diff --git a/doc/plans/2026-10-02-legacy-document-skill-repair.json b/doc/plans/2026-10-02-legacy-document-skill-repair.json new file mode 100644 index 00000000000..b215bc08b82 --- /dev/null +++ b/doc/plans/2026-10-02-legacy-document-skill-repair.json @@ -0,0 +1,1187 @@ +{ + "schema": "paperclip.legacy-document-skill-repair.safe-comparison/v1", + "asOf": "2026-10-02T21:22:00Z", + "partial": false, + "expectedCellsPerVariant": 4, + "expectedProviderTurnsPerVariant": 4, + "candidateSha": "abd0b628ca642c09a54a4edc56a5227402f6686e", + "baselineSha": "bc83fe030234439ac51279502a28803958963e2e", + "originalMachinePairedDelta": { + "newFailures": 0, + "newPasses": 1, + "unchangedPasses": 1, + "unchangedFailures": 2, + "pending": 0 + }, + "observedBehavioralDeteriorations": [ + "OpenCode explicit case: candidate lacks a clickable link, while baseline provided a clickable API document URL. Both fail original UI-only oracle; equal machine verdicts hide this presentation deterioration." + ], + "baseline": [ + { + "executionId": "stock-harness.legacy-claude.local.assigned-skill-explicit-invocation", + "profileId": "legacy-claude", + "caseId": "assigned-skill-explicit-invocation", + "status": "failed", + "model": "claude-sonnet-4-6", + "durationMs": 34592, + "cleanup": "passed", + "billing": { + "llm": { + "runCount": 1, + "runsWithTokenUsage": 1, + "runsWithReportedCost": 1, + "inputTokens": 28439, + "outputTokens": 1137, + "cachedInputTokens": 143829, + "totalTokens": 173405, + "reportedCostUsd": 0.16683944999999997, + "costStatus": "reported" + }, + "runtime": { + "provider": "local", + "agentRunDurationMs": 23032, + "leaseDurationMs": null, + "leaseCount": 0, + "costStatus": "not_metered", + "costSource": "local_not_metered" + }, + "reportedCostUsd": 0.16683944999999997, + "estimatedRuntimeCostUsd": 0, + "observedAndEstimatedCostUsd": 0.16683944999999997, + "complete": true + }, + "sourceSha": "bc83fe030234439ac51279502a28803958963e2e", + "providerDurationMs": 23032, + "resultSha256": "67bcbe152852280b58fec97754601c57c416c2cb7965de55557240b8448dca63", + "contextEvidenceSha256": "9bc59db9678a3f5cfc207d1aeeec1f868f94429a7a10d8d214621f286532a1b7", + "failedBehaviorChecks": [ + "single-durable-output" + ], + "publicDocumentCount": 0, + "savedRevisionEvidence": false, + "runPurposes": [ + "assignment" + ], + "prerequisites": { + "passed": true, + "sourceSha": "bc83fe030234439ac51279502a28803958963e2e", + "sourceFingerprint": "27fbbc1792dca136c9c72dd492ff16c18bce0638ce4979985d798cde85175201", + "providerCallsBeforeAdmission": 0, + "executedChecks": 571 + } + }, + { + "executionId": "stock-harness.legacy-claude.local.assigned-skill-paperclip-document", + "profileId": "legacy-claude", + "caseId": "assigned-skill-paperclip-document", + "status": "passed", + "model": "claude-sonnet-4-6", + "durationMs": 50778, + "cleanup": "passed", + "billing": { + "llm": { + "runCount": 1, + "runsWithTokenUsage": 1, + "runsWithReportedCost": 1, + "inputTokens": 27675, + "outputTokens": 2062, + "cachedInputTokens": 216852, + "totalTokens": 246589, + "reportedCostUsd": 0.19975709999999997, + "costStatus": "reported" + }, + "runtime": { + "provider": "local", + "agentRunDurationMs": 34999, + "leaseDurationMs": null, + "leaseCount": 0, + "costStatus": "not_metered", + "costSource": "local_not_metered" + }, + "reportedCostUsd": 0.19975709999999997, + "estimatedRuntimeCostUsd": 0, + "observedAndEstimatedCostUsd": 0.19975709999999997, + "complete": true + }, + "sourceSha": "bc83fe030234439ac51279502a28803958963e2e", + "providerDurationMs": 34999, + "resultSha256": "bca45a760fc5ec30ae8f154d9ec52ab4073b3ce2375256b8718f91200cb79d4e", + "contextEvidenceSha256": "473f3b417e71ee823ae9aad2596c7f1379b3fd5cf8d8fd540c30a4f3a2e10e21", + "failedBehaviorChecks": [], + "publicDocumentCount": 1, + "savedRevisionEvidence": true, + "runPurposes": [ + "assignment" + ], + "prerequisites": { + "passed": true, + "sourceSha": "bc83fe030234439ac51279502a28803958963e2e", + "sourceFingerprint": "27fbbc1792dca136c9c72dd492ff16c18bce0638ce4979985d798cde85175201", + "providerCallsBeforeAdmission": 0, + "executedChecks": 571 + } + }, + { + "executionId": "stock-harness.legacy-opencode.local.assigned-skill-explicit-invocation", + "profileId": "legacy-opencode", + "caseId": "assigned-skill-explicit-invocation", + "status": "failed", + "model": "openrouter/deepseek/deepseek-v4-flash-0731", + "durationMs": 47625, + "cleanup": "passed", + "billing": { + "llm": { + "runCount": 1, + "runsWithTokenUsage": 1, + "runsWithReportedCost": 1, + "inputTokens": 24068, + "outputTokens": 1960, + "cachedInputTokens": 149248, + "totalTokens": 175276, + "reportedCostUsd": 0.0033927116000000003, + "costStatus": "reported" + }, + "runtime": { + "provider": "local", + "agentRunDurationMs": 36770, + "leaseDurationMs": null, + "leaseCount": 0, + "costStatus": "not_metered", + "costSource": "local_not_metered" + }, + "reportedCostUsd": 0.0033927116000000003, + "estimatedRuntimeCostUsd": 0, + "observedAndEstimatedCostUsd": 0.0033927116000000003, + "complete": true + }, + "sourceSha": "bc83fe030234439ac51279502a28803958963e2e", + "providerDurationMs": 36770, + "resultSha256": "a322e292fd7688c714cc8e57e7f64ca70c19f0784dc242b3e809a6e4b0be224e", + "contextEvidenceSha256": "3e295ec42b9c7208a7e4b17bf8329d9e1f01ba984c4a586539d7a703ffd59bf6", + "failedBehaviorChecks": [ + "single-durable-output" + ], + "publicDocumentCount": 0, + "savedRevisionEvidence": false, + "runPurposes": [ + "assignment" + ], + "prerequisites": { + "passed": true, + "sourceSha": "bc83fe030234439ac51279502a28803958963e2e", + "sourceFingerprint": "27fbbc1792dca136c9c72dd492ff16c18bce0638ce4979985d798cde85175201", + "providerCallsBeforeAdmission": 0, + "executedChecks": 571 + } + }, + { + "executionId": "stock-harness.legacy-opencode.local.assigned-skill-paperclip-document", + "profileId": "legacy-opencode", + "caseId": "assigned-skill-paperclip-document", + "status": "failed", + "model": "openrouter/deepseek/deepseek-v4-flash-0731", + "durationMs": 217620, + "cleanup": "passed", + "billing": { + "llm": { + "runCount": 1, + "runsWithTokenUsage": 1, + "runsWithReportedCost": 1, + "inputTokens": 78795, + "outputTokens": 4331, + "cachedInputTokens": 846848, + "totalTokens": 929974, + "reportedCostUsd": 0.0102644593, + "costStatus": "reported" + }, + "runtime": { + "provider": "local", + "agentRunDurationMs": 206191, + "leaseDurationMs": null, + "leaseCount": 0, + "costStatus": "not_metered", + "costSource": "local_not_metered" + }, + "reportedCostUsd": 0.0102644593, + "estimatedRuntimeCostUsd": 0, + "observedAndEstimatedCostUsd": 0.0102644593, + "complete": true + }, + "sourceSha": "bc83fe030234439ac51279502a28803958963e2e", + "providerDurationMs": 206191, + "resultSha256": "60dfb068ea80d0155d93fdace77a29fd19d64bdb0740e6e6b63f7e83ccfb5ded", + "contextEvidenceSha256": "23eed2766b5e8db9cea0ba5c76da235cf2d4222fa5718420cc60d6e13c86fa06", + "failedBehaviorChecks": [ + "saved-document-link" + ], + "publicDocumentCount": 1, + "savedRevisionEvidence": true, + "runPurposes": [ + "assignment" + ], + "prerequisites": { + "passed": true, + "sourceSha": "bc83fe030234439ac51279502a28803958963e2e", + "sourceFingerprint": "27fbbc1792dca136c9c72dd492ff16c18bce0638ce4979985d798cde85175201", + "providerCallsBeforeAdmission": 0, + "executedChecks": 571 + }, + "observedHandoff": "clickable API document URL; rejected by UI-only link oracle" + } + ], + "candidate": [ + { + "executionId": "stock-harness.legacy-claude.local.assigned-skill-explicit-invocation", + "profileId": "legacy-claude", + "caseId": "assigned-skill-explicit-invocation", + "status": "passed", + "model": "claude-sonnet-4-6", + "durationMs": 46702, + "cleanup": "passed", + "billing": { + "llm": { + "runCount": 1, + "runsWithTokenUsage": 1, + "runsWithReportedCost": 1, + "inputTokens": 40330, + "outputTokens": 1336, + "cachedInputTokens": 185090, + "totalTokens": 226756, + "reportedCostUsd": 0.22679475, + "costStatus": "reported" + }, + "runtime": { + "provider": "local", + "agentRunDurationMs": 27702, + "leaseDurationMs": null, + "leaseCount": 0, + "costStatus": "not_metered", + "costSource": "local_not_metered" + }, + "reportedCostUsd": 0.22679475, + "estimatedRuntimeCostUsd": 0, + "observedAndEstimatedCostUsd": 0.22679475, + "complete": true + }, + "sourceSha": "abd0b628ca642c09a54a4edc56a5227402f6686e", + "providerDurationMs": 27702, + "resultSha256": "2f66f62dc2f16756400a0f5b0b0d55e5c9f5f7cb2590d84ba05f3d4ee6b4ad66", + "contextEvidenceSha256": "c5619df489213873db64d35424b08f9852b0099bab07e92d3a71f993008de389", + "failedBehaviorChecks": [], + "publicDocumentCount": 1, + "savedRevisionEvidence": true, + "runPurposes": [ + "assignment" + ], + "prerequisites": { + "passed": true, + "sourceSha": "abd0b628ca642c09a54a4edc56a5227402f6686e", + "sourceFingerprint": "441999ae2618f1f1e9f430c099bbef63d2948495599f0d8b342d8e5f027a7f4c", + "providerCallsBeforeAdmission": 0, + "executedChecks": 571 + } + }, + { + "executionId": "stock-harness.legacy-claude.local.assigned-skill-paperclip-document", + "profileId": "legacy-claude", + "caseId": "assigned-skill-paperclip-document", + "status": "passed", + "model": "claude-sonnet-4-6", + "durationMs": 56928, + "cleanup": "passed", + "billing": { + "llm": { + "runCount": 1, + "runsWithTokenUsage": 1, + "runsWithReportedCost": 1, + "inputTokens": 42017, + "outputTokens": 2468, + "cachedInputTokens": 205226, + "totalTokens": 249711, + "reportedCostUsd": 0.25614254999999997, + "costStatus": "reported" + }, + "runtime": { + "provider": "local", + "agentRunDurationMs": 40513, + "leaseDurationMs": null, + "leaseCount": 0, + "costStatus": "not_metered", + "costSource": "local_not_metered" + }, + "reportedCostUsd": 0.25614254999999997, + "estimatedRuntimeCostUsd": 0, + "observedAndEstimatedCostUsd": 0.25614254999999997, + "complete": true + }, + "sourceSha": "abd0b628ca642c09a54a4edc56a5227402f6686e", + "providerDurationMs": 40513, + "resultSha256": "8a0ef693477a03925d8166d1872b5db4f5547ae48eb9df934beabcce9eccf147", + "contextEvidenceSha256": "f7aba5c9653567a0c3ee252d01fbcee20c81c432b4402d76c7e00221b6917f2e", + "failedBehaviorChecks": [], + "publicDocumentCount": 1, + "savedRevisionEvidence": true, + "runPurposes": [ + "assignment" + ], + "prerequisites": { + "passed": true, + "sourceSha": "abd0b628ca642c09a54a4edc56a5227402f6686e", + "sourceFingerprint": "441999ae2618f1f1e9f430c099bbef63d2948495599f0d8b342d8e5f027a7f4c", + "providerCallsBeforeAdmission": 0, + "executedChecks": 571 + } + }, + { + "executionId": "stock-harness.legacy-opencode.local.assigned-skill-explicit-invocation", + "profileId": "legacy-opencode", + "caseId": "assigned-skill-explicit-invocation", + "status": "failed", + "model": "openrouter/deepseek/deepseek-v4-flash-0731", + "durationMs": 79578, + "cleanup": "passed", + "billing": { + "llm": { + "runCount": 2, + "runsWithTokenUsage": 2, + "runsWithReportedCost": 2, + "inputTokens": 54946, + "outputTokens": 2264, + "cachedInputTokens": 169472, + "totalTokens": 226682, + "reportedCostUsd": 0.0047148437000000005, + "costStatus": "reported" + }, + "runtime": { + "provider": "local", + "agentRunDurationMs": 68032, + "leaseDurationMs": null, + "leaseCount": 0, + "costStatus": "not_metered", + "costSource": "local_not_metered" + }, + "reportedCostUsd": 0.0047148437000000005, + "estimatedRuntimeCostUsd": 0, + "observedAndEstimatedCostUsd": 0.0047148437000000005, + "complete": true + }, + "sourceSha": "abd0b628ca642c09a54a4edc56a5227402f6686e", + "providerDurationMs": 68032, + "resultSha256": "f807c235250d8675d29aa34fc97d0e89be2c7ea885edfc2f3845627037552c0b", + "contextEvidenceSha256": "0dc98ad8ef1f22d69c3a3d7344082a6d5a0effc8d9c73c5d4c1267c92800b57a", + "failedBehaviorChecks": [ + "single-durable-output" + ], + "publicDocumentCount": 0, + "savedRevisionEvidence": false, + "runPurposes": [ + "assignment", + "issue_disposition_repair" + ], + "prerequisites": { + "passed": true, + "sourceSha": "abd0b628ca642c09a54a4edc56a5227402f6686e", + "sourceFingerprint": "441999ae2618f1f1e9f430c099bbef63d2948495599f0d8b342d8e5f027a7f4c", + "providerCallsBeforeAdmission": 0, + "executedChecks": 571 + } + }, + { + "executionId": "stock-harness.legacy-opencode.local.assigned-skill-paperclip-document", + "profileId": "legacy-opencode", + "caseId": "assigned-skill-paperclip-document", + "status": "failed", + "model": "openrouter/deepseek/deepseek-v4-flash-0731", + "durationMs": 59941, + "cleanup": "passed", + "billing": { + "llm": { + "runCount": 1, + "runsWithTokenUsage": 1, + "runsWithReportedCost": 1, + "inputTokens": 33724, + "outputTokens": 2597, + "cachedInputTokens": 147200, + "totalTokens": 183521, + "reportedCostUsd": 0.0042468724, + "costStatus": "reported" + }, + "runtime": { + "provider": "local", + "agentRunDurationMs": 46300, + "leaseDurationMs": null, + "leaseCount": 0, + "costStatus": "not_metered", + "costSource": "local_not_metered" + }, + "reportedCostUsd": 0.0042468724, + "estimatedRuntimeCostUsd": 0, + "observedAndEstimatedCostUsd": 0.0042468724, + "complete": true + }, + "sourceSha": "abd0b628ca642c09a54a4edc56a5227402f6686e", + "providerDurationMs": 46300, + "resultSha256": "cff6e5bf4b6b646ae6f9b5a6a2934c444c37a14f50b591f5226ccac0e4de1dda", + "contextEvidenceSha256": "6c943d78291c1e493cb29ca3a52dcb96d186fd18ed51e4cfb17055649162f6f2", + "failedBehaviorChecks": [ + "saved-document-link" + ], + "publicDocumentCount": 1, + "savedRevisionEvidence": true, + "runPurposes": [ + "assignment" + ], + "prerequisites": { + "passed": true, + "sourceSha": "abd0b628ca642c09a54a4edc56a5227402f6686e", + "sourceFingerprint": "441999ae2618f1f1e9f430c099bbef63d2948495599f0d8b342d8e5f027a7f4c", + "providerCallsBeforeAdmission": 0, + "executedChecks": 571 + }, + "observedHandoff": "code-formatted prefix-less UI document path; no clickable anchor" + } + ], + "matchedProfileProof": { + "schema": "paperclip.legacy-document-skill-repair.fixture-match-proof/v1", + "cells": [ + { + "executionId": "stock-harness.legacy-claude.local.assigned-skill-explicit-invocation", + "fixtureConfigSha256": "6c21be9bb3fcec6274372fc586d88e42d82bc0d438e26a69ef614f8e6fcb8519", + "model": "claude-sonnet-4-6" + }, + { + "executionId": "stock-harness.legacy-claude.local.assigned-skill-paperclip-document", + "fixtureConfigSha256": "ea1cd67b0f68ef3830c011768d1a1133b5828c602d5ae3e021889dcd72708344", + "model": "claude-sonnet-4-6" + }, + { + "executionId": "stock-harness.legacy-opencode.local.assigned-skill-explicit-invocation", + "fixtureConfigSha256": "dd673f469cab583d26bc2e4b484f18b27884ec14a59542177026389181e2da2c", + "model": "openrouter/deepseek/deepseek-v4-flash-0731" + }, + { + "executionId": "stock-harness.legacy-opencode.local.assigned-skill-paperclip-document", + "fixtureConfigSha256": "b5173358d664ab6b3daa67d8758f18d59ec99cb9c485c02ccf953a9f8375e01e", + "model": "openrouter/deepseek/deepseek-v4-flash-0731" + } + ] + }, + "identicalSourceCount": 203, + "skillSources": [ + { + "path": "skills/paperclip/SKILL.md", + "candidate": { + "present": true, + "sha256": "65e379cab319bfef5911c0f6c9423cb262c9b8f1fef3fd08d45c1e4f32f8971a" + }, + "baseline": { + "present": true, + "sha256": "77aafc0c7b16a82254af642f0ebff723dcef7d7fbb6d3354efc0c11032e0344d" + } + }, + { + "path": "skills/paperclip/references/issue-documents.md", + "candidate": { + "present": true, + "sha256": "353845a406b1b4948487c348c86eb7b8c89a55ae67afe203100eee10aee46336" + }, + "baseline": { + "present": false, + "sha256": null + } + } + ], + "toolDeliveryAudit": { + "schema": "paperclip.legacy-document-repair.tool-audit/v1", + "rows": [ + { + "variant": "candidate", + "case": "assigned-skill-paperclip-document", + "sourceSha": "abd0b628ca642c09a54a4edc56a5227402f6686e", + "runOrdinal": 1, + "runPurpose": "assignment", + "startedAt": "2026-10-02T20:37:40.215Z", + "contextEvidenceSha256": "6c943d78291c1e493cb29ca3a52dcb96d186fd18ed51e4cfb17055649162f6f2", + "toolCalls": [ + { + "tool": "skill", + "skill": "context-integrity-output-7b55f897b3b41", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "skill", + "skill": "paperclip", + "operationalSkillLoad": true, + "earlyRecipeVisible": true, + "newReferencePointerVisible": true, + "newReferenceRead": false, + "outputTruncated": true, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "read", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": true, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": true, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": true + } + ] + }, + { + "variant": "candidate", + "case": "assigned-skill-explicit-invocation", + "sourceSha": "abd0b628ca642c09a54a4edc56a5227402f6686e", + "runOrdinal": 1, + "runPurpose": "assignment", + "startedAt": "2026-10-02T20:37:37.587Z", + "contextEvidenceSha256": "0dc98ad8ef1f22d69c3a3d7344082a6d5a0effc8d9c73c5d4c1267c92800b57a", + "toolCalls": [ + { + "tool": "skill", + "skill": "context-integrity-output-60e03959432b1", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "write", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": true, + "documentApiCall": false, + "statusApiCall": false + } + ] + }, + { + "variant": "candidate", + "case": "assigned-skill-explicit-invocation", + "sourceSha": "abd0b628ca642c09a54a4edc56a5227402f6686e", + "runOrdinal": 2, + "runPurpose": "issue_disposition_repair", + "startedAt": "2026-10-02T20:37:52.592Z", + "contextEvidenceSha256": "0dc98ad8ef1f22d69c3a3d7344082a6d5a0effc8d9c73c5d4c1267c92800b57a", + "toolCalls": [ + { + "tool": "skill", + "skill": "paperclip", + "operationalSkillLoad": true, + "earlyRecipeVisible": true, + "newReferencePointerVisible": true, + "newReferenceRead": false, + "outputTruncated": true, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": true + } + ] + }, + { + "variant": "baseline", + "case": "assigned-skill-explicit-invocation", + "sourceSha": "bc83fe030234439ac51279502a28803958963e2e", + "runOrdinal": 1, + "runPurpose": "assignment", + "startedAt": "2026-10-02T20:37:57.110Z", + "contextEvidenceSha256": "3e295ec42b9c7208a7e4b17bf8329d9e1f01ba984c4a586539d7a703ffd59bf6", + "toolCalls": [ + { + "tool": "skill", + "skill": "context-integrity-output-816e6b4cdd6a1", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "write", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": true, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "skill", + "skill": "paperclip", + "operationalSkillLoad": true, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": true, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": true + }, + { + "tool": "write", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": true + } + ] + }, + { + "variant": "baseline", + "case": "assigned-skill-paperclip-document", + "sourceSha": "bc83fe030234439ac51279502a28803958963e2e", + "runOrdinal": 1, + "runPurpose": "assignment", + "startedAt": "2026-10-02T20:38:02.031Z", + "contextEvidenceSha256": "23eed2766b5e8db9cea0ba5c76da235cf2d4222fa5718420cc60d6e13c86fa06", + "toolCalls": [ + { + "tool": "skill", + "skill": "context-integrity-output-06ce9804653e1", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "skill", + "skill": "paperclip", + "operationalSkillLoad": true, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": true, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "read", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": true, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "read", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": true, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "read", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": true, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": true, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false + }, + { + "tool": "bash", + "skill": null, + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": true, + "statusApiCall": true + } + ] + } + ] + }, + "limitations": [ + "Tiny manual/shared prompts and merged native Codex base fix remain fixed; only the two operational skill sources differ.", + "Original task-document wording is storage-ambiguous; explicit case requests a usable link but its oracle requires the canonical UI route, so the baseline API-link failure has an oracle/request ambiguity.", + "Provider-free source review confirms a board-readable GET document endpoint and an unprefixed UI redirect preserving the hash; actual authenticated browser navigation in the cleaned-up measured instance was not replayed.", + "One paired trial cannot establish broad performance, causality, timing or spending improvement.", + "Candidate OpenCode original has an additional automatic disposition-repair run; all nine actual runs and costs are retained.", + "Local runtime is unmetered; reported LLM cost is not a statement of actual external billing.", + "Subsequent generated capability metadata and pre-provider drift checks are derived maintenance after these measured frozen revisions, not a new provider qualification." + ] +} diff --git a/doc/plans/2026-10-02-legacy-document-skill-repair.md b/doc/plans/2026-10-02-legacy-document-skill-repair.md new file mode 100644 index 00000000000..728e0b7c489 --- /dev/null +++ b/doc/plans/2026-10-02-legacy-document-skill-repair.md @@ -0,0 +1,51 @@ +# Legacy document skill repair qualification — 2026-10-02 + +**TL;DR: one newly passing case (Claude original document delivery), one unchanged pass (Claude explicit storage), two unchanged OpenCode failures, and no pending pairs or new overall machine failures. OpenCode's explicit-case clickable handoff nevertheless got worse:** baseline supplied a clickable API document URL, while candidate supplied only a code-formatted path. Both fail the original UI-only link oracle, whose intended route was narrower than the request's "usable link" wording. This result does not establish general non-regression. + +This focused repair keeps the eight-word manual, reduced shared prompts and merged Codex base fix #14920 fixed. It varies only the early operational Paperclip skill recipe and the presence of the new `issue-documents.md` reference. The earlier combined prompt-removal comparison retains its two new classic Claude/OpenCode storage failures; this trial repairs Claude's original case but leaves OpenCode's original regression unresolved. + +| Classic profile | Case | Pre-fix skill → repaired skill | Retained evidence | +| --- | --- | --- | --- | +| Claude | Original assigned skill | Fail → Pass | Baseline lacks a durable task document; candidate saves one containing the skill marker. | +| Claude | Explicit Paperclip document | Pass → Pass | Both save the document/revision and provide a canonical UI link. | +| OpenCode | Original assigned skill | Fail → Fail | Both write a workspace file rather than a public task document. Candidate requires automatic disposition recovery. | +| OpenCode | Explicit Paperclip document | Fail → Fail | Both save the document/revision. Baseline provides a clickable API URL; candidate gives a code-formatted UI path without an anchor. Original UI-link checks fail in both. | + +Original machine verdicts and evidence hashes are retained in the [safe evidence projection](2026-10-02-legacy-document-skill-repair.json). No model was rerun to improve these results. + +## Frozen sources and public reports + +- [Repaired skill campaign](https://github.com/paperclipai/paperclip/actions/runs/37060885547), source `abd0b628ca642c09a54a4edc56a5227402f6686e`, branch `codex/legacy-document-repaired-skill`. [Published candidate report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37060885547-1/index.html). +- [Pre-fix skill campaign](https://github.com/paperclipai/paperclip/actions/runs/37060888047), source `bc83fe030234439ac51279502a28803958963e2e`, branch `codex/legacy-document-pre-fix-skill`. [Published baseline report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37060888047-1/index.html). +- Both trusted default-branch workflows ran concurrently on separate concurrency targets. All eight cells pass their exact-source prerequisite gate before providers; all eight clean up successfully. Failed campaigns are published successfully and remain failed. +- The four model/effort/auth/tool/permission/environment configurations match. Models are `claude-sonnet-4-6` and `openrouter/deepseek/deepseek-v4-flash-0731`. The comparison records 203 byte-identical fixture/behavior sources and explicit present/absent skill-source fingerprints. The new reference is absent from the baseline, rather than silently introduced into it. +- Local admission preparation passed 909 E2E support tests, E2E typecheck and four-cell discovery; the candidate exact-source gate passed 571 checks, including one Rust check. Initial sandbox and obsolete catalog-count attempts remain retained. The original request and behavioral oracle were not weakened. + +## What OpenCode actually received + +The repaired original assignment invokes only its assigned Context integrity output skill, then writes `task-document.md`. There is no operational Paperclip skill load or issue-document reference read before that local write. A separate automatic `issue_disposition_repair` run then loads Paperclip. Its returned content contains the new early saved-document paragraph and reference pointer even though the overall skill result is marked truncated; it never reads the reference or saves a document, and only finalizes the issue's status. + +This is evidence of operational-skill selection after work, not evidence that the assignment saw and ignored the new recipe. The pre-fix original run also writes locally before loading the old Paperclip skill. Whether the adapter should activate operational Paperclip guidance before work is a production-delivery question for review; expanding a skill the first assignment never opened would not address the observed sequence. + +The explicit candidate loads the assigned skill, then Paperclip, receives the new early paragraph, reads `issue-documents.md`, and calls the document API successfully. Its saved document has a persisted revision. The remaining defect is its comment's non-clickable path. Baseline supplied an actual Markdown link to the issue-document API GET endpoint. Read-only source review confirms that this endpoint returns the document in board context, and the unprefixed UI issue route redirects through the selected company's prefix while preserving the document hash. Actual authenticated browser navigation against these cleaned-up measured instances was not replayed, so neither URL is classified as a proven broken route. Candidate's missing clickable anchor is directly observable. + +A future explicit fixture should name a clickable Paperclip UI document link if that is the intended contract. This clarification would not retroactively pass or fail either original result. A short canonical Markdown UI-link example can be considered separately from operational-skill delivery. No further production instruction change or paid rerun was made for this diagnosis. + +## Runs, timing and cost + +Eight provider turns were expected; nine actual runs are retained. Candidate OpenCode's original assignment succeeds but leaves the task `in_progress`, triggering automatic disposition recovery. The recovery succeeds and marks it done, without creating the missing document. Both runs, their time and cost are counted rather than selecting a better attempt. + +| Profile | Case | Baseline provider seconds | Candidate provider seconds | Baseline reported USD | Candidate reported USD | +| --- | --- | ---: | ---: | ---: | ---: | +| legacy-claude | Original | 23.032 | 27.702 | 0.1668394500 | 0.2267947500 | +| legacy-claude | Explicit storage | 34.999 | 40.513 | 0.1997571000 | 0.2561425500 | +| legacy-opencode | Original | 36.770 | 68.032 | 0.0033927116 | 0.0047148437 | +| legacy-opencode | Explicit storage | 206.191 | 46.300 | 0.0102644593 | 0.0042468724 | + +Reported LLM totals are $0.3802537209 baseline and $0.4918990161 candidate. All four baseline and five candidate ledger rows contain token usage and reported cost. Local runtime is unmetered and actual external billing is not established. Single trials, the additional recovery and baseline OpenCode's long explicit-case execution prevent a general timing or spending conclusion. + +## PR maintenance and remaining qualifications + +The skill insertion shifted generated capability heading anchors. General PR CI detected stale `capabilities.yaml`; the paid workflow's narrower build and every exact-source admission still passed before provider access. Both canonical capability metadata inventories are regenerated after measurement, include the new reference where declared, and adds a credential-free admission test that rejects stale/missing manifests. These maintenance changes do not modify the frozen measured branches or the skill bytes used in this comparison. + +Draft [PR #14948](https://github.com/paperclipai/paperclip/pull/14948) keeps this repair and the tiny manual; draft [PR #14961](https://github.com/paperclipai/paperclip/pull/14961) measures native completion documentation separately. OpenCode original delivery remains unresolved, and existing Claude chat-memory and legacy ACP credential/receipt findings remain separate. No broad matrix rerun was launched, and merging remains user-controlled. diff --git a/doc/plans/2026-10-02-opencode-skill-routing-link-qualification.json b/doc/plans/2026-10-02-opencode-skill-routing-link-qualification.json new file mode 100644 index 00000000000..f909e31edb8 --- /dev/null +++ b/doc/plans/2026-10-02-opencode-skill-routing-link-qualification.json @@ -0,0 +1,1418 @@ +{ + "schema": "paperclip.opencode-skill-routing-link-comparison/v1", + "status": "complete", + "candidateSha": "fe9dc1e3c518825242ed889ab9c8352986f8c2ed", + "baselineSha": "0d7ecfa96d72fba79b7f0a25052b42c0686c0488", + "pairedSummary": { + "newFailures": 0, + "newPasses": 0, + "unchangedPasses": 2, + "unchangedFailures": 0, + "pendingPairs": 0, + "limitation": "Both original cases have deficient handoff links outside their machine oracle; candidate uses incorrect PAP prefix, baseline has bare unprefixed slug path." + }, + "sourceProof": { + "identicalTrackedFiles": 8242, + "changedTrackedPaths": [ + "skills/paperclip/SKILL.md", + "skills/paperclip/references/issue-documents.md", + "tests/runner-e2e/opencode-skill-routing-comparison.json" + ], + "onlyProductionDifferences": [ + { + "path": "skills/paperclip/SKILL.md", + "candidateSha256": "6ca593effe18704f8014ac4e8155f7513c0701aa3c61d3cf0b96d3366c92f0e1", + "baselineSha256": "768522e081c1ec43dee4a5794539f7b597ee3708c634d05e388c874df283cc53" + }, + { + "path": "skills/paperclip/references/issue-documents.md", + "candidateSha256": "a6141e8ba4510b683418197d58598d5a14db2641c49c869ae8aca4e2b05aca97", + "baselineSha256": "353845a406b1b4948487c348c86eb7b8c89a55ae67afe203100eee10aee46336" + } + ], + "baselineReceipt": { + "schema": "paperclip.opencode-skill-routing.baseline/v1", + "candidateSha": "fe9dc1e3c518825242ed889ab9c8352986f8c2ed", + "priorSkillSha": "b85668c4065d66e08026218a6bd48a3d22ddd4c9", + "expectedCellsPerVariant": 2, + "expectedProviderTurnsPerVariant": 2, + "heldConstant": [ + "tiny manual/shared prompts", + "native Codex additive base fix", + "future explicit clickable UI-link fixture and calibrated grader", + "models/effort/auth/permissions/tools/budgets", + "all other tracked repository source bytes" + ], + "onlyChangedProductionSources": [ + { + "path": "skills/paperclip/SKILL.md", + "candidateSha256": "6ca593effe18704f8014ac4e8155f7513c0701aa3c61d3cf0b96d3366c92f0e1", + "baselineSha256": "768522e081c1ec43dee4a5794539f7b597ee3708c634d05e388c874df283cc53" + }, + { + "path": "skills/paperclip/references/issue-documents.md", + "candidateSha256": "a6141e8ba4510b683418197d58598d5a14db2641c49c869ae8aca4e2b05aca97", + "baselineSha256": "353845a406b1b4948487c348c86eb7b8c89a55ae67afe203100eee10aee46336" + } + ] + }, + "fixtureConfigurations": { + "schema": "paperclip.opencode-skill-routing.fixture-match-proof/v1", + "cells": [ + { + "executionId": "stock-harness.legacy-opencode.local.assigned-skill-explicit-invocation", + "fixtureConfigSha256": "dd673f469cab583d26bc2e4b484f18b27884ec14a59542177026389181e2da2c", + "model": "openrouter/deepseek/deepseek-v4-flash-0731" + }, + { + "executionId": "stock-harness.legacy-opencode.local.assigned-skill-paperclip-document", + "fixtureConfigSha256": "4924cbbcb8768fdbff70d9f53c20835dba28626278da77fb5edcc09c1700fd5f", + "model": "openrouter/deepseek/deepseek-v4-flash-0731" + } + ] + } + }, + "cells": [ + { + "variant": "baseline", + "caseId": "assigned-skill-explicit-invocation", + "source": { + "sha": "0d7ecfa96d72fba79b7f0a25052b42c0686c0488", + "ref": "refs/heads/codex/opencode-pre-routing-skill", + "workflowRunUrl": "https://github.com/paperclipai/paperclip/actions/runs/37069552374" + }, + "status": "passed", + "attempt": 1, + "suiteDefinitionHash": "18a8eb87f11ebaa4d5be89f878de669bc32eee6f58a47c3f19d15bb879519dd8", + "model": "openrouter/deepseek/deepseek-v4-flash-0731", + "runtimeMode": "legacy", + "durationMs": 46285, + "providerDurationMs": 25870, + "runCount": 1, + "recoveryRuns": 0, + "cleanup": "passed", + "matcherResults": [ + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.assigned-skill-present", + "expected": true + }, + "passed": true, + "detail": "The task run must receive one pinned skill version through the public assignment state." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.skill-request-explicit", + "expected": true + }, + "passed": true, + "detail": "The task request must explicitly name the assigned skill with a supported slash or dollar reference." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.skill-source-marker", + "expected": true + }, + "passed": true, + "detail": "The assigned pinned skill source must contain the output marker." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.marker-not-in-request", + "expected": true + }, + "passed": true, + "detail": "The output marker must originate from the assigned skill, not the task request or comments." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.single-durable-output", + "expected": true + }, + "passed": true, + "detail": "Exactly one durable task document must contain the skill's marker." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.completed-task", + "expected": true + }, + "passed": true, + "detail": "Final task status: done." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.successful-runs", + "expected": true + }, + "passed": true, + "detail": "All recorded context-integrity runs must succeed." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.budget-hard-stops", + "expected": true + }, + "passed": true, + "detail": "Public company and agent records must both retain the 1,000-cent monthly hard stops." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.default-hire-bundle", + "expected": true + }, + "passed": true, + "detail": "The public managed bundle must contain only the eight-word shipped identity, without an injected QA manual." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.provider-runs-present", + "expected": true + }, + "passed": true, + "detail": "The lifecycle oracle must reach actual provider runs; missing runs cannot pass instruction delivery." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.invocation-evidence-complete", + "expected": true + }, + "passed": true, + "detail": "Every legacy run must have a public adapter.invoke event with its actual nonempty prompt." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.generic-procedures-absent", + "expected": true + }, + "passed": true, + "detail": "Neither startup nor continuation may reintroduce the removed generic manual." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.fresh-default-delivered", + "expected": true + }, + "passed": true, + "detail": "At least one fresh invocation must carry default identity and connection guidance." + } + ], + "billing": { + "llm": { + "runCount": 1, + "runsWithTokenUsage": 1, + "runsWithReportedCost": 1, + "inputTokens": 34011, + "outputTokens": 2533, + "cachedInputTokens": 192512, + "totalTokens": 229056, + "reportedCostUsd": 0.0058472545, + "costStatus": "reported" + }, + "runtime": { + "provider": "local", + "agentRunDurationMs": 25870, + "leaseDurationMs": null, + "leaseCount": 0, + "costStatus": "not_metered", + "costSource": "local_not_metered" + }, + "reportedCostUsd": 0.0058472545, + "estimatedRuntimeCostUsd": 0, + "observedAndEstimatedCostUsd": 0.0058472545, + "complete": true + }, + "preflight": { + "sourceSha": "0d7ecfa96d72fba79b7f0a25052b42c0686c0488", + "sourceFingerprint": "0315602ac6788cf0d65561a73cc85d790768ef90c6815257077a2381f961b96e", + "passed": true, + "providerCalls": 0, + "assertions": 587 + }, + "delivery": { + "documentCount": 1, + "documentKey": "task-document", + "savedRevisionNumber": 1, + "savedRevisionPresent": true, + "markerPresent": true, + "markdownLinkTargets": [], + "expectedDocumentUiTarget": "/RUN/issues/RUN-1#document-task-document", + "hasExactDocumentUiLink": false, + "originalCaseRequiresLink": false, + "originalBaselineBarePath": true, + "originalCandidateWrongPrefix": false + }, + "operationalSkillReadOrder": [ + { + "tool": "skill", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 1, + "skill": "assigned-output" + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 2, + "skill": null + }, + { + "tool": "write", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": true, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 3, + "skill": null + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 4, + "skill": null + }, + { + "tool": "skill", + "operationalSkillLoad": true, + "earlyRecipeVisible": true, + "newReferencePointerVisible": true, + "newReferenceRead": false, + "outputTruncated": true, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 5, + "skill": "paperclip" + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 6, + "skill": null + }, + { + "tool": "read", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": true, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 7, + "skill": null + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": true, + "statusApiCall": false, + "ordinal": 8, + "skill": null + }, + { + "tool": "read", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 9, + "skill": null + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 10, + "skill": null + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": true, + "ordinal": 11, + "skill": null + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 12, + "skill": null + } + ], + "evidenceHashes": { + "result": "a4953654cba1d6bd3911ad3cfb000d8bfb21041ef3e3ee409e2c19d14af30f1e", + "context": "c3fb84a16e6fe1dead74ffece90035bc99aa247ffa65b4665d5288acbe8a5515", + "preflight": "a0756b19a5c20c1bf600a9abeb1b62307c99f37ffc2ff02063709a870bbe2986", + "finalStateScreenshot": "49068a59e35b2fb0207bd81ca36d4db639d17c8639c0ce885ac5869fc83320e9" + }, + "screenshotInspected": true + }, + { + "variant": "baseline", + "caseId": "assigned-skill-paperclip-document", + "source": { + "sha": "0d7ecfa96d72fba79b7f0a25052b42c0686c0488", + "ref": "refs/heads/codex/opencode-pre-routing-skill", + "workflowRunUrl": "https://github.com/paperclipai/paperclip/actions/runs/37069552374" + }, + "status": "passed", + "attempt": 1, + "suiteDefinitionHash": "18a8eb87f11ebaa4d5be89f878de669bc32eee6f58a47c3f19d15bb879519dd8", + "model": "openrouter/deepseek/deepseek-v4-flash-0731", + "runtimeMode": "legacy", + "durationMs": 94804, + "providerDurationMs": 73984, + "runCount": 1, + "recoveryRuns": 0, + "cleanup": "passed", + "matcherResults": [ + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.assigned-skill-present", + "expected": true + }, + "passed": true, + "detail": "The task run must receive one pinned skill version through the public assignment state." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.skill-request-explicit", + "expected": true + }, + "passed": true, + "detail": "The task request must explicitly name the assigned skill with a supported slash or dollar reference." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.skill-source-marker", + "expected": true + }, + "passed": true, + "detail": "The assigned pinned skill source must contain the output marker." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.marker-not-in-request", + "expected": true + }, + "passed": true, + "detail": "The output marker must originate from the assigned skill, not the task request or comments." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.single-durable-output", + "expected": true + }, + "passed": true, + "detail": "Exactly one durable task document must contain the skill's marker." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.saved-document-revision", + "expected": true + }, + "passed": true, + "detail": "The public saved document must have a persisted revision and the requested content." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.saved-document-link", + "expected": true + }, + "passed": true, + "detail": "An agent completion comment must link the exact saved document on this task; a local path or claimed URL does not suffice." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.completed-task", + "expected": true + }, + "passed": true, + "detail": "Final task status: done." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.successful-runs", + "expected": true + }, + "passed": true, + "detail": "All recorded context-integrity runs must succeed." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.budget-hard-stops", + "expected": true + }, + "passed": true, + "detail": "Public company and agent records must both retain the 1,000-cent monthly hard stops." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.default-hire-bundle", + "expected": true + }, + "passed": true, + "detail": "The public managed bundle must contain only the eight-word shipped identity, without an injected QA manual." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.provider-runs-present", + "expected": true + }, + "passed": true, + "detail": "The lifecycle oracle must reach actual provider runs; missing runs cannot pass instruction delivery." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.invocation-evidence-complete", + "expected": true + }, + "passed": true, + "detail": "Every legacy run must have a public adapter.invoke event with its actual nonempty prompt." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.generic-procedures-absent", + "expected": true + }, + "passed": true, + "detail": "Neither startup nor continuation may reintroduce the removed generic manual." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.fresh-default-delivered", + "expected": true + }, + "passed": true, + "detail": "At least one fresh invocation must carry default identity and connection guidance." + } + ], + "billing": { + "llm": { + "runCount": 1, + "runsWithTokenUsage": 1, + "runsWithReportedCost": 1, + "inputTokens": 62834, + "outputTokens": 2127, + "cachedInputTokens": 130304, + "totalTokens": 195265, + "reportedCostUsd": 0.004943647, + "costStatus": "reported" + }, + "runtime": { + "provider": "local", + "agentRunDurationMs": 73984, + "leaseDurationMs": null, + "leaseCount": 0, + "costStatus": "not_metered", + "costSource": "local_not_metered" + }, + "reportedCostUsd": 0.004943647, + "estimatedRuntimeCostUsd": 0, + "observedAndEstimatedCostUsd": 0.004943647, + "complete": true + }, + "preflight": { + "sourceSha": "0d7ecfa96d72fba79b7f0a25052b42c0686c0488", + "sourceFingerprint": "0315602ac6788cf0d65561a73cc85d790768ef90c6815257077a2381f961b96e", + "passed": true, + "providerCalls": 0, + "assertions": 587 + }, + "delivery": { + "documentCount": 1, + "documentKey": "context-integrity-output", + "savedRevisionNumber": 1, + "savedRevisionPresent": true, + "markerPresent": true, + "markdownLinkTargets": [ + "/RUN/issues/RUN-1#document-context-integrity-output" + ], + "expectedDocumentUiTarget": "/RUN/issues/RUN-1#document-context-integrity-output", + "hasExactDocumentUiLink": true, + "originalCaseRequiresLink": true, + "originalBaselineBarePath": false, + "originalCandidateWrongPrefix": false + }, + "operationalSkillReadOrder": [ + { + "tool": "skill", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 1, + "skill": "assigned-output" + }, + { + "tool": "skill", + "operationalSkillLoad": true, + "earlyRecipeVisible": true, + "newReferencePointerVisible": true, + "newReferenceRead": false, + "outputTruncated": true, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 2, + "skill": "paperclip" + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 3, + "skill": null + }, + { + "tool": "read", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": true, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 4, + "skill": null + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 5, + "skill": null + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 6, + "skill": null + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": true, + "statusApiCall": false, + "ordinal": 7, + "skill": null + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": true, + "ordinal": 8, + "skill": null + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 9, + "skill": null + } + ], + "evidenceHashes": { + "result": "b09f82a96976fdcd5ddfb1655d783db5076a19fde0a5d751ff3df995cd5fac59", + "context": "2c630300b0f5737c74a9921d25ed67d2ab2612957f843d9709ebfd57fb84306f", + "preflight": "9e6e55c1ca71312f7bb53708bd2f9727e5395c9da8586b285d29e1fee4fe50fe", + "finalStateScreenshot": "128e6e0c7be8b8ca0e3e90601bce8c3d33df8c503636d09ffe514df03881173b" + }, + "screenshotInspected": true + }, + { + "variant": "candidate", + "caseId": "assigned-skill-explicit-invocation", + "source": { + "sha": "fe9dc1e3c518825242ed889ab9c8352986f8c2ed", + "ref": "refs/heads/codex/opencode-stock-routing-qualified", + "workflowRunUrl": "https://github.com/paperclipai/paperclip/actions/runs/37069547401" + }, + "status": "passed", + "attempt": 1, + "suiteDefinitionHash": "62cfca8303c66bf5e29b0665c72b7bb717acf9f96d7a3ef768407afa5bc9870f", + "model": "openrouter/deepseek/deepseek-v4-flash-0731", + "runtimeMode": "legacy", + "durationMs": 89455, + "providerDurationMs": 67897, + "runCount": 1, + "recoveryRuns": 0, + "cleanup": "passed", + "matcherResults": [ + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.assigned-skill-present", + "expected": true + }, + "passed": true, + "detail": "The task run must receive one pinned skill version through the public assignment state." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.skill-request-explicit", + "expected": true + }, + "passed": true, + "detail": "The task request must explicitly name the assigned skill with a supported slash or dollar reference." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.skill-source-marker", + "expected": true + }, + "passed": true, + "detail": "The assigned pinned skill source must contain the output marker." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.marker-not-in-request", + "expected": true + }, + "passed": true, + "detail": "The output marker must originate from the assigned skill, not the task request or comments." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.single-durable-output", + "expected": true + }, + "passed": true, + "detail": "Exactly one durable task document must contain the skill's marker." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.completed-task", + "expected": true + }, + "passed": true, + "detail": "Final task status: done." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.successful-runs", + "expected": true + }, + "passed": true, + "detail": "All recorded context-integrity runs must succeed." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.budget-hard-stops", + "expected": true + }, + "passed": true, + "detail": "Public company and agent records must both retain the 1,000-cent monthly hard stops." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.default-hire-bundle", + "expected": true + }, + "passed": true, + "detail": "The public managed bundle must contain only the eight-word shipped identity, without an injected QA manual." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.provider-runs-present", + "expected": true + }, + "passed": true, + "detail": "The lifecycle oracle must reach actual provider runs; missing runs cannot pass instruction delivery." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.invocation-evidence-complete", + "expected": true + }, + "passed": true, + "detail": "Every legacy run must have a public adapter.invoke event with its actual nonempty prompt." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.generic-procedures-absent", + "expected": true + }, + "passed": true, + "detail": "Neither startup nor continuation may reintroduce the removed generic manual." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.fresh-default-delivered", + "expected": true + }, + "passed": true, + "detail": "At least one fresh invocation must carry default identity and connection guidance." + } + ], + "billing": { + "llm": { + "runCount": 1, + "runsWithTokenUsage": 1, + "runsWithReportedCost": 1, + "inputTokens": 71068, + "outputTokens": 1716, + "cachedInputTokens": 97792, + "totalTokens": 170576, + "reportedCostUsd": 0.00413837, + "costStatus": "reported" + }, + "runtime": { + "provider": "local", + "agentRunDurationMs": 67897, + "leaseDurationMs": null, + "leaseCount": 0, + "costStatus": "not_metered", + "costSource": "local_not_metered" + }, + "reportedCostUsd": 0.00413837, + "estimatedRuntimeCostUsd": 0, + "observedAndEstimatedCostUsd": 0.00413837, + "complete": true + }, + "preflight": { + "sourceSha": "fe9dc1e3c518825242ed889ab9c8352986f8c2ed", + "sourceFingerprint": "aeb792549146cc29eda779e83ae22d336276e31162f738a1dde85a82e7833902", + "passed": true, + "providerCalls": 0, + "assertions": 587 + }, + "delivery": { + "documentCount": 1, + "documentKey": "context-integrity-output", + "savedRevisionNumber": 1, + "savedRevisionPresent": true, + "markerPresent": true, + "markdownLinkTargets": [ + "/PAP/issues/RUN-1#document-context-integrity-output" + ], + "expectedDocumentUiTarget": "/RUN/issues/RUN-1#document-context-integrity-output", + "hasExactDocumentUiLink": false, + "originalCaseRequiresLink": false, + "originalBaselineBarePath": false, + "originalCandidateWrongPrefix": true + }, + "operationalSkillReadOrder": [ + { + "tool": "skill", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 1, + "skill": "assigned-output" + }, + { + "tool": "skill", + "operationalSkillLoad": true, + "earlyRecipeVisible": true, + "newReferencePointerVisible": true, + "newReferenceRead": false, + "outputTruncated": true, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 2, + "skill": "paperclip" + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 3, + "skill": null + }, + { + "tool": "read", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": true, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 4, + "skill": null + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": true, + "statusApiCall": false, + "ordinal": 5, + "skill": null + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 6, + "skill": null + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 7, + "skill": null + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": true, + "ordinal": 8, + "skill": null + } + ], + "evidenceHashes": { + "result": "7f13c7f84dbbd662d5689b82ef02d5e9806dc43237604599c183ba807097f403", + "context": "16aa44354be2763149fe4c98d5907230025f909ec2d7eafa93767959c183f6a1", + "preflight": "cd5c451ec7c01ac145b8f20a11facfe4410b6c6ea621358b0d5bfcaa339260cf", + "finalStateScreenshot": "123a366f1d9372bf5249e9a46d16dc322538f7a0c2669504f02a29d97b77cf16" + }, + "screenshotInspected": true + }, + { + "variant": "candidate", + "caseId": "assigned-skill-paperclip-document", + "source": { + "sha": "fe9dc1e3c518825242ed889ab9c8352986f8c2ed", + "ref": "refs/heads/codex/opencode-stock-routing-qualified", + "workflowRunUrl": "https://github.com/paperclipai/paperclip/actions/runs/37069547401" + }, + "status": "passed", + "attempt": 1, + "suiteDefinitionHash": "62cfca8303c66bf5e29b0665c72b7bb717acf9f96d7a3ef768407afa5bc9870f", + "model": "openrouter/deepseek/deepseek-v4-flash-0731", + "runtimeMode": "legacy", + "durationMs": 102901, + "providerDurationMs": 82751, + "runCount": 1, + "recoveryRuns": 0, + "cleanup": "passed", + "matcherResults": [ + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.assigned-skill-present", + "expected": true + }, + "passed": true, + "detail": "The task run must receive one pinned skill version through the public assignment state." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.skill-request-explicit", + "expected": true + }, + "passed": true, + "detail": "The task request must explicitly name the assigned skill with a supported slash or dollar reference." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.skill-source-marker", + "expected": true + }, + "passed": true, + "detail": "The assigned pinned skill source must contain the output marker." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.marker-not-in-request", + "expected": true + }, + "passed": true, + "detail": "The output marker must originate from the assigned skill, not the task request or comments." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.single-durable-output", + "expected": true + }, + "passed": true, + "detail": "Exactly one durable task document must contain the skill's marker." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.saved-document-revision", + "expected": true + }, + "passed": true, + "detail": "The public saved document must have a persisted revision and the requested content." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.saved-document-link", + "expected": true + }, + "passed": true, + "detail": "An agent completion comment must link the exact saved document on this task; a local path or claimed URL does not suffice." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.completed-task", + "expected": true + }, + "passed": true, + "detail": "Final task status: done." + }, + { + "matcher": { + "kind": "json_path", + "path": "contextIntegrity.successful-runs", + "expected": true + }, + "passed": true, + "detail": "All recorded context-integrity runs must succeed." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.budget-hard-stops", + "expected": true + }, + "passed": true, + "detail": "Public company and agent records must both retain the 1,000-cent monthly hard stops." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.default-hire-bundle", + "expected": true + }, + "passed": true, + "detail": "The public managed bundle must contain only the eight-word shipped identity, without an injected QA manual." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.provider-runs-present", + "expected": true + }, + "passed": true, + "detail": "The lifecycle oracle must reach actual provider runs; missing runs cannot pass instruction delivery." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.invocation-evidence-complete", + "expected": true + }, + "passed": true, + "detail": "Every legacy run must have a public adapter.invoke event with its actual nonempty prompt." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.generic-procedures-absent", + "expected": true + }, + "passed": true, + "detail": "Neither startup nor continuation may reintroduce the removed generic manual." + }, + { + "matcher": { + "kind": "json_path", + "path": "stockHarness.fresh-default-delivered", + "expected": true + }, + "passed": true, + "detail": "At least one fresh invocation must carry default identity and connection guidance." + } + ], + "billing": { + "llm": { + "runCount": 1, + "runsWithTokenUsage": 1, + "runsWithReportedCost": 1, + "inputTokens": 92370, + "outputTokens": 2691, + "cachedInputTokens": 185856, + "totalTokens": 280917, + "reportedCostUsd": 0.006644079, + "costStatus": "reported" + }, + "runtime": { + "provider": "local", + "agentRunDurationMs": 82751, + "leaseDurationMs": null, + "leaseCount": 0, + "costStatus": "not_metered", + "costSource": "local_not_metered" + }, + "reportedCostUsd": 0.006644079, + "estimatedRuntimeCostUsd": 0, + "observedAndEstimatedCostUsd": 0.006644079, + "complete": true + }, + "preflight": { + "sourceSha": "fe9dc1e3c518825242ed889ab9c8352986f8c2ed", + "sourceFingerprint": "aeb792549146cc29eda779e83ae22d336276e31162f738a1dde85a82e7833902", + "passed": true, + "providerCalls": 0, + "assertions": 587 + }, + "delivery": { + "documentCount": 1, + "documentKey": "context-integrity-output", + "savedRevisionNumber": 1, + "savedRevisionPresent": true, + "markerPresent": true, + "markdownLinkTargets": [ + "/RUN/issues/RUN-1#document-context-integrity-output" + ], + "expectedDocumentUiTarget": "/RUN/issues/RUN-1#document-context-integrity-output", + "hasExactDocumentUiLink": true, + "originalCaseRequiresLink": true, + "originalBaselineBarePath": false, + "originalCandidateWrongPrefix": false + }, + "operationalSkillReadOrder": [ + { + "tool": "skill", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 1, + "skill": "assigned-output" + }, + { + "tool": "skill", + "operationalSkillLoad": true, + "earlyRecipeVisible": true, + "newReferencePointerVisible": true, + "newReferenceRead": false, + "outputTruncated": true, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 2, + "skill": "paperclip" + }, + { + "tool": "read", + "operationalSkillLoad": true, + "earlyRecipeVisible": true, + "newReferencePointerVisible": true, + "newReferenceRead": false, + "outputTruncated": true, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 3, + "skill": null + }, + { + "tool": "read", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": true, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 4, + "skill": null + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 5, + "skill": null + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 6, + "skill": null + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 7, + "skill": null + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 8, + "skill": null + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": true, + "statusApiCall": false, + "ordinal": 9, + "skill": null + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": false, + "ordinal": 10, + "skill": null + }, + { + "tool": "bash", + "operationalSkillLoad": false, + "earlyRecipeVisible": false, + "newReferencePointerVisible": false, + "newReferenceRead": false, + "outputTruncated": false, + "localDocumentWrite": false, + "documentApiCall": false, + "statusApiCall": true, + "ordinal": 11, + "skill": null + } + ], + "evidenceHashes": { + "result": "d363c4c394629706e30959682ad9b5ec301330636ec5f8c0dab319082ac3d8e8", + "context": "208a34086b904864fb9d115a9bae910ed49256b812b9e2ba9688459594921530", + "preflight": "08763f3fda4e6c60a23986ebdbba4973e6c10565575593e5797fd24ec8330d89", + "finalStateScreenshot": "982e5645377a39a6aa853795229d6b3e03afe00d567e03bd90a695c0f9115169" + }, + "screenshotInspected": true + } + ], + "providerCalls": 4, + "additionalPaidRuns": 0, + "limitations": [ + "One trial per cell cannot establish fix causality, robust non-regression or general coding quality.", + "Both original case link defects are outside the preserved durable-document oracle.", + "Explicit wording now requires clickable company-prefixed UI link and is not a direct repeat of prior usable-link cohorts.", + "Skill metadata improves stock selection but does not guarantee invocation on every harness.", + "API writes/revisions/comments and screenshot rendering were inspected; authenticated click navigation against cleaned-up instances was not replayed.", + "Runtime is unmetered; reported LLM receipts do not establish actual external bill.", + "Historical combined prompt-removal and early-recipe failures are preserved, alongside unrelated chat/credential findings." + ] +} diff --git a/doc/plans/2026-10-02-opencode-skill-routing-link-qualification.md b/doc/plans/2026-10-02-opencode-skill-routing-link-qualification.md new file mode 100644 index 00000000000..79f0c4b5193 --- /dev/null +++ b/doc/plans/2026-10-02-opencode-skill-routing-link-qualification.md @@ -0,0 +1,75 @@ +# OpenCode skill routing and clickable delivery — 2026-10-02 + +**TL;DR: 0 newly failing cells, 0 newly passing cells, 2 unchanged passes, and 0 pending pairs in this matched trial. Both document cases pass in both variants, so this does not establish that the metadata/link change caused the earlier failures to recover. Handoff remains imperfect in the original case:** candidate uses a clickable link with the wrong `PAP` company prefix; baseline supplies only a bare prefix-less slug path. Neither defect is scored by the preserved original document-storage oracle. The explicit clickable-link cases deliver the correct `RUN` link in both variants. + +The narrow correction expands the operational skill's stock discovery description to include Paperclip task/heartbeat work and document/file delivery. The early API recipe asks for a clickable Markdown link, and the reference shows a canonical UI-link example. The full skill stays on disk; the eight-word manual and shared prompts remain reduced. No full-body injection, adapter policy override or native tool guidance is added to legacy runs. + +| Classic OpenCode case | Pre-correction skill | Corrected skill | Independent evidence and limitation | +| --- | --- | --- | --- | +| Original assigned skill, verbatim request | Pass | Pass | Exactly one saved document with skill-only marker and persisted revision in each. Both handoff paths are deficient and outside this case's link-free oracle. | +| Explicit Paperclip storage with clarified clickable UI link | Pass | Pass | Saved revision/content and exact clickable `/RUN/issues/RUN-1#document-context-integrity-output` in both. | + +Original machine results, source/configuration proof, tool-read sequence, costs and evidence hashes are preserved in the [safe evidence projection](2026-10-02-opencode-skill-routing-link-qualification.json). All four cells clean up successfully; none uses automatic disposition recovery or an extra attempt. No provider was rerun to improve these results. + +## Sources, admission and inspectable reports + +- Candidate `fe9dc1e3c518825242ed889ab9c8352986f8c2ed`, frozen branch `codex/opencode-stock-routing-qualified`: [workflow](https://github.com/paperclipai/paperclip/actions/runs/37069547401), [published dashboard and screenshots](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37069547401-1/index.html). +- Baseline `0d7ecfa96d72fba79b7f0a25052b42c0686c0488`, frozen branch `codex/opencode-pre-routing-skill`: [workflow](https://github.com/paperclipai/paperclip/actions/runs/37069552374), [published dashboard and screenshots](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37069552374-1/index.html). +- Trusted default-branch workflows ran concurrently on separate targets. Each of four measured cells passed its own exact-source, credential-free 587-assertion prerequisite before provider credentials. Candidate source fingerprint is `aeb792549146cc29eda779e83ae22d336276e31162f738a1dde85a82e7833902`; baseline is `0315602ac6788cf0d65561a73cc85d790768ef90c6815257077a2381f961b96e`. +- Git tree comparison verifies 8,242 identical tracked files. Only the two operational skill sources and an explicit baseline provenance receipt differ. Models, effort, tools, auth, permissions, budgets, manual/shared prompts and behavioral fixtures/graders are fixed; both fixture configuration hashes match. Suite/source digest differences correctly include the changed skill bytes. +- Local source preparation passed 918 E2E support tests, eight canonical inventory calibrations, skill validation, E2E typecheck, two-cell discovery and both exact-source prerequisites. Both capability inventories use the same declared source list. Initial setup mistakes remain retained and did not reach providers. + +The explicit fixture now asks for a clickable company-prefixed Paperclip UI document link, matching its intended oracle. Bare paths and code-formatted paths fail calibration; correct relative/same-app absolute links pass. The original assigned-skill request and procedure remain verbatim. Earlier API-link grades are not retroactively changed, and these explicit results are not a direct repeat of the older “usable link” cohort. + +## Which guidance was read before delivery + +In the candidate original assignment, OpenCode loads the assigned Context integrity output skill, then operational Paperclip before any document write. The returned Paperclip skill is marked truncated, but the early saved-document recipe and reference pointer are visible. It reads `issue-documents.md`, then writes the public task document through the API. This differs from the previously failing candidate, which first loaded Paperclip only in a separate disposition-recovery run after its local-only output. + +The current baseline original assignment first loads the assigned output skill and writes a workspace Markdown file. It later loads Paperclip and reads the issue-document reference, then saves a public document before completing the **same** assignment. Thus its successful delivery did not require improved metadata; stock selection timing varies between trials. Both explicit variants load Paperclip and read the reference before the document API call. Four rendered final-state screenshots were inspected, and public document revisions/content and completion comments agree with the retained results. + +The candidate original comment has an actual Markdown href `/PAP/issues/RUN-1#document-context-integrity-output`, despite the measured company prefix being `RUN`. Baseline's original comment has a bare `/issues/runner-e2e-…#document-task-document` path without a Markdown anchor. The candidate's literal prefix matches the reference's example, making example copying a plausible cause, but this trial does not isolate that cause. Authenticated click navigation was not replayed after cleanup; neither original link is presented as a verified usable handoff. A minimal follow-up can show an issue-derived prefix instead of a literal company example. These original machine passes must not be presented as fully correct links. + +## Timing, usage and cost + +Four provider turns were expected and four assignment runs occurred. All four ledger receipts contain token usage and reported LLM cost. Local runtime remains unmetered; actual external billing is not established. + +| Case | Baseline provider seconds | Candidate provider seconds | Baseline cell seconds | Candidate cell seconds | Baseline reported USD | Candidate reported USD | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| Original | 25.870 | 67.897 | 46.285 | 89.455 | 0.0058472545 | 0.0041383700 | +| Explicit clickable storage | 73.984 | 82.751 | 94.804 | 102.901 | 0.0049436470 | 0.0066440790 | + +Reported LLM totals are **$0.0107909015 baseline** and **$0.0107824490 candidate**. The candidate original case takes longer in this trial; these single observations, different cache/token receipts and unmetered runtime do not establish general speed, cost or quality equivalence. + +## Preserved failures and scope + +The [original combined prompt-removal report](2026-10-02-stock-harness-live-comparison.md) still records two new classic Claude/OpenCode document-delivery failures, two OpenCode ordered-case improvements, and separate unchanged credential/chat failures. The [first recipe repair report](2026-10-02-legacy-document-skill-repair.md) still records Claude's improvement, OpenCode's unresolved local-only output, and the candidate's worse non-clickable explicit handoff. New successes do not erase those earlier observations or establish robustness. + +This comparison qualifies only classic OpenCode's two selected journeys. Existing classic Codex/Claude/OpenCode and ACP Codex skill mounts were audited; ACP Claude's names/root-only metadata and unsupported custom ACP delivery remain separate. Hermes/Pi/OpenClaw are not live-qualified here. Native completion descriptions and hiring templates have separate changes and measurements. Both PRs remain draft; no merge or further paid breadth is part of this report. + +## Provider-free follow-up after measurement + +The approved reference-only follow-up replaces the literal `PAP` link with a +JavaScript construction using the current `issue.identifier` and the successful +write receipt's `saved.key`. It is a later source change, **not live-qualified** +by the frozen candidate above. The independent storage grader now calibrates a +wrong-company link as failure, and executes the shipped recipe against two +other company prefixes plus a locked-write redirected document key. All 25 +focused document/source-digest tests, eight inventory calibrations, canonical +generated metadata checks and E2E typecheck pass. An initial sandboxed support attempt denied localhost/tsx +pipe creation; the failed attempt is retained, with its affected files rerun +unchanged under permitted local execution: 906 assertions passed in the original +921-assertion attempt, and all 82 selected assertions passed in the permitted +retry, including every originally failed assertion. No new model calls or old verdict +changes are part of this follow-up. + +Fresh native-PR review caught a supported edge case in that later recipe: +`issue.identifier` may be null. The recipe now uses the current issue ID in the +unprefixed UI route when no identifier exists. Provider-free tests cover both +null and absent identifiers, preserving the returned document key. Source review +confirms that the board resolves the loaded issue's actual company and preserves +the document hash; no agent-side company fetch is needed. This route logic also +corrects wrong prefixes, so the frozen candidate's `PAP` href is noncanonical, +not a proven broken link. No authenticated click replay or further model run was +made. The previous 4/5 review and local setup/timing failures remain retained; +current focused document/source/manifest/admission checks pass 56 assertions and +E2E typecheck passes. This correction still has no live qualification. diff --git a/doc/plans/2026-10-02-stock-harness-live-comparison.json b/doc/plans/2026-10-02-stock-harness-live-comparison.json new file mode 100644 index 00000000000..7512c947605 --- /dev/null +++ b/doc/plans/2026-10-02-stock-harness-live-comparison.json @@ -0,0 +1,357 @@ +{ + "schema": "paperclip.stock-harness.safe-comparison/v1", + "candidateSha": "f02d8d0df327abb43b20c7e7beb86798239abbf5", + "baselineSha": "12c5433c67dc2e62916b879349c7ba2b6e0431f0", + "nativeCodexFixHeldConstant": "408f70e69f9c5e49cb4377f4886ac2001bfa67a2", + "comparisonManifest": { + "schema": "paperclip.stock-harness-comparison.v1", + "variant": "previous-instructions", + "candidateSha": "f02d8d0df327abb43b20c7e7beb86798239abbf5", + "restoredInstructionsFrom": "e00d10d5d5594f6e2d1e8cf4e0ec81c074b0bd88", + "nativeCodex14920": "held-constant; no before/after performance claim", + "productionDifferences": [ + "server/src/onboarding-assets/default/AGENTS.md", + "packages/adapter-utils/src/server-utils.ts" + ], + "structuralOracle": "Historical manual exact content and observed generic procedures; independent behavioral graders unchanged", + "expectedMatrixCells": 24, + "expectedProviderTurns": 48, + "maxInitialCampaignsPerVariant": 1, + "behavioralSources": { + "context-integrity-cases.ts": "201d05f8257c1121a94dd9d8a22f08b9db80555e388013ffeabb65e301499fd2", + "context-integrity-scoring.ts": "50d370044e45386ba87fb23ca63c228b96bb8373d67bb7e82e8b2a89eb23f252", + "context-integrity-flow.ts": "63bcb7bdba4c80b15ff5872f16383844456eab0c5a10627da5d3487a26821c6a", + "chat-cases.ts": "754b251bdd42605e147da9081b663329234260d2b13df0df3d44562e362b14ca", + "chat-flow.ts": "11fc506559da61315ce04a877f6b38a055e71e532fecfcf62c749a1d3bf985e7", + "live-fixtures.ts": "1ef80209cc53cc514a8e58af4c2225e38ec15b20acf17ca7564506ca6617ac3e", + "runner.spec.ts": "14d57e0f9131b3f1981ac593464738d01362008eefe85ea836075c794681fa85", + "launch.ts": "5ada6ca5484542f028b29fc603dc7992213d4c06fa9046834bd7d23c4511b847", + "stock-harness-admission.ts": "c9775d77d48f8d59ba1197a7e5769f14db0c7d33f5e2a2968a04466e083d63ff", + "types.ts": "c45dc3b2366457449ec9e7d413f930605dec88ea93a14ce5c5a5f445a27d148c" + }, + "instructionSha256": { + "manual": "e4d2375d722602cd744403292d99f6e6c9b7e8014d9cdfa6f9811ace03428e5f", + "shared": "819026782b92f37064fca354ddbca685b3c05f1869ec560e40d8a757b0a8aee9" + } + }, + "modelsEffortToolsPermissionsCredentials": "same fixture/source configuration; effort inherits provider default when unspecified", + "expectedCellsPerVariant": 24, + "expectedProviderTurnsPerVariant": 48, + "baseline": [ + {"executionId":"stock-harness.legacy-acp-claude.local.assigned-skill-explicit-invocation","profile":"legacy-acp-claude","case":"assigned-skill-explicit-invocation","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"failed","failureClass":"secret_leak","causeCodes":["persisted_credential_guard"],"failedMatcherPaths":[],"model":"claude-sonnet-4-6","runtimeMode":"legacy","durationMs":59075,"providerDurationMs":42037,"cleanup":"passed","billing":{"llm":{"runCount":1,"runsWithTokenUsage":1,"runsWithReportedCost":1,"inputTokens":15184,"outputTokens":2261,"cachedInputTokens":218341,"totalTokens":235786,"reportedCostUsd":0.16114005000000003,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":42037,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.16114005000000003,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.16114005000000003,"complete":true},"billingTypes":["metered_api"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"historical-generic-procedures-observed":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":14137,"reportedPromptChars":13607,"freshPromptChars":4688,"retrievalClipped":false}],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"78f6add6f11a45f58bfc6fee81600871f888750986c27497563a79eba39a101c","evidence-manifest.json":"14f67d036a3663f606cf269890f5a36a18221e5c7dcb466f513f761d882fb6eb","snapshots/context-integrity.json":"93bba435c845af31707d4e4b84b2330cd8ff14045a76c2d1267b958c8133950b","snapshots/stock-harness-hire.json":"e8b85aa909fc6af6a72b180c79572d70a26dd6ee6307c9b985f80b928eb96170","snapshots/stock-harness.json":"fa0eb10eb3939b91d83fed8a8c9f9915bb407ab03a5a9bd7e5df99aa593ffb99","snapshots/stock-harness-preflight.json":"2ac069369e2d26b0e4c213966770962c0282f9a7665fa1297fe3c70df2c73960"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":7,"failed":0}}, + {"executionId":"stock-harness.legacy-acp-claude.local.continuity-restart","profile":"legacy-acp-claude","case":"continuity-restart","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"failed","failureClass":"secret_leak","causeCodes":["persisted_credential_guard","structural_receipt_incomplete"],"failedMatcherPaths":["stockHarness.fresh-default-delivered"],"model":"claude-sonnet-4-6","runtimeMode":"legacy","durationMs":36961,"providerDurationMs":19492,"cleanup":"passed","billing":{"llm":{"runCount":2,"runsWithTokenUsage":2,"runsWithReportedCost":2,"inputTokens":19069,"outputTokens":329,"cachedInputTokens":55518,"totalTokens":74916,"reportedCostUsd":0.1039979,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":19492,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.1039979,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.1039979,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"historical-generic-procedures-observed":true,"fresh-default-delivered":false},"invocationMetrics":[{"capturedPromptChars":16395,"reportedPromptChars":17606,"freshPromptChars":1462,"retrievalClipped":true},{"capturedPromptChars":16395,"reportedPromptChars":17638,"freshPromptChars":1462,"retrievalClipped":true}],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"b00296e015e0d35a213e031aede08732f900cf8aaa0f67dd6adc30310871a9f4","evidence-manifest.json":"101cb2a051bb64d3fc12e3a5389c2f39be4acb85e120b9bca1557163643f32e8","snapshots/context-integrity.json":null,"snapshots/stock-harness-hire.json":"df3b7ce86f99b1f31ce3061e637e936d40a68b63d49aa307cf4a9dcfa75b3983","snapshots/stock-harness.json":"cffaf9caec42246841497abf2059ee2d11c23a67971ad14112fa96beb23d302a","snapshots/stock-harness-preflight.json":"7da7c7be4d0c33c6d1c3fadeb500eebe3fceba797edd0f24a9db187091896b6b"}}, + {"executionId":"stock-harness.legacy-acp-claude.local.ordered-comment-continuation","profile":"legacy-acp-claude","case":"ordered-comment-continuation","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"failed","failureClass":"secret_leak","causeCodes":["persisted_credential_guard","structural_receipt_incomplete"],"failedMatcherPaths":["stockHarness.fresh-default-delivered"],"model":"claude-sonnet-4-6","runtimeMode":"legacy","durationMs":169818,"providerDurationMs":150645,"cleanup":"passed","billing":{"llm":{"runCount":2,"runsWithTokenUsage":2,"runsWithReportedCost":2,"inputTokens":49679,"outputTokens":7557,"cachedInputTokens":465247,"totalTokens":522483,"reportedCostUsd":0.45116585000000003,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":150645,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.45116585000000003,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.45116585000000003,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"historical-generic-procedures-observed":true,"fresh-default-delivered":false},"invocationMetrics":[{"capturedPromptChars":16433,"reportedPromptChars":21108,"freshPromptChars":4688,"retrievalClipped":true},{"capturedPromptChars":14499,"reportedPromptChars":14021,"freshPromptChars":4688,"retrievalClipped":false}],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"9994318ef5b3cac9143215d580f7551600777864f3d54a0cb2c48941a0799523","evidence-manifest.json":"79ba61a1c2ac3adde8d1905f09a051d0b711be7c618285b728d13f3650fe9d64","snapshots/context-integrity.json":"0566e89e716c742ee4452f5d9bd719354dba968459809e3e362794f55135b705","snapshots/stock-harness-hire.json":"05919c3be635604a40c0ea9bb76ab1ce1a1ce6545417e66e02c5a26881f082ee","snapshots/stock-harness.json":"6fdbee98a58b20e131a46a5bf9220c1d9e6c4cfc02d64baeee700627869b137c","snapshots/stock-harness-preflight.json":"cb4abccc23ebbe0cd4884ca1d5238eef14b82e15987a2d9e7083ccb608504189"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":12,"failed":0}}, + {"executionId":"stock-harness.legacy-acp-codex.local.assigned-skill-explicit-invocation","profile":"legacy-acp-codex","case":"assigned-skill-explicit-invocation","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"failed","failureClass":"secret_leak","causeCodes":["persisted_credential_guard"],"failedMatcherPaths":[],"model":"gpt-5.6-sol","runtimeMode":"legacy","durationMs":59328,"providerDurationMs":35548,"cleanup":"passed","billing":{"llm":{"runCount":1,"runsWithTokenUsage":1,"runsWithReportedCost":0,"inputTokens":511,"outputTokens":30,"cachedInputTokens":27646,"totalTokens":28187,"reportedCostUsd":0,"costStatus":"unpriced"},"runtime":{"provider":"local","agentRunDurationMs":35548,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":false},"billingTypes":["metered_api"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"historical-generic-procedures-observed":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":13618,"reportedPromptChars":13642,"freshPromptChars":4687,"retrievalClipped":false}],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"743b4b283ee73f34db4434facefb26012145fa528a1912856725c4f13e5b9247","evidence-manifest.json":"fe7f46cac1d8722c6137ec813638ecd0c5944f1ea25cc24092ff8d22236c807b","snapshots/context-integrity.json":"d247901367c17626a5738743dc300566c758db50fa7c0b60e5f74be5b7083245","snapshots/stock-harness-hire.json":"15866eaa21a98137bd2c4088dd53256360cdf65b27c8475d49d98cd49788fcf6","snapshots/stock-harness.json":"639cd4b7fec29383a18016e5b8be2227f68d8a5c37035318089ace252408a231","snapshots/stock-harness-preflight.json":"f52edc51ee627e9f5f02d3049a83a9b99fc073fc504ca9f238cf95bc691a1ce8"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":7,"failed":0}}, + {"executionId":"stock-harness.legacy-acp-codex.local.continuity-restart","profile":"legacy-acp-codex","case":"continuity-restart","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"failed","failureClass":"secret_leak","causeCodes":["persisted_credential_guard","structural_receipt_incomplete"],"failedMatcherPaths":["stockHarness.invocation-evidence-complete","stockHarness.fresh-default-delivered"],"model":"gpt-5.6-sol","runtimeMode":"legacy","durationMs":44754,"providerDurationMs":22118,"cleanup":"passed","billing":{"llm":{"runCount":3,"runsWithTokenUsage":1,"runsWithReportedCost":0,"inputTokens":662,"outputTokens":29,"cachedInputTokens":29955,"totalTokens":30646,"reportedCostUsd":0,"costStatus":"unpriced"},"runtime":{"provider":"local","agentRunDurationMs":22118,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":false},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":false,"historical-generic-procedures-observed":true,"fresh-default-delivered":false},"invocationMetrics":[{"capturedPromptChars":16383,"reportedPromptChars":17671,"freshPromptChars":1461,"retrievalClipped":true}],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"3ba472d29268b3e2d2170cd9947d7c89fbe685d902fedacf5c74a1952a97ce3a","evidence-manifest.json":"a5c263cc9afe3836a3c9f46cda537bb07a374c30fd7ad6f136088680b3ade01c","snapshots/context-integrity.json":null,"snapshots/stock-harness-hire.json":"8dd9d388cc265387d960fd435c19874601a5f7ca209b328943f8bacb145c5d89","snapshots/stock-harness.json":"f2b69dda7e26a76c952f0b36f10cfd7e9fb364ae3930585961d84093d17c2979","snapshots/stock-harness-preflight.json":"5fc3c8235143766fc6be8c7e12b31b92222594698c966462f944f13fb5929113"}}, + {"executionId":"stock-harness.legacy-acp-codex.local.ordered-comment-continuation","profile":"legacy-acp-codex","case":"ordered-comment-continuation","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"failed","failureClass":"secret_leak","causeCodes":["persisted_credential_guard","structural_receipt_incomplete"],"failedMatcherPaths":["contextIntegrity.ordered-comments","contextIntegrity.changed-scope-preserved","contextIntegrity.packing-report-order","contextIntegrity.final-scope-applied","contextIntegrity.continuation-run-count","contextIntegrity.continuation-wake-comment-ids","contextIntegrity.single-durable-output","contextIntegrity.completed-task","contextIntegrity.successful-runs","stockHarness.invocation-evidence-complete"],"model":"gpt-5.6-sol","runtimeMode":"legacy","durationMs":50896,"providerDurationMs":39443,"cleanup":"passed","billing":{"llm":{"runCount":4,"runsWithTokenUsage":1,"runsWithReportedCost":0,"inputTokens":992,"outputTokens":46,"cachedInputTokens":24349,"totalTokens":25387,"reportedCostUsd":0,"costStatus":"unpriced"},"runtime":{"provider":"local","agentRunDurationMs":39443,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":false},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":false,"historical-generic-procedures-observed":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":14032,"reportedPromptChars":14056,"freshPromptChars":4687,"retrievalClipped":false}],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"33ac59f0a690ed76618f695e658575751e718f53f9d828d8b52c690df9c15afb","evidence-manifest.json":"92c048e01012a77b94003e38f3f73888d725f6942327f7fbf3e6d2a42d12184d","snapshots/context-integrity.json":"2f9a803657068a67bcbf3b190370395187ef13b7a1487705d0305e121339eac7","snapshots/stock-harness-hire.json":"f47bb35d606ae00e6057a2a77769e2091e8ccb12eca08db3a17e1ed297a8ccf4","snapshots/stock-harness.json":"05582c37956b55f6a2bd7cad0c55bc0fa1dbff875d7dbed12b83352f6bd4e04c","snapshots/stock-harness-preflight.json":"badf01a591f5327f0dc2e48865711f8f4d7327888a7688668728173a29b273a8"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":3,"failed":9}}, + {"executionId":"stock-harness.legacy-claude.local.assigned-skill-explicit-invocation","profile":"legacy-claude","case":"assigned-skill-explicit-invocation","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"claude-sonnet-4-6","runtimeMode":"legacy","durationMs":49054,"providerDurationMs":29420,"cleanup":"passed","billing":{"llm":{"runCount":1,"runsWithTokenUsage":1,"runsWithReportedCost":1,"inputTokens":29883,"outputTokens":1348,"cachedInputTokens":225730,"totalTokens":256961,"reportedCostUsd":0.19999125,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":29420,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.19999125,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.19999125,"complete":true},"billingTypes":["metered_api"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"historical-generic-procedures-observed":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":7512,"reportedPromptChars":7512,"freshPromptChars":4684,"retrievalClipped":false}],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"4fc73f7b1226b9a41b209a2f6dfa2a58805ddc1a7e7b699ac2b42b2e74a189a7","evidence-manifest.json":"440ee14cca3f8558fb47ecf27b17693c0f9dbecdcf5c8532d7e2ccbefe31700b","snapshots/context-integrity.json":"4545847b9c4ab2a4fa3053a384b5e990133705fc9da0d1e9cd98f1275ec05f44","snapshots/stock-harness-hire.json":"a6abfc40d1c788e4b897f2375506d59b2caba9fab0d4df019e7137fe9f66177f","snapshots/stock-harness.json":"755cc9d5f656dc068a08c8c141203eaa5d3d60b53edd9d516887aae108119f45","snapshots/stock-harness-preflight.json":"f52730b4555ab31de952c8c1e38db4067fb40756b6befd36ed4b686e1a273d73"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":7,"failed":0}}, + {"executionId":"stock-harness.legacy-claude.local.continuity-restart","profile":"legacy-claude","case":"continuity-restart","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"failed","failureClass":"candidate_failure","causeCodes":["chat_memory_assertion"],"failedMatcherPaths":[],"model":"claude-sonnet-4-6","runtimeMode":"legacy","durationMs":40202,"providerDurationMs":23072,"cleanup":"passed","billing":{"llm":{"runCount":2,"runsWithTokenUsage":2,"runsWithReportedCost":2,"inputTokens":39036,"outputTokens":829,"cachedInputTokens":116171,"totalTokens":156036,"reportedCostUsd":0.1936638,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":23072,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.1936638,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.1936638,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"historical-generic-procedures-observed":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":11451,"reportedPromptChars":11451,"freshPromptChars":1458,"retrievalClipped":false},{"capturedPromptChars":11508,"reportedPromptChars":11508,"freshPromptChars":1458,"retrievalClipped":false}],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"f8e377f617204eabdb6b4e0c24ab3280c51664b429051f7ea21a4746790820a6","evidence-manifest.json":"45818efcb8ed6d663f7ffab6d53605d5191fb983a68e422dadebdae875ed5b4c","snapshots/context-integrity.json":null,"snapshots/stock-harness-hire.json":"e3a8735ad4d5f89dc3899668c36791579562157fa25aa32a6e59aa098d0c079a","snapshots/stock-harness.json":"f3d24452ade2f232b0f671768d46e4d65a6a0c2a0740bb1daa17a4871a09aa7b","snapshots/stock-harness-preflight.json":"b85c96ada9801ce6c6569338b68bc810494a0711f01af39a63932e288f7a81a3"}}, + {"executionId":"stock-harness.legacy-claude.local.ordered-comment-continuation","profile":"legacy-claude","case":"ordered-comment-continuation","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"claude-sonnet-4-6","runtimeMode":"legacy","durationMs":142017,"providerDurationMs":119179,"cleanup":"passed","billing":{"llm":{"runCount":2,"runsWithTokenUsage":2,"runsWithReportedCost":2,"inputTokens":66601,"outputTokens":6610,"cachedInputTokens":783643,"totalTokens":856854,"reportedCostUsd":0.58397715,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":119179,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.58397715,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.58397715,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"historical-generic-procedures-observed":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":14473,"reportedPromptChars":14447,"freshPromptChars":4684,"retrievalClipped":false},{"capturedPromptChars":7926,"reportedPromptChars":7926,"freshPromptChars":4684,"retrievalClipped":false}],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"9161c05c288e8620e5a748a28c901e1582efc240ca640203e1d8f7527e22a651","evidence-manifest.json":"296246a83eb0870a53e70f359f1859719f30b655e6ed20908abbfe4213605280","snapshots/context-integrity.json":"17779f4ebc38ed08071e40ac4b939149a18dfceb76c0f1f4f6ed21bf5d0153fa","snapshots/stock-harness-hire.json":"1f5a97e38555f57f47ee1900440d88ee521de75e7589c163f7688c852b661dc8","snapshots/stock-harness.json":"499a0b49c042b7c44f992c56983d207677cf619306ed6f6db9975ef1b84d455b","snapshots/stock-harness-preflight.json":"8f92c4b703533a645196bf03d1353be7196271d25638be7cb3fae6ec92c69c52"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":12,"failed":0}}, + {"executionId":"stock-harness.legacy-codex.local.assigned-skill-explicit-invocation","profile":"legacy-codex","case":"assigned-skill-explicit-invocation","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"gpt-5.6-sol","runtimeMode":"legacy","durationMs":50078,"providerDurationMs":27658,"cleanup":"passed","billing":{"llm":{"runCount":1,"runsWithTokenUsage":1,"runsWithReportedCost":0,"inputTokens":128946,"outputTokens":1727,"cachedInputTokens":97854,"totalTokens":228527,"reportedCostUsd":0,"costStatus":"unpriced"},"runtime":{"provider":"local","agentRunDurationMs":27658,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":false},"billingTypes":["metered_api"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"historical-generic-procedures-observed":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":12332,"reportedPromptChars":12332,"freshPromptChars":4683,"retrievalClipped":false}],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"ea06fc8b1ec166c9932de8634d08e528e90f7b734388ccad6d47b4b96df9ff6d","evidence-manifest.json":"45e71226e99eaee9bea0b0fae2cca35de573c2ab9d5add13159f321c8480ac0b","snapshots/context-integrity.json":"4cf10f812f258a77b472abd74f6691224dfcba4b9a2dec9c0f2c35c5ae741d7a","snapshots/stock-harness-hire.json":"19db78dbf51a67ff4bc307ef378da9cc024a0cae1c977951b49d089c1ec47fe4","snapshots/stock-harness.json":"a3cb7e2bf4e775cf9058b6be33d8df5321e312df2aed39736719454647610a8f","snapshots/stock-harness-preflight.json":"e9a116f8605a2630ff46075598ac6bae30817892d76efcd9a25891649cac0856"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":7,"failed":0}}, + {"executionId":"stock-harness.legacy-codex.local.continuity-restart","profile":"legacy-codex","case":"continuity-restart","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"gpt-5.6-sol","runtimeMode":"legacy","durationMs":85550,"providerDurationMs":29332,"cleanup":"passed","billing":{"llm":{"runCount":3,"runsWithTokenUsage":3,"runsWithReportedCost":0,"inputTokens":468597,"outputTokens":2919,"cachedInputTokens":373648,"totalTokens":845164,"reportedCostUsd":0,"costStatus":"unpriced"},"runtime":{"provider":"local","agentRunDurationMs":29332,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":false},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"historical-generic-procedures-observed":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":2787,"reportedPromptChars":2787,"freshPromptChars":0,"retrievalClipped":false},{"capturedPromptChars":2730,"reportedPromptChars":2730,"freshPromptChars":0,"retrievalClipped":false},{"capturedPromptChars":16326,"reportedPromptChars":16326,"freshPromptChars":1457,"retrievalClipped":false}],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"db7be8127de96e697a9d274b7a18db40695daa234684ec4d8e28afd8665c1f3e","evidence-manifest.json":"2e9151536b8ade2bfbb3c465438e8ace21417258c4037b6923920ee0784b0e47","snapshots/context-integrity.json":null,"snapshots/stock-harness-hire.json":"af8b2fe11402a8b87a056818df29a62829ec9425f8a2d955e95692980e15e043","snapshots/stock-harness.json":"739e642582faf89b74c522774c64ea26c6af623539b7a8f99ce332066c965600","snapshots/stock-harness-preflight.json":"8fe9f32d7629a1f6bd632d58e22e39e47baff2b8bc3b74e3593e6411ab3c9a2b"}}, + {"executionId":"stock-harness.legacy-codex.local.ordered-comment-continuation","profile":"legacy-codex","case":"ordered-comment-continuation","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"gpt-5.6-sol","runtimeMode":"legacy","durationMs":74114,"providerDurationMs":57150,"cleanup":"passed","billing":{"llm":{"runCount":2,"runsWithTokenUsage":2,"runsWithReportedCost":0,"inputTokens":628626,"outputTokens":7098,"cachedInputTokens":563211,"totalTokens":1198935,"reportedCostUsd":0,"costStatus":"unpriced"},"runtime":{"provider":"local","agentRunDurationMs":57150,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":false},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"historical-generic-procedures-observed":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":9811,"reportedPromptChars":9785,"freshPromptChars":0,"retrievalClipped":false},{"capturedPromptChars":12746,"reportedPromptChars":12746,"freshPromptChars":4683,"retrievalClipped":false}],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"56a457fcc7a36809eb4bfcedeff2fa079f5272754ff52fc453e8bf8d813e52f3","evidence-manifest.json":"bf17fb7d2a05f7e99e0723907bdc57f0a88bf6838d68f31f4b84dc6ebc62d3c7","snapshots/context-integrity.json":"a9156b810e32fa1d9d07826276a00c62ebe4f25ac3dc284ada157f2b2d72d800","snapshots/stock-harness-hire.json":"5fc8abd51d401c96d701b0862ac2dc674eb36244fdc25c49b67ba3947ba5a2fb","snapshots/stock-harness.json":"c72b8c53e1e50a82ab6ff815b363942936440cfff75f8f8ca5f54185c723889d","snapshots/stock-harness-preflight.json":"b14d304196cdc3146436c66199243c6da04784c13e45c9bd31e0ef83e97a29bc"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":12,"failed":0}}, + {"executionId":"stock-harness.legacy-opencode.local.assigned-skill-explicit-invocation","profile":"legacy-opencode","case":"assigned-skill-explicit-invocation","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"openrouter/deepseek/deepseek-v4-flash-0731","runtimeMode":"legacy","durationMs":86384,"providerDurationMs":68956,"cleanup":"passed","billing":{"llm":{"runCount":1,"runsWithTokenUsage":1,"runsWithReportedCost":1,"inputTokens":57002,"outputTokens":2365,"cachedInputTokens":223232,"totalTokens":282599,"reportedCostUsd":0.0044563934000000005,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":68956,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.0044563934000000005,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.0044563934000000005,"complete":true},"billingTypes":["unknown"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"historical-generic-procedures-observed":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":12335,"reportedPromptChars":12335,"freshPromptChars":4686,"retrievalClipped":false}],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"20dbb34bb975f73c3ac2695c047b6cbc8f6382757939ae6210b08a0f13ba9e9d","evidence-manifest.json":"67c3e042be8e0a5a0cd7cca82d1b0887e1392a2806c3b4db67baa3e99365b6be","snapshots/context-integrity.json":"decf2d93769acce2e2990de27c81ee32fd805ca53d3c2be68c56f22a013a7ac0","snapshots/stock-harness-hire.json":"187598ed72837719be8d6f0b77c5f8d35cfc871f8cf41d4fdc5b259f88171025","snapshots/stock-harness.json":"98424edd880cd05e0cbf8c32973fe9fbfc055037a74e76c20f893c3ee263f6a0","snapshots/stock-harness-preflight.json":"86d9f4080a2045ae3fa68468d0ffaf68baa26f27f0528274f13762f093b4c113"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":7,"failed":0}}, + {"executionId":"stock-harness.legacy-opencode.local.continuity-restart","profile":"legacy-opencode","case":"continuity-restart","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"openrouter/deepseek/deepseek-v4-flash-0731","runtimeMode":"legacy","durationMs":126589,"providerDurationMs":78910,"cleanup":"passed","billing":{"llm":{"runCount":3,"runsWithTokenUsage":3,"runsWithReportedCost":3,"inputTokens":64286,"outputTokens":1046,"cachedInputTokens":28672,"totalTokens":94004,"reportedCostUsd":0.0022020907,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":78910,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.0022020907,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.0022020907,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"historical-generic-procedures-observed":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":7611,"reportedPromptChars":7611,"freshPromptChars":0,"retrievalClipped":false},{"capturedPromptChars":7554,"reportedPromptChars":7554,"freshPromptChars":0,"retrievalClipped":false},{"capturedPromptChars":16335,"reportedPromptChars":16335,"freshPromptChars":1460,"retrievalClipped":false}],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"29dce6418aa35bb8070b07433046bbf39c9a8367a198cdb0aa28a0bea480032c","evidence-manifest.json":"439dc46130b607b7488e9df7736a711b267efa53689fe9fdad2f822525677215","snapshots/context-integrity.json":null,"snapshots/stock-harness-hire.json":"7f7ad64827bbe430e06a7e01b1377443653793584264c0fa00ac88a742bd0f49","snapshots/stock-harness.json":"4dd536951a950e7507710a9eb44cdbcd354907397f1e95b8297b28a98fd75dbc","snapshots/stock-harness-preflight.json":"8fa2ec342ad583fb20fed4db88890fc31eab3ef5898a952a85371e53dd433aaf"}}, + {"executionId":"stock-harness.legacy-opencode.local.ordered-comment-continuation","profile":"legacy-opencode","case":"ordered-comment-continuation","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37048838402","cellAttempt":1,"status":"failed","failureClass":"candidate_failure","causeCodes":["deadline_or_run_start_timeout"],"failedMatcherPaths":["contextIntegrity.comments-queued-as-batch","contextIntegrity.initial-run-started","contextIntegrity.ordered-comments","contextIntegrity.changed-scope-preserved","contextIntegrity.packing-report-order","contextIntegrity.final-scope-applied","contextIntegrity.continuation-run-count","contextIntegrity.continuation-wake-comment-ids","contextIntegrity.single-durable-output","contextIntegrity.completed-task","contextIntegrity.successful-runs"],"model":"openrouter/deepseek/deepseek-v4-flash-0731","runtimeMode":"legacy","durationMs":720616,"providerDurationMs":442902,"cleanup":"passed","billing":{"llm":{"runCount":3,"runsWithTokenUsage":2,"runsWithReportedCost":2,"inputTokens":51446,"outputTokens":8911,"cachedInputTokens":967936,"totalTokens":1028293,"reportedCostUsd":0.030657645200000007,"costStatus":"partial"},"runtime":{"provider":"local","agentRunDurationMs":442902,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.030657645200000007,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.030657645200000007,"complete":false},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"historical-generic-procedures-observed":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":16392,"reportedPromptChars":16461,"freshPromptChars":0,"retrievalClipped":true},{"capturedPromptChars":16406,"reportedPromptChars":16879,"freshPromptChars":4686,"retrievalClipped":true},{"capturedPromptChars":12749,"reportedPromptChars":12749,"freshPromptChars":4686,"retrievalClipped":false}],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"6d0e5d54b1742b02eb37346747510c66cb5cd0b80262a6c9590fd2a28861dcb2","evidence-manifest.json":"758931a4d30213d1b320f58cd2aa338e8a6a8aed579b878e14c5e52848bd5f13","snapshots/context-integrity.json":"bcaa4ee278361a90347625044aa49086b27093356abc0f3700b0363d28c8653c","snapshots/stock-harness-hire.json":"21431fa5e7cb6486037c670fc1e4aa64fb24716ad8ab325601ff294ada40263f","snapshots/stock-harness.json":"29c55e52f4adbbc9259cb9e221c08bfbbd7ad4c5f8247c24b3f029362e9000de","snapshots/stock-harness-preflight.json":"f341723074c996c79d8a6c30b9f12270634b20b40da452b13e4ab9ce91b47cac"},"taskDocumentCount":null,"behaviorCheckCounts":{"passed":1,"failed":11}}, + {"executionId":"stock-harness.runner-acpx-claude.local.assigned-skill-explicit-invocation","profile":"runner-acpx-claude","case":"assigned-skill-explicit-invocation","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"claude-sonnet-5","runtimeMode":"native","durationMs":46264,"providerDurationMs":25372,"cleanup":"passed","billing":{"llm":{"runCount":1,"runsWithTokenUsage":1,"runsWithReportedCost":1,"inputTokens":10,"outputTokens":1042,"cachedInputTokens":141254,"totalTokens":142306,"reportedCostUsd":0,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":25372,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":true},"billingTypes":["unknown"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true},"invocationMetrics":[],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"073085f804a58d191bd44972edfb08d914fe622a86f236d0c4dc1b2bf8a03322","evidence-manifest.json":"4fad428c1df72abbfee24a4f7338e25284fc874cff1220e6b631fa86eb2d0bed","snapshots/context-integrity.json":"77e27b8ea01a83e0b65f3d5410e0787fbf84a7f6ea02f4401bd77488999dd4a2","snapshots/stock-harness-hire.json":"8c6c5d56fd068c15f1e19ce9a0b62204443c894d204311bd949d5e22c624fd3d","snapshots/stock-harness.json":"2a6860d7c29ad0d03240bda0a779b83d0455b8ce9fcfb1cf9e6ad559c4dec112","snapshots/stock-harness-preflight.json":"79b11e544d2821d1025476f5fd0d737e822ae5ab731ab32047d1fbf6b76015e4"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":7,"failed":0}}, + {"executionId":"stock-harness.runner-acpx-claude.local.continuity-restart","profile":"runner-acpx-claude","case":"continuity-restart","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"claude-sonnet-5","runtimeMode":"native","durationMs":77424,"providerDurationMs":36820,"cleanup":"passed","billing":{"llm":{"runCount":3,"runsWithTokenUsage":3,"runsWithReportedCost":3,"inputTokens":6,"outputTokens":94,"cachedInputTokens":94997,"totalTokens":95097,"reportedCostUsd":0,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":36820,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true},"invocationMetrics":[],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"be582fe8af5f19543c16c8590697c5aa9dc688fdb653b0550aba0b55174becd1","evidence-manifest.json":"cfbd1cb13917a96b08dd46aae1fcc63e2b27af5d1d22f4952974d3415c74ffb8","snapshots/context-integrity.json":null,"snapshots/stock-harness-hire.json":"254f18a8e8e1d6a5a2e7731728756671c1374cd4d1210469172bed9c2cd6e4c2","snapshots/stock-harness.json":"8efa84d6907de220dd7b9fc2c384c61aae0606e7f5311f0e926e26ae1617d869","snapshots/stock-harness-preflight.json":"c7e3f2342ef9fc824b28e3f22ffbbb326d2ba1ff84e36c4deedc56a7abefbe07"}}, + {"executionId":"stock-harness.runner-acpx-claude.local.ordered-comment-continuation","profile":"runner-acpx-claude","case":"ordered-comment-continuation","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"claude-sonnet-5","runtimeMode":"native","durationMs":132587,"providerDurationMs":108895,"cleanup":"passed","billing":{"llm":{"runCount":2,"runsWithTokenUsage":2,"runsWithReportedCost":2,"inputTokens":26,"outputTokens":7996,"cachedInputTokens":565030,"totalTokens":573052,"reportedCostUsd":0,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":108895,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true},"invocationMetrics":[],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"62c6c55047813b0b5aa5334e29dbe3f90225c83ac436eab14afd97d78ed9ec65","evidence-manifest.json":"7d53ad7f80b65a1c7af3fad4fa3d0c652ae25cdaeae3d41cbee1bb888b853fb5","snapshots/context-integrity.json":"2081459b761a84469399bec377d7380c172b64fddae48735c5329771656e28a7","snapshots/stock-harness-hire.json":"f11c90fdae215949458d61a6e1ca6565fe277ea30d52fd922c05cffdf4867e5c","snapshots/stock-harness.json":"f303a3fb5c5e55cefb696d46204aafe6f1cefc5b0a6b07a728f2bf0dbf1dfdcf","snapshots/stock-harness-preflight.json":"79768aa58996d28ee80c8a9e8d5c4ba1a42eba58a408163cd8a6037ac6005b26"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":12,"failed":0}}, + {"executionId":"stock-harness.runner-codex.local.assigned-skill-explicit-invocation","profile":"runner-codex","case":"assigned-skill-explicit-invocation","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"gpt-5.6-sol","runtimeMode":"native","durationMs":53751,"providerDurationMs":29850,"cleanup":"passed","billing":{"llm":{"runCount":1,"runsWithTokenUsage":1,"runsWithReportedCost":1,"inputTokens":162041,"outputTokens":1296,"cachedInputTokens":137822,"totalTokens":301159,"reportedCostUsd":0,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":29850,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":true},"billingTypes":["unknown"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true},"invocationMetrics":[],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"ffcc5203624c19b0acda50dfd1e283c595ae3ed6072e4aaf9781fef4bf08fa45","evidence-manifest.json":"1460fae9f00a241a3ddbe7f623a4c1c6cf0f6434e73e47d0de7f98dc00dec459","snapshots/context-integrity.json":"63d16c6558efcf03097fd2ef8caf1f01c39a38127aee2904eba0cf0edfb711e6","snapshots/stock-harness-hire.json":"61630a0e0521f8ccca30753bde385b731f5c8c15cf6280b000947e80acf3decf","snapshots/stock-harness.json":"bc95a45aab085ce3eee988a9b235281b847cd8cb73c4d294ce576b97e73d879f","snapshots/stock-harness-preflight.json":"4daf4cd83a8b1539ab8a8db2fd5cf204746884d8cde04f6507acfff5be75ef11"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":7,"failed":0}}, + {"executionId":"stock-harness.runner-codex.local.continuity-restart","profile":"runner-codex","case":"continuity-restart","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"gpt-5.6-sol","runtimeMode":"native","durationMs":84362,"providerDurationMs":35132,"cleanup":"passed","billing":{"llm":{"runCount":3,"runsWithTokenUsage":3,"runsWithReportedCost":3,"inputTokens":315100,"outputTokens":2339,"cachedInputTokens":231408,"totalTokens":548847,"reportedCostUsd":0,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":35132,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true},"invocationMetrics":[],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"dbf2e14fcda52d2a33808e5a4c226d45f864752274c823c26beb7c2ea5409f03","evidence-manifest.json":"86a34f1a25854354bee0d9f0ea88cab9ba9b51476cf274d12f5b491a9910188a","snapshots/context-integrity.json":null,"snapshots/stock-harness-hire.json":"a38cab643c78fde9d61d15074b82297b94bf46ae5c3296aa06e6958f67188eda","snapshots/stock-harness.json":"2d148d3e55c57619bde56558f7e87e85cbb1305e46eba83bedb21c3ee4b43b54","snapshots/stock-harness-preflight.json":"9517ef436cc83da9078dd75587f01be8a28685131c37773dc092cf89c137ca0c"}}, + {"executionId":"stock-harness.runner-codex.local.ordered-comment-continuation","profile":"runner-codex","case":"ordered-comment-continuation","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"gpt-5.6-sol","runtimeMode":"native","durationMs":92663,"providerDurationMs":70832,"cleanup":"passed","billing":{"llm":{"runCount":2,"runsWithTokenUsage":2,"runsWithReportedCost":2,"inputTokens":428674,"outputTokens":3591,"cachedInputTokens":372482,"totalTokens":804747,"reportedCostUsd":0,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":70832,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true},"invocationMetrics":[],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"8c7a81172e0d519ffc8aca9a3025c88166d93d12a1744e560813c66063d6d130","evidence-manifest.json":"b495ad141c12f2458e3b66123cb673dec68511bcb5c2d59af78400dc67c417eb","snapshots/context-integrity.json":"166eb5a26fa389529287aa940c4e3caa85665ea67fcce019bc69a8bb8d9f7b41","snapshots/stock-harness-hire.json":"2fe4b0d207648628658312e97b491336929197f826c00dce012b6afc43537564","snapshots/stock-harness.json":"79b83fed61c69589d38b48552b04dce41bf41dca27f6e7f5dd15229024b6338f","snapshots/stock-harness-preflight.json":"689ce3141b2c6970eac5468885e5111daa730bcda25a1459e7bac4177650045e"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":12,"failed":0}}, + {"executionId":"stock-harness.runner-opencode.local.assigned-skill-explicit-invocation","profile":"runner-opencode","case":"assigned-skill-explicit-invocation","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"openrouter/deepseek/deepseek-v4-flash-0731","runtimeMode":"native","durationMs":90297,"providerDurationMs":68028,"cleanup":"passed","billing":{"llm":{"runCount":1,"runsWithTokenUsage":1,"runsWithReportedCost":1,"inputTokens":44259,"outputTokens":1165,"cachedInputTokens":62976,"totalTokens":108400,"reportedCostUsd":0.0020380985,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":68028,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.0020380985,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.0020380985,"complete":true},"billingTypes":["unknown"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true},"invocationMetrics":[],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"3ffa3bff50fbade8801da7c26ffed426eee9bda9a7e3640793f03a220f74ea82","evidence-manifest.json":"6df0c3b49ef54154721d4d0ba434f893fcba372ba8588fef2bd1a5947284b1d3","snapshots/context-integrity.json":"41b60c88adb0c529148981a55ab3b18964035b47786da50942183b58e8884fed","snapshots/stock-harness-hire.json":"c9a2fe249e217fcd5c205f1dc768446333932de74d87dc6f4a1e7855c5e4d815","snapshots/stock-harness.json":"474b0d6aecd8e8bce23f78c260ad4dc83c286d1d39917ea0920ad4b402d9404b","snapshots/stock-harness-preflight.json":"baeb38686ea4d3233906d83a889fdf3d805846b27f25c6c04da68842822a496d"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":7,"failed":0}}, + {"executionId":"stock-harness.runner-opencode.local.continuity-restart","profile":"runner-opencode","case":"continuity-restart","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"openrouter/deepseek/deepseek-v4-flash-0731","runtimeMode":"native","durationMs":155833,"providerDurationMs":101201,"cleanup":"passed","billing":{"llm":{"runCount":3,"runsWithTokenUsage":3,"runsWithReportedCost":3,"inputTokens":48028,"outputTokens":96,"cachedInputTokens":23808,"totalTokens":71932,"reportedCostUsd":0.0004892436,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":101201,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.0004892436,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.0004892436,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true},"invocationMetrics":[],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"57fe32615f16ff3706044419bb81362a67ec35aed7a5f74a5b8207b5831a5cba","evidence-manifest.json":"07e3be863cc20a1352a094f38e6133004dbfd673f4fd59599431a022ce061ff1","snapshots/context-integrity.json":null,"snapshots/stock-harness-hire.json":"d127504476aee2130312fdf092a523060188066abe9f969ae9b178b57374cced","snapshots/stock-harness.json":"0df941763ed917e2c553a6c6a4dfb0c87bdb7181d42a28f614c765906b74f1e9","snapshots/stock-harness-preflight.json":"f57655c970472d2a9b9bebe100d8c31788ed76e06da4363046a5a6d3f09269bb"}}, + {"executionId":"stock-harness.runner-opencode.local.ordered-comment-continuation","profile":"runner-opencode","case":"ordered-comment-continuation","sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceRef":"refs/heads/codex/stock-harness-previous-instructions","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042864888","cellAttempt":1,"status":"failed","failureClass":"candidate_failure","causeCodes":["deadline_or_run_start_timeout"],"failedMatcherPaths":["contextIntegrity.comments-queued-as-batch","contextIntegrity.initial-run-started","contextIntegrity.ordered-comments","contextIntegrity.changed-scope-preserved","contextIntegrity.packing-report-order","contextIntegrity.final-scope-applied","contextIntegrity.continuation-run-count","contextIntegrity.continuation-wake-comment-ids","contextIntegrity.single-durable-output","contextIntegrity.completed-task","contextIntegrity.successful-runs"],"model":"openrouter/deepseek/deepseek-v4-flash-0731","runtimeMode":"native","durationMs":721578,"providerDurationMs":446042,"cleanup":"passed","billing":{"llm":{"runCount":11,"runsWithTokenUsage":11,"runsWithReportedCost":11,"inputTokens":159917,"outputTokens":7291,"cachedInputTokens":1077987,"totalTokens":1245195,"reportedCostUsd":0.015645790399999998,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":446042,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.015645790399999998,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.015645790399999998,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true},"invocationMetrics":[],"prerequisite":{"sourceSha":"12c5433c67dc2e62916b879349c7ba2b6e0431f0","sourceFingerprint":"2e5f22848450179ac2d573662c718300f86a935d09d15204c9d97e98f65b53f4","passed":true,"providerCallsBeforeAdmission":0,"passedTests":536},"evidenceHashes":{"result.json":"f727dd5f76162c4b995e649e1a7bf55a52381efa550e3bda46ec0a51a899d178","evidence-manifest.json":"6a550c598b67f60fcea70d56907c4a2c84239bc0d01e58737e99a0014144bb70","snapshots/context-integrity.json":"14212826847fbd9d6d63c7b70afb102fdb5d0100f7d8f1368d46ff6f5e32db07","snapshots/stock-harness-hire.json":"ab6dd2377fc251e44ed8f3680d40d10a315b635f3003e45501c7496037fb21ca","snapshots/stock-harness.json":"38759bca194a8038f0d2be4598e50a04d61215ecabdfec95cd7f6cfa4eeec61c","snapshots/stock-harness-preflight.json":"312611057436fd8cbdcc5179ffa5e62144800a6a9244e646bb6e0b3a9a0cb551"},"taskDocumentCount":null,"behaviorCheckCounts":{"passed":1,"failed":11}} + ], + "candidate": [ + {"executionId":"stock-harness.legacy-acp-claude.local.assigned-skill-explicit-invocation","profile":"legacy-acp-claude","case":"assigned-skill-explicit-invocation","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/reduce-default-agent-manual","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042856368","cellAttempt":1,"status":"failed","failureClass":"secret_leak","causeCodes":["persisted_credential_guard","no_single_paperclip_document"],"failedMatcherPaths":["contextIntegrity.single-durable-output"],"model":"claude-sonnet-4-6","runtimeMode":"legacy","durationMs":123175,"providerDurationMs":111928,"cleanup":"passed","billing":{"llm":{"runCount":2,"runsWithTokenUsage":2,"runsWithReportedCost":2,"inputTokens":25333,"outputTokens":5771,"cachedInputTokens":316215,"totalTokens":347319,"reportedCostUsd":0.2838505,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":111928,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.2838505,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.2838505,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"generic-procedures-absent":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":10966,"reportedPromptChars":10449,"freshPromptChars":787,"retrievalClipped":false},{"capturedPromptChars":6028,"reportedPromptChars":5498,"freshPromptChars":787,"retrievalClipped":false}],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"2a5fed84ee0f95a67e2c2b0d4247ffae33e16a7c2aa11dc57b11228e0db90965","evidence-manifest.json":"7e31c707e866e15bf44a276ac252c423a47a63364e76ad85de9a8d5b6a4e1dfd","snapshots/context-integrity.json":"3c36c18d5e257e788514c02b50d63c7c69885201df0a07750129d55c0f583676","snapshots/stock-harness-hire.json":"2d128826b4e5068ad6ed819512bcb17160346becd9344b8ab8e70c984c3241b7","snapshots/stock-harness.json":"49b8da640ab0b9dd2b209ce0cf9f3bec6deceb801bfe98c0166d7702a9732a80","snapshots/stock-harness-preflight.json":"a12fdf9b8b7d761ff223e30dcaeb835ad9bf32ef7f1c8b8983a638d175c2ab5f"},"taskDocumentCount":0,"behaviorCheckCounts":{"passed":6,"failed":1}}, + {"executionId":"stock-harness.legacy-acp-claude.local.continuity-restart","profile":"legacy-acp-claude","case":"continuity-restart","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/reduce-default-agent-manual","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042856368","cellAttempt":1,"status":"failed","failureClass":"secret_leak","causeCodes":["persisted_credential_guard"],"failedMatcherPaths":[],"model":"claude-sonnet-4-6","runtimeMode":"legacy","durationMs":34813,"providerDurationMs":18798,"cleanup":"passed","billing":{"llm":{"runCount":2,"runsWithTokenUsage":2,"runsWithReportedCost":2,"inputTokens":16797,"outputTokens":329,"cachedInputTokens":54375,"totalTokens":71501,"reportedCostUsd":0.092943,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":18798,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.092943,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.092943,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"generic-procedures-absent":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":13201,"reportedPromptChars":12723,"freshPromptChars":787,"retrievalClipped":false},{"capturedPromptChars":13233,"reportedPromptChars":12755,"freshPromptChars":787,"retrievalClipped":false}],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"ecab7c9f83668fd4712702bb906adf38e188e34f7f07bd7d746da685339a2ecd","evidence-manifest.json":"544a726ddf3ac212671f7c3d73dafcf94cd5b3abb9a3ad5197d6c793e9d7aaa0","snapshots/context-integrity.json":null,"snapshots/stock-harness-hire.json":"78f5c467789921db4d2194ba80459524b04644f2d8d8a3b8678fdf7686004e8f","snapshots/stock-harness.json":"61641e5e25351e748afb2e4533d70b1cc34a8b78e1fbc56a27d96b2017bd0f45","snapshots/stock-harness-preflight.json":"4e522117d8489a16263c3d66ca513207217a7861733bfd11ef2f7149aef680e9"}}, + {"executionId":"stock-harness.legacy-acp-claude.local.ordered-comment-continuation","profile":"legacy-acp-claude","case":"ordered-comment-continuation","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/reduce-default-agent-manual","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042856368","cellAttempt":1,"status":"failed","failureClass":"secret_leak","causeCodes":["persisted_credential_guard"],"failedMatcherPaths":[],"model":"claude-sonnet-4-6","runtimeMode":"legacy","durationMs":154765,"providerDurationMs":139409,"cleanup":"passed","billing":{"llm":{"runCount":2,"runsWithTokenUsage":2,"runsWithReportedCost":2,"inputTokens":61445,"outputTokens":6979,"cachedInputTokens":562903,"totalTokens":631327,"reportedCostUsd":0.5120724,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":139409,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.5120724,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.5120724,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"generic-procedures-absent":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":13419,"reportedPromptChars":12915,"freshPromptChars":787,"retrievalClipped":false},{"capturedPromptChars":6390,"reportedPromptChars":5912,"freshPromptChars":787,"retrievalClipped":false}],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"9c23fec76e718c1fa5f775ae8cdcee1e5433039ff74d7ffbf306b5be6693700d","evidence-manifest.json":"5284aba88b923322b1f162045d20e36236fb731689527a1a1d06d5305efd4a64","snapshots/context-integrity.json":"8fef43dac791a84db62bfea8977f586afe3e5dc5c36225eaab79faaa15b4dfca","snapshots/stock-harness-hire.json":"4961a6103fcb6b58d5d65981c68afa178ea9e1eb6038ad3b3049d47e4bc737ab","snapshots/stock-harness.json":"4e11dd1b37581fed8628b7ebd953ebf3b6fe389d357ffd2efcd9e6b976e4d64a","snapshots/stock-harness-preflight.json":"7d962ba46f4f02f111ed4e423705009f38ee99096d180b693f4c41ed4c805126"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":12,"failed":0}}, + {"executionId":"stock-harness.legacy-acp-codex.local.assigned-skill-explicit-invocation","profile":"legacy-acp-codex","case":"assigned-skill-explicit-invocation","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/reduce-default-agent-manual","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042856368","cellAttempt":1,"status":"failed","failureClass":"secret_leak","causeCodes":["persisted_credential_guard"],"failedMatcherPaths":[],"model":"gpt-5.6-sol","runtimeMode":"legacy","durationMs":65745,"providerDurationMs":49358,"cleanup":"passed","billing":{"llm":{"runCount":1,"runsWithTokenUsage":1,"runsWithReportedCost":0,"inputTokens":1379,"outputTokens":34,"cachedInputTokens":41987,"totalTokens":43400,"reportedCostUsd":0,"costStatus":"unpriced"},"runtime":{"provider":"local","agentRunDurationMs":49358,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":false},"billingTypes":["metered_api"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"generic-procedures-absent":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":5509,"reportedPromptChars":5533,"freshPromptChars":786,"retrievalClipped":false}],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"81c5c35a942c78a2c9d7d77064264504311a191c6dd421e656d4f0751670b6ff","evidence-manifest.json":"bff59a88d627186d18282fe39822dd5a57d3a2396ae9aca5fa178a3b4ac3cf60","snapshots/context-integrity.json":"3809130fd34f0b6035ed5b17bd9e354d19aac02fc2f64aea01811803151c7f82","snapshots/stock-harness-hire.json":"4e0468ee05d56f08faa9ffbe3ed6cf28c07490a8b27a08e80875264dc66daac5","snapshots/stock-harness.json":"c5a82192ff53c6eec7a5497602a9969af3c21ea7fde8fdff4309025f5ccf80d2","snapshots/stock-harness-preflight.json":"12944d93aac9f8688d2832369f7eadc10a9546a6fb6bc2f2d8b3caef38b0cc2f"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":7,"failed":0}}, + {"executionId":"stock-harness.legacy-acp-codex.local.continuity-restart","profile":"legacy-acp-codex","case":"continuity-restart","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/reduce-default-agent-manual","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042856368","cellAttempt":1,"status":"failed","failureClass":"secret_leak","causeCodes":["persisted_credential_guard","structural_receipt_incomplete"],"failedMatcherPaths":["stockHarness.invocation-evidence-complete"],"model":"gpt-5.6-sol","runtimeMode":"legacy","durationMs":35779,"providerDurationMs":20059,"cleanup":"passed","billing":{"llm":{"runCount":3,"runsWithTokenUsage":1,"runsWithReportedCost":0,"inputTokens":711,"outputTokens":30,"cachedInputTokens":31118,"totalTokens":31859,"reportedCostUsd":0,"costStatus":"unpriced"},"runtime":{"provider":"local","agentRunDurationMs":20059,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":false},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":false,"generic-procedures-absent":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":12764,"reportedPromptChars":12788,"freshPromptChars":786,"retrievalClipped":false}],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"1c59fb6b3845ef6bc4f9cc549826e45334b3220ce5832f2fe4b9d23fd45136b6","evidence-manifest.json":"4cdd7cc5bb0a025e7626e947dd72558c13203aa9dc9765f5afbfed5acc907de1","snapshots/context-integrity.json":null,"snapshots/stock-harness-hire.json":"dcbe621edcd9437c47361c74676db38a240fdcaff2401d41213a185e7cf9b785","snapshots/stock-harness.json":"8908438e0c7565ebca6d4753e619f87bd0c015210214fbb8c39876d7e13cebaa","snapshots/stock-harness-preflight.json":"3b2de5b161384bebe03f7cad3b3268f98cca291cd8fbf4b8a0c830536daecf87"}}, + {"executionId":"stock-harness.legacy-acp-codex.local.ordered-comment-continuation","profile":"legacy-acp-codex","case":"ordered-comment-continuation","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/stock-harness-candidate-cancelled-cells","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37045368302","cellAttempt":1,"status":"failed","failureClass":"secret_leak","causeCodes":["persisted_credential_guard"],"failedMatcherPaths":["contextIntegrity.comments-queued-as-batch","contextIntegrity.initial-run-started","contextIntegrity.ordered-comments","contextIntegrity.changed-scope-preserved","contextIntegrity.packing-report-order","contextIntegrity.final-scope-applied","contextIntegrity.continuation-run-count","contextIntegrity.continuation-wake-comment-ids","contextIntegrity.single-durable-output","contextIntegrity.completed-task","contextIntegrity.successful-runs"],"model":"gpt-5.6-sol","runtimeMode":"legacy","durationMs":720307,"providerDurationMs":46665,"cleanup":"passed","billing":{"llm":{"runCount":1,"runsWithTokenUsage":1,"runsWithReportedCost":0,"inputTokens":540,"outputTokens":28,"cachedInputTokens":32395,"totalTokens":32963,"reportedCostUsd":0,"costStatus":"unpriced"},"runtime":{"provider":"local","agentRunDurationMs":46665,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":false},"billingTypes":["metered_api"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"generic-procedures-absent":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":5923,"reportedPromptChars":5947,"freshPromptChars":786,"retrievalClipped":false}],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"a1285f2fecccdcdac85360784e4be82188b564d3135f7adf6606fa1451307376","evidence-manifest.json":"39864272e9331c88c1266f2b5f05b755aacdc46c2c7015c4d4df8539c0a22718","snapshots/context-integrity.json":"5e663656093c672fb294719edb8636dd15a5ed1441912f32a32c78254663b797","snapshots/stock-harness-hire.json":"35d859103bf2acb7c7f6d212a8ce277e42726da388f7c52ee0b05d600dc09c78","snapshots/stock-harness.json":"483b4efedcc34570ad0ad364d92ba0212186cbc36a704639106e181402470468","snapshots/stock-harness-preflight.json":"5bb04e0c93a3f4ac5d9f41a5775fba85e0687a8161e255257545d54151421c62"},"taskDocumentCount":null,"behaviorCheckCounts":{"passed":1,"failed":11}}, + {"executionId":"stock-harness.legacy-claude.local.assigned-skill-explicit-invocation","profile":"legacy-claude","case":"assigned-skill-explicit-invocation","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/reduce-default-agent-manual","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042856368","cellAttempt":1,"status":"failed","failureClass":"candidate_failure","causeCodes":["no_single_paperclip_document"],"failedMatcherPaths":["contextIntegrity.single-durable-output"],"model":"claude-sonnet-4-6","runtimeMode":"legacy","durationMs":37720,"providerDurationMs":25763,"cleanup":"passed","billing":{"llm":{"runCount":1,"runsWithTokenUsage":1,"runsWithReportedCost":1,"inputTokens":27376,"outputTokens":1281,"cachedInputTokens":184039,"totalTokens":212696,"reportedCostUsd":0.17707545,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":25763,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.17707545,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.17707545,"complete":true},"billingTypes":["metered_api"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"generic-procedures-absent":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":3611,"reportedPromptChars":3611,"freshPromptChars":783,"retrievalClipped":false}],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"8636a50a54d39fd66b1439f13ce5ac125b79f16b2826f76fc297427a1a2ab701","evidence-manifest.json":"d1d7f47272981406d104f5ae6455d5b0a4f6113f3c3d02fc8bc787d7c8fc70fc","snapshots/context-integrity.json":"eac7d7b889e5cc3861b937ffa87c7b92d25ade5b0f2bb73bdd29b5764c2a085a","snapshots/stock-harness-hire.json":"545030ff52d9196a604e099b81b52a3071f582f2467284c695eb62d96589c67c","snapshots/stock-harness.json":"c9ca463a792c7dccd00c0795c1e405495d9e1dff30b2cbae2b5e7bac7114fdd2","snapshots/stock-harness-preflight.json":"c1b5a843459d80e074759600c830c92bb00a0d2430255cda7bed5c22cabc7680"},"taskDocumentCount":0,"behaviorCheckCounts":{"passed":6,"failed":1}}, + {"executionId":"stock-harness.legacy-claude.local.continuity-restart","profile":"legacy-claude","case":"continuity-restart","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/reduce-default-agent-manual","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042856368","cellAttempt":1,"status":"failed","failureClass":"candidate_failure","causeCodes":["chat_memory_assertion"],"failedMatcherPaths":[],"model":"claude-sonnet-4-6","runtimeMode":"legacy","durationMs":28251,"providerDurationMs":11843,"cleanup":"passed","billing":{"llm":{"runCount":2,"runsWithTokenUsage":2,"runsWithReportedCost":2,"inputTokens":19490,"outputTokens":298,"cachedInputTokens":50061,"totalTokens":69849,"reportedCostUsd":0.09257055,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":11843,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.09257055,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.09257055,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"generic-procedures-absent":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":10776,"reportedPromptChars":10776,"freshPromptChars":783,"retrievalClipped":false},{"capturedPromptChars":10833,"reportedPromptChars":10833,"freshPromptChars":783,"retrievalClipped":false}],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"86af4c61a239c718181ab58fd815cf7a200e7473b2fe9390b58bc15ff9d6ed4c","evidence-manifest.json":"e1c69d7c6044b0e8ba65fa1b961de84989facb3fd65f6bba42632b22d2a1bab9","snapshots/context-integrity.json":null,"snapshots/stock-harness-hire.json":"9a320857e3d1fe14a41b0154b8c1d2cfc81f869ed69790bd729fe7c86e9c54fb","snapshots/stock-harness.json":"9a2d546c992bf9d59711e37e81d841b7b1c73e6e1d62a6f27b5302f1d8ea3980","snapshots/stock-harness-preflight.json":"13318522befc6c0ffcc0d6ca7a0dc12112902427ddc1a7ac36f3ac43e7f83d8d"}}, + {"executionId":"stock-harness.legacy-claude.local.ordered-comment-continuation","profile":"legacy-claude","case":"ordered-comment-continuation","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/reduce-default-agent-manual","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042856368","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"claude-sonnet-4-6","runtimeMode":"legacy","durationMs":158440,"providerDurationMs":139386,"cleanup":"passed","billing":{"llm":{"runCount":2,"runsWithTokenUsage":2,"runsWithReportedCost":2,"inputTokens":64318,"outputTokens":7515,"cachedInputTokens":669889,"totalTokens":741722,"reportedCostUsd":0.5548662,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":139386,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.5548662,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.5548662,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"generic-procedures-absent":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":11079,"reportedPromptChars":11053,"freshPromptChars":783,"retrievalClipped":false},{"capturedPromptChars":4025,"reportedPromptChars":4025,"freshPromptChars":783,"retrievalClipped":false}],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"59885705b15fc33517ab7f7a82d9ada2e411c918b10fa5b4ce928498a801e739","evidence-manifest.json":"c2a5be637da0d7ff9436b732d8db72ab1fdce788caf6a40c5973c14a27bcc08a","snapshots/context-integrity.json":"fbee7c1a37b964bcc327ebbf37602bc227bccb7fafdc4296614e178e66016599","snapshots/stock-harness-hire.json":"cf2f8ff4b1eb13c5e5f854ecac27c25826e7a4dbafbc277e8702b8beee6d913f","snapshots/stock-harness.json":"3e8e7c06f0918a896341a38e4613cc1f4caaad4b5ab12c03a8c3e41c4707a82a","snapshots/stock-harness-preflight.json":"6d6716a08fc2ea7d27ee08840fd3290a04a8e85dfed5950b690f4c2267d20a6e"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":12,"failed":0}}, + {"executionId":"stock-harness.legacy-codex.local.assigned-skill-explicit-invocation","profile":"legacy-codex","case":"assigned-skill-explicit-invocation","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/reduce-default-agent-manual","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042856368","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"gpt-5.6-sol","runtimeMode":"legacy","durationMs":44075,"providerDurationMs":27042,"cleanup":"passed","billing":{"llm":{"runCount":1,"runsWithTokenUsage":1,"runsWithReportedCost":0,"inputTokens":241944,"outputTokens":2132,"cachedInputTokens":215273,"totalTokens":459349,"reportedCostUsd":0,"costStatus":"unpriced"},"runtime":{"provider":"local","agentRunDurationMs":27042,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":false},"billingTypes":["metered_api"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"generic-procedures-absent":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":4223,"reportedPromptChars":4223,"freshPromptChars":782,"retrievalClipped":false}],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"b0b26f37d2f0c0342a9a275b22716be190891d1b55e79fa8d33e3f495649a26b","evidence-manifest.json":"446abc91b7366507c4100741d71ace903de0b78149b33e45fa90d39183f71b75","snapshots/context-integrity.json":"85197eb92d34334cc2de658d09d7a7e54d44ab3d2fd81ad87732c88c6610b6b2","snapshots/stock-harness-hire.json":"568453725988fc61d459f96efda07d19ec5c00935d4c4a9ff9557cd0062aef29","snapshots/stock-harness.json":"0c8626740b520687d08b0b1f0aab8a9ead2ebd58290252f05d890f524636f461","snapshots/stock-harness-preflight.json":"9a15351a65076deeb77ccd58216def768a658fc743fd57dc89c3234ff6b36542"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":7,"failed":0}}, + {"executionId":"stock-harness.legacy-codex.local.continuity-restart","profile":"legacy-codex","case":"continuity-restart","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/reduce-default-agent-manual","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042856368","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"gpt-5.6-sol","runtimeMode":"legacy","durationMs":62742,"providerDurationMs":10028,"cleanup":"passed","billing":{"llm":{"runCount":3,"runsWithTokenUsage":3,"runsWithReportedCost":0,"inputTokens":91929,"outputTokens":475,"cachedInputTokens":45089,"totalTokens":137493,"reportedCostUsd":0,"costStatus":"unpriced"},"runtime":{"provider":"local","agentRunDurationMs":10028,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":false},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"generic-procedures-absent":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":2787,"reportedPromptChars":2787,"freshPromptChars":0,"retrievalClipped":false},{"capturedPromptChars":2730,"reportedPromptChars":2730,"freshPromptChars":0,"retrievalClipped":false},{"capturedPromptChars":11443,"reportedPromptChars":11443,"freshPromptChars":782,"retrievalClipped":false}],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"1ec35f3857350d86d3888c4becf8ce898a59641f3eb9825a0d800970ec7090ad","evidence-manifest.json":"908761dcb779dcd0b95ef78b5e9addb77b211acbc502a1c8fdf793202d309472","snapshots/context-integrity.json":null,"snapshots/stock-harness-hire.json":"b7b6f4db7cdc84a795ccfa24df58dd8131fe8a44b3ef1cdaf36dfc6685556da7","snapshots/stock-harness.json":"9a0045bbf4240ddbf08ec0444e08ef70e3ef76399db09075b5316b1fc51ac5bf","snapshots/stock-harness-preflight.json":"cfbae9842e2a045b815a9a7f1fa471f8f9494ec999c91e9808933cbc013c84a4"}}, + {"executionId":"stock-harness.legacy-codex.local.ordered-comment-continuation","profile":"legacy-codex","case":"ordered-comment-continuation","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/reduce-default-agent-manual","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37042856368","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"gpt-5.6-sol","runtimeMode":"legacy","durationMs":66403,"providerDurationMs":44175,"cleanup":"passed","billing":{"llm":{"runCount":2,"runsWithTokenUsage":2,"runsWithReportedCost":0,"inputTokens":416426,"outputTokens":4483,"cachedInputTokens":353673,"totalTokens":774582,"reportedCostUsd":0,"costStatus":"unpriced"},"runtime":{"provider":"local","agentRunDurationMs":44175,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":false},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"generic-procedures-absent":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":8456,"reportedPromptChars":8430,"freshPromptChars":0,"retrievalClipped":false},{"capturedPromptChars":4637,"reportedPromptChars":4637,"freshPromptChars":782,"retrievalClipped":false}],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"71f99ec1b45a57ae40acbd42c55e835f63d9d2329752f6bffcabd4488ecb2a96","evidence-manifest.json":"0c3347aaec6d7c3498e62c19938762c35a6069d31385e5d617174695ab60462e","snapshots/context-integrity.json":"5c08b2af1b27e9255505e917958e7b97856cfbf48cce3bef3b8e792275823b6e","snapshots/stock-harness-hire.json":"ddc6b182c4d53c7cd7f63c906d2a64e6286c54c8b6e9991e0c78da578586d865","snapshots/stock-harness.json":"11a737f54107eb9a9e55234890bce90003f84b258f6d5c8512d65f080edfc52f","snapshots/stock-harness-preflight.json":"d72a98df5a4e00bfa766e85ef28b866d32b89fe2aa9a67e3000ad50ddea47599"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":12,"failed":0}}, + {"executionId":"stock-harness.legacy-opencode.local.assigned-skill-explicit-invocation","profile":"legacy-opencode","case":"assigned-skill-explicit-invocation","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/stock-harness-candidate-cancelled-cells","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37045368302","cellAttempt":1,"status":"failed","failureClass":"candidate_failure","causeCodes":["no_single_paperclip_document"],"failedMatcherPaths":["contextIntegrity.single-durable-output"],"model":"openrouter/deepseek/deepseek-v4-flash-0731","runtimeMode":"legacy","durationMs":212558,"providerDurationMs":199337,"cleanup":"passed","billing":{"llm":{"runCount":2,"runsWithTokenUsage":2,"runsWithReportedCost":2,"inputTokens":96260,"outputTokens":6708,"cachedInputTokens":172288,"totalTokens":275256,"reportedCostUsd":0.010608548800000001,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":199337,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.010608548800000001,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.010608548800000001,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"generic-procedures-absent":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":8426,"reportedPromptChars":8439,"freshPromptChars":0,"retrievalClipped":false},{"capturedPromptChars":4226,"reportedPromptChars":4226,"freshPromptChars":785,"retrievalClipped":false}],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"435f708b59438d1d344130bc9e23b8ddbc93adf74d4e5ea8f049e0f1d06efa72","evidence-manifest.json":"8591f98df496159f99b98659ffdcbdf2e86718dccb4545d461f592bab0626b05","snapshots/context-integrity.json":"e4271e26bb891c3c15f2d54dd5cfba6605ebb4431456d70b4f146bb8eb4e1788","snapshots/stock-harness-hire.json":"6bb95cb92fbb14976b5f6796f530823f6aacdcb1448f89cb8b2ec093125266e2","snapshots/stock-harness.json":"b8a5a04730324a038869e28f79d55dd06ebf12abc2e6366bbfe9218995a6748b","snapshots/stock-harness-preflight.json":"c17087412694993475276a69162a171859dc54afae5b08ab89573cf0637d6e19"},"taskDocumentCount":0,"behaviorCheckCounts":{"passed":6,"failed":1}}, + {"executionId":"stock-harness.legacy-opencode.local.continuity-restart","profile":"legacy-opencode","case":"continuity-restart","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/stock-harness-candidate-cancelled-cells","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37045368302","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"openrouter/deepseek/deepseek-v4-flash-0731","runtimeMode":"legacy","durationMs":69761,"providerDurationMs":19747,"cleanup":"passed","billing":{"llm":{"runCount":3,"runsWithTokenUsage":3,"runsWithReportedCost":3,"inputTokens":12219,"outputTokens":116,"cachedInputTokens":0,"totalTokens":12335,"reportedCostUsd":0.0003185752,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":19747,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.0003185752,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.0003185752,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"generic-procedures-absent":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":3403,"reportedPromptChars":3403,"freshPromptChars":0,"retrievalClipped":false},{"capturedPromptChars":3346,"reportedPromptChars":3346,"freshPromptChars":0,"retrievalClipped":false},{"capturedPromptChars":11452,"reportedPromptChars":11452,"freshPromptChars":785,"retrievalClipped":false}],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"0e56bb0c46a05efeeffa6feb09f54fe13a33c2c96ab4e7a7ec548a0d80a1a235","evidence-manifest.json":"0e601f9bde1bdb7f44f000704078343a1478046c9eb488f2c8a785cd1d4cebec","snapshots/context-integrity.json":null,"snapshots/stock-harness-hire.json":"ab7d6af088306e1d43d4901a370dffa46b16dd4c17c2838a8e9e48a2b8e7a62f","snapshots/stock-harness.json":"2a6eabd1657c89568f890de12840747b544c7385c4363b3016879b07028961fa","snapshots/stock-harness-preflight.json":"539f106844e9421c284944c0b1526d74b0cc8e0905c5960be14cfa77ee17a023"}}, + {"executionId":"stock-harness.legacy-opencode.local.ordered-comment-continuation","profile":"legacy-opencode","case":"ordered-comment-continuation","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/stock-harness-candidate-cancelled-cells","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37045368302","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"openrouter/deepseek/deepseek-v4-flash-0731","runtimeMode":"legacy","durationMs":212184,"providerDurationMs":188508,"cleanup":"passed","billing":{"llm":{"runCount":2,"runsWithTokenUsage":2,"runsWithReportedCost":2,"inputTokens":177025,"outputTokens":7495,"cachedInputTokens":256512,"totalTokens":441032,"reportedCostUsd":0.011804638700000002,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":188508,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.011804638700000002,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.011804638700000002,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true,"invocation-evidence-complete":true,"generic-procedures-absent":true,"fresh-default-delivered":true},"invocationMetrics":[{"capturedPromptChars":10775,"reportedPromptChars":10736,"freshPromptChars":0,"retrievalClipped":false},{"capturedPromptChars":4640,"reportedPromptChars":4640,"freshPromptChars":785,"retrievalClipped":false}],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"8ca5a9a446dda03100d8ee66c5b9e02fd19c335dfc8cc34f83662aa1181240b9","evidence-manifest.json":"496a392c39d3a75f0287c19a6fc25e7577dee6bfef0e3c7f16801f61574cfa99","snapshots/context-integrity.json":"da023ad1b2e22b1659573e1829cfb273a4d3e329d1b324506dbc73104901ecf7","snapshots/stock-harness-hire.json":"ec6d9274b80d291e8949c71a9bfb1f6fdc88cd78a2094dcad1c0a599ce9b78cd","snapshots/stock-harness.json":"680d3af8a66f473a95905920123f0760aff53eb4ce09cd01393a78b6292178cd","snapshots/stock-harness-preflight.json":"5b79fae54bd6036dab2ad8bf8e8bacb9087e702b363dbbe4af2c58115eea8ead"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":12,"failed":0}}, + {"executionId":"stock-harness.runner-acpx-claude.local.assigned-skill-explicit-invocation","profile":"runner-acpx-claude","case":"assigned-skill-explicit-invocation","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/stock-harness-candidate-cancelled-cells","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37045368302","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"claude-sonnet-5","runtimeMode":"native","durationMs":50059,"providerDurationMs":28899,"cleanup":"passed","billing":{"llm":{"runCount":1,"runsWithTokenUsage":1,"runsWithReportedCost":1,"inputTokens":10,"outputTokens":1332,"cachedInputTokens":136292,"totalTokens":137634,"reportedCostUsd":0,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":28899,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":true},"billingTypes":["unknown"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true},"invocationMetrics":[],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"526b8881ac8dd2b51a788ca6ae535f7cd856e38066fadbd70737071c4f898abf","evidence-manifest.json":"40dc320ecab36c004d5e33bba8ebe1fb4d8444fd23d7ef0f055b188b474fffca","snapshots/context-integrity.json":"0b0b2b42376e1c5cf58b6ae1e9e0960b6b2bc82218e20589e04ca44639207ecb","snapshots/stock-harness-hire.json":"7f47075f11209d29c92091925bb61b2e2e22b06e1ce49b7dba69060242d62fc5","snapshots/stock-harness.json":"06398532782d5e4c9eb76ef55cdc87ed8d49c6f31dc06f09ce28715aa09f76fa","snapshots/stock-harness-preflight.json":"cb1980515ce11e2b55fcea5799277f511d0c6d01bf26b78e93be3ae17fa9b08d"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":7,"failed":0}}, + {"executionId":"stock-harness.runner-acpx-claude.local.continuity-restart","profile":"runner-acpx-claude","case":"continuity-restart","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/stock-harness-candidate-cancelled-cells","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37045368302","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"claude-sonnet-5","runtimeMode":"native","durationMs":93902,"providerDurationMs":43211,"cleanup":"passed","billing":{"llm":{"runCount":3,"runsWithTokenUsage":3,"runsWithReportedCost":3,"inputTokens":6,"outputTokens":101,"cachedInputTokens":92222,"totalTokens":92329,"reportedCostUsd":0,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":43211,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true},"invocationMetrics":[],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"332ba05ee7cda6971777af3454e8589652f742f45cd6350da4396a420e8c549f","evidence-manifest.json":"eaff7dd508a80dffef9d638a39e13b21d2ef427086ae478c10520c2b336fe881","snapshots/context-integrity.json":null,"snapshots/stock-harness-hire.json":"e8a854b3a0d886b01cb910e5b05baa5f1c791ef57b9a04168a0ce65cc60df7ce","snapshots/stock-harness.json":"1e53449e403bc079887cd183e2b556fbecd238a20130ef07c0c5decf499ff4ae","snapshots/stock-harness-preflight.json":"02283c369f44ab85a6400e0659da00e48bee84727fcebc35a8a691c4529c7cc0"}}, + {"executionId":"stock-harness.runner-acpx-claude.local.ordered-comment-continuation","profile":"runner-acpx-claude","case":"ordered-comment-continuation","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/stock-harness-candidate-cancelled-cells","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37045368302","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"claude-sonnet-5","runtimeMode":"native","durationMs":93050,"providerDurationMs":70393,"cleanup":"passed","billing":{"llm":{"runCount":2,"runsWithTokenUsage":2,"runsWithReportedCost":2,"inputTokens":18,"outputTokens":3794,"cachedInputTokens":332880,"totalTokens":336692,"reportedCostUsd":0,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":70393,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true},"invocationMetrics":[],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"3aae7a7ca62c39c74ac6d12e71b6b75223c76112ea018b8d08db1da52ea72e6a","evidence-manifest.json":"f08a3eee09a8fd4b279865dbe8f9ab2a66cc1a79addb216fe484561fcb9e2467","snapshots/context-integrity.json":"400df6ee5a9294998d4306f6eab6816e86155d8ecf0f39a6fc672381d47d423f","snapshots/stock-harness-hire.json":"604c41ac65944ef25cb7bbbf19941401139ed422342d7d6ebf9a07b79ecf7a75","snapshots/stock-harness.json":"f17d4f47657a0151dbfc585de64e338b7f79deae052f454794c2b934a7d398a2","snapshots/stock-harness-preflight.json":"fa502f0e3a3467dab6b04f810e6001ca635ce2d4bb5bae42a50e7a85f520a356"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":12,"failed":0}}, + {"executionId":"stock-harness.runner-codex.local.assigned-skill-explicit-invocation","profile":"runner-codex","case":"assigned-skill-explicit-invocation","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/stock-harness-candidate-cancelled-cells","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37045368302","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"gpt-5.6-sol","runtimeMode":"native","durationMs":48064,"providerDurationMs":26079,"cleanup":"passed","billing":{"llm":{"runCount":1,"runsWithTokenUsage":1,"runsWithReportedCost":1,"inputTokens":133429,"outputTokens":1102,"cachedInputTokens":110284,"totalTokens":244815,"reportedCostUsd":0,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":26079,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":true},"billingTypes":["unknown"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true},"invocationMetrics":[],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"690736dbf71179913631255763fd7b8aecd825236532154e54a3e1e53240d6b2","evidence-manifest.json":"75135aa0e7020617f54b5f2f593a5d48669ac7ecc864064d1eefc91af86d431c","snapshots/context-integrity.json":"46e65857a4d82f6ad900bc7e89166c35f2543faa368e48130f83dcea526ca65e","snapshots/stock-harness-hire.json":"2bf564606e974bb38baf50e363edd6d11aa5009fd429602a44fecb337bf9036c","snapshots/stock-harness.json":"2bf943a7cba0fc3886a04fa96f5646677f427e516232b35f6538384f4e6ed39f","snapshots/stock-harness-preflight.json":"635be5d6d578d7ad95b0e87697b939ed10f3cf2d3338caedc94350a1ed67ccc9"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":7,"failed":0}}, + {"executionId":"stock-harness.runner-codex.local.continuity-restart","profile":"runner-codex","case":"continuity-restart","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/stock-harness-candidate-cancelled-cells","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37045368302","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"gpt-5.6-sol","runtimeMode":"native","durationMs":89776,"providerDurationMs":43862,"cleanup":"passed","billing":{"llm":{"runCount":3,"runsWithTokenUsage":3,"runsWithReportedCost":3,"inputTokens":460440,"outputTokens":2336,"cachedInputTokens":378421,"totalTokens":841197,"reportedCostUsd":0,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":43862,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true},"invocationMetrics":[],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"b32c6c2c10d1eaabd0e2408db313bf9f30a7a60413733ef8ddfd8114ff6f2fdb","evidence-manifest.json":"34a60f60f021a731a98fab708ac31e344d8ecfa5a1d7dc72daa8c957028cbcea","snapshots/context-integrity.json":null,"snapshots/stock-harness-hire.json":"855403ffdcaa2f1542a5ebbc0dbd7590c609c170afc657884eeb9a1e79a51310","snapshots/stock-harness.json":"d4997c371e7db1a9e1c13536279480035eee7873e92a3ab3300969d508339481","snapshots/stock-harness-preflight.json":"642913765f2dfb9bbae2f46401448e4c3b6d3f9134cba2a073178d530c395efc"}}, + {"executionId":"stock-harness.runner-codex.local.ordered-comment-continuation","profile":"runner-codex","case":"ordered-comment-continuation","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/stock-harness-candidate-cancelled-cells","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37045368302","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"gpt-5.6-sol","runtimeMode":"native","durationMs":85296,"providerDurationMs":64569,"cleanup":"passed","billing":{"llm":{"runCount":2,"runsWithTokenUsage":2,"runsWithReportedCost":2,"inputTokens":359345,"outputTokens":2627,"cachedInputTokens":307813,"totalTokens":669785,"reportedCostUsd":0,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":64569,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true},"invocationMetrics":[],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"1a1a5f9b53dfc8cd2d96bafb8b34f6b1ac006514da70963ea3d8328f1c594309","evidence-manifest.json":"5220dd721a29958a2dbc6139a33bed2c94765801181c795abca6118722204f7b","snapshots/context-integrity.json":"100d34aa2c461b540164a00442100d5f6f438f706d14fb7316d80bf07d680cd8","snapshots/stock-harness-hire.json":"103296d40bc0b4500fa9b45ffb1e8192b770a1d4f87f8d0dd4f9e309384d407c","snapshots/stock-harness.json":"8d1eea8270fd2c29e284db9af321bebe85e635f6ec77d56674466d1480012ba8","snapshots/stock-harness-preflight.json":"be022954d2b5f5ef0bdae4cd892f06163f9d0863d0201aa92a5bab12f56a4dac"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":12,"failed":0}}, + {"executionId":"stock-harness.runner-opencode.local.assigned-skill-explicit-invocation","profile":"runner-opencode","case":"assigned-skill-explicit-invocation","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/stock-harness-candidate-cancelled-cells","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37045368302","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"openrouter/deepseek/deepseek-v4-flash-0731","runtimeMode":"native","durationMs":69432,"providerDurationMs":45144,"cleanup":"passed","billing":{"llm":{"runCount":1,"runsWithTokenUsage":1,"runsWithReportedCost":1,"inputTokens":22870,"outputTokens":1171,"cachedInputTokens":105472,"totalTokens":129513,"reportedCostUsd":0.0021534242,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":45144,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.0021534242,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.0021534242,"complete":true},"billingTypes":["unknown"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true},"invocationMetrics":[],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"69ac8808e939a796c40ac3665d56b152b43747fa91b7bbc84fd0a0f2fc68f5d9","evidence-manifest.json":"a0af2ef7ba4653d7ec825ac12de2a9e5f8b7fd7042bcfbe215e8af1b75ad38f8","snapshots/context-integrity.json":"e82ea3b31d4c6c73a95715c1d0e31338eb31a50f760bb6ea99b1d21032c6da32","snapshots/stock-harness-hire.json":"87ec7c1456344be2ed5cf68d4813920e8204fc483c6a34b1b62116bd721d9073","snapshots/stock-harness.json":"11cb4a18967ee59a7d5c202ee590a8dcfa7a9c145ab5ac87cde8da6a69f25497","snapshots/stock-harness-preflight.json":"c044805c4bb0db4c3cac0a4bbf54b937da7d1d376a4ac319f4d2baf3c636f51c"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":7,"failed":0}}, + {"executionId":"stock-harness.runner-opencode.local.continuity-restart","profile":"runner-opencode","case":"continuity-restart","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/stock-harness-candidate-cancelled-cells","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37045368302","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"openrouter/deepseek/deepseek-v4-flash-0731","runtimeMode":"native","durationMs":152945,"providerDurationMs":93715,"cleanup":"passed","billing":{"llm":{"runCount":3,"runsWithTokenUsage":3,"runsWithReportedCost":3,"inputTokens":26520,"outputTokens":106,"cachedInputTokens":45312,"totalTokens":71938,"reportedCostUsd":0.0005020232,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":93715,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.0005020232,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.0005020232,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true},"invocationMetrics":[],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"ae12b2eee65f45b8e2f530c037002e91db351134b6848da89e80bf7c15eb25a3","evidence-manifest.json":"dce1906b35fbbe6ac471ef922a2f0b1bae9da9438d56e78894e3651cba402aaa","snapshots/context-integrity.json":null,"snapshots/stock-harness-hire.json":"a57da515e6454ee8b82b9bf633a8e07097217f30555606538e135c7979de71c0","snapshots/stock-harness.json":"fa0501dbd07065bbbbd9a1f4c6bd82ba27f0fa45238790d531db4587c000309d","snapshots/stock-harness-preflight.json":"a20db2a5478fb7bd568bf6fcfbe7702dbe2a7a0390d9b36ec0fe2a6d07954d1b"}}, + {"executionId":"stock-harness.runner-opencode.local.ordered-comment-continuation","profile":"runner-opencode","case":"ordered-comment-continuation","sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceRef":"refs/heads/codex/stock-harness-candidate-cancelled-cells","workflowRunUrl":"https://github.com/paperclipai/paperclip/actions/runs/37045368302","cellAttempt":1,"status":"passed","failureClass":null,"causeCodes":[],"failedMatcherPaths":[],"model":"openrouter/deepseek/deepseek-v4-flash-0731","runtimeMode":"native","durationMs":95671,"providerDurationMs":72942,"cleanup":"passed","billing":{"llm":{"runCount":2,"runsWithTokenUsage":2,"runsWithReportedCost":2,"inputTokens":24355,"outputTokens":2690,"cachedInputTokens":160512,"totalTokens":187557,"reportedCostUsd":0.0043860217,"costStatus":"reported"},"runtime":{"provider":"local","agentRunDurationMs":72942,"leaseDurationMs":null,"leaseCount":0,"costStatus":"not_metered","costSource":"local_not_metered"},"reportedCostUsd":0.0043860217,"estimatedRuntimeCostUsd":0,"observedAndEstimatedCostUsd":0.0043860217,"complete":true},"billingTypes":["unavailable"],"stockChecks":{"budget-hard-stops":true,"default-hire-bundle":true,"provider-runs-present":true},"invocationMetrics":[],"prerequisite":{"sourceSha":"f02d8d0df327abb43b20c7e7beb86798239abbf5","sourceFingerprint":"ea488f1e5f67865c0e08e3628eb3081b86c6b3a8e40631702697ffac08359211","passed":true,"providerCallsBeforeAdmission":0,"passedTests":537},"evidenceHashes":{"result.json":"54f3029b581b82be87bd30b7670029bd5722915917e4bcab2e8ef0682bc3c0be","evidence-manifest.json":"b793fa707ccc2125b192816d59e922a0d43348c11eb516b8dff7e5d445b5545b","snapshots/context-integrity.json":"105ff5eb813ac3779e3ba04a8115ce1495c185011de4735782d3c4adba99787e","snapshots/stock-harness-hire.json":"4dde5055f48fff1b4cf466b4307739c967e2595e3c2e93d4e07230545741fa16","snapshots/stock-harness.json":"b0ce3eb0edce09ca4ef61af6f622cddd7c9bb1a3286c9ea7756dc228d00c10fa","snapshots/stock-harness-preflight.json":"78f2298b3bb4357b2a3ce6d75a922fc34d6449ef43378bb0dcc40476469daa01"},"taskDocumentCount":1,"behaviorCheckCounts":{"passed":12,"failed":0}} + ], + "totals": { + "baseline": { + "results": 24, + "passed": 15, + "failed": 9, + "missing": 0, + "cleanupPassed": 24, + "durationMs": 3250195, + "providerDurationMs": 2108034, + "llm": { + "runCount": 58, + "runsWithTokenUsage": 52, + "runsWithReportedCost": 43, + "inputTokens": 2778581, + "outputTokens": 68015, + "cachedInputTokens": 6908917, + "reportedCostUsd": 1.7494252618000001 + } + }, + "candidate": { + "results": 24, + "passed": 15, + "failed": 9, + "missing": 0, + "cleanupPassed": 24, + "durationMs": 2804913, + "providerDurationMs": 1540860, + "llm": { + "runCount": 47, + "runsWithTokenUsage": 45, + "runsWithReportedCost": 36, + "inputTokens": 2280185, + "outputTokens": 58933, + "cachedInputTokens": 4655025, + "reportedCostUsd": 1.7431513318 + } + } + }, + "matchedPassingTiming": { + "baseline": { + "cells": 13, + "medianCellDurationMs": 85550, + "medianProviderDurationMs": 57150 + }, + "candidate": { + "cells": 13, + "medianCellDurationMs": 69761, + "medianProviderDurationMs": 43862 + } + }, + "missingBaseline": [], + "cancellationRecovery": { + "sourceSha": "f02d8d0df327abb43b20c7e7beb86798239abbf5", + "originalRun": 37042856368, + "reason": "superseding pilot cancelled incomplete cells; completed behavior failures not rerun", + "executionIds": [ + "stock-harness.legacy-acp-codex.local.ordered-comment-continuation", + "stock-harness.legacy-opencode.local.ordered-comment-continuation", + "stock-harness.legacy-opencode.local.continuity-restart", + "stock-harness.runner-acpx-claude.local.continuity-restart", + "stock-harness.legacy-opencode.local.assigned-skill-explicit-invocation", + "stock-harness.runner-acpx-claude.local.assigned-skill-explicit-invocation", + "stock-harness.runner-acpx-claude.local.ordered-comment-continuation", + "stock-harness.runner-codex.local.continuity-restart", + "stock-harness.runner-codex.local.ordered-comment-continuation", + "stock-harness.runner-codex.local.assigned-skill-explicit-invocation", + "stock-harness.runner-opencode.local.assigned-skill-explicit-invocation", + "stock-harness.runner-opencode.local.continuity-restart", + "stock-harness.runner-opencode.local.ordered-comment-continuation" + ] + }, + "baselineInfrastructureRecovery": { + "schema": "paperclip.stock-harness.infra-recovery/v1", + "sourceSha": "12c5433c67dc2e62916b879349c7ba2b6e0431f0", + "originalRun": 37042864888, + "originalJob": 110958906481, + "reason": "AWS runner shutdown, no result artifact; no completed model failure rerun", + "executionIds": [ + "stock-harness.legacy-opencode.local.ordered-comment-continuation" + ] + }, + "layoutRecovery": { + "resultEdits": 0, + "oracleEdits": 0, + "originalEvidenceRetained": true, + "manifests": [ + { + "campaign": "candidate-original", + "sha256": "9135457cdc339d2fcaeea52cd06622ecaea6a74561df73d4d7a0baa0d8c0e7cc" + }, + { + "campaign": "baseline-final", + "sha256": "85223428a0d473ba83e324621e80d711725ebe069c4f055665b0bf987d1f0c4c" + }, + { + "campaign": "candidate-recovery", + "sha256": "7788f879d0cf1d140d452cfa78fe085b34bc23d298d7c31cac9d630b12b2572f" + }, + { + "campaign": "baseline-recovery-final", + "sha256": "e39d62dd60f092e76f8776e844041f2458fca1989854d57ab13932a36c937ceb" + } + ] + }, + "limitations": [ + "Single trial per cell; not general coding quality or statistical proof.", + "Skill task document storage wording is ambiguous; original graders unchanged.", + "Persisted credential guard failures are not equivalent to poor task behavior.", + "Report costs are incomplete; zero reported native cost has unknown billing type.", + "Local runtime and interrupted partial attempts are not metered.", + "Historical public prompt retrieval is clipped in three ACP cells; missing suffix does not prove provider omission.", + "Reconstructed reports use separate copies with directory-layout-only repair; original public reports retain infrastructure failures." + ], + "fixtureMatchProof": { + "schema": "paperclip.stock-harness.fixture-match-proof/v1", + "cells": [ + { + "executionId": "stock-harness.legacy-codex.local.ordered-comment-continuation", + "fixtureConfigSha256": "f727e68de06dec501c49efc41ab941bd6d63826db82e615d5e60f2d20133b488", + "model": "gpt-5.6-sol" + }, + { + "executionId": "stock-harness.legacy-codex.local.assigned-skill-explicit-invocation", + "fixtureConfigSha256": "ca51e8b29aab8e6efa32f2433c0f3960e43f9fc2f5edcd0660e753810d71f30d", + "model": "gpt-5.6-sol" + }, + { + "executionId": "stock-harness.legacy-codex.local.continuity-restart", + "fixtureConfigSha256": "8b1669039f3285f89a7f93c57e1c0dc12f071fbb2bbc8dcef6bf70bc14a4949a", + "model": "gpt-5.6-sol" + }, + { + "executionId": "stock-harness.legacy-claude.local.ordered-comment-continuation", + "fixtureConfigSha256": "94db17927ceddb54fe7766a362adcf78783726c68e1e136e407572aef6fed319", + "model": "claude-sonnet-4-6" + }, + { + "executionId": "stock-harness.legacy-claude.local.assigned-skill-explicit-invocation", + "fixtureConfigSha256": "6c21be9bb3fcec6274372fc586d88e42d82bc0d438e26a69ef614f8e6fcb8519", + "model": "claude-sonnet-4-6" + }, + { + "executionId": "stock-harness.legacy-claude.local.continuity-restart", + "fixtureConfigSha256": "695071e5453c90288659f54f40fb307486f64d5fb8a401b767d2e9974faab15f", + "model": "claude-sonnet-4-6" + }, + { + "executionId": "stock-harness.runner-codex.local.ordered-comment-continuation", + "fixtureConfigSha256": "9e19859684aad0246fd009061101eec59c41179e51e92a9467f506af22558126", + "model": "gpt-5.6-sol" + }, + { + "executionId": "stock-harness.runner-codex.local.assigned-skill-explicit-invocation", + "fixtureConfigSha256": "8fc61211aefb339a310469695d716c3faea15c2a547c543d70c599857a7e071d", + "model": "gpt-5.6-sol" + }, + { + "executionId": "stock-harness.runner-codex.local.continuity-restart", + "fixtureConfigSha256": "024e90354d760f90a40e2b3923468b72d36a9fabdb4fc58c10a7939a09e5cbc3", + "model": "gpt-5.6-sol" + }, + { + "executionId": "stock-harness.runner-opencode.local.ordered-comment-continuation", + "fixtureConfigSha256": "9dd6fef8fea77b07a67681acc1d7866fac9b4f16e23aca69eda32a4656cc260b", + "model": "openrouter/deepseek/deepseek-v4-flash-0731" + }, + { + "executionId": "stock-harness.runner-opencode.local.assigned-skill-explicit-invocation", + "fixtureConfigSha256": "891eb6dd154aa5926a3de2eae79780661b436618b6eac7733998750c6c341ba3", + "model": "openrouter/deepseek/deepseek-v4-flash-0731" + }, + { + "executionId": "stock-harness.runner-opencode.local.continuity-restart", + "fixtureConfigSha256": "441c2fa8bdd26cf14ac2440311b99a71b7dd6202698cafc3d01ed7a91dc4b004", + "model": "openrouter/deepseek/deepseek-v4-flash-0731" + }, + { + "executionId": "stock-harness.runner-acpx-claude.local.ordered-comment-continuation", + "fixtureConfigSha256": "77cc32714b23852d3ef67c687202d89f7513275500b3c522a0a1de3d745c9c4e", + "model": "claude-sonnet-5" + }, + { + "executionId": "stock-harness.runner-acpx-claude.local.assigned-skill-explicit-invocation", + "fixtureConfigSha256": "c190a1e731ef450ea6ae0c24b5a8ae0f21db725b47350645cdbb9273415c5acd", + "model": "claude-sonnet-5" + }, + { + "executionId": "stock-harness.runner-acpx-claude.local.continuity-restart", + "fixtureConfigSha256": "40876df6f88f34ba8232ccba8816bf7e1173be0042a90a174f79cddcca346c4c", + "model": "claude-sonnet-5" + }, + { + "executionId": "stock-harness.legacy-acp-codex.local.ordered-comment-continuation", + "fixtureConfigSha256": "22500c39874cda62a2512ac3d8a43309ce5879f55f8515e31dd8ac632edb6c93", + "model": "gpt-5.6-sol" + }, + { + "executionId": "stock-harness.legacy-acp-codex.local.assigned-skill-explicit-invocation", + "fixtureConfigSha256": "896d8602d2990bf7517a4392d66909c6649e0f00f79a5ea0bb8f0e370b7b597a", + "model": "gpt-5.6-sol" + }, + { + "executionId": "stock-harness.legacy-acp-codex.local.continuity-restart", + "fixtureConfigSha256": "7b66845e231674cc6e78f1d86e863dc03005998c2674316af50db60ec53b7e4a", + "model": "gpt-5.6-sol" + }, + { + "executionId": "stock-harness.legacy-acp-claude.local.ordered-comment-continuation", + "fixtureConfigSha256": "6c1b9d7debb306af405640eb381e1709ae7185e7925bc2bd3ef31acf98fbff8f", + "model": "claude-sonnet-4-6" + }, + { + "executionId": "stock-harness.legacy-acp-claude.local.assigned-skill-explicit-invocation", + "fixtureConfigSha256": "53f308d6fbdc30adc60370279ab38679b5d87075899be4db6c0730593e1632c4", + "model": "claude-sonnet-4-6" + }, + { + "executionId": "stock-harness.legacy-acp-claude.local.continuity-restart", + "fixtureConfigSha256": "80ec22c5080ef1a71fc97547c3e78f4bc5f54f121f46f982506f2f9f773b27d0", + "model": "claude-sonnet-4-6" + }, + { + "executionId": "stock-harness.legacy-opencode.local.ordered-comment-continuation", + "fixtureConfigSha256": "e015ded0177d8d4c893acc658e25699b28937d97cb9d58da023ddff7ba7eac0b", + "model": "openrouter/deepseek/deepseek-v4-flash-0731" + }, + { + "executionId": "stock-harness.legacy-opencode.local.assigned-skill-explicit-invocation", + "fixtureConfigSha256": "dd673f469cab583d26bc2e4b484f18b27884ec14a59542177026389181e2da2c", + "model": "openrouter/deepseek/deepseek-v4-flash-0731" + }, + { + "executionId": "stock-harness.legacy-opencode.local.continuity-restart", + "fixtureConfigSha256": "3dadc83dff00ef2919cd24c6328ea9507f971e7d294e9ece699ff2ee6fa98230", + "model": "openrouter/deepseek/deepseek-v4-flash-0731" + } + ] + }, + "workflowProvenance": [ + { + "run": 37042856368, + "workflowSha": "7d59de6113caf5219d34d65e1fb5f0c1a7fbdc2a", + "workflowUrl": "https://github.com/paperclipai/paperclip/actions/runs/37042856368" + }, + { + "run": 37042864888, + "workflowSha": "7d59de6113caf5219d34d65e1fb5f0c1a7fbdc2a", + "workflowUrl": "https://github.com/paperclipai/paperclip/actions/runs/37042864888" + }, + { + "run": 37045368302, + "workflowSha": "7d59de6113caf5219d34d65e1fb5f0c1a7fbdc2a", + "workflowUrl": "https://github.com/paperclipai/paperclip/actions/runs/37045368302" + }, + { + "run": 37044967981, + "workflowSha": "7d59de6113caf5219d34d65e1fb5f0c1a7fbdc2a", + "workflowUrl": "https://github.com/paperclipai/paperclip/actions/runs/37044967981" + }, + { + "run": 37048838402, + "workflowSha": "7d59de6113caf5219d34d65e1fb5f0c1a7fbdc2a", + "workflowUrl": "https://github.com/paperclipai/paperclip/actions/runs/37048838402" + } + ] +} diff --git a/doc/plans/2026-10-02-stock-harness-live-comparison.md b/doc/plans/2026-10-02-stock-harness-live-comparison.md new file mode 100644 index 00000000000..42c7ae3fa33 --- /dev/null +++ b/doc/plans/2026-10-02-stock-harness-live-comparison.md @@ -0,0 +1,110 @@ +# Stock-harness live comparison — 2026-10-02 + +**TL;DR:** Two newly failing paired cases: classic Claude/OpenCode document delivery. Two newly passing cases: classic and native OpenCode ordered continuation. Seven unchanged failures and 13 unchanged passes. The extra ACP Claude storage symptom occurs beneath an existing credential-guard failure. All 48 results are available; no general performance equivalence is established. + +The reduced instructions are **not yet qualified for merge**. Classic Claude and classic OpenCode pass the historical skill-output case but fail with the reduced instructions: they finish without saving a Paperclip task document. Native Codex and native Claude pass all three journeys in both variants. Single trials and ambiguous fixture storage wording limit causal attribution; the failed oracle remains unchanged. + +The reductions and evaluation setup are in draft [PR #14948](https://github.com/paperclipai/paperclip/pull/14948). This report and its [safe evidence projection](2026-10-02-stock-harness-live-comparison.json) record the measured revisions rather than claiming the final documentation head was run through the full matrix. + +## Revisions and matched inputs + +- Candidate: `f02d8d0df327abb43b20c7e7beb86798239abbf5`, eight-word hire manual plus reduced shared startup/resume instructions. +- Historical baseline: `12c5433c67dc2e62916b879349c7ba2b6e0431f0`, the same evaluation harness with only the prior default manual and shared prompt source restored from `e00d10d5d5594f6e2d1e8cf4e0ec81c074b0bd88`. Its manifest records ten identical behavioral/fixture source hashes. +- Native Codex fix [#14920](https://github.com/paperclipai/paperclip/pull/14920) is held constant. This comparison does not measure its before/after performance. +- Same eight profiles, three journeys, provider models, effort configuration, skills, tools, permissions, credentials, local runtime, and independent graders. Effort inherits provider defaults where the existing profile does not set it. +- Each variant selects 24 cells and initially budgets 48 turns, with existing bounded retries and 1,000-cent company/agent hard stops. Recorded run-ledger counts can differ after early failures, retries and cleanup. + +## Per-profile and case outcomes + +These are recorded overall qualifications, including security and receipt failures. A credential-guard failure does not imply every behavioral matcher failed. + +| Profile | Model | Journey | Historical | Reduced | +| --- | --- | --- | --- | --- | +| `legacy-codex` | `gpt-5.6-sol` | `assigned-skill-explicit-invocation` | Pass | Pass | +| `legacy-codex` | `gpt-5.6-sol` | `ordered-comment-continuation` | Pass | Pass | +| `legacy-codex` | `gpt-5.6-sol` | `continuity-restart` | Pass | Pass | +| `legacy-claude` | `claude-sonnet-4-6` | `assigned-skill-explicit-invocation` | Pass | Fail: no Paperclip document | +| `legacy-claude` | `claude-sonnet-4-6` | `ordered-comment-continuation` | Pass | Pass | +| `legacy-claude` | `claude-sonnet-4-6` | `continuity-restart` | Fail: chat memory | Fail: chat memory | +| `legacy-opencode` | `openrouter/deepseek/deepseek-v4-flash-0731` | `assigned-skill-explicit-invocation` | Pass | Fail: no Paperclip document | +| `legacy-opencode` | `openrouter/deepseek/deepseek-v4-flash-0731` | `ordered-comment-continuation` | Fail: deadline/run-start timeout | Pass | +| `legacy-opencode` | `openrouter/deepseek/deepseek-v4-flash-0731` | `continuity-restart` | Pass | Pass | +| `legacy-acp-codex` | `gpt-5.6-sol` | `assigned-skill-explicit-invocation` | Fail: credential guard | Fail: credential guard | +| `legacy-acp-codex` | `gpt-5.6-sol` | `ordered-comment-continuation` | Fail: credential guard, receipt incomplete | Fail: credential guard | +| `legacy-acp-codex` | `gpt-5.6-sol` | `continuity-restart` | Fail: credential guard, receipt incomplete | Fail: credential guard, receipt incomplete | +| `legacy-acp-claude` | `claude-sonnet-4-6` | `assigned-skill-explicit-invocation` | Fail: credential guard | Fail: credential guard, no Paperclip document | +| `legacy-acp-claude` | `claude-sonnet-4-6` | `ordered-comment-continuation` | Fail: credential guard, receipt incomplete | Fail: credential guard | +| `legacy-acp-claude` | `claude-sonnet-4-6` | `continuity-restart` | Fail: credential guard, receipt incomplete | Fail: credential guard | +| `runner-codex` | `gpt-5.6-sol` | `assigned-skill-explicit-invocation` | Pass | Pass | +| `runner-codex` | `gpt-5.6-sol` | `ordered-comment-continuation` | Pass | Pass | +| `runner-codex` | `gpt-5.6-sol` | `continuity-restart` | Pass | Pass | +| `runner-acpx-claude` | `claude-sonnet-5` | `assigned-skill-explicit-invocation` | Pass | Pass | +| `runner-acpx-claude` | `claude-sonnet-5` | `ordered-comment-continuation` | Pass | Pass | +| `runner-acpx-claude` | `claude-sonnet-5` | `continuity-restart` | Pass | Pass | +| `runner-opencode` | `openrouter/deepseek/deepseek-v4-flash-0731` | `assigned-skill-explicit-invocation` | Pass | Pass | +| `runner-opencode` | `openrouter/deepseek/deepseek-v4-flash-0731` | `ordered-comment-continuation` | Fail: deadline/run-start timeout | Pass | +| `runner-opencode` | `openrouter/deepseek/deepseek-v4-flash-0731` | `continuity-restart` | Pass | Pass | + +## Findings and next step + +The classic Claude and classic OpenCode skill runs read the pinned skill marker and complete the task, but save zero Paperclip documents; both historical runs save exactly one. The legacy ACP Claude skill run has the same document difference under its credential-guard failure. The earlier reduced-instruction Claude pilot also failed this document oracle. The pinned skill asks for a “task document” without explicitly naming Paperclip storage, so this may reveal both reliance on the former manual and a fixture ambiguity. It remains a failed delivery result, not a regraded pass. + +Keep the agreed tiny manual. Next, verify that legacy agents discover and follow the existing **Paperclip skill/API path for durable task documents and artifacts**. The skill already prohibits file-only handoff, lists issue-document routes and gives a plan-document example; audit that delivery before adding a concise generic-document recipe if needed. Run the same preserved case plus a clearly storage-specific case on both variants. Legacy agents do not receive native `paperclip_finish`/`paperclip_block` tools. Native tool descriptions are a separate item 2.3 change; guidance must follow the actual runtime capability. No production or skill instructions were changed to make these runs pass. + +Classic Claude chat fails the memory assertion in both variants. Six legacy ACP cells per variant fail the persisted-credential guard: the scanner finds a scoped provider credential in persisted ACP session state. Raw session files and credential values are not published. These are existing qualification failures requiring their own investigation; they cannot be waived or interpreted as instruction-reduction quality evidence. + +Historical native OpenCode ordered comments hit the case deadline without the required continuation outcome. The initial historical classic OpenCode ordered cell lost its AWS runner before result upload; its one unchanged-source recovery also hits the 12-minute deadline. Both OpenCode ordered cases pass in the candidate. Completed failures are never rerun to select a better result. Those candidate passes against historical timeouts are observed differences, not proof of a general improvement. + +Historical public prompt retrieval is clipped in three legacy ACP cells (four invocations), including identity/connection suffix text. Those structural receipt failures cannot establish that the provider omitted instructions. The candidate ACP Codex chat run also lacks a complete per-run invocation receipt. Instruction presence/absence claims are bounded by the retained public receipts. + +## Timing, usage and cost + +| Variant | Retained cells | Pass / fail / missing | Cell duration sum | Provider duration sum | Recorded runs with tokens / with reported cost / total | Reported LLM subtotal | +| --- | ---: | --- | ---: | ---: | --- | ---: | +| baseline | 24 | 15 / 9 / 0 | 3250.195s | 2108.034s | 52 / 43 / 58 | $1.749425 | +| candidate | 24 | 15 / 9 / 0 | 2804.913s | 1540.860s | 45 / 36 / 47 | $1.743151 | + +For the 13 cells that pass both variants, median cell time is 85.550s historical versus 69.761s reduced; median provider time is 57.150s versus 43.862s. This excludes failures and is descriptive, not a reliable speedup estimate. + +Provider billing is incomplete. Legacy Codex/ACP usage is unpriced in several runs; native Codex/Claude often report zero with billing type `unknown`. Reported subtotals do not prove zero native spending or invoice totals. Local runner costs and interrupted partial runs are not metered. Diagnostic/pilot costs and repeated interrupted work are outside the matched-cell subtotal and remain retained. No aggregate cost-saving claim is justified. + +## Campaign and evidence history + +| Campaign | Source | Outcome / evidence | +| --- | --- | --- | +| [Initial diagnostic Codex](https://github.com/paperclipai/paperclip/actions/runs/37034213743) | `a63437069` | Skill passes; predates mandatory admission. | +| [Cold full attempt](https://github.com/paperclipai/paperclip/actions/runs/37037105491) | `36e987246` | Missing SDK build; cancelled before provider admission. | +| [SDK pilot](https://github.com/paperclipai/paperclip/actions/runs/37039240025) | `4163dbfd0` | Missing daemon; stops before providers. | +| [Daemon pilot](https://github.com/paperclipai/paperclip/actions/runs/37040493183) | `ac6ddefb5` | Missing fake Codex fixture; stops before providers. | +| [Complete-build Claude pilot](https://github.com/paperclipai/paperclip/actions/runs/37041741124) | `f02d8d0df` | 537 prerequisites pass; skill document oracle fails; reported $0.175308. | +| [Initial candidate matrix](https://github.com/paperclipai/paperclip/actions/runs/37042856368) | `f02d8d0df` | 4 pass, 7 fail, 13 cancelled when a same-target-branch pilot superseded it. Original attempts retained. | +| [Candidate cancelled-cell recovery](https://github.com/paperclipai/paperclip/actions/runs/37045368302) | `f02d8d0df` | Only the 13 cancelled cells: 11 pass, 2 fail. Combined candidate has 24 results, 15 pass / 9 fail. | +| [Historical matrix](https://github.com/paperclipai/paperclip/actions/runs/37042864888) | `12c5433c6` | 15 pass, 8 recorded failures, one AWS-runner shutdown without result upload. | +| [Historical interrupted-cell recovery](https://github.com/paperclipai/paperclip/actions/runs/37048838402) | `12c5433c6` | Only legacy OpenCode ordered comments: 720.616s deadline failure; evidence and cleanup valid. Historical cohort now has 24 results, 15 pass / 9 fail. | +| [Current packaging pilot](https://github.com/paperclipai/paperclip/actions/runs/37044967981) | `1eb5ba420` | Attempt 1 stopped before cell/provider execution; attempt 2 passes, evidence valid and cleanup pass. | + +The same-target-branch workflow concurrency rule caused the candidate interruption; that orchestration mistake was acknowledged and corrected with a separate unchanged-source recovery branch. No completed model failure was retried. The historical missing cell is recovered once for a runner shutdown. Partial/failed attempts remain part of the history and unknown-spend accounting. + +The measured matrix packages contain a prerequisite folder beside the exact campaign root. The trusted selector correctly rejects this layout, so their original published reports synthesize missing-result infrastructure failures. A separate local reconstruction nests that prerequisite folder under the campaign root and verifies byte identity for every copied file; result, scorer, usage, screenshot and prerequisite content are unchanged. The unchanged trusted selector then selects 11/24 original candidate, 13/13 recovery 23/24 original historical packages and 1/1 interrupted-cell recovery; recovered normalization validates every retained result’s evidence. Original missing packages stay missing in their original campaign; the separate recovery result has its own provenance. This is declared directory-layout recovery, not a canonical successful publication or a change to the oracle. The safe JSON records result/receipt/scoring hashes and recovery-manifest hashes. + +The fix at `1eb5ba420` writes prerequisites beneath the exact campaign root. Its single selected Codex skill cell passes through the protected report and publication pipeline: [interactive pilot report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37044967981-2/index.html), [GitHub evidence](https://github.com/paperclipai/paperclip/actions/runs/37044967981#artifacts). It has `evidenceValid=true`, cleanup pass, 27.149s provider / 49.523s cell, and a reported zero subtotal with unknown billing type. It does not replace the full matched cohort or qualify untested providers. + +Original historical/recovery dashboards retain the publication failure: [historical](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37042864888-1/index.html), [candidate recovery](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37045368302-1/index.html). Inspect their access-controlled raw artifacts for measured cell evidence; do not read synthetic missing-result rows as task performance. + +## Verification and remaining qualification + +At `1eb5ba420`, all 557 credential-free prerequisites pass (556 TypeScript plus one Rust), along with 895 E2E support tests, E2E typecheck and 24-cell discovery. The cold builder prepares SDK/shared outputs and both daemon/fake protocol binaries, records hashes, and fails closed before credentials. The 313 filtered native tests are not counted as passing coverage. This head has 52 passing CI gates and two skipped gates; Greptile reviewed that exact head at 5/5 with zero open threads. Later documentation heads require fresh checks/review; measured sources remain explicit above. + +Repository typecheck and build pass. The complete local Vitest run has 14,870 passed, 83 skipped and three unrelated timing failures; the three affected files pass unchanged narrow reruns (25 auth, 16 reviewed-chat binding, four webhook tests). Original failures are retained rather than calling the first full run green. + +PR #14948 stays draft while document delivery is unresolved. Unrepresented harnesses, legacy ACP credential safety, public-receipt clipping, the independent ACP base-replacement issue, existing Claude chat memory behavior and saved-session migration are not qualified away by these results. This small matrix measures skill/context/chat behavior, not broad coding quality or statistical equivalence. + +## Retained delivery diagnosis and approved repair + +Both failed classic agents loaded the operational Paperclip skill. This is not an observed skill-discovery failure. In the reduced Claude run, the agent wrote a workspace Markdown file before loading Paperclip and then completed the issue without a document API write. The reduced OpenCode run loaded Paperclip, accessed the assigned skill through the public skills API, and wrote a workspace task-document file; it made no document, attachment, or work-product delivery write. Both corresponding historical runs saved a Paperclip issue document. + +Claude received the full operational skill, including its existing no-local-only delivery guidance. OpenCode's skill result was explicitly truncated in both variants: the early artifact rule survived, but the later generic document endpoint and plan-only write example were omitted. That truncation is existing behavior, not a newly measured regression. Removing always-on delivery reminders may interact with it, but the combined manual/shared reduction and ambiguous assigned-skill wording prevent a causal attribution to one layer. + +Dotta approved a small early API-runtime skill recipe and a generic issue-document reference. The tiny manual remains eight words. The recipe checks the successful write's returned saved revision/content and links the returned key; it does not require another GET after a clear valid receipt. It preserves explicit destinations, downloadable-file delivery, and native document-tool boundaries. + +The focused repair comparison will hold the tiny manual/shared prompts fixed and vary only `skills/paperclip/SKILL.md` plus the new `references/issue-documents.md`. Classic Claude and OpenCode each run the preserved original case and an added explicitly Paperclip-storage case: four cells per variant, eight expected provider turns in total. This measures the skill repair separately from the original reduction. The explicit case independently checks public saved body/revision and an agent comment linking the exact same-app document; local-only, missing-revision, wrong-document, local-path, and other-origin outcomes cannot pass. Current state: fixture preparation and source freeze, no repair provider run yet. diff --git a/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md b/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md index f71067ff070..38141901dce 100644 --- a/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md +++ b/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md @@ -1,7 +1,17 @@ # Stock harness, with Paperclip: working checklist -Created: 2026-10-02. Status: item 1 implemented for native Codex app-server; -PR validation in progress. Other implementation items remain open. +Created: 2026-10-02. Status: item 1 merged for native Codex app-server in +[PR #14920](https://github.com/paperclipai/paperclip/pull/14920). Item 2's default +hire manual is reduced to identity only, and common legacy startup/resume +instructions are in [PR #14948](https://github.com/paperclipai/paperclip/pull/14948). +GitHub live qualification retains earlier document-delivery failures; the latest +focused OpenCode comparison passes both cases in both variants but still exposes +deficient original-case handoff links. PR #14948 remains draft. The original +[live report](2026-10-02-stock-harness-live-comparison.md), completed +[skill repair](2026-10-02-legacy-document-skill-repair.md) and latest +[stock-selection/link comparison](2026-10-02-opencode-skill-routing-link-qualification.md) +retain their separate sources and limitations. Additional carriers and fixed +native instructions remain open; hiring templates are being reduced separately. Goal: keep the agent's stock harness behavior and add only what it needs to work with Paperclip. Apply this across legacy adapters, the new Runner, and their @@ -12,7 +22,11 @@ record the proposed behavior and Dotta's direction, make a bounded change, and verify the affected paths before moving on. New findings get stable IDs in the ledger below so they do not disappear into conversation history. -**Current item: 1 — validate and review the native Codex instruction fix.** +For every change, record its executable test/eval coverage before continuing. +Distinguish coverage setup, deterministic results, and measured live results; +configured cells alone do not qualify behavior. + +**Current item: 2 — reduce the default operating manual and shared prompt layers.** ## Agreed direction and boundaries @@ -43,7 +57,7 @@ ledger below so they do not disappear into conversation history. - [x] Remove default replacement of stock base instructions across those paths. - [x] Verify the actual app-server request and retained session instructions, including resume; checking only a prompt builder is insufficient. -- [ ] Verify Paperclip task context, tools, auth, assigned skills, and completion +- [x] Verify Paperclip task context, tools, auth, assigned skills, and completion still work. Record applicable regressions and eval results. Starting points: [Codex backend](../../packages/paperclip-runner/src/backends/codex-native-backend.ts), @@ -61,14 +75,20 @@ and need a provider session reset. Do not reset active sessions automatically. The separate Codex-through-ACP dependency patch remains a coverage follow-up under item 6; this change covers the native app-server path. +Verification: 412 focused TypeScript/Rust tests, repository typecheck/build, +55 passing PR checks, and Greptile 5/5. The original full local test command +was stopped after setup failures; affected suites passed on rerun. No paid +live campaign was run, and no task-quality improvement is claimed. + ## 2. Reduce the default operating manual and shared prompt layers - [ ] Inventory what an agent actually receives: hire instructions, shared prompt template, wake context, runtime prompt, bootstrap, and loaded skills. Separate always-present text from content loaded on demand. -- [ ] Agree on the tiny common contract and the coordination details that each - runtime still requires. Legacy API coordination and native semantic tools - need appropriate instructions for their respective interfaces. +- [x] Agree on the tiny default: identity only, with no skill or native-tool + pointers. The harness supplies coordination instructions. +- [x] Reduce the generic default hire `AGENTS.md` to the agreed sentence and + update the existing creation test to expect the minimal bundle. - [ ] Remove repeated workflow rules and stock coding/style/autonomy guidance. - [ ] Move detailed planning, hiring, artifacts, and exceptional procedures to discoverable references or tools where feasible. @@ -80,7 +100,130 @@ Starting points: [default hire instructions](../../server/src/onboarding-assets/ [native runtime contract](../../packages/paperclip-runner/src/contracts/runtime-context.ts), [Paperclip operational skill](../../skills/paperclip/SKILL.md). -Decision: exact retained paragraph and on-demand boundaries pending. +Decision: Dotta confirmed the harness already handles skill and tool delivery; +the default hire manual is now only "You are an agent in a Paperclip company." +This replaces 602 words with eight. The shared loader supplies this default +across instruction-bundle-capable adapters for non-CEO hires without explicit +instructions. Existing saved bundles retain their content; CEO, first-agent, +and role/team templates remain separate work under item 3. The common +prompt/wake reduction is complete locally below; additional carriers, native +instructions, and skill/reference corrections remain open. + +Verification: the existing agent-skills route suite passed all 54 tests, +including default creation, custom bundles, and CEO/first-agent paths. The +onboarding asset suite passed all 10 tests. This initially had only deterministic +coverage. Dotta subsequently requested behavioral eval coverage for every change; +the dedicated production-default-hire suite below closes the custom QA manual +gap. Full repository typecheck/build/test were not rerun for this narrow asset +and existing-test update. + +### Shared prompt follow-ups + +- [x] **2.1 Reduce common legacy startup/resume instructions.** Keep + identity and connection guidance in the shared task/chat defaults; remove the + generic resumed-wake execution contract. Preserve current task/event data, + specialized wake contracts, custom templates, skills, and auth. +- [ ] **2.2 Review additional legacy carriers.** Reduce Hermes local/gateway + wrappers, review Pi system delivery, and check OpenClaw fresh-wake framing. + Preserve transport facts and user configuration. Dotta deferred this item on + 2026-10-02 for a later revisit; it remains open. +- [ ] **2.3 Reduce native Runner instructions and constraints.** Improve + discoverable tool documentation first, then shorten fixed guidance and + consolidate completion rules. Verify prompt revisions, digests, and session + compatibility across native Codex, ACPX, and OpenCode. Dotta approved the + native tool-documentation slice on 2026-10-02 in a separate worktree. Native + `paperclip_finish`/`paperclip_block` guidance must not leak into legacy + completion paths, which use the operational skill and API. + +### Follow-up 2.1 implementation and verification + +The common task and conversation defaults now share the same identity sentence +and unchanged connection guidance: 113 words each, down from 661 and 205. +Connection guidance accounts for 108 of those words. The ordinary resumed-wake +execution contract is removed (172 words), including the old opt-in used by +OpenClaw on fresh turns. `includeExecutionContract` remains accepted as a +deprecated no-op for adapter/plugin source compatibility. + +Current task facts, ordered comments, work modes, approval/review state, +checkout, holds/blockers, recovery/watchdog roles, and external-chat contracts +remain in their existing owners. Skill delivery, runtime authentication, +custom `promptTemplate`, and `bootstrapPromptTemplate` mechanics are unchanged. +Hermes's own wrappers and Pi's system carrier remain follow-up 2.2; native +fixed instructions and full-turn constraints remain follow-up 2.3. + +The later read-only 2.2 audit found another OpenClaw-owned HTTP identity, +checkout/status, delegation, plan-approval and task-discovery wrapper in +`buildWakeText`. Its short conversation branch is selected when optional task +Markdown is present, including on ordinary task dispatch. Existing dispatch +coverage does not assert the absence of conversation waiting language. This is +a framing concern to verify, not a measured failure. Pi already uses additive +`--append-system-prompt` delivery and suppresses the duplicate user-prompt copy; +its next step is a delivery audit with minimal changes. These findings are +deferred with 2.2 and are not implemented in PR #14948. + +The new defaults apply when prompts are assembled after deployment. Existing +provider sessions can retain earlier startup instructions in their history +until reset; no active session or saved custom template is rewritten here. + +Verification: 563 distinct focused tests passed across 18 suites: + +- Shared prompt selection/rendering, operational-skill selection, and retained + specialized wake contracts (130 tests). +- ACPX, Pi, OpenCode, Cursor Cloud, and Codex local execution, including + task/chat fresh/resumed/reset/fallback delivery, custom prompts, skill + mounting, and authentication (259 tests). +- Hermes local prompt rendering and gateway execution (41 tests). +- Claude, Gemini, Cursor local, Grok, Kimi, OpenClaw, and Claude/Codex ACP + fallback execution (113 tests). +- Native execution input, preserving shared wake compatibility (20 tests). + +`@paperclipai/adapter-utils` typecheck and build passed; `git diff --check` +passed. Full repository typecheck/build/test and live behavioral evals were not +run for this local bounded change. These checks prove prompt and runtime +mechanics, not an improvement in coding-task quality. + +### Remaining shared layers inspected (2026-10-02) + +The table records the initial inspection before follow-up 2.1. Word counts +cover fixed source text, not complete assembled prompts or token counts. They +exclude task data, assigned agent instructions, and skills. This inspection +establishes instruction mechanics, not a performance result. + +| Layer | Delivery at initial inspection | Disposition | +| --- | --- | --- | +| Shared legacy task/chat templates | `DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE` was 661 words and the conversation template was 205, including connection guidance. Used by Claude, Codex, Cursor local/cloud, Gemini, Grok, Kimi, OpenCode, Pi, and Hermes local. | Completed in 2.1: both defaults now 113 words. Repeated procedures removed, connection guidance retained, no replacement skill pointer. | +| Generic resumed wake contract | `renderPaperclipWakePrompt` added a 172-word execution contract on ordinary resumed turns. OpenClaw requested it on fresh turns too because it has no shared template. | Completed in 2.1: generic contract removed, including legacy opt-ins. Task/event data and conditional runtime contracts retained. | +| Additional legacy carriers | Hermes local adds a 171-word identity/API/curl wrapper before the shared task template; Hermes gateway supplies its own four-rule execution contract. Pi carries shared defaults in its system extension and suppresses the wake copy when that extension owns policy. | Separate follow-up 2.2; changing the shared constants alone does not remove every wrapper. Preserve transport facts and custom configuration. | +| Native fixed prompt | The 262-word `paperclip-execution.v5` prompt carries hiring, delegation/dependencies, connection setup, and finalization guidance across native Codex, ACPX, and OpenCode backends. | Keep one short native completion/dependency contract. Relocate uncommon procedures to the relevant tool documentation, improving that documentation where needed before removing instructions. Any change needs prompt revision/digest/session-compatibility verification. | +| Native full-turn constraints | `nativeTaskConstraints` adds assigned-skill limits, current agent-file paths, document/file delivery, answered-question handling, and accepted completion/final-response sequencing. Codex/OpenCode add another completion reminder. Prepared constraints reach the model envelope through `native-session-runtime`; compact continuations avoid replaying prior task constraints. | Consolidate duplicate completion instructions. Keep current paths, answer scope, and the required result protocol. Move detailed file/document procedure to discoverable tool descriptions where adequately covered. | +| Task/event-specific framing | Server task Markdown and wake rendering supply work mode, ordered comments, approved revisions, checkout/holds/blockers, recovery/review/watchdog roles, external-chat bookkeeping/delivery, and conversation handoff. | Retain the facts and mode boundaries in the first reduction. Use one authoritative owner for shared mode directives; review specialized wording separately. | + +Custom `promptTemplate` and fresh-only `bootstrapPromptTemplate` configuration +are distinct from shipped defaults and should not be silently rewritten. +The built-in process adapter invokes a configured command; HTTP passes context +as JSON. They do not use the shared text template. External adapter plugins and +hosted configuration variants still require the broader item 6 audit. + +An accepted-plan discrepancy needs resolution: task Markdown permits cohesive +implementation on the current issue, while the wake renderer's planning-mode +accepted-confirmation branch requires children and prohibits implementation on +the source issue. Existing tests assert both versions. Production reachability +after the server changes work mode has not yet been established; do not claim +this proves a live plan-continuation failure. + +Dotta selected follow-up 2.1 first: reduce the common legacy startup/resume +copies, preserving conditional context. Additional carriers and native fixed +instructions remain distinct follow-ups 2.2 and 2.3. No shared runtime text was +changed by the initial inspection. + +Coverage identified during inspection includes shared prompt selection/rendering, +adapter execution tests, native backend/runtime context tests, and server +task-context/native-input tests. Follow-up 2.1 extended the shared and adapter +checks; its results are above. Follow-ups 2.2 and 2.3 need their corresponding +carrier/protocol assertions. Product E2E context-integrity, completion-updates, +and direct blocker guidance are candidate lifecycle checks; their current scope +does not establish coding-task quality. The initial inspection itself ran no +tests or live evals. ## 3. Review every hiring and role template @@ -107,7 +250,61 @@ Starting points: [onboarding assets](../../server/src/onboarding-assets/), [hiring skill and references](../../skills/paperclip-create-agent/), [teams catalog](../../packages/teams-catalog/catalog/). -Decision: per-template disposition and migration policy pending. +Dotta approved and merged [PR #14985](https://github.com/paperclipai/paperclip/pull/14985) +at `862a5758ba0e88a33232c1f1fa645e85c38a3113`, after corrected source +`57dcee147ed0b2d2e3cc657cd9e50fb16bf9ec25` passed full CI and fresh Greptile 5/5. +The hiring skill/references, CEO/CoS assets and team catalog are now on master. +Their live qualification remains separate from unit/CI verification. +[Prompt diffs and preserved scope](https://github.com/paperclipai/paperclip/blob/57dcee147ed0b2d2e3cc657cd9e50fb16bf9ec25/doc/plans/2026-10-02-hiring-template-prompt-diff.md). + +The read-only audit found four onboarding CEO files (1,897 words), a 164-word +CoS manual, four role examples (coder 652, QA 619, UX 1,325, security 1,724 words), +and eight team-catalog manuals. Hiring step 6, the 60–150-line role guide and +review checklist would regenerate the removed operating policies. + +- [x] **3.1** Reduce role drafting rules, coder/adapted hire examples and matching + catalog coder, preserving metadata, auth, permissions and skill selections. +- [x] **3.2** Reduce the CEO bundle/copy, including forced delegation, hiring, + memory and repeated API recipes. +- [x] **3.3** Review remaining roles/catalog and CoS copy. +- [ ] **3.4** Review specialized built-in/plugin contracts separately; retain + product-required behavior rather than assuming every rule is redundant. + +Custom and existing bundles and governance remain deliberate rollout boundaries. +Specialized Summarizer, Reflection Coach and Wiki Maintainer prompts remain +unchanged for separate product-contract review; optional Content Lead was already +tiny. The generic non-CEO fallback belongs to PR #14948. PR #14985 does not reduce +every shipped specialized agent. + +Its explicit `hiring-templates` suite selects native local Codex and ACPX Claude +`hire-coder-template-reuse` cells: a real default CEO, explicitly requested +production hiring skill/coder reference, independently scored saved JSON +artifacts and session reuse. Outcome and source/read coverage are separate; +missing or redacted receipts do not prove no regression. The existing first-task +suite covers actual wizard CoS snapshot selection. The matched union holds the +reduced manual/shared/operational skill and native completion guidance constant, +restoring only 21 historical hiring production/derived sources in baseline. +Frozen candidate `9f5404ad3aacbe76777952759414d34fd381e674` and baseline +`296a4df85e8bcc97a160fc78c291b17adb828196` passed 705 exact-source credential-free +prerequisites each (704 TypeScript + 1 Rust; zero providers), with 8,242 identical +other tracked files and matching fixture/model/auth/permissions configuration. +Only four candidate-specific single-file CEO selection assertions are filtered +symmetrically; each variant's bundle/source hashes and canonical catalog are +independently admitted. Two cells per variant, five expected turns each: +four cells / 20 turns, 15 minutes each, no automatic retries or broader selection. +The [candidate](https://github.com/paperclipai/paperclip/actions/runs/37075466208) +and [baseline](https://github.com/paperclipai/paperclip/actions/runs/37075469463) +protected campaigns are dispatched. No graded live pairs yet; pending evidence +cannot establish non-regression. +[Inspect the partial report](https://github.com/paperclipai/paperclip/blob/e3720f369df070036526dd295a7a68e49488c51a/doc/plans/2026-10-02-hiring-template-live-comparison.md). +Report-only updates preserve the measured refs and merged hiring PR. + +The default-manual PR is replayed on the merged hiring base, preserving custom +CEO bundle checks, the minimal generic boundary and both eval suites. Its prior +browser failure remains retained: a deterministic process fixture replayed its +last plan command on an asynchronous `chat_task_completed` wake and wrote a +second, text-identical source plan revision. It was not paid/provider execution; +fresh rebased CI remains required before readiness. ## 4. Fix repository context while retaining Paperclip configuration @@ -154,7 +351,7 @@ does not apply. | Legacy Codex / Claude | Local adapters, managed auth and isolated configuration | Initial inspection only | | Other legacy local adapters | ACPX, OpenCode, Pi, Cursor, Gemini, Grok, Kimi, Hermes | Pending | | Other adapter transports | Cursor Cloud, Hermes/OpenClaw gateways, process, HTTP, external adapter plugins | Pending | -| Runner Codex | App-server driver, runnerd bridge, Rust provider, direct/fallback paths | Additive instruction fix implemented; PR validation pending | +| Runner Codex | App-server driver, runnerd bridge, Rust provider, direct/fallback paths | Additive instruction fix merged in PR #14920 | | Runner ACPX | Enabled profiles, especially Claude/Grok; declared or pending profiles tracked separately | Claude initial inspection; remaining audit pending | | Runner OpenCode | Native provider and configuration paths | Pending | | Hosted/remote providers | Claude Managed and AWS AgentCore; identify their own baseline rather than assuming CLI semantics | Pending | @@ -172,15 +369,18 @@ does not apply. ## Verification and evals — apply to each item -- [ ] Map existing coverage before adding cases; use [doc/evals.md](../evals.md). +- [x] Map existing coverage for all implemented changes before adding cases; + see the SH-1–SH-3 map below and [doc/evals.md](../evals.md). Keep Runner protocol evals and Product E2E evals distinct. -- [ ] Start with narrow deterministic checks for instruction layering, effective +- [x] Set up narrow deterministic checks for the implemented instruction layering, + hire, and shared-prompt changes. Future changes still need their own map for effective configuration, skill/auth delivery, and session behavior where appropriate. -- [ ] Use Runner evals for provider/session/tool protocol changes; use Product - E2E for real hiring, repository context, task lifecycle, and artifact delivery. -- [ ] Review existing context-integrity, hiring, completion-updates, and blocker - suites for reusable coverage; record gaps rather than claiming coverage from - a similarly named case. +- [x] Reuse existing protocol tests for native Codex request layering and add Product + E2E for production-default hiring, skill delivery, task lifecycle, and chat + continuity. Repository context and additional artifact cases remain future work. +- [x] Review existing suites for reusable coverage. Their custom QA manuals did + not exercise the tiny default hire; the new suite deliberately omits those + bundles and reuses independently graded skill, continuation, and chat journeys. - [ ] Compare task quality as well as Paperclip protocol compliance when claiming that fewer instructions improve agent performance. Keep model, effort, permissions, tools, and fixture comparable. @@ -189,19 +389,71 @@ does not apply. - [ ] Run the relevant checks for each change and the repository's required full verification before a PR-ready handoff. Record unrun checks and their reasons. +### Coverage for changes implemented so far + +The [stock-harness runbook](../../tests/runner-e2e/STOCK-HARNESS.md) defines a +credential-free prerequisite plus 24 explicit local Product E2E cells across +eight legacy/native profiles. Each profile runs assigned-skill invocation, +ordered comment continuation, and chat continuity across restart. Public receipts +verify the identity-only managed bundle before provider execution; final receipts +check actual legacy prompts and both budget hard stops. The existing lifecycle +oracles still own task/chat success. Models, auth, skills, permissions, and +secret-reference plumbing are inherited from existing profiles. + +| Coverage ID | Implemented change | Executable coverage | Live status | +| --- | --- | --- | --- | +| SH-1 | Native Codex additive developer instructions, including start/resume/recovery | TypeScript driver, runnerd transport, Runner Lab/live-session, and Rust provider tests in `pnpm test:e2e:runner:stock-harness`; native Codex cells exercise real hires. | Native Codex passes all three matched journeys in both variants; #14920 held constant, so task success is not before/after vendor-base proof. | +| SH-2 | Eight-word default hire `AGENTS.md` | Public creation/onboarding tests; exact independent public bundle oracle before and after provider execution for every stock-harness cell. | Candidate public tiny-bundle and budget receipts pass in all 24 retained cells. Behavioral delivery is not fully qualified; see F14–F17. | +| SH-3 | Reduced shared task/chat defaults and removed generic resume contract | Shared renderer and ACPX/Codex/OpenCode/Pi/Hermes/Cursor Cloud regressions; actual legacy invocation prompts checked in the new suite. | Actual legacy prompts measured. Document-delivery regressions and clipped/missing receipts retained; additional carriers remain deferred item 2.2. | + +The suite is explicit-only and excluded from `--all`; it does not add paid work +to ordinary campaigns. Negative calibration covers manual regrowth, missing or +malformed prompt receipts, old startup/resume procedures, missing connection +guidance, wrong budgets, and skipped prerequisite assertions. Source revisions, +attempts, cost, partial failures, cleanup, and sanitized evidence use the existing +report pipeline. No paid providers were launched during the initial setup. +Subsequent GitHub measurements are recorded below; no broad coding-quality +improvement or full live qualification is claimed. Unrepresented harnesses remain +unqualified; Codex-through-ACP vendor-base preservation is still F7. + +Local setup verification: 478 prerequisite tests passed (477 TypeScript plus one +Rust), 860 Product E2E support tests passed, E2E typecheck passed, and all 24 cells +were discovered. The 313 unrelated native tests filtered by the gate are not +passing coverage. Final prerequisite evidence is retained at +`tests/runner-e2e/results/stock-harness-preflight-2026-10-02T16-15-39.064Z/preflight.json`; +earlier interrupted, missing-Hermes-discovery, and setup/test-timeout attempts are +retained as failures. Hermes is checked through its package config because the +root Vitest project list omits it. Later exact-head prerequisites expanded to +557 passed checks (556 TypeScript and one Rust), and E2E support to 895 tests. +Repository typecheck/build passed. The complete local Vitest run retained three +unrelated timing failures; all three files passed unchanged reruns. At +`1eb5ba420`, 52 CI gates pass and two skip; Greptile is 5/5 with zero open +threads. Later documentation heads require fresh checks. See the live report +for exact source hashes, campaigns, failures and cost limitations. + ## Findings ledger Confirmed mechanics below do not by themselves establish an effect on task quality. | ID | Finding | Work item / disposition | | --- | --- | --- | -| F1 | Native Codex sent Paperclip text as `baseInstructions`. Probes on codex-cli 0.153.4 showed replacement; additive `developerInstructions` retained the stock base on start and cold resume. | 1; app-server fix implemented, PR validation pending | +| F1 | Native Codex sent Paperclip text as `baseInstructions`. Probes on codex-cli 0.153.4 showed replacement; additive `developerInstructions` retained the stock base on start and cold resume. | 1; app-server fix merged in PR #14920 | | F2 | Default hires and role templates prescribe substantial operating procedures; common prompt and wake layers add further coordination text. | 2–3; mechanics confirmed, performance effect unmeasured | | F3 | Hiring references require legacy Paperclip skill/comment procedures, while native Runner intentionally omits that operational skill and uses semantic tools. | 2–3, 5; reconcile runtime contracts | | F4 | Local Claude appends instructions; Runner Claude preserves the Claude Code preset. Runner isolation excludes project/local settings, which can also exclude repository instruction discovery. | 4; selective context fix to design | | F5 | Some Codex capability settings differ between the direct driver and daemon path; an intermediate configuration does not prove the final provider behavior. | 6; effective-path audit pending | | F6 | Omitting or nulling `baseInstructions` on an old Codex thread's resume preserves its saved replacement; an empty string produces an empty base. | 1; document the required provider session reset; no automatic migration in this PR | | F7 | The isolated Codex-through-ACP dependency patch also sets `baseInstructions` on start/resume. It is a separate path from the native app-server backend. | 6; follow-up patch/profile audit pending | +| F8 | The operational skill says target-bound confirmations default `supersedeOnUserComment` to true; the default manual and server normalizer say false. The server uses false. | 2; correct stale skill/reference guidance in a follow-up | +| F9 | The default manual required a comment on every task, while the operational skill's verified external-chat shortcut delegates comments and lifecycle bookkeeping to the harness. | 2; unconditional manual rule removed; shared layers still need review | +| F10 | Removing the default manual left the 661-word legacy task template and its 172-word resumed-wake execution contract. Hermes and Pi have additional policy carriers. | 2.1 complete locally: task/chat defaults 113 words, no generic resume contract; 2.2 wrappers pending | +| F11 | Task Markdown and the wake renderer prescribe different accepted-plan behavior for planning-mode accepted-confirmation payloads; tests currently expect both. | 2; unify the directive owner; production reachability still to trace | +| F12 | Native fixed instructions and full-turn constraints repeat completion and uncommon procedures, while reserved finish/block tool descriptions are only one sentence each. | 2; improve tool documentation before removing needed native protocol guidance | +| F13 | Existing context-integrity/chat fixtures injected a QA manual, so their green results did not qualify the production tiny hire default. | Dedicated stock-harness suite measured real default hires; retained failures prevent blanket qualification. | +| F14 | Historical classic Claude/OpenCode skill runs save one Paperclip document; reduced runs save none while completing the task. Legacy ACP Claude has the same behavior under a separate guard failure. Pinned skill storage wording is ambiguous. | Measured delivery failures; keep original oracle. Improve legacy skill/API delivery guidance separately, then compare both variants with preserved and storage-specific cases. | +| F15 | Classic Claude restart chat fails its memory assertion in both variants. | Existing behavior, not attributable to this reduction from these trials. | +| F16 | All six legacy ACP cells per variant fail the persisted-provider-credential guard. Three historical ACP cells also have clipped public prompt retrieval, and candidate ACP Codex chat lacks a complete invocation receipt. | Security/receipt qualification follow-ups; do not waive guard, expose raw sessions, or infer missing provider instructions from clipped receipts. | +| F17 | Prerequisites beside the campaign root violate trusted artifact selection. A same-target pilot superseded 13 candidate cells; one historical AWS runner shut down without upload. | Packaging fixed at `1eb5ba420` and pilot passes. Directory-layout-only copies preserve every byte; missing cells alone recovered at unchanged source, completed failures never rerun. | Append new findings with evidence, affected paths, and the numbered item that will address them. Record intentional behavior explicitly rather than as a bug. @@ -214,6 +466,26 @@ will address them. Record intentional behavior explicitly rather than as a bug. | 2026-10-02 | Paperclip-owned MCP isolation and configuration changes/session resets are acceptable. | Preserve Paperclip auth and assigned skills while fixing repository context. | | 2026-10-02 | Dotta requested implementation and a PR for item 1. Native app-server paths now use additive developer instructions. | 139 targeted TypeScript tests and 91 Rust provider tests passed; repository typecheck/build passed. Remaining test/review results to record. | | 2026-10-02 | Verified actual Codex instruction layering using a localhost Responses stub, without paid inference. | codex-cli 0.153.4 sent identical 14,732-character stock base instructions on start and cold resume, with the Paperclip marker retained in developer input. This is protocol evidence, not a task-quality eval. | +| 2026-10-02 | Dotta requested and confirmed the merge of item 1. | PR #14920 merged at `408f70e69f9c5e49cb4377f4886ac2001bfa67a2`, with all 55 checks passing and Greptile 5/5. | +| 2026-10-02 | Dotta directed an identity-only default manual with no skill or runtime pointers; the harness already handles coordination. | Reduced the default to eight words. Existing hire and onboarding suites passed all 64 tests. Shared prompts and role templates remain pending. | +| 2026-10-02 | Dotta requested three explicit shared-prompt follow-ups and selected common legacy startup/resume reduction first. | Follow-ups 2.1–2.3 recorded. 2.1 complete locally: both defaults 113 words; generic resume contract removed. 563 focused tests and shared utility typecheck/build passed. Extra carriers and native instruction reduction remain pending. | +| 2026-10-02 | Dotta requested executable eval coverage for everything implemented so far and all subsequent changes before continuing. | Added SH-1–SH-3 coverage map, credential-free prerequisite, and 24 production-default-hire cells. 478 prerequisite and 860 support tests passed; E2E typecheck/discovery passed. Live provider results remain `not_run`; item 2.2/2.3 unchanged. | +| 2026-10-02 | Dotta requested PRs and GitHub-runner before/after qualification while discussing subsequent work separately. | Draft [PR #14948](https://github.com/paperclipai/paperclip/pull/14948); native Codex diagnostic [passed](https://github.com/paperclipai/paperclip/actions/runs/37034213743), 34.745 s provider / 56.660 s cell. Comparison restores only the prior manual/shared prompts and holds #14920 constant; no general coding-quality claim. | +| 2026-10-02 | Full candidate cold setup failed before provider admission; stopped and retained the attempt. | [Run 37037105491](https://github.com/paperclipai/paperclip/actions/runs/37037105491), target `36e987246`: prerequisite imports lacked the plugin SDK build. Added ordinary dependency setup before credential-free prerequisites and exact-source coverage of connection guidance. Cold pilot and full matched campaigns pending. | +| 2026-10-02 | Dotta deferred 2.2 additional legacy carriers and approved the first 2.3 native tool-description slice separately. | Keep wrappers open; native finish/block documentation must not be supplied to legacy skill/API completion paths. | +| 2026-10-02 | Matched default-manual/shared-prompt campaigns measured failures, with #14920 constant. | Candidate `f02d8d0df`: 15/24 pass. Historical `12c5433c6`: 15/24 pass after the one runner-shutdown recovery also timed out. Classic Claude/OpenCode Paperclip document delivery regressed in observed trials; keep PR #14948 draft. [Live report](2026-10-02-stock-harness-live-comparison.md). | +| 2026-10-02 | Cold prerequisite and packaging faults were repaired without weakening admission or behavioral graders. | Three setup attempts stopped before providers. Current `1eb5ba420` pilot passes 557 prerequisite checks and protected report publication; source/hash/cost evidence retained. Candidate cancellation and historical runner shutdown recovered only for missing cells. | +| 2026-10-02 | Final bounded recovery completed; no further model reruns. | Both matched cohorts have all 24 results, 15 pass and nine fail. Two classic skill deliveries regress; two OpenCode ordered cases pass only with reduced instructions. Overall parity is not behavioral equivalence. All 48 retained result/receipt projections are hashed and sanitized; original interruptions and partial unknown spend remain recorded. | +| 2026-10-02 | Dotta approved the measured legacy delivery repair and narrow follow-up qualification. | Early operational skill PUT/receipt/link guidance plus generic issue-document reference; eight-word manual retained. Focused original + explicit Paperclip-storage cases on classic Claude/OpenCode, with only the two skill sources varied. All four pairs completed: Claude original Fail → Pass, Claude explicit Pass → Pass, both OpenCode cases Fail → Fail. Explicit OpenCode handoff worsened beneath the unchanged machine grade (clickable API URL → code-formatted path). [Repair report](2026-10-02-legacy-document-skill-repair.md); no full matrix rerun. Claude chat-memory (F15) and ACP credential/receipt failures (F16) remain separate and unresolved. | +| 2026-10-02 | Native tool-description comparison completed all six paired cases. | Zero newly failing cases, three unchanged completion passes, three unchanged blocker failures. Claude/OpenCode blocker API matchers pass in both variants; UI matcher wrongly demanded marker-only replies. Codex's exact-action punctuation failure is unchanged. Original failures retained; corrected 10-case browser calibration and separate retained-DOM replay pass all six visible replies. Exact Codex action failures remain. No native fixed-prompt removal measured. | +| 2026-10-02 | Dotta approved minimal operational skill selection/link correction after retained OpenCode diagnosis. | Candidate `fe9dc1e3c` and baseline `0d7ecfa96d` have two matched Pass → Pass cases, zero new failures/passes and no pending pairs. All four exact-source gates pass 587 checks, all four provider runs and cleanup pass. Candidate original loads Paperclip/reference before delivery, but uses the wrong PAP prefix; baseline original saves publicly later within the same assignment and gives a bare path. Explicit clickable UI links are correct in both. [Complete report](2026-10-02-opencode-skill-routing-link-qualification.md); prior failures retained, no causal or broad quality claim. | + +- F18: Native blocker browser assertion required a marker-only reply despite asking for owner/action/reason. Correct marker-plus-explanation checks symmetrically, calibrate contradictory and future-condition replies, retain original verdicts. +- F19: Original prompt-removal cohorts loaded Paperclip; OpenCode skill truncation omitted the late API recipe. The early repair fixed Claude in one paired trial. In the repaired original OpenCode assignment, Paperclip was first loaded only during disposition recovery after a local-file write. Shared legacy operational-skill delivery/selection needs review before more recipe expansion. +- F20: Explicit repaired OpenCode reads the early recipe/reference and saves a public document/revision, but hands off a code-formatted path rather than a clickable anchor. Baseline provides a clickable API URL, rejected by the UI-only oracle. Preserve both grades; clarify future clickable UI-link fixture and review the smallest link example separately. +- F22: Legacy operational skill mounting is already mandatory, but its discovery description omitted ordinary task/heartbeat work and document delivery. The approved narrow fix expands stock metadata selection and adds a real Markdown link example in the existing reference. Original + clarified-explicit OpenCode pairs both pass in both variants, so improvement causality is not established. Candidate loads the operational skill/reference before public delivery; baseline original writes locally first, then saves publicly in the same assignment. Original candidate copies the example PAP prefix into its link while baseline supplies a bare slug path; neither is scored by the original link-free oracle. No full-body injection or native-tool leakage. ACP Claude names-only metadata, custom ACP unsupported delivery, OpenClaw wrappers and Pi HOME differences remain distinct follow-ups. +- F23: The new original OpenCode candidate copies `/PAP/issues/RUN-1#document-context-integrity-output` from the literal example despite actual prefix RUN. It passes durable-document grading but has an incorrect-prefix handoff. Baseline original also lacks a clickable canonical link. Approved reference-only correction now derives the link from `issue.identifier` and the saved receipt key. Independent provider-free calibration checks two other prefixes, redirected keys and rejects a wrong-company link; 25 focused document/source tests, canonical metadata checks and E2E typecheck pass. The recipe now also handles null/absent identifiers through the supported issue-ID route; fresh review caught the initial nullable edge. 56 focused checks pass. Source review shows the UI corrects wrong prefixes, so the frozen PAP href is noncanonical rather than proven broken. These later corrections are not live-qualified by the preserved runs; no further paid rerun. +- F21: Skill heading insertion shifted both generated capability inventories. General CI caught stale metadata after the frozen repair campaign’s narrower build admitted providers. Regenerate canonical derived files and require stale-manifest/inventory checks before future provider admission. For each completed item, add the chosen behavior, changed paths, verification results, remaining exceptions, and follow-ups here before checking it off. diff --git a/package.json b/package.json index 2fbd9d95a99..29223ed74ec 100644 --- a/package.json +++ b/package.json @@ -74,6 +74,7 @@ "test:e2e:browser-context": "playwright test --config tests/e2e/playwright-company-context.config.ts", "test:e2e:runner:browser-support": "playwright test --config tests/runner-e2e/playwright-support.config.ts", "test:e2e:runner:unit": "vitest run --config tests/runner-e2e/vitest.config.ts", + "test:e2e:runner:stock-harness": "node tests/runner-e2e/stock-harness-checks.mjs", "test:e2e:runner:typecheck": "tsc -p tests/runner-e2e/tsconfig.json", "test:e2e:runner:report": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/report.ts", "test:runner-workflow-evals": "pnpm --filter @paperclipai/paperclip-eval-kernel build && pnpm --filter @paperclipai/paperclip-runner test:runner-workflow-evals", diff --git a/packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.test.ts b/packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.test.ts new file mode 100644 index 00000000000..40d13e7b403 --- /dev/null +++ b/packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.test.ts @@ -0,0 +1,165 @@ +import { mkdtemp, readFile, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { createRuntimeStore, type AcpSessionRecord, type AcpSessionStore } from "acpx/runtime"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { createEphemeralSessionEnvironmentStore } from "./ephemeral-session-environment.js"; + +const currentEnvironment = Object.freeze({ + PAPERCLIP_API_KEY: "current-credential-fixture", + PAPERCLIP_RUN_ID: "current-run-fixture", + PATH: "/fixture/current-bin", +}); +const oldEnvironment = { PAPERCLIP_API_KEY: "old-credential-fixture", OLD_RUN_PATH: "/fixture/old-run" }; + +function record(): AcpSessionRecord { + return { + schema: "acpx.session.v1", + acpxRecordId: "fixture-record", + acpSessionId: "fixture-session", + agentSessionId: "fixture-agent-session", + agentCommand: "fixture-provider", + cwd: "/fixture/project", + name: "fixture-name", + createdAt: "2026-10-03T00:00:00.000Z", + lastUsedAt: "2026-10-03T00:00:01.000Z", + lastSeq: 3, + eventLog: { active_path: "/fixture/events.jsonl", segment_count: 1, max_segment_bytes: 1024, max_segments: 2 }, + messages: [{ User: { id: "message-1", content: [{ Text: "Retained conversation" }] } }], + updated_at: "2026-10-03T00:00:01.000Z", + cumulative_token_usage: { input_tokens: 11, output_tokens: 7 }, + request_token_usage: { request_1: { input_tokens: 11 } }, + importedFrom: { recordId: "origin-record", cwdOriginal: "/fixture/origin", exportedBy: "fixture-caller", exportedAt: "2026-10-03T00:00:00.000Z" }, + acpx: { + current_mode_id: "code", + desired_config_options: { effort: "high" }, + session_options: { model: "fixture-model", allowed_tools: ["Read", "Write"], max_turns: 4, system_prompt: { append: "Retained caller option" }, env: { ...currentEnvironment } }, + }, + }; +} +function freeze(value: T): T { + for (const child of Object.values(value)) if (typeof child === "object" && child !== null) freeze(child); + return Object.freeze(value); +} +function stubStore(loaded: AcpSessionRecord | undefined) { + const load = vi.fn().mockResolvedValue(loaded); + const save = vi.fn().mockResolvedValue(undefined); + return { load, save }; +} +const directories: string[] = []; +afterEach(async () => { await Promise.all(directories.splice(0).map(directory => rm(directory, { recursive: true, force: true }))); }); +async function diskStore() { + const stateDir = await mkdtemp(join(tmpdir(), "paperclip-session-env-")); directories.push(stateDir); + return { store: createRuntimeStore({ stateDir }), file: join(stateDir, "sessions", "fixture-record.json") }; +} + +describe("ephemeral ACPX session environment", () => { + it("saves a fresh session to the actual file store without persisting its live environment", async () => { + const disk = await diskStore(), input = freeze(record()); + const wrapper = createEphemeralSessionEnvironmentStore(disk.store, currentEnvironment); + await wrapper.save(input); + const text = await readFile(disk.file, "utf8"), saved = JSON.parse(text); + expect(saved.acpx.session_options).not.toHaveProperty("env"); + for (const value of Object.values(currentEnvironment)) expect(text).not.toContain(value); + expect(input.acpx?.session_options?.env).toEqual(currentEnvironment); + expect(saved.acpx.current_mode_id).toBe("code"); + expect(saved.acpx.desired_config_options).toEqual(input.acpx?.desired_config_options); + expect(saved.acpx.session_options).toEqual({ model: "fixture-model", allowed_tools: ["Read", "Write"], max_turns: 4, system_prompt: { append: "Retained caller option" } }); + expect(saved.messages).toEqual(input.messages); + expect(saved.imported_from.record_id).toBe("origin-record"); + const loaded = await wrapper.load(input.acpxRecordId); + expect(loaded?.acpx?.session_options?.env).toEqual(currentEnvironment); + expect(loaded?.messages).toEqual(input.messages); + }); + + it("loads an old env-bearing file with rotated credentials without writing, then saves neither environment", async () => { + const disk = await diskStore(), legacy = record(); legacy.acpx!.session_options!.env = { ...oldEnvironment }; + await disk.store.save(legacy); + const before = await readFile(disk.file, "utf8"); + const wrapper = createEphemeralSessionEnvironmentStore(disk.store, currentEnvironment); + const loaded = await wrapper.load(legacy.acpxRecordId); + expect(loaded?.acpx?.session_options?.env).toEqual(currentEnvironment); + expect(loaded?.acpx?.session_options?.env).not.toHaveProperty("OLD_RUN_PATH"); + expect(await readFile(disk.file, "utf8")).toBe(before); + expect(legacy.acpx?.session_options?.env).toEqual(oldEnvironment); + await wrapper.save(loaded!); + const after = await readFile(disk.file, "utf8"); + for (const value of [...Object.values(oldEnvironment), ...Object.values(currentEnvironment)]) expect(after).not.toContain(value); + expect(JSON.parse(after).acpx.session_options).not.toHaveProperty("env"); + expect(loaded?.acpx?.session_options?.env).toEqual(currentEnvironment); + }); + + it("loads an env-free persisted session into the current environment without writing", async () => { + const disk = await diskStore(), input = record(); delete input.acpx!.session_options!.env; + await disk.store.save(input); + const before = await readFile(disk.file, "utf8"); + const loaded = await createEphemeralSessionEnvironmentStore(disk.store, currentEnvironment).load(input.acpxRecordId); + expect(loaded?.acpx?.session_options?.env).toEqual(currentEnvironment); + expect(loaded?.acpx?.session_options?.model).toBe("fixture-model"); + expect(await readFile(disk.file, "utf8")).toBe(before); + }); + + it("copies loaded records, options and current env without mutating stored or caller-owned objects", async () => { + const stored = record(); stored.acpx!.session_options!.env = { ...oldEnvironment }; freeze(stored); + const backing = stubStore(stored), wrapper = createEphemeralSessionEnvironmentStore(backing, currentEnvironment); + const loaded = await wrapper.load("fixture-record"); + expect(backing.load).toHaveBeenCalledWith("fixture-record"); expect(backing.save).not.toHaveBeenCalled(); + expect(loaded).not.toBe(stored); expect(loaded?.acpx).not.toBe(stored.acpx); + expect(loaded?.acpx?.session_options).not.toBe(stored.acpx?.session_options); + expect(loaded?.acpx?.session_options?.env).not.toBe(currentEnvironment); + expect(stored.acpx?.session_options?.env).toEqual(oldEnvironment); + loaded!.acpx!.session_options!.env!.PAPERCLIP_API_KEY = "isolated-loaded-fixture"; + expect(currentEnvironment.PAPERCLIP_API_KEY).toBe("current-credential-fixture"); + }); + + it("saves only a copied env-free projection, preserving unrelated and caller metadata", async () => { + const input = freeze(Object.assign(record(), { caller_metadata: { keep: "caller-state", env: { preserve: "unrelated caller metadata" } } })); + const backing = stubStore(undefined), wrapper = createEphemeralSessionEnvironmentStore(backing, currentEnvironment); + await wrapper.save(input); + const saved = backing.save.mock.calls[0]![0]; + const { env: _environment, ...expectedOptions } = input.acpx!.session_options!; + expect(saved).toEqual({ ...input, acpx: { ...input.acpx, session_options: expectedOptions } }); + expect(saved).not.toBe(input); expect(saved.acpx).not.toBe(input.acpx); + expect(saved.acpx?.session_options).not.toBe(input.acpx?.session_options); + expect(saved.acpx?.session_options).not.toHaveProperty("env"); + expect(input.acpx?.session_options?.env).toEqual(currentEnvironment); + }); + + for (const [name, acpx] of [ + ["absent acpx", undefined], + ["explicit undefined acpx", undefined], + ["absent session options", { current_mode_id: "code" }], + ["undefined session options", { session_options: undefined }], + ["empty session options", { session_options: {} }], + ["undefined env", { session_options: { env: undefined, model: "fixture-model" } }], + ] as const) it(`preserves ${name} when saving and retains existing load injection semantics`, async () => { + const input = record(); + if (name === "absent acpx") delete input.acpx; else input.acpx = acpx; + freeze(input); + const backing = stubStore(input), wrapper = createEphemeralSessionEnvironmentStore(backing, currentEnvironment); + await wrapper.save(input); + const saved = backing.save.mock.calls[0]![0]; + if (name === "undefined env") expect(saved.acpx?.session_options).toEqual({ model: "fixture-model" }); + else expect(saved).toEqual(input); + expect(Object.hasOwn(saved, "acpx")).toBe(Object.hasOwn(input, "acpx")); + if (saved.acpx && input.acpx) expect(Object.hasOwn(saved.acpx, "session_options")).toBe(Object.hasOwn(input.acpx, "session_options")); + const loaded = await wrapper.load(input.acpxRecordId); + expect(loaded?.acpx?.session_options?.env).toEqual(currentEnvironment); + expect(backing.save).toHaveBeenCalledTimes(1); + }); + + it("returns a missing record without saving or creating conversation state", async () => { + const backing = stubStore(undefined); + expect(await createEphemeralSessionEnvironmentStore(backing, currentEnvironment).load("missing")).toBeUndefined(); + expect(backing.save).not.toHaveBeenCalled(); + }); + + it("propagates load and save errors unchanged", async () => { + const failure = new Error("fixture store failure"), backing = stubStore(undefined); + backing.load.mockRejectedValue(failure); backing.save.mockRejectedValue(failure); + const wrapper = createEphemeralSessionEnvironmentStore(backing, currentEnvironment); + await expect(wrapper.load("fixture-record")).rejects.toBe(failure); + const input = freeze(record()); await expect(wrapper.save(input)).rejects.toBe(failure); + expect(input.acpx?.session_options?.env).toEqual(currentEnvironment); + }); +}); diff --git a/packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.ts b/packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.ts new file mode 100644 index 00000000000..3bd59942ec9 --- /dev/null +++ b/packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.ts @@ -0,0 +1,32 @@ +import type { AcpSessionStore } from "acpx/runtime"; + +/** Keep the current run's environment in memory, outside persisted conversation state. */ +export function createEphemeralSessionEnvironmentStore( + persistedStore: AcpSessionStore, + currentEnvironment: Readonly>, +): AcpSessionStore { + return { + async load(id) { + const record = await persistedStore.load(id); + if (!record) return undefined; + return { + ...record, + acpx: { + ...record.acpx, + session_options: { + ...record.acpx?.session_options, + env: { ...currentEnvironment }, + }, + }, + }; + }, + save(record) { + const persisted = { ...record }; + if (record.acpx?.session_options !== undefined) { + const { env: _environment, ...sessionOptions } = record.acpx.session_options; + persisted.acpx = { ...record.acpx, session_options: sessionOptions }; + } + return persistedStore.save(persisted); + }, + }; +} diff --git a/packages/adapter-utils/src/acpx-engine/execute.test.ts b/packages/adapter-utils/src/acpx-engine/execute.test.ts index ffc724b4d9f..03a42f96455 100644 --- a/packages/adapter-utils/src/acpx-engine/execute.test.ts +++ b/packages/adapter-utils/src/acpx-engine/execute.test.ts @@ -2,7 +2,7 @@ import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import type { AcpRuntimeOptions } from "acpx/runtime"; +import { createRuntimeStore, type AcpRuntimeOptions, type AcpSessionRecord, type AcpSessionStore } from "acpx/runtime"; import type { AdapterExecutionContext, AdapterRuntimeMcpAccess } from "@paperclipai/adapter-utils"; import { DEFAULT_REMOTE_SANDBOX_ADAPTER_TIMEOUT_SEC, @@ -377,6 +377,42 @@ const ALLOWED_TURN_SPAN_ATTRIBUTE_KEYS = new Set([ ]); describe("shared ACPX engine runtime behavior", () => { + it("integrates the env-free store while delivering current credentials and options to the provider", async () => { + const root = await makeTempRoot(); + const stateDir = path.join(root, "state"); + const first = await runExecutor({ agent: "claude", agentCommand: "node ./fake-acp.js", cwd: root, stateDir, + env: { ANTHROPIC_API_KEY: "first-fixture-credential" } }); + const liveEnvironment = (first.sessionInputs[0]!.sessionOptions as { env: Record }).env; + expect(liveEnvironment.ANTHROPIC_API_KEY).toBe("first-fixture-credential"); + const store = first.runtimeOptions[0]!.sessionStore as AcpSessionStore; + const record: AcpSessionRecord = { + schema: "acpx.session.v1", acpxRecordId: "integration-record", acpSessionId: "integration-session", + agentCommand: "fixture-provider", cwd: root, name: "fixture", createdAt: "2026-10-03T00:00:00.000Z", + lastUsedAt: "2026-10-03T00:00:01.000Z", lastSeq: 0, + eventLog: { active_path: path.join(stateDir, "fixture.jsonl"), segment_count: 1, max_segment_bytes: 1024, max_segments: 2 }, + messages: [], updated_at: "2026-10-03T00:00:01.000Z", cumulative_token_usage: {}, request_token_usage: {}, + acpx: { session_options: { env: { ...liveEnvironment }, model: "retained-model", max_turns: 4 } }, + }; + await store.save(record); + const file = path.join(stateDir, "sessions", "integration-record.json"); + const saved = await fs.readFile(file, "utf8"); + expect(saved).not.toContain("first-fixture-credential"); + expect(JSON.parse(saved).acpx.session_options).toEqual({ model: "retained-model", max_turns: 4 }); + expect(record.acpx?.session_options?.env).toEqual(liveEnvironment); + // A file written by the prior engine is loaded with the current run's env, + // without altering the provider launch or deleting unrelated options. + record.acpx!.session_options!.env = { ANTHROPIC_API_KEY: "legacy-fixture-credential" }; + await createRuntimeStore({ stateDir }).save(record); + const second = await runExecutor({ agent: "claude", agentCommand: "node ./fake-acp.js", cwd: root, stateDir, + env: { ANTHROPIC_API_KEY: "rotated-fixture-credential" } }, { runtime: { sessionParams: first.result.sessionParams } }); + expect((second.sessionInputs[0]!.sessionOptions as { env: Record }).env.ANTHROPIC_API_KEY).toBe("rotated-fixture-credential"); + const rotatedStore = second.runtimeOptions[0]!.sessionStore as AcpSessionStore; + const loaded = await rotatedStore.load(record.acpxRecordId); + expect(loaded?.acpx?.session_options).toMatchObject({ env: { ANTHROPIC_API_KEY: "rotated-fixture-credential" }, model: "retained-model", max_turns: 4 }); + await rotatedStore.save(loaded!); + const resaved = await fs.readFile(file, "utf8"); + for (const value of ["first-fixture-credential", "legacy-fixture-credential", "rotated-fixture-credential"]) expect(resaved).not.toContain(value); + }); it.each(["claude", "codex", "gemini", "kimi", "custom"])("defaults the legacy %s engine to full auto on fresh and resumed runs", async (agent) => { const root = await makeTempRoot(); const config = { @@ -597,6 +633,9 @@ describe("shared ACPX engine runtime behavior", () => { const fresh = await runExecutor(config, { context }); expect(fresh.turnInputs).toHaveLength(1); const prompt = String(fresh.turnInputs[0]?.text ?? ""); + expect(prompt).toContain("You are agent agent-1"); + expect(prompt).toContain("Connection tools:"); + expect(prompt).not.toContain("Execution contract:"); expect(prompt).toContain(context.paperclipTaskMarkdownAssignment); expect(prompt).toContain(context.paperclipTaskCommunicationGuidance); expect(prompt).not.toContain('"objective":'); @@ -610,10 +649,18 @@ describe("shared ACPX engine runtime behavior", () => { const resumed = await runExecutor(config, { context, runtime: { sessionParams: fresh.result.sessionParams } }); expect(resumed.sessionInputs[0]?.resumeSessionId).toBe(fresh.result.sessionId); const resumedPrompt = String(resumed.turnInputs[0]?.text ?? ""); + expect(resumedPrompt).not.toContain("Execution contract:"); expect(resumedPrompt).toContain(context.paperclipTaskMarkdownAssignmentCompact); expect(resumedPrompt).not.toContain(context.paperclipTaskCommunicationGuidance); expect(resumedPrompt).not.toContain('"id":"comment-first"'); expect(resumedPrompt).toContain('"id":"comment-second"'); + const reset = await runExecutor(config, { context }); + expect(reset.sessionInputs[0]?.resumeSessionId).toBeUndefined(); + const resetPrompt = String(reset.turnInputs[0]?.text ?? ""); + expect(resetPrompt).toContain("You are agent agent-1"); + expect(resetPrompt).toContain("Connection tools:"); + expect(resetPrompt).toContain(context.paperclipTaskMarkdownAssignment); + expect(resetPrompt).not.toContain("Execution contract:"); }); it("includes Paperclip env and API access notes in the ACPX prompt without leaking the token", async () => { @@ -747,10 +794,10 @@ describe("shared ACPX engine runtime behavior", () => { expect(prompt).not.toContain("Create child issues"); expect(prompt).not.toContain("Use child issues"); } - expect(String(fresh.meta[0]?.prompt)).toContain(custom ? "Custom agent instructions." : "Continue your Paperclip conversation"); - expect(String(reset.meta[0]?.prompt)).toContain(custom ? "Custom agent instructions." : "Continue your Paperclip conversation"); + expect(String(fresh.meta[0]?.prompt)).toContain(custom ? "Custom agent instructions." : "You are agent agent-1"); + expect(String(reset.meta[0]?.prompt)).toContain(custom ? "Custom agent instructions." : "You are agent agent-1"); const ordinary = await runExecutor({ ...config, promptTemplate: "" }, { context: { ...context, conversationMode: false } }); - expect(String(ordinary.meta[0]?.prompt)).toContain("Execution contract:"); + expect(String(ordinary.meta[0]?.prompt)).not.toContain("Execution contract:"); expect(String(ordinary.meta[0]?.prompt)).toContain("Create child issues from the approved plan"); }); @@ -3773,9 +3820,54 @@ describe("ACPX engine Claude skill bundle staging (remote ACP lane)", () => { }; } + it("advertises folded routing metadata and exact staged paths without injecting skill bodies", async () => { + const root = await makeTempRoot(); + const cwd = path.join(root, "worktree"); + const stateDir = path.join(root, "state"); + await fs.mkdir(cwd, { recursive: true }); + const skill = await createSkill(path.join(root, "sources"), "paperclip--versioned", [ + "---", "description: >-", " Use for Paperclip-managed tasks", " and durable task documents.", + "---", "PRIVATE_SKILL_BODY_RECIPE", "", + ].join("\n")); + const { meta, result } = await runExecutor({ + agent: "claude", agentCommand: "node ./fake-acp.js", stateDir, cwd, + paperclipRuntimeSkills: [skill], paperclipSkillSync: { desiredSkills: [skill.key] }, + }); + const bundle = await onlyChildDir(path.join(stateDir, "runtime-skills", "claude")); + const stagedFile = path.join(bundle, ".claude", "skills", skill.runtimeName, "SKILL.md"); + const prompt = String(meta[0]?.prompt ?? ""); + expect(prompt).toContain(`- ${skill.runtimeName}: Use for Paperclip-managed tasks and durable task documents. (file: ${stagedFile})`); + expect(prompt).not.toContain("PRIVATE_SKILL_BODY_RECIPE"); + expect(prompt).not.toContain(skill.source); + expect(result.sessionParams?.skills).toMatchObject({ selectedSkills: [skill.runtimeName] }); + await expect(fs.readFile(stagedFile, "utf8")).resolves.toContain("PRIVATE_SKILL_BODY_RECIPE"); + }); + + it("bounds routing descriptions and keeps files discoverable when metadata is absent or malformed", async () => { + const root = await makeTempRoot(); + const cwd = path.join(root, "worktree"); + const stateDir = path.join(root, "state"); + await fs.mkdir(cwd, { recursive: true }); + const sources = path.join(root, "sources"); + const bounded = await createSkill(sources, "bounded", `---\ndescription: ${"x".repeat(600)}\n---\nBOUNDED_BODY\n`); + const absent = await createSkill(sources, "absent", "# ABSENT_BODY\n"); + const malformed = await createSkill(sources, "malformed", "---\ndescription: incomplete header\nMALFORMED_BODY\n"); + const skills = [bounded, absent, malformed]; + const { meta } = await runExecutor({ + agent: "claude", agentCommand: "node ./fake-acp.js", stateDir, cwd, + paperclipRuntimeSkills: skills, paperclipSkillSync: { desiredSkills: skills.map((skill) => skill.key) }, + }); + const prompt = String(meta[0]?.prompt ?? ""); + expect(prompt).toContain(`- bounded: ${"x".repeat(512)} (file: `); + expect(prompt).not.toContain("x".repeat(513)); + expect(prompt).toMatch(/- absent \(file: .*\/absent\/SKILL\.md\)/); + expect(prompt).toMatch(/- malformed \(file: .*\/malformed\/SKILL\.md\)/); + for (const body of ["BOUNDED_BODY", "ABSENT_BODY", "MALFORMED_BODY"]) expect(prompt).not.toContain(body); + }); + it("rewrites the prompt and skill identity onto the in-sandbox skill root once the bundle is staged", async () => { const { stateDir, localCwd, executionTarget } = await setupRemoteSandbox(); - const skill = await createSkill(path.join(localCwd, "skills"), "review"); + const skill = await createSkill(path.join(localCwd, "skills"), "review", "---\ndescription: Review requested task documents.\n---\n# review\nREMOTE_BODY_ONLY\n"); const { meta, result } = await runExecutor( { @@ -3798,6 +3890,8 @@ describe("ACPX engine Claude skill bundle staging (remote ACP lane)", () => { const inSandboxSkillsRoot = prompt.match(/Skill root: (\S+)/)![1]!; expect(inSandboxSkillsRoot).not.toBe(hostSkillsHome); + expect(prompt).toContain(`- review: Review requested task documents. (file: ${path.join(inSandboxSkillsRoot, "review", "SKILL.md")})`); + expect(prompt).not.toContain("REMOTE_BODY_ONLY"); await expect( fs.readFile(path.join(inSandboxSkillsRoot, skill.runtimeName, "SKILL.md"), "utf8"), ).resolves.toContain("# review"); diff --git a/packages/adapter-utils/src/acpx-engine/execute.ts b/packages/adapter-utils/src/acpx-engine/execute.ts index 5c575ca8c5b..9f25c6ba5fa 100644 --- a/packages/adapter-utils/src/acpx-engine/execute.ts +++ b/packages/adapter-utils/src/acpx-engine/execute.ts @@ -7,6 +7,7 @@ import path from "node:path"; import { execFile } from "node:child_process"; import { promisify } from "node:util"; import { createHash, randomUUID } from "node:crypto"; +import { parseFrontmatterMarkdown } from "@paperclipai/shared"; import { fileURLToPath } from "node:url"; import type { AdapterBillingType, @@ -85,6 +86,7 @@ import { type PaperclipSkillEntry, } from "@paperclipai/adapter-utils/server-utils"; import { shellQuote } from "@paperclipai/adapter-utils/ssh"; +import { createEphemeralSessionEnvironmentStore } from "./ephemeral-session-environment.js"; import { createAcpRuntime, createAgentRegistry, @@ -100,7 +102,6 @@ import { type AcpRuntimeTurnResult, type AcpRuntimeUsageBreakdown, type AcpRuntimeUsageCost, - type AcpSessionStore, } from "acpx/runtime"; import { ACPX_DUPLEX_LOSS_CANCEL_DEADLINE_MS, @@ -1183,11 +1184,24 @@ async function prepareClaudeSkillRuntime(input: { } const selectedNames = materializedNames.sort(); + const skillDescriptions = await Promise.all(selectedNames.map(async (name) => { + const file = path.join(skillsHome, name, "SKILL.md"); + let description = ""; + try { + const parsed = parseFrontmatterMarkdown(await fs.readFile(file, "utf8")); + description = asString(parsed.frontmatter.description, "").replace(/\s+/g, " ").trim().slice(0, 512); + } catch { + // A readable skill without valid routing metadata remains available. + // Do not substitute its body for a missing description. + } + return `- ${name}${description ? `: ${description}` : ""} (file: ${file})`; + })); const promptInstructions = selectedNames.length > 0 ? [ "Paperclip has materialized selected runtime skills for this ACPX Claude session.", `Skill root: ${skillsHome}`, `Selected skills: ${selectedNames.join(", ")}`, + ...skillDescriptions, "When a task calls for one of these skills, read its SKILL.md from that root and follow it.", ].join("\n") : ""; @@ -4249,26 +4263,9 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { flushChildStderr(childStderrState); childStderrState.logPath = prepared.childStderrLogPath; const persistedRuntimeStore = createRuntimeStore({ stateDir: prepared.stateDir }); - const runtimeStore: AcpSessionStore = { - async load(id) { - const record = await persistedRuntimeStore.load(id); - if (!record) return undefined; - // ACPX resumes from the stored session options rather than the - // options passed to ensureSession. Keep conversation state, but - // launch the provider with this run's credentials and scratch paths. - return { - ...record, - acpx: { - ...record.acpx, - session_options: { - ...record.acpx?.session_options, - env: { ...prepared.env }, - }, - }, - }; - }, - save: (record) => persistedRuntimeStore.save(record), - }; + // Resume with this run's launch environment; keep it out of the saved + // conversation record without mutating the live runtime's options. + const runtimeStore = createEphemeralSessionEnvironmentStore(persistedRuntimeStore, prepared.env); const runtimeOptions: PaperclipAcpRuntimeOptions = { cwd: prepared.cwd, // Host-only spawn cwd for the relay proxy on the remote process-session diff --git a/packages/adapter-utils/src/prompt-sections.test.ts b/packages/adapter-utils/src/prompt-sections.test.ts index 6c8e67192a6..bc660ccfde8 100644 --- a/packages/adapter-utils/src/prompt-sections.test.ts +++ b/packages/adapter-utils/src/prompt-sections.test.ts @@ -3,13 +3,14 @@ import { selectPaperclipPromptSections as selectSections } from "./server-utils. import { createPromptContextFixture } from "./test-fixtures/prompt-context.js"; describe("task and event section ownership", () => { - it("lets a separate instruction carrier own the execution contract on a resumed turn", () => { + it("preserves resumed wake data without reintroducing generic procedures", () => { const context = createPromptContextFixture(); - const sections = selectSections(context, { resumedSession: true, includeExecutionContract: false }); - expect(sections.taskContextNote).toBe(context.paperclipTaskMarkdownAssignmentCompact); - expect(sections.wakePrompt).toContain('"id":"comment-second"'); - expect(sections.wakePrompt).not.toContain("Execution contract:"); - expect(selectSections(context, { resumedSession: true }).wakePrompt).toContain("Execution contract:"); + for (const includeExecutionContract of [undefined, false, true]) { + const sections = selectSections(context, { resumedSession: true, includeExecutionContract }); + expect(sections.taskContextNote).toBe(context.paperclipTaskMarkdownAssignmentCompact); + expect(sections.wakePrompt).toContain('"id":"comment-second"'); + expect(sections.wakePrompt).not.toContain("Execution contract:"); + } }); it("preserves user repetition and distinct same-body comments under their source owners", () => { diff --git a/packages/adapter-utils/src/server-utils.test.ts b/packages/adapter-utils/src/server-utils.test.ts index 95a2085bdf6..c35e4e2a8f6 100644 --- a/packages/adapter-utils/src/server-utils.test.ts +++ b/packages/adapter-utils/src/server-utils.test.ts @@ -925,7 +925,7 @@ describe("renderPaperclipWakePrompt", () => { fallbackFetchNeeded: false, }; const ordinary = renderPaperclipWakePrompt(payload, { resumedSession: true }); - expect(ordinary).toContain("Execution contract:"); + expect(ordinary).not.toContain("Execution contract:"); expect(ordinary).toContain("Create child issues from the approved plan"); for (const resumedSession of [false, true]) { const chat = renderPaperclipWakePrompt(payload, { @@ -1171,7 +1171,7 @@ describe("renderPaperclipWakePrompt", () => { expect(incompleteResumePrompt).toContain( "[continuation summary truncated]", ); - expect(incompleteResumePrompt).toContain( + expect(incompleteResumePrompt).not.toContain( "a successful process exit or final response is not sufficient", ); }); @@ -1410,76 +1410,18 @@ describe("renderPaperclipWakePrompt", () => { ); }); - it("keeps the default local-agent prompt action-oriented", () => { - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain( - "Start actionable work in this heartbeat", - ); - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain( - "do not stop at a plan", - ); - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain( - "clear final disposition", - ); - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain( - "evidence, not valid liveness paths by themselves", - ); - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain( - "keep `in_progress` only when a live continuation path exists", - ); - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain( - "Prefer the smallest verification that proves the change", - ); - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain( - "After 2 consecutive failures of the same control-plane write", - ); - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain( - "adapter/runtime status channel as the sanctioned fallback", - ); - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain( - "Use child issues", - ); - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain( - "instead of polling agents, sessions, or processes", - ); - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain( - "Create child issues directly when you know what needs to be done", - ); - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain( - "POST /api/issues/$PAPERCLIP_TASK_ID/interactions", - ); - // URL paths in prompt text carry real ids or env vars, never brace - // placeholders: agents paste these lines verbatim, and a literal {issueId} - // reaches the server as /api/issues/%7BissueId%7D and 404s. - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).not.toContain( - "/api/issues/{issueId}", - ); - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).not.toContain( - "/api/issues/{id}", - ); - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain( - "kind suggest_tasks, ask_user_questions, or request_confirmation", - ); - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain( - "Use continuationPolicy wake_assignee when you need to resume after a response (it wakes on acceptance and rejection alike; only expiry does not wake); use wake_assignee_on_accept when you want to resume only after acceptance", - ); - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).not.toContain( - "for request_confirmation this resumes only after acceptance", - ); - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain( - "Never create probe or throwaway issue-thread interactions to discover the interactions API shape or your permissions", - ); - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain( - "confirmation:{issueId}:plan:{revisionId}", - ); - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain( - "Wait for acceptance before creating implementation subtasks", - ); - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain( - "Respect budget, pause/cancel, approval gates, and company boundaries", - ); + it("keeps task and chat defaults to identity and connection guidance", () => { + for (const template of [ + DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE, + DEFAULT_PAPERCLIP_CONVERSATION_PROMPT_TEMPLATE, + ]) { + expect(template).toBe( + `You are agent {{agent.id}} ({{agent.name}}).\n\n${CONNECTION_INTENT_AGENT_GUIDANCE}`, + ); + } }); - it("leaves the execution contract to the heartbeat template on fresh scoped wake prompts", () => { + it("keeps current task data without generic procedures on fresh scoped wake prompts", () => { const prompt = renderPaperclipWakePrompt({ reason: "issue_assigned", issue: { @@ -1499,12 +1441,12 @@ describe("renderPaperclipWakePrompt", () => { expect(prompt).toContain("## Paperclip Wake Payload"); expect(prompt).not.toContain("Execution contract:"); - expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain( + expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).not.toContain( "Execution contract:", ); }); - it("adds the execution contract to resume delta prompts and opted-in fresh prompts", () => { + it("does not restore generic procedures on resume or with the legacy opt-in", () => { const payload = { reason: "issue_assigned", issue: { @@ -1522,36 +1464,19 @@ describe("renderPaperclipWakePrompt", () => { fallbackFetchNeeded: false, }; - for (const prompt of [ - renderPaperclipWakePrompt(payload, { resumedSession: true }), - renderPaperclipWakePrompt(payload, { includeExecutionContract: true }), - ]) { - expect(prompt).toContain( - "Execution contract: take concrete action in this heartbeat", - ); - expect(prompt).toContain("clear final disposition"); - expect(prompt).toContain( - "Immediately before returning, verify that Paperclip records one of those dispositions", - ); - expect(prompt).toContain( - "a successful process exit or final response is not sufficient", - ); - expect(prompt).toContain( - "If no valid disposition is recorded, record it now and do not end the run", - ); - expect(prompt).toContain( - "After 2 consecutive failures of the same control-plane write", - ); - expect(prompt).toContain( - "adapter/runtime status channel as the sanctioned fallback", - ); - expect(prompt).toContain( - "evidence, not valid liveness paths by themselves", - ); - expect(prompt).toContain( - "Use child issues for long or parallel delegated work instead of polling", - ); - expect(prompt).toContain("named unblock owner/action"); + for (const resumedSession of [false, true]) { + for (const includeExecutionContract of [undefined, false, true]) { + const prompt = renderPaperclipWakePrompt(payload, { + resumedSession, includeExecutionContract, + }); + expect(prompt).toContain(resumedSession ? "## Paperclip Resume Delta" : "## Paperclip Wake Payload"); + expect(prompt).toContain("- reason: issue_assigned"); + expect(prompt).toContain("- issue: PAP-1580 Update prompts"); + expect(prompt).toContain("- issue status: in_progress"); + expect(prompt).not.toContain("Execution contract:"); + expect(prompt).not.toContain("clear final disposition"); + expect(prompt).not.toContain("do not end the run"); + } } }); @@ -1947,7 +1872,7 @@ describe("renderPaperclipWakePrompt", () => { ); }); - it("keeps exactly one execution contract in a composed fresh heartbeat prompt", () => { + it("keeps connection guidance without a generic manual in a composed fresh prompt", () => { const wakePrompt = renderPaperclipWakePrompt({ reason: "issue_assigned", issue: { @@ -1967,7 +1892,8 @@ describe("renderPaperclipWakePrompt", () => { const composed = [wakePrompt, DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE].join( "\n\n", ); - expect(composed.match(/Execution contract/g)).toHaveLength(1); + expect(composed).toContain(CONNECTION_INTENT_AGENT_GUIDANCE); + expect(composed).not.toContain("Execution contract:"); }); it("trims comment-batch boilerplate on fresh wakes with zero pending comments", () => { diff --git a/packages/adapter-utils/src/server-utils.ts b/packages/adapter-utils/src/server-utils.ts index 71fda10d4a4..6c9b2a2b138 100644 --- a/packages/adapter-utils/src/server-utils.ts +++ b/packages/adapter-utils/src/server-utils.ts @@ -208,41 +208,15 @@ export function resolvePaperclipInstanceRootForAdapter( } export const DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE = [ - "You are agent {{agent.id}} ({{agent.name}}). Continue your Paperclip work.", - "", - "Execution contract:", - "- Start actionable work in this heartbeat; do not stop at a plan unless the issue asks for planning.", - "- Leave durable progress in comments, documents, or work products, then update the issue to a clear final disposition before ending the heartbeat.", - "- Comments, documents, screenshots, work products, and `Remaining` bullets are evidence, not valid liveness paths by themselves.", - "- Final disposition checklist: mark `done` when complete; use `in_review` only with a real reviewer, approval, interaction, or monitor path; use `blocked` only with first-class blockers or a named unblock owner/action; create delegated follow-up issues with blockers when another agent owns the next step; keep `in_progress` only when a live continuation path exists.", - "- Prefer the smallest verification that proves the change; do not default to full workspace typecheck/build/test on every heartbeat unless the task scope warrants it.", - "- After 2 consecutive failures of the same control-plane write, stop retrying that write for the rest of the heartbeat. Continue useful work, report the failure in the final response, and rely on the adapter/runtime status channel as the sanctioned fallback.", - "- Use child issues for parallel or long delegated work instead of polling agents, sessions, or processes.", - "- If woken by a human comment on a dependency-blocked issue, respond or triage the comment without treating the blocked deliverable work as unblocked.", - "- Create child issues directly when you know what needs to be done; use issue-thread interactions when the board/user must choose suggested tasks, answer structured questions, or confirm a proposal.", - "- Use `PAPERCLIP_SCRATCH_DIR` / `PAPERCLIP_RUN_SCRATCH_DIR` for temporary scratch files instead of ad hoc `/tmp` paths; Paperclip removes that run-owned directory after the run ends.", - "- To ask for that input, create an interaction on the current issue with POST /api/issues/$PAPERCLIP_TASK_ID/interactions using kind suggest_tasks, ask_user_questions, or request_confirmation. Use continuationPolicy wake_assignee when you need to resume after a response (it wakes on acceptance and rejection alike; only expiry does not wake); use wake_assignee_on_accept when you want to resume only after acceptance.", - "- Never create probe or throwaway issue-thread interactions to discover the interactions API shape or your permissions; schema discovery goes through the OpenAPI spec and explicit validation errors, not placeholder cards. Every ask_user_questions, suggest_tasks, or request_confirmation you post must carry a real, answerable prompt; withdraw one you no longer need instead of leaving it pending.", - "- When you intentionally restart follow-up work on a completed assigned issue, include structured `resume: true` with the POST /api/issues/$PAPERCLIP_TASK_ID/comments or PATCH /api/issues/$PAPERCLIP_TASK_ID comment payload (substitute that issue's real id when it is not the current task). Generic agent comments on closed issues are inert by default.", - "- For plan approval, update the plan document first, then create request_confirmation targeting the latest plan revision with idempotencyKey confirmation:{issueId}:plan:{revisionId}. Wait for acceptance before creating implementation subtasks, and create a fresh confirmation after superseding board/user comments if approval is still needed.", - "- If blocked, mark the issue blocked and name the unblock owner and action.", - "- Respect budget, pause/cancel, approval gates, and company boundaries.", - "- When the server-authenticated wake payload includes an External chat response contract, that narrower contract replaces the generic Paperclip comment, status, checkout, and final-disposition steps above for that turn. Follow the external-chat contract exactly; it does not relax any permission, approval, execution-policy, containment, budget, pause/cancel, or company boundary.", + "You are agent {{agent.id}} ({{agent.name}}).", "", CONNECTION_INTENT_AGENT_GUIDANCE, ].join("\n"); -// Chat behavior is supplied centrally by the server's task-context markdown. -// Keep the ordinary task's completion/delegation contract out of this template. -export const DEFAULT_PAPERCLIP_CONVERSATION_PROMPT_TEMPLATE = [ - "You are agent {{agent.id}} ({{agent.name}}). Continue your Paperclip conversation using the supplied chat mode directive.", - "Use available tools and assigned skills as needed; respect budget, pause/cancel, approval gates, and company boundaries.", - "Prefer the smallest verification that proves the action. Use PAPERCLIP_SCRATCH_DIR / PAPERCLIP_RUN_SCRATCH_DIR for temporary scratch files.", - "After 2 consecutive failures of the same control-plane write, stop retrying that write for the rest of the turn. Report the failure honestly; never claim an unconfirmed mutation succeeded.", - "Never create probe or throwaway issue-thread interactions. Every interaction must carry a real, answerable prompt; withdraw one you no longer need.", - "", - CONNECTION_INTENT_AGENT_GUIDANCE, -].join("\n"); +// Task/chat modes arrive in server-owned context; operational procedures live +// in the harness-delivered Paperclip skill rather than either default template. +export const DEFAULT_PAPERCLIP_CONVERSATION_PROMPT_TEMPLATE = + DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE; export const WATCHDOG_DEFAULT_MANDATE = [ "You are running as a task watchdog, not as the original deliverable worker.", @@ -2242,6 +2216,7 @@ export function selectPaperclipPromptSections( options: { resumedSession?: boolean; includeCommunicationGuidance?: boolean; + /** @deprecated Generic execution instructions are no longer injected. */ includeExecutionContract?: boolean; nativeWakeReaderAvailable?: boolean; } = {}, @@ -2277,6 +2252,7 @@ function renderPaperclipWakePromptBody( value: unknown, options: { resumedSession?: boolean; + /** @deprecated Generic execution instructions are no longer injected. */ includeExecutionContract?: boolean; // Conversation policy arrives in the server-owned task markdown. Generic // task disposition and child-delegation instructions conflict with it. @@ -2301,12 +2277,6 @@ function renderPaperclipWakePromptBody( externalChatTurn || externalChatReaderTurn || externalChatQuestionResponseTurn; - // The heartbeat prompt template already carries the execution contract on - // fresh sessions; only resume deltas (which replace the template) and - // template-less adapters need the wake-payload copy. An explicit false means - // another delivery carrier owns the contract, including on resume. - const includeExecutionContract = options.conversationMode !== true && - (options.includeExecutionContract ?? resumedSession); const hasWakeCommentBatch = normalized.comments.length > 0 || normalized.includedCount > 0 || @@ -2397,7 +2367,7 @@ function renderPaperclipWakePromptBody( } }; - const executionContractLines = externalChatContract + const wakeContractLines = externalChatContract ? [ externalChatQuestionResponseTurn ? "## External chat answered-question contract" @@ -2444,12 +2414,7 @@ function renderPaperclipWakePromptBody( `Fallback preference order: (1) send back to ${originalAssigneeLabel} with a retry instruction; (2) fix the runtime/adapter/workspace problem, then send it back; (3) reassign to another agent with the right specialty; (4) convert to an explicit manual-review state for the board.`, "", ] - : includeExecutionContract - ? [ - "Execution contract: take concrete action in this heartbeat when the issue is actionable; do not stop at a plan unless planning was requested. Leave durable progress and then give the issue a clear final disposition before ending the heartbeat: `done`, `in_review` with a real reviewer/approval/interaction path, `blocked` with first-class blockers or a named unblock owner/action, delegated follow-up issues with blockers, or `in_progress` only when a live continuation path exists. Immediately before returning, verify that Paperclip records one of those dispositions; a successful process exit or final response is not sufficient. If no valid disposition is recorded, record it now and do not end the run. After 2 consecutive failures of the same control-plane write, stop retrying it for the rest of the heartbeat, continue useful work, report the failure in the final response, and rely on the adapter/runtime status channel as the sanctioned fallback. Use child issues for long or parallel delegated work instead of polling. Comments, documents, screenshots, work products, and `Remaining` bullets are evidence, not valid liveness paths by themselves.", - "", - ] - : []; + : []; const wakeSummaryLines = [ `- reason: ${normalized.reason ?? "unknown"}`, `- issue: ${normalized.issue?.identifier ?? normalized.issue?.id ?? "unknown"}${normalized.issue?.title ? ` ${normalized.issue.title}` : ""}`, @@ -2507,7 +2472,7 @@ function renderPaperclipWakePromptBody( ]), "", ...externalInteractionContinuationLines, - ...executionContractLines, + ...wakeContractLines, ...wakeSummaryLines, ] : [ @@ -2537,7 +2502,7 @@ function renderPaperclipWakePromptBody( : []), "", ...externalInteractionContinuationLines, - ...executionContractLines, + ...wakeContractLines, ...wakeSummaryLines, ]; diff --git a/packages/adapters/cursor-cloud/src/server/execute.test.ts b/packages/adapters/cursor-cloud/src/server/execute.test.ts index db4617fa816..b57f8207814 100644 --- a/packages/adapters/cursor-cloud/src/server/execute.test.ts +++ b/packages/adapters/cursor-cloud/src/server/execute.test.ts @@ -164,7 +164,7 @@ describe("cursor_cloud execute", () => { expect(result.exitCode).toBe(0); const prompt = String(sdkAgent.send.mock.calls[0]?.[0]); expect(prompt).toContain(directive); - expect(prompt).toContain(custom ? "Do the work for" : "Continue your Paperclip conversation"); + expect(prompt).toContain(custom ? "Do the work for" : "You are agent agent-1"); expect(prompt).not.toContain("Execution contract:"); expect(prompt).not.toContain("Create child issues"); }); @@ -178,6 +178,9 @@ describe("cursor_cloud execute", () => { expect(result.exitCode).toBe(0); const prompt = String(sdkAgent.send.mock.calls[0]?.[0]); expect(prompt).toContain("## Owned assignment"); + expect(prompt).toContain("You are agent agent-1 (Cursor Cloud Agent)."); + expect(prompt).toContain("Connection tools:"); + expect(prompt).not.toContain("Execution contract:"); expect(prompt.indexOf("Append the same ledger entry.")).toBeLessThan(prompt.indexOf("Change the final scope to the launch checklist.")); expect(prompt.split("Append the same ledger entry.")).toHaveLength(3); }); diff --git a/packages/adapters/hermes/src/server/prompt-rendering.test.ts b/packages/adapters/hermes/src/server/prompt-rendering.test.ts index d12da187587..b47dca03619 100644 --- a/packages/adapters/hermes/src/server/prompt-rendering.test.ts +++ b/packages/adapters/hermes/src/server/prompt-rendering.test.ts @@ -82,7 +82,7 @@ test("renders standard assignment wake with task authority and no backlog discov expect(prompt).toContain("Paperclip task context:"); expect(prompt).toContain("Add focused unit tests for assignment wake and custom prompt rendering."); expect(prompt).toContain("The harness already checked out this issue for the current run."); - expect(prompt).toContain("clear final disposition"); + expect(prompt).not.toContain("clear final disposition"); expect(prompt).not.toContain("check for unassigned issues"); expect(prompt).not.toContain("status=backlog"); }); @@ -130,8 +130,8 @@ test("renders scoped planning wake authority before the Hermes default workflow" expect(prompt).toContain("- checkout: already claimed by the harness for this run"); expect(prompt).toContain("The harness already checked out this issue for the current run."); expect(prompt).toContain("Issue description:\n```text\nUse the wake payload as runtime authority.\n```"); - expect(prompt).toContain("clear final disposition"); - expect(prompt).toContain("keep `in_progress` only when a live continuation path exists"); + expect(prompt).not.toContain("clear final disposition"); + expect(prompt).not.toContain("keep `in_progress` only when a live continuation path exists"); expect(prompt).not.toContain("check for unassigned issues"); expect(prompt).not.toContain("status=backlog"); }); diff --git a/packages/adapters/opencode-local/src/server/execute.test.ts b/packages/adapters/opencode-local/src/server/execute.test.ts index d1480d95b2d..a754b9d4b97 100644 --- a/packages/adapters/opencode-local/src/server/execute.test.ts +++ b/packages/adapters/opencode-local/src/server/execute.test.ts @@ -77,7 +77,7 @@ describe("OpenCode local skill injection", () => { }); expect(result.exitCode).toBe(0); expect(prompt).toContain(directive); - expect(prompt).toContain(custom ? "Custom agent instruction." : "Continue your Paperclip conversation"); + expect(prompt).toContain(custom ? "Custom agent instruction." : "You are agent agent-1"); expect(prompt).not.toContain("Execution contract:"); expect(prompt).not.toContain("Create child issues"); }); @@ -102,6 +102,9 @@ describe("OpenCode local skill injection", () => { expect(prompts).toHaveLength(2); expect(prompts[0]).toContain("## Compact assignment"); expect(prompts[1]).toContain("## Owned assignment"); + for (const prompt of prompts) expect(prompt).not.toContain("Execution contract:"); + expect(prompts[1]).toContain("You are agent agent-1 (OpenCode)."); + expect(prompts[1]).toContain("Connection tools:"); }); it("injects runtime skills into the configured child HOME", async () => { diff --git a/packages/adapters/pi-local/src/server/execute.remote.test.ts b/packages/adapters/pi-local/src/server/execute.remote.test.ts index 99cd81345c0..6f443902ff3 100644 --- a/packages/adapters/pi-local/src/server/execute.remote.test.ts +++ b/packages/adapters/pi-local/src/server/execute.remote.test.ts @@ -696,7 +696,7 @@ describe("pi remote execution", () => { expect(userPrompt).toBe("CUSTOM POLICY run-custom-policy"); }); - it("keeps the resumed default execution contract in the system carrier only", async () => { + it("keeps resumed default identity and connection guidance in the system carrier", async () => { const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-pi-resumed-policy-")); cleanupDirs.push(rootDir); const sessionPath = path.join(rootDir, "session.jsonl"); @@ -726,11 +726,13 @@ describe("pi remote execution", () => { const args = call?.[2] ?? []; const systemPrompt = args[args.indexOf("--append-system-prompt") + 1] ?? ""; const userPrompt = args.at(-1) ?? ""; - expect(systemPrompt).toContain("Execution contract:"); + expect(systemPrompt).toContain("You are agent agent-1 (Pi Builder)."); + expect(systemPrompt).toContain("Connection tools:"); + expect(systemPrompt).not.toContain("Execution contract:"); expect(userPrompt).not.toContain("Execution contract:"); }); - it("keeps the default contract in system input when custom prompt uses loaded instructions", async () => { + it("keeps default identity in system input when custom prompt uses loaded instructions", async () => { const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-pi-instructions-policy-")); cleanupDirs.push(rootDir); const sessionPath = path.join(rootDir, "session.jsonl"); @@ -769,7 +771,9 @@ describe("pi remote execution", () => { const systemPrompt = args[args.indexOf("--append-system-prompt") + 1] ?? ""; const userPrompt = args.at(-1) ?? ""; expect(systemPrompt).toContain("Loaded instructions for this run."); - expect(systemPrompt).toContain("Execution contract:"); + expect(systemPrompt).toContain("You are agent agent-1 (Pi Builder)."); + expect(systemPrompt).toContain("Connection tools:"); + expect(systemPrompt).not.toContain("Execution contract:"); expect(userPrompt).not.toContain("CUSTOM POLICY run-instructions-policy"); expect(userPrompt).not.toContain("Execution contract:"); }); diff --git a/packages/paperclip-runner/docs/capability-contract.md b/packages/paperclip-runner/docs/capability-contract.md index 15e6d7d4c1b..cf14bc59966 100644 --- a/packages/paperclip-runner/docs/capability-contract.md +++ b/packages/paperclip-runner/docs/capability-contract.md @@ -8,9 +8,9 @@ The skill/reference inventory and eval cases are the only normative behavior sou ## Baseline Counts -- Skill/reference headings: 157 +- Skill/reference headings: 160 - Eval cases: 106 across 16 groups -- Total normative rows: 263 +- Total normative rows: 266 - Legacy MCP aliases folded into normative rows: 42 | Eval group | Cases | @@ -43,91 +43,35 @@ The skill/reference inventory and eval cases are the only normative behavior sou | --- | --- | --- | | skill:skills/paperclip/SKILL.md:paperclip-skill:10 | optional_agent_tool | skills/paperclip/SKILL.md:10 | | skill:skills/paperclip/SKILL.md:terminology:14 | optional_agent_tool | skills/paperclip/SKILL.md:14 | -| skill:skills/paperclip/SKILL.md:authentication:18 | control_plane_owned | skills/paperclip/SKILL.md:18 | -| skill:skills/paperclip/SKILL.md:conversation-tasks:30 | optional_agent_tool | skills/paperclip/SKILL.md:30 | -| skill:skills/paperclip/SKILL.md:server-verified-external-chat-turns:47 | control_plane_owned | skills/paperclip/SKILL.md:47 | -| skill:skills/paperclip/SKILL.md:the-heartbeat-procedure:87 | optional_agent_tool | skills/paperclip/SKILL.md:87 | -| skill:skills/paperclip/SKILL.md:generated-artifacts-and-work-products:196 | always_agent_tool | skills/paperclip/SKILL.md:196 | -| skill:skills/paperclip/SKILL.md:status-quick-guide:244 | control_plane_owned | skills/paperclip/SKILL.md:244 | -| skill:skills/paperclip/SKILL.md:monitors-and-watchers-say-only-what-you-actually-scheduled:254 | optional_agent_tool | skills/paperclip/SKILL.md:254 | -| skill:skills/paperclip/SKILL.md:delegating-review-tasks:267 | always_agent_tool | skills/paperclip/SKILL.md:267 | -| skill:skills/paperclip/SKILL.md:managing-a-user-s-inbox:278 | control_plane_owned | skills/paperclip/SKILL.md:278 | -| skill:skills/paperclip/SKILL.md:issue-dependencies-blockers:286 | control_plane_owned | skills/paperclip/SKILL.md:286 | -| skill:skills/paperclip/SKILL.md:requesting-board-approval:311 | optional_agent_tool | skills/paperclip/SKILL.md:311 | -| skill:skills/paperclip/SKILL.md:issue-thread-interactions:332 | optional_agent_tool | skills/paperclip/SKILL.md:332 | -| skill:skills/paperclip/SKILL.md:standalone-decisions:361 | optional_agent_tool | skills/paperclip/SKILL.md:361 | -| skill:skills/paperclip/SKILL.md:mcp-tool-approval-gates:465 | optional_agent_tool | skills/paperclip/SKILL.md:465 | -| skill:skills/paperclip/SKILL.md:niche-workflow-pointers:507 | optional_agent_tool | skills/paperclip/SKILL.md:507 | -| skill:skills/paperclip/SKILL.md:cases:517 | optional_agent_tool | skills/paperclip/SKILL.md:517 | -| skill:skills/paperclip/SKILL.md:company-skills-workflow:522 | optional_agent_tool | skills/paperclip/SKILL.md:522 | -| skill:skills/paperclip/SKILL.md:routines:533 | optional_agent_tool | skills/paperclip/SKILL.md:533 | -| skill:skills/paperclip/SKILL.md:issue-workspace-runtime-controls:544 | optional_agent_tool | skills/paperclip/SKILL.md:544 | -| skill:skills/paperclip/SKILL.md:proposing-credentials-safely:551 | optional_agent_tool | skills/paperclip/SKILL.md:551 | -| skill:skills/paperclip/SKILL.md:reading-granted-secrets:558 | optional_agent_tool | skills/paperclip/SKILL.md:558 | -| skill:skills/paperclip/SKILL.md:critical-rules:584 | optional_agent_tool | skills/paperclip/SKILL.md:584 | -| skill:skills/paperclip/SKILL.md:comment-style-required:605 | always_agent_tool | skills/paperclip/SKILL.md:605 | -| skill:skills/paperclip/SKILL.md:update:637 | optional_agent_tool | skills/paperclip/SKILL.md:637 | -| skill:skills/paperclip/SKILL.md:planning-required-when-planning-requested:647 | optional_agent_tool | skills/paperclip/SKILL.md:647 | -| skill:skills/paperclip/SKILL.md:key-endpoints-hot-routes:680 | optional_agent_tool | skills/paperclip/SKILL.md:680 | -| skill:skills/paperclip/SKILL.md:searching-issues:709 | optional_agent_tool | skills/paperclip/SKILL.md:709 | -| skill:skills/paperclip/SKILL.md:full-reference:719 | optional_agent_tool | skills/paperclip/SKILL.md:719 | -| skill:skills/paperclip/SKILL.md:conversational-confirmation-answers:723 | always_agent_tool | skills/paperclip/SKILL.md:723 | -| skill:skills/paperclip/references/artifacts.md:generated-artifacts-and-work-products:1 | always_agent_tool | skills/paperclip/references/artifacts.md:1 | -| skill:skills/paperclip/references/artifacts.md:workspace-only-file-references:15 | optional_agent_tool | skills/paperclip/references/artifacts.md:15 | -| skill:skills/paperclip/references/cases.md:cases:1 | optional_agent_tool | skills/paperclip/references/cases.md:1 | -| skill:skills/paperclip/references/cases.md:core-model:12 | optional_agent_tool | skills/paperclip/references/cases.md:12 | -| skill:skills/paperclip/references/cases.md:upsert-semantics:29 | optional_agent_tool | skills/paperclip/references/cases.md:29 | -| skill:skills/paperclip/references/cases.md:read-and-search:67 | optional_agent_tool | skills/paperclip/references/cases.md:67 | -| skill:skills/paperclip/references/cases.md:documents:90 | always_agent_tool | skills/paperclip/references/cases.md:90 | -| skill:skills/paperclip/references/cases.md:fields:118 | optional_agent_tool | skills/paperclip/references/cases.md:118 | -| skill:skills/paperclip/references/cases.md:issue-links:151 | optional_agent_tool | skills/paperclip/references/cases.md:151 | -| skill:skills/paperclip/references/cases.md:child-cases:176 | optional_agent_tool | skills/paperclip/references/cases.md:176 | -| skill:skills/paperclip/references/cases.md:attachments:195 | optional_agent_tool | skills/paperclip/references/cases.md:195 | -| skill:skills/paperclip/references/cases.md:lifecycle:208 | optional_agent_tool | skills/paperclip/references/cases.md:208 | -| skill:skills/paperclip/references/cases.md:worked-blog-post-example:222 | optional_agent_tool | skills/paperclip/references/cases.md:222 | -| skill:skills/paperclip/references/company-skills.md:company-skills-workflow:1 | optional_agent_tool | skills/paperclip/references/company-skills.md:1 | -| skill:skills/paperclip/references/company-skills.md:what-exists:5 | optional_agent_tool | skills/paperclip/references/company-skills.md:5 | -| skill:skills/paperclip/references/company-skills.md:permission-model:22 | optional_agent_tool | skills/paperclip/references/company-skills.md:22 | -| skill:skills/paperclip/references/company-skills.md:core-endpoints:29 | optional_agent_tool | skills/paperclip/references/company-skills.md:29 | -| skill:skills/paperclip/references/company-skills.md:install-a-skill-into-the-company:65 | optional_agent_tool | skills/paperclip/references/company-skills.md:65 | -| skill:skills/paperclip/references/company-skills.md:app-shipped-catalog:75 | optional_agent_tool | skills/paperclip/references/company-skills.md:75 | -| skill:skills/paperclip/references/company-skills.md:external-source-import:101 | optional_agent_tool | skills/paperclip/references/company-skills.md:101 | -| skill:skills/paperclip/references/company-skills.md:source-types-in-order-of-preference:105 | optional_agent_tool | skills/paperclip/references/company-skills.md:105 | -| skill:skills/paperclip/references/company-skills.md:example-skills-sh-import-preferred:116 | optional_agent_tool | skills/paperclip/references/company-skills.md:116 | -| skill:skills/paperclip/references/company-skills.md:example-github-import:138 | optional_agent_tool | skills/paperclip/references/company-skills.md:138 | -| skill:skills/paperclip/references/company-skills.md:inspect-what-was-installed:164 | optional_agent_tool | skills/paperclip/references/company-skills.md:164 | -| skill:skills/paperclip/references/company-skills.md:assign-skills-to-an-existing-agent:181 | optional_agent_tool | skills/paperclip/references/company-skills.md:181 | -| skill:skills/paperclip/references/company-skills.md:include-skills-during-hire-or-create:216 | optional_agent_tool | skills/paperclip/references/company-skills.md:216 | -| skill:skills/paperclip/references/company-skills.md:notes:256 | optional_agent_tool | skills/paperclip/references/company-skills.md:256 | -| skill:skills/paperclip/references/issue-workspaces.md:issue-workspace-runtime-controls:1 | optional_agent_tool | skills/paperclip/references/issue-workspaces.md:1 | -| skill:skills/paperclip/references/issue-workspaces.md:discover-the-workspace:5 | optional_agent_tool | skills/paperclip/references/issue-workspaces.md:5 | -| skill:skills/paperclip/references/issue-workspaces.md:control-services:23 | optional_agent_tool | skills/paperclip/references/issue-workspaces.md:23 | -| skill:skills/paperclip/references/issue-workspaces.md:start-all-configured-services-waits-for-configured-readiness-checks:28 | optional_agent_tool | skills/paperclip/references/issue-workspaces.md:28 | -| skill:skills/paperclip/references/issue-workspaces.md:restart-all-configured-services:36 | optional_agent_tool | skills/paperclip/references/issue-workspaces.md:36 | -| skill:skills/paperclip/references/issue-workspaces.md:stop-all-running-services:44 | optional_agent_tool | skills/paperclip/references/issue-workspaces.md:44 | -| skill:skills/paperclip/references/issue-workspaces.md:read-the-url:63 | optional_agent_tool | skills/paperclip/references/issue-workspaces.md:63 | -| skill:skills/paperclip/references/issue-workspaces.md:mcp-tools:72 | optional_agent_tool | skills/paperclip/references/issue-workspaces.md:72 | -| skill:skills/paperclip/references/routines.md:paperclip-routines:1 | optional_agent_tool | skills/paperclip/references/routines.md:1 | -| skill:skills/paperclip/references/routines.md:lifecycle:16 | optional_agent_tool | skills/paperclip/references/routines.md:16 | -| skill:skills/paperclip/references/routines.md:creating-a-routine:27 | optional_agent_tool | skills/paperclip/references/routines.md:27 | -| skill:skills/paperclip/references/routines.md:concurrency-policies:64 | optional_agent_tool | skills/paperclip/references/routines.md:64 | -| skill:skills/paperclip/references/routines.md:catch-up-policies:76 | optional_agent_tool | skills/paperclip/references/routines.md:76 | -| skill:skills/paperclip/references/routines.md:activity-gated-scheduled-runs:87 | optional_agent_tool | skills/paperclip/references/routines.md:87 | -| skill:skills/paperclip/references/routines.md:example-skip-quiet-nights:107 | optional_agent_tool | skills/paperclip/references/routines.md:107 | -| skill:skills/paperclip/references/routines.md:adding-triggers:126 | optional_agent_tool | skills/paperclip/references/routines.md:126 | -| skill:skills/paperclip/references/routines.md:schedule-cron:136 | optional_agent_tool | skills/paperclip/references/routines.md:136 | -| skill:skills/paperclip/references/routines.md:webhook:150 | optional_agent_tool | skills/paperclip/references/routines.md:150 | -| skill:skills/paperclip/references/routines.md:api-manual-only:167 | optional_agent_tool | skills/paperclip/references/routines.md:167 | -| skill:skills/paperclip/references/routines.md:updating-and-deleting-triggers:179 | optional_agent_tool | skills/paperclip/references/routines.md:179 | -| skill:skills/paperclip/references/routines.md:manual-run:196 | optional_agent_tool | skills/paperclip/references/routines.md:196 | -| skill:skills/paperclip/references/routines.md:updating-a-routine:212 | optional_agent_tool | skills/paperclip/references/routines.md:212 | -| skill:skills/paperclip/references/routines.md:reading-routines-and-runs:223 | optional_agent_tool | skills/paperclip/references/routines.md:223 | -| skill:skills/paperclip/references/workflows.md:paperclip-workflow-playbooks:1 | optional_agent_tool | skills/paperclip/references/workflows.md:1 | -| skill:skills/paperclip/references/workflows.md:project-setup-ceo-manager:7 | optional_agent_tool | skills/paperclip/references/workflows.md:7 | -| skill:skills/paperclip/references/workflows.md:openclaw-invite-ceo:22 | optional_agent_tool | skills/paperclip/references/workflows.md:22 | -| skill:skills/paperclip/references/workflows.md:setting-agent-instructions-path:50 | optional_agent_tool | skills/paperclip/references/workflows.md:50 | -| skill:skills/paperclip/references/workflows.md:company-import-export:79 | optional_agent_tool | skills/paperclip/references/workflows.md:79 | -| skill:skills/paperclip/references/workflows.md:self-test-playbook-app-level:106 | optional_agent_tool | skills/paperclip/references/workflows.md:106 | +| skill:skills/paperclip/SKILL.md:authentication:20 | control_plane_owned | skills/paperclip/SKILL.md:20 | +| skill:skills/paperclip/SKILL.md:conversation-tasks:32 | optional_agent_tool | skills/paperclip/SKILL.md:32 | +| skill:skills/paperclip/SKILL.md:server-verified-external-chat-turns:49 | control_plane_owned | skills/paperclip/SKILL.md:49 | +| skill:skills/paperclip/SKILL.md:the-heartbeat-procedure:89 | optional_agent_tool | skills/paperclip/SKILL.md:89 | +| skill:skills/paperclip/SKILL.md:generated-artifacts-and-work-products:198 | always_agent_tool | skills/paperclip/SKILL.md:198 | +| skill:skills/paperclip/SKILL.md:status-quick-guide:246 | control_plane_owned | skills/paperclip/SKILL.md:246 | +| skill:skills/paperclip/SKILL.md:monitors-and-watchers-say-only-what-you-actually-scheduled:256 | optional_agent_tool | skills/paperclip/SKILL.md:256 | +| skill:skills/paperclip/SKILL.md:delegating-review-tasks:269 | always_agent_tool | skills/paperclip/SKILL.md:269 | +| skill:skills/paperclip/SKILL.md:managing-a-user-s-inbox:280 | control_plane_owned | skills/paperclip/SKILL.md:280 | +| skill:skills/paperclip/SKILL.md:issue-dependencies-blockers:288 | control_plane_owned | skills/paperclip/SKILL.md:288 | +| skill:skills/paperclip/SKILL.md:requesting-board-approval:313 | optional_agent_tool | skills/paperclip/SKILL.md:313 | +| skill:skills/paperclip/SKILL.md:issue-thread-interactions:334 | optional_agent_tool | skills/paperclip/SKILL.md:334 | +| skill:skills/paperclip/SKILL.md:standalone-decisions:363 | optional_agent_tool | skills/paperclip/SKILL.md:363 | +| skill:skills/paperclip/SKILL.md:mcp-tool-approval-gates:467 | optional_agent_tool | skills/paperclip/SKILL.md:467 | +| skill:skills/paperclip/SKILL.md:niche-workflow-pointers:509 | optional_agent_tool | skills/paperclip/SKILL.md:509 | +| skill:skills/paperclip/SKILL.md:cases:519 | optional_agent_tool | skills/paperclip/SKILL.md:519 | +| skill:skills/paperclip/SKILL.md:company-skills-workflow:524 | optional_agent_tool | skills/paperclip/SKILL.md:524 | +| skill:skills/paperclip/SKILL.md:routines:535 | optional_agent_tool | skills/paperclip/SKILL.md:535 | +| skill:skills/paperclip/SKILL.md:issue-workspace-runtime-controls:546 | optional_agent_tool | skills/paperclip/SKILL.md:546 | +| skill:skills/paperclip/SKILL.md:proposing-credentials-safely:553 | optional_agent_tool | skills/paperclip/SKILL.md:553 | +| skill:skills/paperclip/SKILL.md:reading-granted-secrets:560 | optional_agent_tool | skills/paperclip/SKILL.md:560 | +| skill:skills/paperclip/SKILL.md:critical-rules:586 | optional_agent_tool | skills/paperclip/SKILL.md:586 | +| skill:skills/paperclip/SKILL.md:comment-style-required:607 | always_agent_tool | skills/paperclip/SKILL.md:607 | +| skill:skills/paperclip/SKILL.md:update:639 | optional_agent_tool | skills/paperclip/SKILL.md:639 | +| skill:skills/paperclip/SKILL.md:planning-required-when-planning-requested:649 | optional_agent_tool | skills/paperclip/SKILL.md:649 | +| skill:skills/paperclip/SKILL.md:key-endpoints-hot-routes:682 | optional_agent_tool | skills/paperclip/SKILL.md:682 | +| skill:skills/paperclip/SKILL.md:searching-issues:711 | optional_agent_tool | skills/paperclip/SKILL.md:711 | +| skill:skills/paperclip/SKILL.md:full-reference:721 | optional_agent_tool | skills/paperclip/SKILL.md:721 | +| skill:skills/paperclip/SKILL.md:conversational-confirmation-answers:725 | always_agent_tool | skills/paperclip/SKILL.md:725 | | skill:skills/paperclip/references/api-reference.md:paperclip-api-reference:1 | optional_agent_tool | skills/paperclip/references/api-reference.md:1 | | skill:skills/paperclip/references/api-reference.md:response-schemas:9 | optional_agent_tool | skills/paperclip/references/api-reference.md:9 | | skill:skills/paperclip/references/api-reference.md:agent-record-get-api-agents-me-or-get-api-agents-agentid:11 | optional_agent_tool | skills/paperclip/references/api-reference.md:11 | @@ -198,6 +142,65 @@ The skill/reference inventory and eval cases are the only normative behavior sou | skill:skills/paperclip/references/api-reference.md:agent-secret-proposals:1521 | optional_agent_tool | skills/paperclip/references/api-reference.md:1521 | | skill:skills/paperclip/references/api-reference.md:agent-secret-access:1621 | optional_agent_tool | skills/paperclip/references/api-reference.md:1621 | | skill:skills/paperclip/references/api-reference.md:common-mistakes:1661 | optional_agent_tool | skills/paperclip/references/api-reference.md:1661 | +| skill:skills/paperclip/references/artifacts.md:generated-artifacts-and-work-products:1 | always_agent_tool | skills/paperclip/references/artifacts.md:1 | +| skill:skills/paperclip/references/artifacts.md:workspace-only-file-references:15 | optional_agent_tool | skills/paperclip/references/artifacts.md:15 | +| skill:skills/paperclip/references/cases.md:cases:1 | optional_agent_tool | skills/paperclip/references/cases.md:1 | +| skill:skills/paperclip/references/cases.md:core-model:12 | optional_agent_tool | skills/paperclip/references/cases.md:12 | +| skill:skills/paperclip/references/cases.md:upsert-semantics:29 | optional_agent_tool | skills/paperclip/references/cases.md:29 | +| skill:skills/paperclip/references/cases.md:read-and-search:67 | optional_agent_tool | skills/paperclip/references/cases.md:67 | +| skill:skills/paperclip/references/cases.md:documents:90 | always_agent_tool | skills/paperclip/references/cases.md:90 | +| skill:skills/paperclip/references/cases.md:fields:118 | optional_agent_tool | skills/paperclip/references/cases.md:118 | +| skill:skills/paperclip/references/cases.md:issue-links:151 | optional_agent_tool | skills/paperclip/references/cases.md:151 | +| skill:skills/paperclip/references/cases.md:child-cases:176 | optional_agent_tool | skills/paperclip/references/cases.md:176 | +| skill:skills/paperclip/references/cases.md:attachments:195 | optional_agent_tool | skills/paperclip/references/cases.md:195 | +| skill:skills/paperclip/references/cases.md:lifecycle:208 | optional_agent_tool | skills/paperclip/references/cases.md:208 | +| skill:skills/paperclip/references/cases.md:worked-blog-post-example:222 | optional_agent_tool | skills/paperclip/references/cases.md:222 | +| skill:skills/paperclip/references/company-skills.md:company-skills-workflow:1 | optional_agent_tool | skills/paperclip/references/company-skills.md:1 | +| skill:skills/paperclip/references/company-skills.md:what-exists:5 | optional_agent_tool | skills/paperclip/references/company-skills.md:5 | +| skill:skills/paperclip/references/company-skills.md:permission-model:22 | optional_agent_tool | skills/paperclip/references/company-skills.md:22 | +| skill:skills/paperclip/references/company-skills.md:core-endpoints:29 | optional_agent_tool | skills/paperclip/references/company-skills.md:29 | +| skill:skills/paperclip/references/company-skills.md:install-a-skill-into-the-company:65 | optional_agent_tool | skills/paperclip/references/company-skills.md:65 | +| skill:skills/paperclip/references/company-skills.md:app-shipped-catalog:75 | optional_agent_tool | skills/paperclip/references/company-skills.md:75 | +| skill:skills/paperclip/references/company-skills.md:external-source-import:101 | optional_agent_tool | skills/paperclip/references/company-skills.md:101 | +| skill:skills/paperclip/references/company-skills.md:source-types-in-order-of-preference:105 | optional_agent_tool | skills/paperclip/references/company-skills.md:105 | +| skill:skills/paperclip/references/company-skills.md:example-skills-sh-import-preferred:116 | optional_agent_tool | skills/paperclip/references/company-skills.md:116 | +| skill:skills/paperclip/references/company-skills.md:example-github-import:138 | optional_agent_tool | skills/paperclip/references/company-skills.md:138 | +| skill:skills/paperclip/references/company-skills.md:inspect-what-was-installed:164 | optional_agent_tool | skills/paperclip/references/company-skills.md:164 | +| skill:skills/paperclip/references/company-skills.md:assign-skills-to-an-existing-agent:181 | optional_agent_tool | skills/paperclip/references/company-skills.md:181 | +| skill:skills/paperclip/references/company-skills.md:include-skills-during-hire-or-create:216 | optional_agent_tool | skills/paperclip/references/company-skills.md:216 | +| skill:skills/paperclip/references/company-skills.md:notes:256 | optional_agent_tool | skills/paperclip/references/company-skills.md:256 | +| skill:skills/paperclip/references/issue-documents.md:issue-documents-through-the-api:1 | always_agent_tool | skills/paperclip/references/issue-documents.md:1 | +| skill:skills/paperclip/references/issue-documents.md:create-a-document:8 | always_agent_tool | skills/paperclip/references/issue-documents.md:8 | +| skill:skills/paperclip/references/issue-documents.md:update-or-resolve-an-unclear-write:49 | optional_agent_tool | skills/paperclip/references/issue-documents.md:49 | +| skill:skills/paperclip/references/issue-workspaces.md:issue-workspace-runtime-controls:1 | optional_agent_tool | skills/paperclip/references/issue-workspaces.md:1 | +| skill:skills/paperclip/references/issue-workspaces.md:discover-the-workspace:5 | optional_agent_tool | skills/paperclip/references/issue-workspaces.md:5 | +| skill:skills/paperclip/references/issue-workspaces.md:control-services:23 | optional_agent_tool | skills/paperclip/references/issue-workspaces.md:23 | +| skill:skills/paperclip/references/issue-workspaces.md:start-all-configured-services-waits-for-configured-readiness-checks:28 | optional_agent_tool | skills/paperclip/references/issue-workspaces.md:28 | +| skill:skills/paperclip/references/issue-workspaces.md:restart-all-configured-services:36 | optional_agent_tool | skills/paperclip/references/issue-workspaces.md:36 | +| skill:skills/paperclip/references/issue-workspaces.md:stop-all-running-services:44 | optional_agent_tool | skills/paperclip/references/issue-workspaces.md:44 | +| skill:skills/paperclip/references/issue-workspaces.md:read-the-url:63 | optional_agent_tool | skills/paperclip/references/issue-workspaces.md:63 | +| skill:skills/paperclip/references/issue-workspaces.md:mcp-tools:72 | optional_agent_tool | skills/paperclip/references/issue-workspaces.md:72 | +| skill:skills/paperclip/references/routines.md:paperclip-routines:1 | optional_agent_tool | skills/paperclip/references/routines.md:1 | +| skill:skills/paperclip/references/routines.md:lifecycle:16 | optional_agent_tool | skills/paperclip/references/routines.md:16 | +| skill:skills/paperclip/references/routines.md:creating-a-routine:27 | optional_agent_tool | skills/paperclip/references/routines.md:27 | +| skill:skills/paperclip/references/routines.md:concurrency-policies:64 | optional_agent_tool | skills/paperclip/references/routines.md:64 | +| skill:skills/paperclip/references/routines.md:catch-up-policies:76 | optional_agent_tool | skills/paperclip/references/routines.md:76 | +| skill:skills/paperclip/references/routines.md:activity-gated-scheduled-runs:87 | optional_agent_tool | skills/paperclip/references/routines.md:87 | +| skill:skills/paperclip/references/routines.md:example-skip-quiet-nights:107 | optional_agent_tool | skills/paperclip/references/routines.md:107 | +| skill:skills/paperclip/references/routines.md:adding-triggers:126 | optional_agent_tool | skills/paperclip/references/routines.md:126 | +| skill:skills/paperclip/references/routines.md:schedule-cron:136 | optional_agent_tool | skills/paperclip/references/routines.md:136 | +| skill:skills/paperclip/references/routines.md:webhook:150 | optional_agent_tool | skills/paperclip/references/routines.md:150 | +| skill:skills/paperclip/references/routines.md:api-manual-only:167 | optional_agent_tool | skills/paperclip/references/routines.md:167 | +| skill:skills/paperclip/references/routines.md:updating-and-deleting-triggers:179 | optional_agent_tool | skills/paperclip/references/routines.md:179 | +| skill:skills/paperclip/references/routines.md:manual-run:196 | optional_agent_tool | skills/paperclip/references/routines.md:196 | +| skill:skills/paperclip/references/routines.md:updating-a-routine:212 | optional_agent_tool | skills/paperclip/references/routines.md:212 | +| skill:skills/paperclip/references/routines.md:reading-routines-and-runs:223 | optional_agent_tool | skills/paperclip/references/routines.md:223 | +| skill:skills/paperclip/references/workflows.md:paperclip-workflow-playbooks:1 | optional_agent_tool | skills/paperclip/references/workflows.md:1 | +| skill:skills/paperclip/references/workflows.md:project-setup-ceo-manager:7 | optional_agent_tool | skills/paperclip/references/workflows.md:7 | +| skill:skills/paperclip/references/workflows.md:openclaw-invite-ceo:22 | optional_agent_tool | skills/paperclip/references/workflows.md:22 | +| skill:skills/paperclip/references/workflows.md:setting-agent-instructions-path:50 | optional_agent_tool | skills/paperclip/references/workflows.md:50 | +| skill:skills/paperclip/references/workflows.md:company-import-export:79 | optional_agent_tool | skills/paperclip/references/workflows.md:79 | +| skill:skills/paperclip/references/workflows.md:self-test-playbook-app-level:106 | optional_agent_tool | skills/paperclip/references/workflows.md:106 | ## Legacy MCP Alias Index diff --git a/packages/paperclip-runner/generated/capability/capabilities.yaml b/packages/paperclip-runner/generated/capability/capabilities.yaml index 61bba278fd3..a2557fcd05c 100644 --- a/packages/paperclip-runner/generated/capability/capabilities.yaml +++ b/packages/paperclip-runner/generated/capability/capabilities.yaml @@ -20,261 +20,261 @@ "expectedMockState": "runtime_decision_record" }, { - "id": "skill:skills/paperclip/SKILL.md:18", + "id": "skill:skills/paperclip/SKILL.md:20", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L18:authentication", + "sourceAnchor": "skills/paperclip/SKILL.md#L20:authentication", "heading": "Authentication", "primaryDisposition": "control_plane_owned", "semanticOperation": "runtime_reconciliation", "expectedMockState": "runtime_decision_record" }, { - "id": "skill:skills/paperclip/SKILL.md:30", + "id": "skill:skills/paperclip/SKILL.md:32", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L30:conversation-tasks", + "sourceAnchor": "skills/paperclip/SKILL.md#L32:conversation-tasks", "heading": "Conversation tasks", "primaryDisposition": "control_plane_owned", "semanticOperation": "runtime_reconciliation", "expectedMockState": "runtime_decision_record" }, { - "id": "skill:skills/paperclip/SKILL.md:47", + "id": "skill:skills/paperclip/SKILL.md:49", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L47:server-verified-external-chat-turns", + "sourceAnchor": "skills/paperclip/SKILL.md#L49:server-verified-external-chat-turns", "heading": "Server-Verified External Chat Turns", "primaryDisposition": "control_plane_owned", "semanticOperation": "runtime_reconciliation", "expectedMockState": "runtime_decision_record" }, { - "id": "skill:skills/paperclip/SKILL.md:87", + "id": "skill:skills/paperclip/SKILL.md:89", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L87:the-heartbeat-procedure", + "sourceAnchor": "skills/paperclip/SKILL.md#L89:the-heartbeat-procedure", "heading": "The Heartbeat Procedure", "primaryDisposition": "control_plane_owned", "semanticOperation": "runtime_reconciliation", "expectedMockState": "runtime_decision_record" }, { - "id": "skill:skills/paperclip/SKILL.md:196", + "id": "skill:skills/paperclip/SKILL.md:198", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L196:generated-artifacts-and-work-products", + "sourceAnchor": "skills/paperclip/SKILL.md#L198:generated-artifacts-and-work-products", "heading": "Generated Artifacts and Work Products", "primaryDisposition": "always_agent_tool", "semanticOperation": "register_deliverable", "expectedMockState": "operation_result" }, { - "id": "skill:skills/paperclip/SKILL.md:244", + "id": "skill:skills/paperclip/SKILL.md:246", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L244:status-quick-guide", + "sourceAnchor": "skills/paperclip/SKILL.md#L246:status-quick-guide", "heading": "Status Quick Guide", "primaryDisposition": "control_plane_owned", "semanticOperation": "runtime_reconciliation", "expectedMockState": "runtime_decision_record" }, { - "id": "skill:skills/paperclip/SKILL.md:254", + "id": "skill:skills/paperclip/SKILL.md:256", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L254:monitors-and-watchers-say-only-what-you-actually-scheduled", + "sourceAnchor": "skills/paperclip/SKILL.md#L256:monitors-and-watchers-say-only-what-you-actually-scheduled", "heading": "Monitors and Watchers (say only what you actually scheduled)", "primaryDisposition": "control_plane_owned", "semanticOperation": "runtime_reconciliation", "expectedMockState": "runtime_decision_record" }, { - "id": "skill:skills/paperclip/SKILL.md:267", + "id": "skill:skills/paperclip/SKILL.md:269", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L267:delegating-review-tasks", + "sourceAnchor": "skills/paperclip/SKILL.md#L269:delegating-review-tasks", "heading": "Delegating review tasks", "primaryDisposition": "control_plane_owned", "semanticOperation": "runtime_reconciliation", "expectedMockState": "runtime_decision_record" }, { - "id": "skill:skills/paperclip/SKILL.md:278", + "id": "skill:skills/paperclip/SKILL.md:280", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L278:managing-a-user-s-inbox", + "sourceAnchor": "skills/paperclip/SKILL.md#L280:managing-a-user-s-inbox", "heading": "Managing A User's Inbox", "primaryDisposition": "control_plane_owned", "semanticOperation": "runtime_reconciliation", "expectedMockState": "runtime_decision_record" }, { - "id": "skill:skills/paperclip/SKILL.md:286", + "id": "skill:skills/paperclip/SKILL.md:288", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L286:issue-dependencies-blockers", + "sourceAnchor": "skills/paperclip/SKILL.md#L288:issue-dependencies-blockers", "heading": "Issue Dependencies (Blockers)", "primaryDisposition": "control_plane_owned", "semanticOperation": "runtime_reconciliation", "expectedMockState": "runtime_decision_record" }, { - "id": "skill:skills/paperclip/SKILL.md:311", + "id": "skill:skills/paperclip/SKILL.md:313", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L311:requesting-board-approval", + "sourceAnchor": "skills/paperclip/SKILL.md#L313:requesting-board-approval", "heading": "Requesting Board Approval", "primaryDisposition": "optional_agent_tool", "semanticOperation": "scoped_discovery", "expectedMockState": "operation_result" }, { - "id": "skill:skills/paperclip/SKILL.md:332", + "id": "skill:skills/paperclip/SKILL.md:334", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L332:issue-thread-interactions", + "sourceAnchor": "skills/paperclip/SKILL.md#L334:issue-thread-interactions", "heading": "Issue-Thread Interactions", "primaryDisposition": "always_agent_tool", "semanticOperation": "request_human_input", "expectedMockState": "operation_result" }, { - "id": "skill:skills/paperclip/SKILL.md:361", + "id": "skill:skills/paperclip/SKILL.md:363", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L361:standalone-decisions", + "sourceAnchor": "skills/paperclip/SKILL.md#L363:standalone-decisions", "heading": "Standalone Decisions", "primaryDisposition": "control_plane_owned", "semanticOperation": "runtime_reconciliation", "expectedMockState": "runtime_decision_record" }, { - "id": "skill:skills/paperclip/SKILL.md:465", + "id": "skill:skills/paperclip/SKILL.md:467", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L465:mcp-tool-approval-gates", + "sourceAnchor": "skills/paperclip/SKILL.md#L467:mcp-tool-approval-gates", "heading": "MCP Tool Approval Gates", "primaryDisposition": "optional_agent_tool", "semanticOperation": "scoped_discovery", "expectedMockState": "operation_result" }, { - "id": "skill:skills/paperclip/SKILL.md:507", + "id": "skill:skills/paperclip/SKILL.md:509", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L507:niche-workflow-pointers", + "sourceAnchor": "skills/paperclip/SKILL.md#L509:niche-workflow-pointers", "heading": "Niche Workflow Pointers", "primaryDisposition": "control_plane_owned", "semanticOperation": "runtime_reconciliation", "expectedMockState": "runtime_decision_record" }, { - "id": "skill:skills/paperclip/SKILL.md:517", + "id": "skill:skills/paperclip/SKILL.md:519", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L517:cases", + "sourceAnchor": "skills/paperclip/SKILL.md#L519:cases", "heading": "Cases", "primaryDisposition": "optional_agent_tool", "semanticOperation": "scoped_discovery", "expectedMockState": "operation_result" }, { - "id": "skill:skills/paperclip/SKILL.md:522", + "id": "skill:skills/paperclip/SKILL.md:524", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L522:company-skills-workflow", + "sourceAnchor": "skills/paperclip/SKILL.md#L524:company-skills-workflow", "heading": "Company Skills Workflow", "primaryDisposition": "optional_agent_tool", "semanticOperation": "scoped_discovery", "expectedMockState": "operation_result" }, { - "id": "skill:skills/paperclip/SKILL.md:533", + "id": "skill:skills/paperclip/SKILL.md:535", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L533:routines", + "sourceAnchor": "skills/paperclip/SKILL.md#L535:routines", "heading": "Routines", "primaryDisposition": "optional_agent_tool", "semanticOperation": "scoped_discovery", "expectedMockState": "operation_result" }, { - "id": "skill:skills/paperclip/SKILL.md:544", + "id": "skill:skills/paperclip/SKILL.md:546", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L544:issue-workspace-runtime-controls", + "sourceAnchor": "skills/paperclip/SKILL.md#L546:issue-workspace-runtime-controls", "heading": "Issue Workspace Runtime Controls", "primaryDisposition": "optional_agent_tool", "semanticOperation": "scoped_discovery", "expectedMockState": "operation_result" }, { - "id": "skill:skills/paperclip/SKILL.md:551", + "id": "skill:skills/paperclip/SKILL.md:553", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L551:proposing-credentials-safely", + "sourceAnchor": "skills/paperclip/SKILL.md#L553:proposing-credentials-safely", "heading": "Proposing Credentials Safely", "primaryDisposition": "control_plane_owned", "semanticOperation": "runtime_reconciliation", "expectedMockState": "runtime_decision_record" }, { - "id": "skill:skills/paperclip/SKILL.md:558", + "id": "skill:skills/paperclip/SKILL.md:560", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L558:reading-granted-secrets", + "sourceAnchor": "skills/paperclip/SKILL.md#L560:reading-granted-secrets", "heading": "Reading Granted Secrets", "primaryDisposition": "optional_agent_tool", "semanticOperation": "scoped_discovery", "expectedMockState": "operation_result" }, { - "id": "skill:skills/paperclip/SKILL.md:584", + "id": "skill:skills/paperclip/SKILL.md:586", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L584:critical-rules", + "sourceAnchor": "skills/paperclip/SKILL.md#L586:critical-rules", "heading": "Critical Rules", "primaryDisposition": "control_plane_owned", "semanticOperation": "runtime_reconciliation", "expectedMockState": "runtime_decision_record" }, { - "id": "skill:skills/paperclip/SKILL.md:605", + "id": "skill:skills/paperclip/SKILL.md:607", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L605:comment-style-required", + "sourceAnchor": "skills/paperclip/SKILL.md#L607:comment-style-required", "heading": "Comment Style (Required)", "primaryDisposition": "always_agent_tool", "semanticOperation": "report_progress", "expectedMockState": "operation_result" }, { - "id": "skill:skills/paperclip/SKILL.md:637", + "id": "skill:skills/paperclip/SKILL.md:639", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L637:update", + "sourceAnchor": "skills/paperclip/SKILL.md#L639:update", "heading": "Update", "primaryDisposition": "control_plane_owned", "semanticOperation": "runtime_reconciliation", "expectedMockState": "runtime_decision_record" }, { - "id": "skill:skills/paperclip/SKILL.md:647", + "id": "skill:skills/paperclip/SKILL.md:649", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L647:planning-required-when-planning-requested", + "sourceAnchor": "skills/paperclip/SKILL.md#L649:planning-required-when-planning-requested", "heading": "Planning (Required when planning requested)", "primaryDisposition": "always_agent_tool", "semanticOperation": "write_document", "expectedMockState": "operation_result" }, { - "id": "skill:skills/paperclip/SKILL.md:680", + "id": "skill:skills/paperclip/SKILL.md:682", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L680:key-endpoints-hot-routes", + "sourceAnchor": "skills/paperclip/SKILL.md#L682:key-endpoints-hot-routes", "heading": "Key Endpoints (Hot Routes)", "primaryDisposition": "control_plane_owned", "semanticOperation": "runtime_reconciliation", "expectedMockState": "runtime_decision_record" }, { - "id": "skill:skills/paperclip/SKILL.md:709", + "id": "skill:skills/paperclip/SKILL.md:711", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L709:searching-issues", + "sourceAnchor": "skills/paperclip/SKILL.md#L711:searching-issues", "heading": "Searching Issues", "primaryDisposition": "control_plane_owned", "semanticOperation": "runtime_reconciliation", "expectedMockState": "runtime_decision_record" }, { - "id": "skill:skills/paperclip/SKILL.md:719", + "id": "skill:skills/paperclip/SKILL.md:721", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L719:full-reference", + "sourceAnchor": "skills/paperclip/SKILL.md#L721:full-reference", "heading": "Full Reference", "primaryDisposition": "control_plane_owned", "semanticOperation": "runtime_reconciliation", "expectedMockState": "runtime_decision_record" }, { - "id": "skill:skills/paperclip/SKILL.md:723", + "id": "skill:skills/paperclip/SKILL.md:725", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L723:conversational-confirmation-answers", + "sourceAnchor": "skills/paperclip/SKILL.md#L725:conversational-confirmation-answers", "heading": "Conversational confirmation answers", "primaryDisposition": "always_agent_tool", "semanticOperation": "call_api", @@ -1162,6 +1162,33 @@ "semanticOperation": "runtime_reconciliation", "expectedMockState": "runtime_decision_record" }, + { + "id": "skill:skills/paperclip/references/issue-documents.md:1", + "kind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/issue-documents.md#L1:issue-documents-through-the-api", + "heading": "Issue documents through the API", + "primaryDisposition": "always_agent_tool", + "semanticOperation": "write_document", + "expectedMockState": "operation_result" + }, + { + "id": "skill:skills/paperclip/references/issue-documents.md:8", + "kind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/issue-documents.md#L8:create-a-document", + "heading": "Create a document", + "primaryDisposition": "always_agent_tool", + "semanticOperation": "write_document", + "expectedMockState": "operation_result" + }, + { + "id": "skill:skills/paperclip/references/issue-documents.md:49", + "kind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/issue-documents.md#L49:update-or-resolve-an-unclear-write", + "heading": "Update or resolve an unclear write", + "primaryDisposition": "control_plane_owned", + "semanticOperation": "runtime_reconciliation", + "expectedMockState": "runtime_decision_record" + }, { "id": "skill:skills/paperclip/references/issue-workspaces.md:1", "kind": "skill_heading", diff --git a/packages/paperclip-runner/generated/capability/capability-contract.md b/packages/paperclip-runner/generated/capability/capability-contract.md index 907bf749bd5..4d124613015 100644 --- a/packages/paperclip-runner/generated/capability/capability-contract.md +++ b/packages/paperclip-runner/generated/capability/capability-contract.md @@ -2,9 +2,9 @@ Generated by `scripts/generate-capability-contract.mjs`; do not edit generated files. -- Skill/reference headings: 158 +- Skill/reference headings: 161 - Legacy MCP tools: 42 - Eval cases: 106 across 16 groups -- Deterministic content SHA-256: `2707de482cfbe6c7a483f6dee3bf4a2fc66e2b944bb50c3818ca6d5ef777b1de` +- Deterministic content SHA-256: `e577ca3ad6e6b045a7413ac11008013c8c12c2dc954f5a7876b86d96fbe017e8` Every row has exactly one primary disposition, a source anchor, a semantic operation, and a mock-state expectation. diff --git a/packages/paperclip-runner/scripts/check-capability-inventory.test.mjs b/packages/paperclip-runner/scripts/check-capability-inventory.test.mjs index f81047ccf4c..b15372d3291 100644 --- a/packages/paperclip-runner/scripts/check-capability-inventory.test.mjs +++ b/packages/paperclip-runner/scripts/check-capability-inventory.test.mjs @@ -42,7 +42,7 @@ function validInventories() { schemaVersion: 2, inventoryRole: "normative", generatedFrom: ["skills/paperclip/SKILL.md"], - rows: Array.from({ length: 157 }, (_, index) => row(`capability-${index}`)), + rows: Array.from({ length: 160 }, (_, index) => row(`capability-${index}`)), }, evaluations: { schemaVersion: 2, diff --git a/packages/paperclip-runner/scripts/generate-capability-contract.d.mts b/packages/paperclip-runner/scripts/generate-capability-contract.d.mts new file mode 100644 index 00000000000..401886022fb --- /dev/null +++ b/packages/paperclip-runner/scripts/generate-capability-contract.d.mts @@ -0,0 +1 @@ +export function buildContract(): Promise>; diff --git a/packages/paperclip-runner/scripts/generate-capability-contract.mjs b/packages/paperclip-runner/scripts/generate-capability-contract.mjs index 840b94521c1..7903efd7ca1 100644 --- a/packages/paperclip-runner/scripts/generate-capability-contract.mjs +++ b/packages/paperclip-runner/scripts/generate-capability-contract.mjs @@ -159,7 +159,7 @@ function renderHandoff() { ].join("\n") + "\n"; } -async function buildContract() { +export async function buildContract() { const contract = JSON.parse(await readFile(contractPath, "utf8")); const capabilities = await readSkillHeadings(contract.skillSources); const discoveredTools = await readLegacyTools(); diff --git a/packages/paperclip-runner/scripts/lib/capability-inventory.mjs b/packages/paperclip-runner/scripts/lib/capability-inventory.mjs index be9dd2ce7d9..5f04ef61eee 100644 --- a/packages/paperclip-runner/scripts/lib/capability-inventory.mjs +++ b/packages/paperclip-runner/scripts/lib/capability-inventory.mjs @@ -221,16 +221,16 @@ export async function buildInventories({ repoRoot, evalRoot }) { } export async function buildSkillInventory(repoRoot) { - const skillRoot = resolve(repoRoot, "skills/paperclip"); - const skillFiles = ["SKILL.md", "references/artifacts.md", "references/cases.md", "references/company-skills.md", "references/issue-workspaces.md", "references/routines.md", "references/workflows.md", "references/api-reference.md"]; + const { skillSources: skillFiles } = JSON.parse(await readFile(resolve(repoRoot, + "packages/paperclip-runner/spec/capability/source-contract.json"), "utf8")); const rows = (await Promise.all(skillFiles.map(async (file) => parseSkillHeadings( - await readFile(resolve(skillRoot, file), "utf8"), - `skills/paperclip/${file}`, + await readFile(resolve(repoRoot, file), "utf8"), + file, )))).flat(); return { schemaVersion: 2, inventoryRole: "normative", - generatedFrom: skillFiles.map((file) => `skills/paperclip/${file}`), + generatedFrom: skillFiles, rows, }; } @@ -247,7 +247,7 @@ export async function buildMcpInventory(repoRoot) { export function validateInventories(inventories) { const errors = []; - const expectedCounts = { capabilities: 157, evaluations: 106, legacyMcpAliases: 42 }; + const expectedCounts = { capabilities: 160, evaluations: 106, legacyMcpAliases: 42 }; const normativeNames = ["capabilities", "evaluations"]; const normativeRows = new Map(); const globalNormativeIds = new Set(); diff --git a/packages/paperclip-runner/spec/capability/capabilities.yaml b/packages/paperclip-runner/spec/capability/capabilities.yaml index 5e07d65358f..4170503c4bc 100644 --- a/packages/paperclip-runner/spec/capability/capabilities.yaml +++ b/packages/paperclip-runner/spec/capability/capabilities.yaml @@ -4,13 +4,14 @@ "inventoryRole": "normative", "generatedFrom": [ "skills/paperclip/SKILL.md", + "skills/paperclip/references/api-reference.md", "skills/paperclip/references/artifacts.md", "skills/paperclip/references/cases.md", "skills/paperclip/references/company-skills.md", + "skills/paperclip/references/issue-documents.md", "skills/paperclip/references/issue-workspaces.md", "skills/paperclip/references/routines.md", - "skills/paperclip/references/workflows.md", - "skills/paperclip/references/api-reference.md" + "skills/paperclip/references/workflows.md" ], "rows": [ { @@ -44,9 +45,9 @@ ] }, { - "id": "skill:skills/paperclip/SKILL.md:authentication:18", + "id": "skill:skills/paperclip/SKILL.md:authentication:20", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:18", + "sourceAnchor": "skills/paperclip/SKILL.md:20", "title": "Authentication", "expectedSemantics": "Skill guidance headed “Authentication”.", "primaryDisposition": "control_plane_owned", @@ -55,13 +56,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:18" + "skill:skills/paperclip/SKILL.md:20" ] }, { - "id": "skill:skills/paperclip/SKILL.md:conversation-tasks:30", + "id": "skill:skills/paperclip/SKILL.md:conversation-tasks:32", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:30", + "sourceAnchor": "skills/paperclip/SKILL.md:32", "title": "Conversation tasks", "expectedSemantics": "Skill guidance headed “Conversation tasks”.", "primaryDisposition": "optional_agent_tool", @@ -70,13 +71,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:30" + "skill:skills/paperclip/SKILL.md:32" ] }, { - "id": "skill:skills/paperclip/SKILL.md:server-verified-external-chat-turns:47", + "id": "skill:skills/paperclip/SKILL.md:server-verified-external-chat-turns:49", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:47", + "sourceAnchor": "skills/paperclip/SKILL.md:49", "title": "Server-Verified External Chat Turns", "expectedSemantics": "Skill guidance headed “Server-Verified External Chat Turns”.", "primaryDisposition": "control_plane_owned", @@ -85,13 +86,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:47" + "skill:skills/paperclip/SKILL.md:49" ] }, { - "id": "skill:skills/paperclip/SKILL.md:the-heartbeat-procedure:87", + "id": "skill:skills/paperclip/SKILL.md:the-heartbeat-procedure:89", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:87", + "sourceAnchor": "skills/paperclip/SKILL.md:89", "title": "The Heartbeat Procedure", "expectedSemantics": "Skill guidance headed “The Heartbeat Procedure”.", "primaryDisposition": "optional_agent_tool", @@ -100,13 +101,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:87" + "skill:skills/paperclip/SKILL.md:89" ] }, { - "id": "skill:skills/paperclip/SKILL.md:generated-artifacts-and-work-products:196", + "id": "skill:skills/paperclip/SKILL.md:generated-artifacts-and-work-products:198", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:196", + "sourceAnchor": "skills/paperclip/SKILL.md:198", "title": "Generated Artifacts and Work Products", "expectedSemantics": "Skill guidance headed “Generated Artifacts and Work Products”.", "primaryDisposition": "always_agent_tool", @@ -115,13 +116,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:196" + "skill:skills/paperclip/SKILL.md:198" ] }, { - "id": "skill:skills/paperclip/SKILL.md:status-quick-guide:244", + "id": "skill:skills/paperclip/SKILL.md:status-quick-guide:246", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:244", + "sourceAnchor": "skills/paperclip/SKILL.md:246", "title": "Status Quick Guide", "expectedSemantics": "Skill guidance headed “Status Quick Guide”.", "primaryDisposition": "control_plane_owned", @@ -130,13 +131,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:244" + "skill:skills/paperclip/SKILL.md:246" ] }, { - "id": "skill:skills/paperclip/SKILL.md:monitors-and-watchers-say-only-what-you-actually-scheduled:254", + "id": "skill:skills/paperclip/SKILL.md:monitors-and-watchers-say-only-what-you-actually-scheduled:256", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:254", + "sourceAnchor": "skills/paperclip/SKILL.md:256", "title": "Monitors and Watchers (say only what you actually scheduled)", "expectedSemantics": "Skill guidance headed “Monitors and Watchers (say only what you actually scheduled)”.", "primaryDisposition": "optional_agent_tool", @@ -145,13 +146,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:254" + "skill:skills/paperclip/SKILL.md:256" ] }, { - "id": "skill:skills/paperclip/SKILL.md:delegating-review-tasks:267", + "id": "skill:skills/paperclip/SKILL.md:delegating-review-tasks:269", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:267", + "sourceAnchor": "skills/paperclip/SKILL.md:269", "title": "Delegating review tasks", "expectedSemantics": "Skill guidance headed “Delegating review tasks”.", "primaryDisposition": "always_agent_tool", @@ -160,13 +161,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:267" + "skill:skills/paperclip/SKILL.md:269" ] }, { - "id": "skill:skills/paperclip/SKILL.md:managing-a-user-s-inbox:278", + "id": "skill:skills/paperclip/SKILL.md:managing-a-user-s-inbox:280", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:278", + "sourceAnchor": "skills/paperclip/SKILL.md:280", "title": "Managing A User's Inbox", "expectedSemantics": "Skill guidance headed “Managing A User's Inbox”.", "primaryDisposition": "control_plane_owned", @@ -175,13 +176,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:278" + "skill:skills/paperclip/SKILL.md:280" ] }, { - "id": "skill:skills/paperclip/SKILL.md:issue-dependencies-blockers:286", + "id": "skill:skills/paperclip/SKILL.md:issue-dependencies-blockers:288", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:286", + "sourceAnchor": "skills/paperclip/SKILL.md:288", "title": "Issue Dependencies (Blockers)", "expectedSemantics": "Skill guidance headed “Issue Dependencies (Blockers)”.", "primaryDisposition": "control_plane_owned", @@ -190,13 +191,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:286" + "skill:skills/paperclip/SKILL.md:288" ] }, { - "id": "skill:skills/paperclip/SKILL.md:requesting-board-approval:311", + "id": "skill:skills/paperclip/SKILL.md:requesting-board-approval:313", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:311", + "sourceAnchor": "skills/paperclip/SKILL.md:313", "title": "Requesting Board Approval", "expectedSemantics": "Skill guidance headed “Requesting Board Approval”.", "primaryDisposition": "optional_agent_tool", @@ -205,13 +206,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:311" + "skill:skills/paperclip/SKILL.md:313" ] }, { - "id": "skill:skills/paperclip/SKILL.md:issue-thread-interactions:332", + "id": "skill:skills/paperclip/SKILL.md:issue-thread-interactions:334", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:332", + "sourceAnchor": "skills/paperclip/SKILL.md:334", "title": "Issue-Thread Interactions", "expectedSemantics": "Skill guidance headed “Issue-Thread Interactions”.", "primaryDisposition": "optional_agent_tool", @@ -220,13 +221,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:332" + "skill:skills/paperclip/SKILL.md:334" ] }, { - "id": "skill:skills/paperclip/SKILL.md:standalone-decisions:361", + "id": "skill:skills/paperclip/SKILL.md:standalone-decisions:363", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:361", + "sourceAnchor": "skills/paperclip/SKILL.md:363", "title": "Standalone Decisions", "expectedSemantics": "Skill guidance headed “Standalone Decisions”.", "primaryDisposition": "optional_agent_tool", @@ -235,13 +236,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:361" + "skill:skills/paperclip/SKILL.md:363" ] }, { - "id": "skill:skills/paperclip/SKILL.md:mcp-tool-approval-gates:465", + "id": "skill:skills/paperclip/SKILL.md:mcp-tool-approval-gates:467", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:465", + "sourceAnchor": "skills/paperclip/SKILL.md:467", "title": "MCP Tool Approval Gates", "expectedSemantics": "Skill guidance headed “MCP Tool Approval Gates”.", "primaryDisposition": "optional_agent_tool", @@ -250,13 +251,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:465" + "skill:skills/paperclip/SKILL.md:467" ] }, { - "id": "skill:skills/paperclip/SKILL.md:niche-workflow-pointers:507", + "id": "skill:skills/paperclip/SKILL.md:niche-workflow-pointers:509", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:507", + "sourceAnchor": "skills/paperclip/SKILL.md:509", "title": "Niche Workflow Pointers", "expectedSemantics": "Skill guidance headed “Niche Workflow Pointers”.", "primaryDisposition": "optional_agent_tool", @@ -265,13 +266,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:507" + "skill:skills/paperclip/SKILL.md:509" ] }, { - "id": "skill:skills/paperclip/SKILL.md:cases:517", + "id": "skill:skills/paperclip/SKILL.md:cases:519", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:517", + "sourceAnchor": "skills/paperclip/SKILL.md:519", "title": "Cases", "expectedSemantics": "Skill guidance headed “Cases”.", "primaryDisposition": "optional_agent_tool", @@ -280,13 +281,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:517" + "skill:skills/paperclip/SKILL.md:519" ] }, { - "id": "skill:skills/paperclip/SKILL.md:company-skills-workflow:522", + "id": "skill:skills/paperclip/SKILL.md:company-skills-workflow:524", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:522", + "sourceAnchor": "skills/paperclip/SKILL.md:524", "title": "Company Skills Workflow", "expectedSemantics": "Skill guidance headed “Company Skills Workflow”.", "primaryDisposition": "optional_agent_tool", @@ -295,13 +296,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:522" + "skill:skills/paperclip/SKILL.md:524" ] }, { - "id": "skill:skills/paperclip/SKILL.md:routines:533", + "id": "skill:skills/paperclip/SKILL.md:routines:535", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:533", + "sourceAnchor": "skills/paperclip/SKILL.md:535", "title": "Routines", "expectedSemantics": "Skill guidance headed “Routines”.", "primaryDisposition": "optional_agent_tool", @@ -310,13 +311,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:533" + "skill:skills/paperclip/SKILL.md:535" ] }, { - "id": "skill:skills/paperclip/SKILL.md:issue-workspace-runtime-controls:544", + "id": "skill:skills/paperclip/SKILL.md:issue-workspace-runtime-controls:546", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:544", + "sourceAnchor": "skills/paperclip/SKILL.md:546", "title": "Issue Workspace Runtime Controls", "expectedSemantics": "Skill guidance headed “Issue Workspace Runtime Controls”.", "primaryDisposition": "optional_agent_tool", @@ -325,13 +326,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:544" + "skill:skills/paperclip/SKILL.md:546" ] }, { - "id": "skill:skills/paperclip/SKILL.md:proposing-credentials-safely:551", + "id": "skill:skills/paperclip/SKILL.md:proposing-credentials-safely:553", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:551", + "sourceAnchor": "skills/paperclip/SKILL.md:553", "title": "Proposing Credentials Safely", "expectedSemantics": "Skill guidance headed “Proposing Credentials Safely”.", "primaryDisposition": "optional_agent_tool", @@ -340,13 +341,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:551" + "skill:skills/paperclip/SKILL.md:553" ] }, { - "id": "skill:skills/paperclip/SKILL.md:reading-granted-secrets:558", + "id": "skill:skills/paperclip/SKILL.md:reading-granted-secrets:560", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:558", + "sourceAnchor": "skills/paperclip/SKILL.md:560", "title": "Reading Granted Secrets", "expectedSemantics": "Skill guidance headed “Reading Granted Secrets”.", "primaryDisposition": "optional_agent_tool", @@ -355,13 +356,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:558" + "skill:skills/paperclip/SKILL.md:560" ] }, { - "id": "skill:skills/paperclip/SKILL.md:critical-rules:584", + "id": "skill:skills/paperclip/SKILL.md:critical-rules:586", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:584", + "sourceAnchor": "skills/paperclip/SKILL.md:586", "title": "Critical Rules", "expectedSemantics": "Skill guidance headed “Critical Rules”.", "primaryDisposition": "optional_agent_tool", @@ -370,13 +371,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:584" + "skill:skills/paperclip/SKILL.md:586" ] }, { - "id": "skill:skills/paperclip/SKILL.md:comment-style-required:605", + "id": "skill:skills/paperclip/SKILL.md:comment-style-required:607", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:605", + "sourceAnchor": "skills/paperclip/SKILL.md:607", "title": "Comment Style (Required)", "expectedSemantics": "Skill guidance headed “Comment Style (Required)”.", "primaryDisposition": "always_agent_tool", @@ -385,13 +386,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:605" + "skill:skills/paperclip/SKILL.md:607" ] }, { - "id": "skill:skills/paperclip/SKILL.md:update:637", + "id": "skill:skills/paperclip/SKILL.md:update:639", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:637", + "sourceAnchor": "skills/paperclip/SKILL.md:639", "title": "Update", "expectedSemantics": "Skill guidance headed “Update”.", "primaryDisposition": "optional_agent_tool", @@ -400,13 +401,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:637" + "skill:skills/paperclip/SKILL.md:639" ] }, { - "id": "skill:skills/paperclip/SKILL.md:planning-required-when-planning-requested:647", + "id": "skill:skills/paperclip/SKILL.md:planning-required-when-planning-requested:649", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:647", + "sourceAnchor": "skills/paperclip/SKILL.md:649", "title": "Planning (Required when planning requested)", "expectedSemantics": "Skill guidance headed “Planning (Required when planning requested)”.", "primaryDisposition": "optional_agent_tool", @@ -415,13 +416,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:647" + "skill:skills/paperclip/SKILL.md:649" ] }, { - "id": "skill:skills/paperclip/SKILL.md:key-endpoints-hot-routes:680", + "id": "skill:skills/paperclip/SKILL.md:key-endpoints-hot-routes:682", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:680", + "sourceAnchor": "skills/paperclip/SKILL.md:682", "title": "Key Endpoints (Hot Routes)", "expectedSemantics": "Skill guidance headed “Key Endpoints (Hot Routes)”.", "primaryDisposition": "optional_agent_tool", @@ -430,13 +431,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:680" + "skill:skills/paperclip/SKILL.md:682" ] }, { - "id": "skill:skills/paperclip/SKILL.md:searching-issues:709", + "id": "skill:skills/paperclip/SKILL.md:searching-issues:711", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:709", + "sourceAnchor": "skills/paperclip/SKILL.md:711", "title": "Searching Issues", "expectedSemantics": "Skill guidance headed “Searching Issues”.", "primaryDisposition": "optional_agent_tool", @@ -445,13 +446,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:709" + "skill:skills/paperclip/SKILL.md:711" ] }, { - "id": "skill:skills/paperclip/SKILL.md:full-reference:719", + "id": "skill:skills/paperclip/SKILL.md:full-reference:721", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:719", + "sourceAnchor": "skills/paperclip/SKILL.md:721", "title": "Full Reference", "expectedSemantics": "Skill guidance headed “Full Reference”.", "primaryDisposition": "optional_agent_tool", @@ -460,13 +461,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:719" + "skill:skills/paperclip/SKILL.md:721" ] }, { - "id": "skill:skills/paperclip/SKILL.md:conversational-confirmation-answers:723", + "id": "skill:skills/paperclip/SKILL.md:conversational-confirmation-answers:725", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:723", + "sourceAnchor": "skills/paperclip/SKILL.md:725", "title": "Conversational confirmation answers", "expectedSemantics": "Skill guidance headed “Conversational confirmation answers”.", "primaryDisposition": "always_agent_tool", @@ -475,847 +476,7 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:723" - ] - }, - { - "id": "skill:skills/paperclip/references/artifacts.md:generated-artifacts-and-work-products:1", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/artifacts.md:1", - "title": "Generated Artifacts and Work Products", - "expectedSemantics": "Skill guidance headed “Generated Artifacts and Work Products”.", - "primaryDisposition": "always_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/artifacts.md:1" - ] - }, - { - "id": "skill:skills/paperclip/references/artifacts.md:workspace-only-file-references:15", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/artifacts.md:15", - "title": "Workspace-Only File References", - "expectedSemantics": "Skill guidance headed “Workspace-Only File References”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/artifacts.md:15" - ] - }, - { - "id": "skill:skills/paperclip/references/cases.md:cases:1", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/cases.md:1", - "title": "Cases", - "expectedSemantics": "Skill guidance headed “Cases”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/cases.md:1" - ] - }, - { - "id": "skill:skills/paperclip/references/cases.md:core-model:12", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/cases.md:12", - "title": "Core Model", - "expectedSemantics": "Skill guidance headed “Core Model”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/cases.md:12" - ] - }, - { - "id": "skill:skills/paperclip/references/cases.md:upsert-semantics:29", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/cases.md:29", - "title": "Upsert Semantics", - "expectedSemantics": "Skill guidance headed “Upsert Semantics”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/cases.md:29" - ] - }, - { - "id": "skill:skills/paperclip/references/cases.md:read-and-search:67", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/cases.md:67", - "title": "Read And Search", - "expectedSemantics": "Skill guidance headed “Read And Search”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/cases.md:67" - ] - }, - { - "id": "skill:skills/paperclip/references/cases.md:documents:90", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/cases.md:90", - "title": "Documents", - "expectedSemantics": "Skill guidance headed “Documents”.", - "primaryDisposition": "always_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/cases.md:90" - ] - }, - { - "id": "skill:skills/paperclip/references/cases.md:fields:118", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/cases.md:118", - "title": "Fields", - "expectedSemantics": "Skill guidance headed “Fields”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/cases.md:118" - ] - }, - { - "id": "skill:skills/paperclip/references/cases.md:issue-links:151", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/cases.md:151", - "title": "Issue Links", - "expectedSemantics": "Skill guidance headed “Issue Links”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/cases.md:151" - ] - }, - { - "id": "skill:skills/paperclip/references/cases.md:child-cases:176", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/cases.md:176", - "title": "Child Cases", - "expectedSemantics": "Skill guidance headed “Child Cases”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/cases.md:176" - ] - }, - { - "id": "skill:skills/paperclip/references/cases.md:attachments:195", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/cases.md:195", - "title": "Attachments", - "expectedSemantics": "Skill guidance headed “Attachments”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/cases.md:195" - ] - }, - { - "id": "skill:skills/paperclip/references/cases.md:lifecycle:208", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/cases.md:208", - "title": "Lifecycle", - "expectedSemantics": "Skill guidance headed “Lifecycle”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/cases.md:208" - ] - }, - { - "id": "skill:skills/paperclip/references/cases.md:worked-blog-post-example:222", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/cases.md:222", - "title": "Worked Blog Post Example", - "expectedSemantics": "Skill guidance headed “Worked Blog Post Example”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/cases.md:222" - ] - }, - { - "id": "skill:skills/paperclip/references/company-skills.md:company-skills-workflow:1", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/company-skills.md:1", - "title": "Company Skills Workflow", - "expectedSemantics": "Skill guidance headed “Company Skills Workflow”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/company-skills.md:1" - ] - }, - { - "id": "skill:skills/paperclip/references/company-skills.md:what-exists:5", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/company-skills.md:5", - "title": "What Exists", - "expectedSemantics": "Skill guidance headed “What Exists”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/company-skills.md:5" - ] - }, - { - "id": "skill:skills/paperclip/references/company-skills.md:permission-model:22", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/company-skills.md:22", - "title": "Permission Model", - "expectedSemantics": "Skill guidance headed “Permission Model”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/company-skills.md:22" - ] - }, - { - "id": "skill:skills/paperclip/references/company-skills.md:core-endpoints:29", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/company-skills.md:29", - "title": "Core Endpoints", - "expectedSemantics": "Skill guidance headed “Core Endpoints”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/company-skills.md:29" - ] - }, - { - "id": "skill:skills/paperclip/references/company-skills.md:install-a-skill-into-the-company:65", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/company-skills.md:65", - "title": "Install A Skill Into The Company", - "expectedSemantics": "Skill guidance headed “Install A Skill Into The Company”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/company-skills.md:65" - ] - }, - { - "id": "skill:skills/paperclip/references/company-skills.md:app-shipped-catalog:75", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/company-skills.md:75", - "title": "App-shipped catalog", - "expectedSemantics": "Skill guidance headed “App-shipped catalog”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/company-skills.md:75" - ] - }, - { - "id": "skill:skills/paperclip/references/company-skills.md:external-source-import:101", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/company-skills.md:101", - "title": "External source import", - "expectedSemantics": "Skill guidance headed “External source import”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/company-skills.md:101" - ] - }, - { - "id": "skill:skills/paperclip/references/company-skills.md:source-types-in-order-of-preference:105", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/company-skills.md:105", - "title": "Source types (in order of preference)", - "expectedSemantics": "Skill guidance headed “Source types (in order of preference)”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/company-skills.md:105" - ] - }, - { - "id": "skill:skills/paperclip/references/company-skills.md:example-skills-sh-import-preferred:116", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/company-skills.md:116", - "title": "Example: skills.sh import (preferred)", - "expectedSemantics": "Skill guidance headed “Example: skills.sh import (preferred)”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/company-skills.md:116" - ] - }, - { - "id": "skill:skills/paperclip/references/company-skills.md:example-github-import:138", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/company-skills.md:138", - "title": "Example: GitHub import", - "expectedSemantics": "Skill guidance headed “Example: GitHub import”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/company-skills.md:138" - ] - }, - { - "id": "skill:skills/paperclip/references/company-skills.md:inspect-what-was-installed:164", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/company-skills.md:164", - "title": "Inspect What Was Installed", - "expectedSemantics": "Skill guidance headed “Inspect What Was Installed”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/company-skills.md:164" - ] - }, - { - "id": "skill:skills/paperclip/references/company-skills.md:assign-skills-to-an-existing-agent:181", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/company-skills.md:181", - "title": "Assign Skills To An Existing Agent", - "expectedSemantics": "Skill guidance headed “Assign Skills To An Existing Agent”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/company-skills.md:181" - ] - }, - { - "id": "skill:skills/paperclip/references/company-skills.md:include-skills-during-hire-or-create:216", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/company-skills.md:216", - "title": "Include Skills During Hire Or Create", - "expectedSemantics": "Skill guidance headed “Include Skills During Hire Or Create”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/company-skills.md:216" - ] - }, - { - "id": "skill:skills/paperclip/references/company-skills.md:notes:256", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/company-skills.md:256", - "title": "Notes", - "expectedSemantics": "Skill guidance headed “Notes”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/company-skills.md:256" - ] - }, - { - "id": "skill:skills/paperclip/references/issue-workspaces.md:issue-workspace-runtime-controls:1", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/issue-workspaces.md:1", - "title": "Issue Workspace Runtime Controls", - "expectedSemantics": "Skill guidance headed “Issue Workspace Runtime Controls”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/issue-workspaces.md:1" - ] - }, - { - "id": "skill:skills/paperclip/references/issue-workspaces.md:discover-the-workspace:5", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/issue-workspaces.md:5", - "title": "Discover the Workspace", - "expectedSemantics": "Skill guidance headed “Discover the Workspace”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/issue-workspaces.md:5" - ] - }, - { - "id": "skill:skills/paperclip/references/issue-workspaces.md:control-services:23", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/issue-workspaces.md:23", - "title": "Control Services", - "expectedSemantics": "Skill guidance headed “Control Services”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/issue-workspaces.md:23" - ] - }, - { - "id": "skill:skills/paperclip/references/issue-workspaces.md:start-all-configured-services-waits-for-configured-readiness-checks:28", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/issue-workspaces.md:28", - "title": "Start all configured services; waits for configured readiness checks.", - "expectedSemantics": "Skill guidance headed “Start all configured services; waits for configured readiness checks.”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/issue-workspaces.md:28" - ] - }, - { - "id": "skill:skills/paperclip/references/issue-workspaces.md:restart-all-configured-services:36", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/issue-workspaces.md:36", - "title": "Restart all configured services.", - "expectedSemantics": "Skill guidance headed “Restart all configured services.”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/issue-workspaces.md:36" - ] - }, - { - "id": "skill:skills/paperclip/references/issue-workspaces.md:stop-all-running-services:44", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/issue-workspaces.md:44", - "title": "Stop all running services.", - "expectedSemantics": "Skill guidance headed “Stop all running services.”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/issue-workspaces.md:44" - ] - }, - { - "id": "skill:skills/paperclip/references/issue-workspaces.md:read-the-url:63", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/issue-workspaces.md:63", - "title": "Read the URL", - "expectedSemantics": "Skill guidance headed “Read the URL”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/issue-workspaces.md:63" - ] - }, - { - "id": "skill:skills/paperclip/references/issue-workspaces.md:mcp-tools:72", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/issue-workspaces.md:72", - "title": "MCP Tools", - "expectedSemantics": "Skill guidance headed “MCP Tools”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/issue-workspaces.md:72" - ] - }, - { - "id": "skill:skills/paperclip/references/routines.md:paperclip-routines:1", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/routines.md:1", - "title": "Paperclip Routines", - "expectedSemantics": "Skill guidance headed “Paperclip Routines”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/routines.md:1" - ] - }, - { - "id": "skill:skills/paperclip/references/routines.md:lifecycle:16", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/routines.md:16", - "title": "Lifecycle", - "expectedSemantics": "Skill guidance headed “Lifecycle”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/routines.md:16" - ] - }, - { - "id": "skill:skills/paperclip/references/routines.md:creating-a-routine:27", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/routines.md:27", - "title": "Creating a Routine", - "expectedSemantics": "Skill guidance headed “Creating a Routine”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/routines.md:27" - ] - }, - { - "id": "skill:skills/paperclip/references/routines.md:concurrency-policies:64", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/routines.md:64", - "title": "Concurrency Policies", - "expectedSemantics": "Skill guidance headed “Concurrency Policies”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/routines.md:64" - ] - }, - { - "id": "skill:skills/paperclip/references/routines.md:catch-up-policies:76", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/routines.md:76", - "title": "Catch-Up Policies", - "expectedSemantics": "Skill guidance headed “Catch-Up Policies”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/routines.md:76" - ] - }, - { - "id": "skill:skills/paperclip/references/routines.md:activity-gated-scheduled-runs:87", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/routines.md:87", - "title": "Activity-Gated Scheduled Runs", - "expectedSemantics": "Skill guidance headed “Activity-Gated Scheduled Runs”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/routines.md:87" - ] - }, - { - "id": "skill:skills/paperclip/references/routines.md:example-skip-quiet-nights:107", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/routines.md:107", - "title": "Example: skip quiet nights", - "expectedSemantics": "Skill guidance headed “Example: skip quiet nights”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/routines.md:107" - ] - }, - { - "id": "skill:skills/paperclip/references/routines.md:adding-triggers:126", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/routines.md:126", - "title": "Adding Triggers", - "expectedSemantics": "Skill guidance headed “Adding Triggers”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/routines.md:126" - ] - }, - { - "id": "skill:skills/paperclip/references/routines.md:schedule-cron:136", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/routines.md:136", - "title": "Schedule (cron)", - "expectedSemantics": "Skill guidance headed “Schedule (cron)”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/routines.md:136" - ] - }, - { - "id": "skill:skills/paperclip/references/routines.md:webhook:150", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/routines.md:150", - "title": "Webhook", - "expectedSemantics": "Skill guidance headed “Webhook”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/routines.md:150" - ] - }, - { - "id": "skill:skills/paperclip/references/routines.md:api-manual-only:167", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/routines.md:167", - "title": "API (manual only)", - "expectedSemantics": "Skill guidance headed “API (manual only)”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/routines.md:167" - ] - }, - { - "id": "skill:skills/paperclip/references/routines.md:updating-and-deleting-triggers:179", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/routines.md:179", - "title": "Updating and Deleting Triggers", - "expectedSemantics": "Skill guidance headed “Updating and Deleting Triggers”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/routines.md:179" - ] - }, - { - "id": "skill:skills/paperclip/references/routines.md:manual-run:196", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/routines.md:196", - "title": "Manual Run", - "expectedSemantics": "Skill guidance headed “Manual Run”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/routines.md:196" - ] - }, - { - "id": "skill:skills/paperclip/references/routines.md:updating-a-routine:212", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/routines.md:212", - "title": "Updating a Routine", - "expectedSemantics": "Skill guidance headed “Updating a Routine”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/routines.md:212" - ] - }, - { - "id": "skill:skills/paperclip/references/routines.md:reading-routines-and-runs:223", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/routines.md:223", - "title": "Reading Routines and Runs", - "expectedSemantics": "Skill guidance headed “Reading Routines and Runs”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/routines.md:223" - ] - }, - { - "id": "skill:skills/paperclip/references/workflows.md:paperclip-workflow-playbooks:1", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/workflows.md:1", - "title": "Paperclip Workflow Playbooks", - "expectedSemantics": "Skill guidance headed “Paperclip Workflow Playbooks”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/workflows.md:1" - ] - }, - { - "id": "skill:skills/paperclip/references/workflows.md:project-setup-ceo-manager:7", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/workflows.md:7", - "title": "Project Setup (CEO/Manager)", - "expectedSemantics": "Skill guidance headed “Project Setup (CEO/Manager)”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/workflows.md:7" - ] - }, - { - "id": "skill:skills/paperclip/references/workflows.md:openclaw-invite-ceo:22", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/workflows.md:22", - "title": "OpenClaw Invite (CEO)", - "expectedSemantics": "Skill guidance headed “OpenClaw Invite (CEO)”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/workflows.md:22" - ] - }, - { - "id": "skill:skills/paperclip/references/workflows.md:setting-agent-instructions-path:50", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/workflows.md:50", - "title": "Setting Agent Instructions Path", - "expectedSemantics": "Skill guidance headed “Setting Agent Instructions Path”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/workflows.md:50" - ] - }, - { - "id": "skill:skills/paperclip/references/workflows.md:company-import-export:79", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/workflows.md:79", - "title": "Company Import / Export", - "expectedSemantics": "Skill guidance headed “Company Import / Export”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/workflows.md:79" - ] - }, - { - "id": "skill:skills/paperclip/references/workflows.md:self-test-playbook-app-level:106", - "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/workflows.md:106", - "title": "Self-Test Playbook (App-Level)", - "expectedSemantics": "Skill guidance headed “Self-Test Playbook (App-Level)”.", - "primaryDisposition": "optional_agent_tool", - "requiredGrants": [], - "assertionClasses": [ - "control_plane_invariant" - ], - "evidenceIds": [ - "skill:skills/paperclip/references/workflows.md:106" + "skill:skills/paperclip/SKILL.md:725" ] }, { @@ -2367,6 +1528,891 @@ "evidenceIds": [ "skill:skills/paperclip/references/api-reference.md:1661" ] + }, + { + "id": "skill:skills/paperclip/references/artifacts.md:generated-artifacts-and-work-products:1", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/artifacts.md:1", + "title": "Generated Artifacts and Work Products", + "expectedSemantics": "Skill guidance headed “Generated Artifacts and Work Products”.", + "primaryDisposition": "always_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/artifacts.md:1" + ] + }, + { + "id": "skill:skills/paperclip/references/artifacts.md:workspace-only-file-references:15", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/artifacts.md:15", + "title": "Workspace-Only File References", + "expectedSemantics": "Skill guidance headed “Workspace-Only File References”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/artifacts.md:15" + ] + }, + { + "id": "skill:skills/paperclip/references/cases.md:cases:1", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/cases.md:1", + "title": "Cases", + "expectedSemantics": "Skill guidance headed “Cases”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/cases.md:1" + ] + }, + { + "id": "skill:skills/paperclip/references/cases.md:core-model:12", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/cases.md:12", + "title": "Core Model", + "expectedSemantics": "Skill guidance headed “Core Model”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/cases.md:12" + ] + }, + { + "id": "skill:skills/paperclip/references/cases.md:upsert-semantics:29", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/cases.md:29", + "title": "Upsert Semantics", + "expectedSemantics": "Skill guidance headed “Upsert Semantics”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/cases.md:29" + ] + }, + { + "id": "skill:skills/paperclip/references/cases.md:read-and-search:67", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/cases.md:67", + "title": "Read And Search", + "expectedSemantics": "Skill guidance headed “Read And Search”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/cases.md:67" + ] + }, + { + "id": "skill:skills/paperclip/references/cases.md:documents:90", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/cases.md:90", + "title": "Documents", + "expectedSemantics": "Skill guidance headed “Documents”.", + "primaryDisposition": "always_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/cases.md:90" + ] + }, + { + "id": "skill:skills/paperclip/references/cases.md:fields:118", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/cases.md:118", + "title": "Fields", + "expectedSemantics": "Skill guidance headed “Fields”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/cases.md:118" + ] + }, + { + "id": "skill:skills/paperclip/references/cases.md:issue-links:151", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/cases.md:151", + "title": "Issue Links", + "expectedSemantics": "Skill guidance headed “Issue Links”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/cases.md:151" + ] + }, + { + "id": "skill:skills/paperclip/references/cases.md:child-cases:176", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/cases.md:176", + "title": "Child Cases", + "expectedSemantics": "Skill guidance headed “Child Cases”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/cases.md:176" + ] + }, + { + "id": "skill:skills/paperclip/references/cases.md:attachments:195", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/cases.md:195", + "title": "Attachments", + "expectedSemantics": "Skill guidance headed “Attachments”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/cases.md:195" + ] + }, + { + "id": "skill:skills/paperclip/references/cases.md:lifecycle:208", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/cases.md:208", + "title": "Lifecycle", + "expectedSemantics": "Skill guidance headed “Lifecycle”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/cases.md:208" + ] + }, + { + "id": "skill:skills/paperclip/references/cases.md:worked-blog-post-example:222", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/cases.md:222", + "title": "Worked Blog Post Example", + "expectedSemantics": "Skill guidance headed “Worked Blog Post Example”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/cases.md:222" + ] + }, + { + "id": "skill:skills/paperclip/references/company-skills.md:company-skills-workflow:1", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/company-skills.md:1", + "title": "Company Skills Workflow", + "expectedSemantics": "Skill guidance headed “Company Skills Workflow”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/company-skills.md:1" + ] + }, + { + "id": "skill:skills/paperclip/references/company-skills.md:what-exists:5", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/company-skills.md:5", + "title": "What Exists", + "expectedSemantics": "Skill guidance headed “What Exists”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/company-skills.md:5" + ] + }, + { + "id": "skill:skills/paperclip/references/company-skills.md:permission-model:22", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/company-skills.md:22", + "title": "Permission Model", + "expectedSemantics": "Skill guidance headed “Permission Model”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/company-skills.md:22" + ] + }, + { + "id": "skill:skills/paperclip/references/company-skills.md:core-endpoints:29", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/company-skills.md:29", + "title": "Core Endpoints", + "expectedSemantics": "Skill guidance headed “Core Endpoints”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/company-skills.md:29" + ] + }, + { + "id": "skill:skills/paperclip/references/company-skills.md:install-a-skill-into-the-company:65", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/company-skills.md:65", + "title": "Install A Skill Into The Company", + "expectedSemantics": "Skill guidance headed “Install A Skill Into The Company”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/company-skills.md:65" + ] + }, + { + "id": "skill:skills/paperclip/references/company-skills.md:app-shipped-catalog:75", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/company-skills.md:75", + "title": "App-shipped catalog", + "expectedSemantics": "Skill guidance headed “App-shipped catalog”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/company-skills.md:75" + ] + }, + { + "id": "skill:skills/paperclip/references/company-skills.md:external-source-import:101", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/company-skills.md:101", + "title": "External source import", + "expectedSemantics": "Skill guidance headed “External source import”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/company-skills.md:101" + ] + }, + { + "id": "skill:skills/paperclip/references/company-skills.md:source-types-in-order-of-preference:105", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/company-skills.md:105", + "title": "Source types (in order of preference)", + "expectedSemantics": "Skill guidance headed “Source types (in order of preference)”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/company-skills.md:105" + ] + }, + { + "id": "skill:skills/paperclip/references/company-skills.md:example-skills-sh-import-preferred:116", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/company-skills.md:116", + "title": "Example: skills.sh import (preferred)", + "expectedSemantics": "Skill guidance headed “Example: skills.sh import (preferred)”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/company-skills.md:116" + ] + }, + { + "id": "skill:skills/paperclip/references/company-skills.md:example-github-import:138", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/company-skills.md:138", + "title": "Example: GitHub import", + "expectedSemantics": "Skill guidance headed “Example: GitHub import”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/company-skills.md:138" + ] + }, + { + "id": "skill:skills/paperclip/references/company-skills.md:inspect-what-was-installed:164", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/company-skills.md:164", + "title": "Inspect What Was Installed", + "expectedSemantics": "Skill guidance headed “Inspect What Was Installed”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/company-skills.md:164" + ] + }, + { + "id": "skill:skills/paperclip/references/company-skills.md:assign-skills-to-an-existing-agent:181", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/company-skills.md:181", + "title": "Assign Skills To An Existing Agent", + "expectedSemantics": "Skill guidance headed “Assign Skills To An Existing Agent”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/company-skills.md:181" + ] + }, + { + "id": "skill:skills/paperclip/references/company-skills.md:include-skills-during-hire-or-create:216", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/company-skills.md:216", + "title": "Include Skills During Hire Or Create", + "expectedSemantics": "Skill guidance headed “Include Skills During Hire Or Create”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/company-skills.md:216" + ] + }, + { + "id": "skill:skills/paperclip/references/company-skills.md:notes:256", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/company-skills.md:256", + "title": "Notes", + "expectedSemantics": "Skill guidance headed “Notes”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/company-skills.md:256" + ] + }, + { + "id": "skill:skills/paperclip/references/issue-documents.md:issue-documents-through-the-api:1", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/issue-documents.md:1", + "title": "Issue documents through the API", + "expectedSemantics": "Skill guidance headed “Issue documents through the API”.", + "primaryDisposition": "always_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/issue-documents.md:1" + ] + }, + { + "id": "skill:skills/paperclip/references/issue-documents.md:create-a-document:8", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/issue-documents.md:8", + "title": "Create a document", + "expectedSemantics": "Skill guidance headed “Create a document”.", + "primaryDisposition": "always_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/issue-documents.md:8" + ] + }, + { + "id": "skill:skills/paperclip/references/issue-documents.md:update-or-resolve-an-unclear-write:49", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/issue-documents.md:49", + "title": "Update or resolve an unclear write", + "expectedSemantics": "Skill guidance headed “Update or resolve an unclear write”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/issue-documents.md:49" + ] + }, + { + "id": "skill:skills/paperclip/references/issue-workspaces.md:issue-workspace-runtime-controls:1", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/issue-workspaces.md:1", + "title": "Issue Workspace Runtime Controls", + "expectedSemantics": "Skill guidance headed “Issue Workspace Runtime Controls”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/issue-workspaces.md:1" + ] + }, + { + "id": "skill:skills/paperclip/references/issue-workspaces.md:discover-the-workspace:5", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/issue-workspaces.md:5", + "title": "Discover the Workspace", + "expectedSemantics": "Skill guidance headed “Discover the Workspace”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/issue-workspaces.md:5" + ] + }, + { + "id": "skill:skills/paperclip/references/issue-workspaces.md:control-services:23", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/issue-workspaces.md:23", + "title": "Control Services", + "expectedSemantics": "Skill guidance headed “Control Services”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/issue-workspaces.md:23" + ] + }, + { + "id": "skill:skills/paperclip/references/issue-workspaces.md:start-all-configured-services-waits-for-configured-readiness-checks:28", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/issue-workspaces.md:28", + "title": "Start all configured services; waits for configured readiness checks.", + "expectedSemantics": "Skill guidance headed “Start all configured services; waits for configured readiness checks.”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/issue-workspaces.md:28" + ] + }, + { + "id": "skill:skills/paperclip/references/issue-workspaces.md:restart-all-configured-services:36", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/issue-workspaces.md:36", + "title": "Restart all configured services.", + "expectedSemantics": "Skill guidance headed “Restart all configured services.”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/issue-workspaces.md:36" + ] + }, + { + "id": "skill:skills/paperclip/references/issue-workspaces.md:stop-all-running-services:44", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/issue-workspaces.md:44", + "title": "Stop all running services.", + "expectedSemantics": "Skill guidance headed “Stop all running services.”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/issue-workspaces.md:44" + ] + }, + { + "id": "skill:skills/paperclip/references/issue-workspaces.md:read-the-url:63", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/issue-workspaces.md:63", + "title": "Read the URL", + "expectedSemantics": "Skill guidance headed “Read the URL”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/issue-workspaces.md:63" + ] + }, + { + "id": "skill:skills/paperclip/references/issue-workspaces.md:mcp-tools:72", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/issue-workspaces.md:72", + "title": "MCP Tools", + "expectedSemantics": "Skill guidance headed “MCP Tools”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/issue-workspaces.md:72" + ] + }, + { + "id": "skill:skills/paperclip/references/routines.md:paperclip-routines:1", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/routines.md:1", + "title": "Paperclip Routines", + "expectedSemantics": "Skill guidance headed “Paperclip Routines”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/routines.md:1" + ] + }, + { + "id": "skill:skills/paperclip/references/routines.md:lifecycle:16", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/routines.md:16", + "title": "Lifecycle", + "expectedSemantics": "Skill guidance headed “Lifecycle”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/routines.md:16" + ] + }, + { + "id": "skill:skills/paperclip/references/routines.md:creating-a-routine:27", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/routines.md:27", + "title": "Creating a Routine", + "expectedSemantics": "Skill guidance headed “Creating a Routine”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/routines.md:27" + ] + }, + { + "id": "skill:skills/paperclip/references/routines.md:concurrency-policies:64", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/routines.md:64", + "title": "Concurrency Policies", + "expectedSemantics": "Skill guidance headed “Concurrency Policies”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/routines.md:64" + ] + }, + { + "id": "skill:skills/paperclip/references/routines.md:catch-up-policies:76", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/routines.md:76", + "title": "Catch-Up Policies", + "expectedSemantics": "Skill guidance headed “Catch-Up Policies”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/routines.md:76" + ] + }, + { + "id": "skill:skills/paperclip/references/routines.md:activity-gated-scheduled-runs:87", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/routines.md:87", + "title": "Activity-Gated Scheduled Runs", + "expectedSemantics": "Skill guidance headed “Activity-Gated Scheduled Runs”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/routines.md:87" + ] + }, + { + "id": "skill:skills/paperclip/references/routines.md:example-skip-quiet-nights:107", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/routines.md:107", + "title": "Example: skip quiet nights", + "expectedSemantics": "Skill guidance headed “Example: skip quiet nights”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/routines.md:107" + ] + }, + { + "id": "skill:skills/paperclip/references/routines.md:adding-triggers:126", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/routines.md:126", + "title": "Adding Triggers", + "expectedSemantics": "Skill guidance headed “Adding Triggers”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/routines.md:126" + ] + }, + { + "id": "skill:skills/paperclip/references/routines.md:schedule-cron:136", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/routines.md:136", + "title": "Schedule (cron)", + "expectedSemantics": "Skill guidance headed “Schedule (cron)”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/routines.md:136" + ] + }, + { + "id": "skill:skills/paperclip/references/routines.md:webhook:150", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/routines.md:150", + "title": "Webhook", + "expectedSemantics": "Skill guidance headed “Webhook”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/routines.md:150" + ] + }, + { + "id": "skill:skills/paperclip/references/routines.md:api-manual-only:167", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/routines.md:167", + "title": "API (manual only)", + "expectedSemantics": "Skill guidance headed “API (manual only)”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/routines.md:167" + ] + }, + { + "id": "skill:skills/paperclip/references/routines.md:updating-and-deleting-triggers:179", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/routines.md:179", + "title": "Updating and Deleting Triggers", + "expectedSemantics": "Skill guidance headed “Updating and Deleting Triggers”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/routines.md:179" + ] + }, + { + "id": "skill:skills/paperclip/references/routines.md:manual-run:196", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/routines.md:196", + "title": "Manual Run", + "expectedSemantics": "Skill guidance headed “Manual Run”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/routines.md:196" + ] + }, + { + "id": "skill:skills/paperclip/references/routines.md:updating-a-routine:212", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/routines.md:212", + "title": "Updating a Routine", + "expectedSemantics": "Skill guidance headed “Updating a Routine”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/routines.md:212" + ] + }, + { + "id": "skill:skills/paperclip/references/routines.md:reading-routines-and-runs:223", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/routines.md:223", + "title": "Reading Routines and Runs", + "expectedSemantics": "Skill guidance headed “Reading Routines and Runs”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/routines.md:223" + ] + }, + { + "id": "skill:skills/paperclip/references/workflows.md:paperclip-workflow-playbooks:1", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/workflows.md:1", + "title": "Paperclip Workflow Playbooks", + "expectedSemantics": "Skill guidance headed “Paperclip Workflow Playbooks”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/workflows.md:1" + ] + }, + { + "id": "skill:skills/paperclip/references/workflows.md:project-setup-ceo-manager:7", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/workflows.md:7", + "title": "Project Setup (CEO/Manager)", + "expectedSemantics": "Skill guidance headed “Project Setup (CEO/Manager)”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/workflows.md:7" + ] + }, + { + "id": "skill:skills/paperclip/references/workflows.md:openclaw-invite-ceo:22", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/workflows.md:22", + "title": "OpenClaw Invite (CEO)", + "expectedSemantics": "Skill guidance headed “OpenClaw Invite (CEO)”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/workflows.md:22" + ] + }, + { + "id": "skill:skills/paperclip/references/workflows.md:setting-agent-instructions-path:50", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/workflows.md:50", + "title": "Setting Agent Instructions Path", + "expectedSemantics": "Skill guidance headed “Setting Agent Instructions Path”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/workflows.md:50" + ] + }, + { + "id": "skill:skills/paperclip/references/workflows.md:company-import-export:79", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/workflows.md:79", + "title": "Company Import / Export", + "expectedSemantics": "Skill guidance headed “Company Import / Export”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/workflows.md:79" + ] + }, + { + "id": "skill:skills/paperclip/references/workflows.md:self-test-playbook-app-level:106", + "sourceKind": "skill_heading", + "sourceAnchor": "skills/paperclip/references/workflows.md:106", + "title": "Self-Test Playbook (App-Level)", + "expectedSemantics": "Skill guidance headed “Self-Test Playbook (App-Level)”.", + "primaryDisposition": "optional_agent_tool", + "requiredGrants": [], + "assertionClasses": [ + "control_plane_invariant" + ], + "evidenceIds": [ + "skill:skills/paperclip/references/workflows.md:106" + ] } ] } diff --git a/packages/paperclip-runner/spec/capability/source-contract.json b/packages/paperclip-runner/spec/capability/source-contract.json index ebc924bad31..5ff5ae8ce58 100644 --- a/packages/paperclip-runner/spec/capability/source-contract.json +++ b/packages/paperclip-runner/spec/capability/source-contract.json @@ -7,6 +7,7 @@ "skills/paperclip/references/artifacts.md", "skills/paperclip/references/cases.md", "skills/paperclip/references/company-skills.md", + "skills/paperclip/references/issue-documents.md", "skills/paperclip/references/issue-workspaces.md", "skills/paperclip/references/routines.md", "skills/paperclip/references/workflows.md" diff --git a/packages/paperclip-runner/src/generated/capability-contract.ts b/packages/paperclip-runner/src/generated/capability-contract.ts index 0d145a06c44..70aaf0ac063 100644 --- a/packages/paperclip-runner/src/generated/capability-contract.ts +++ b/packages/paperclip-runner/src/generated/capability-contract.ts @@ -6,9 +6,9 @@ export type CapabilityPrimaryDisposition = | "optional_agent_tool"; export const capabilityInventoryCounts = { - "skillReferenceCapabilities": 157, + "skillReferenceCapabilities": 160, "evalCases": 106, - "normativeRows": 263, + "normativeRows": 266, "legacyMcpAliases": 42 } as const; diff --git a/packages/paperclip-runner/test/capability-contract.test.mjs b/packages/paperclip-runner/test/capability-contract.test.mjs index 234259a91f6..ac0d7e0d13e 100644 --- a/packages/paperclip-runner/test/capability-contract.test.mjs +++ b/packages/paperclip-runner/test/capability-contract.test.mjs @@ -18,7 +18,8 @@ test("generated Capability inventory has full source coverage", async () => { readRows("eval-traceability.yaml"), ]); - assert.equal(capabilities.length, 158); + assert.equal(capabilities.length, 161); + assert.equal(capabilities.filter(row => row.sourceAnchor.startsWith("skills/paperclip/references/issue-documents.md#")).length, 3); assert.equal(tools.length, 42); assert.equal(evals.length, 106); assert.equal(new Set(evals.map((row) => row.group)).size, 16); @@ -51,3 +52,15 @@ test("conversational answer guidance has the same agent-operation classification for (const row of [...generated, ...spec]) assert.equal(row.primaryDisposition, "always_agent_tool"); for (const row of generated) assert.equal(row.semanticOperation, "call_api"); }); + +test("both capability inventories cover every declared skill source", async () => { + const contract = JSON.parse(await readFile(resolve(import.meta.dirname, "../spec/capability/source-contract.json"), "utf8")); + const generated = await readRows("capabilities.yaml"); + const normative = decodeInventory(await readFile(resolve(import.meta.dirname, "../spec/capability/capabilities.yaml"), "utf8")); + assert.deepEqual(normative.generatedFrom, contract.skillSources); + for (const source of contract.skillSources) { + assert.ok(generated.some(row => row.sourceAnchor.startsWith(`${source}#`)), `Missing generated source ${source}`); + assert.ok(normative.rows.some(row => row.sourceAnchor.startsWith(`${source}:`)), `Missing normative source ${source}`); + } + assert.equal(normative.rows.filter(row => row.sourceAnchor.startsWith("skills/paperclip/references/issue-documents.md:")).length, 3); +}); diff --git a/server/src/__tests__/agent-skills-routes.test.ts b/server/src/__tests__/agent-skills-routes.test.ts index cbf6a260f0d..f64229e3d4c 100644 --- a/server/src/__tests__/agent-skills-routes.test.ts +++ b/server/src/__tests__/agent-skills-routes.test.ts @@ -1184,7 +1184,7 @@ describe.sequential("agent skill routes", () => { expect(returnedAgent.adapterConfig).not.toHaveProperty("promptTemplate"); }); - it("materializes the bundled default instruction set for non-CEO agents with no prompt template", async () => { + it("materializes minimal default instructions for non-CEO agents with no prompt template", async () => { const res = await requestApp(await createApp(), (baseUrl) => request(baseUrl) .post("/api/companies/company-1/agents") .send({ @@ -1204,38 +1204,10 @@ describe.sequential("agent skill routes", () => { adapterType: "claude_local", }), expect.objectContaining({ - "AGENTS.md": expect.stringMatching(/Start actionable work in the same heartbeat\.[\s\S]*Keep the work moving until it is done\./), + "AGENTS.md": "You are an agent in a Paperclip company.\n", }), { entryFile: "AGENTS.md", replaceExisting: false }, ); - expect(mockAgentInstructionsService.materializeManagedBundle).toHaveBeenCalledWith( - expect.any(Object), - expect.objectContaining({ - "AGENTS.md": expect.stringContaining('kind: "request_confirmation"'), - }), - expect.any(Object), - ); - expect(mockAgentInstructionsService.materializeManagedBundle).toHaveBeenCalledWith( - expect.any(Object), - expect.objectContaining({ - "AGENTS.md": expect.stringContaining("confirmation:{issueId}:plan:{revisionId}"), - }), - expect.any(Object), - ); - expect(mockAgentInstructionsService.materializeManagedBundle).toHaveBeenCalledWith( - expect.any(Object), - expect.objectContaining({ - "AGENTS.md": expect.stringMatching(/PUT \/issues\/\{id\}\/documents\/plan[\s\S]*Re-`GET \/documents\/plan`, assert it returns `200`[\s\S]*latestRevisionId[\s\S]*target=\{ type: 'issue_document', key: 'plan', revisionId: latestRevisionId \}[\s\S]*Never present a plan only in a thread comment or through `ask_user_questions`/), - }), - expect.any(Object), - ); - expect(mockAgentInstructionsService.materializeManagedBundle).toHaveBeenCalledWith( - expect.any(Object), - expect.objectContaining({ - "AGENTS.md": expect.stringContaining("skills/paperclip/scripts/paperclip-upload-artifact.sh"), - }), - expect.any(Object), - ); }); }); diff --git a/server/src/__tests__/codex-local-execute.test.ts b/server/src/__tests__/codex-local-execute.test.ts index 983fa6e87ef..29a3a2b2255 100644 --- a/server/src/__tests__/codex-local-execute.test.ts +++ b/server/src/__tests__/codex-local-execute.test.ts @@ -1443,9 +1443,8 @@ process.exit(1); expect(invocationPrompt).toContain("baseRevisionId set to that latestRevisionId"); expect(capture.prompt).not.toContain("Execution contract:"); expect(capture.prompt).not.toContain("Use child issues"); - } else { - expect(capture.prompt).toContain("Execution contract:"); } + expect(capture.prompt).not.toContain("Execution contract:"); expect(capture.prompt).not.toContain("Follow the paperclip heartbeat."); if (resumedSession) { expect(capture.prompt).not.toContain("You are managed instructions."); diff --git a/server/src/__tests__/workspace-runtime.test.ts b/server/src/__tests__/workspace-runtime.test.ts index 187135d1959..66b9731dfaa 100644 --- a/server/src/__tests__/workspace-runtime.test.ts +++ b/server/src/__tests__/workspace-runtime.test.ts @@ -6743,35 +6743,44 @@ describeEmbeddedPostgres("workspace runtime service control persistence", () => executionWorkspaceId, serviceName: "web", status: "provisioning", + healthStatus: "unknown", providerRef: null, }); expect(existsSync(markerPath)).toBe(false); await waitForMarker(markerPath); - const startingRow = await waitForPersistedStatus("starting"); - expect(startingRow).toMatchObject({ + // Readiness begins before the PID-bearing transaction commits. Keep the + // listener independent of DB reads and verify the completed records. + const services = await startPromise; + expect(services).toHaveLength(1); + expect(services[0]).toMatchObject({ + id: provisioningRow.id, companyId, projectId, projectWorkspaceId, executionWorkspaceId, issueId, serviceName: "web", - status: "starting", - healthStatus: "unknown", - }); - expect(startingRow.providerRef).toMatch(/^\d+$/); - expect(startingRow.port).toEqual(expect.any(Number)); - - const services = await startPromise; - expect(services).toHaveLength(1); - expect(services[0]).toMatchObject({ - id: startingRow.id, status: "running", healthStatus: "healthy", }); + expect(services[0]!.providerRef).toMatch(/^\d+$/); + expect(services[0]!.port).toEqual(expect.any(Number)); const runningRow = await waitForPersistedStatus("running"); - expect(runningRow.id).toBe(startingRow.id); + expect(runningRow).toMatchObject({ + id: provisioningRow.id, + companyId, + projectId, + projectWorkspaceId, + executionWorkspaceId, + issueId, + serviceName: "web", + status: "running", + healthStatus: "healthy", + providerRef: services[0]!.providerRef, + port: services[0]!.port, + }); await expect(fetch(services[0]!.url!)).resolves.toMatchObject({ ok: true }); const runtimeProvisionOperations = await db .select() diff --git a/server/src/onboarding-assets/default/AGENTS.md b/server/src/onboarding-assets/default/AGENTS.md index 14724839144..8c6df421c78 100644 --- a/server/src/onboarding-assets/default/AGENTS.md +++ b/server/src/onboarding-assets/default/AGENTS.md @@ -1,24 +1 @@ -You are an agent at Paperclip company. - -## Execution Contract - -- Start actionable work in the same heartbeat. Do not stop at a plan unless the issue explicitly asks for planning. -- Keep the work moving until it is done. If you need QA to review it, ask them. If you need your boss to review it, ask them. -- Leave durable progress in task comments, documents, or work products, then update the issue to a clear final disposition before you exit. -- When your work produces a user-inspectable deliverable file, follow the Paperclip skill's "Generated Artifacts and Work Products" workflow before final disposition. Use `skills/paperclip/scripts/paperclip-upload-artifact.sh` when working in this repo, create/update an artifact work product when the file is the deliverable, and link the uploaded attachment in the final comment. Do not rely on local filesystem paths as the only access path. If an important file intentionally remains workspace-only, create/update a work product with `metadata.resourceRef.kind: "workspace_file"` and a workspace-relative path, then name that work product and path in the final comment. Treat browse/search as a fallback for recovering workspace files, not the preferred deliverable path. -- When your work produces or updates an operator-facing engineering output, create/update the matching work product: `pull_request` for opened PRs, `preview_url` for published previews, `runtime_service` for managed preview/dev services, `commit` for notable pushed commits, and `branch` when the branch itself is the handoff. A comment is not a substitute for the work product access path. -- Comments, documents, screenshots, work products, and `Remaining` bullets are evidence, not valid liveness paths by themselves. -- Final disposition checklist: mark `done` when complete and verified; use `in_review` only with a real reviewer, approval, interaction, or monitor path; use `blocked` only with first-class blockers or a named unblock owner/action; create delegated follow-up issues with blockers when another agent owns the next step; keep `in_progress` only when a live continuation path exists. -- Use child issues for parallel or long delegated work instead of polling agents, sessions, or processes. -- Create child issues directly when you know what needs to be done. If the board/user needs to choose suggested tasks, answer structured questions, or confirm a proposal first, create an issue-thread interaction on the current issue with `POST /api/issues/{issueId}/interactions` using `kind: "suggest_tasks"`, `kind: "ask_user_questions"`, or `kind: "request_confirmation"`. -- Use `request_confirmation` instead of asking for yes/no decisions in markdown. Before presenting a plan for review, you MUST complete this publish contract: - 1. `PUT /issues/{id}/documents/plan` with `{ format: 'markdown', body, changeSummary }`. - 2. Re-`GET /documents/plan`, assert it returns `200`, and capture its `latestRevisionId`. - 3. Only then create `request_confirmation` with `target={ type: 'issue_document', key: 'plan', revisionId: latestRevisionId }` and `idempotencyKey=confirmation:{issueId}:plan:{revisionId}`. - 4. Wait for acceptance before creating implementation subtasks. - Never present a plan only in a thread comment or through `ask_user_questions`; comments are supporting context and questions are for gathering input, not plan review. -- `ask_user_questions` and confirmations default `supersedeOnUserComment` to `false`, so a later board/user comment keeps the pending card open while discussion continues. Set it to `true` when a new comment should replace the pending request. If you wake up from a superseding comment, revise the artifact, question set, or proposal and create a fresh interaction if input is still needed. -- For human input, save a pending question/confirmation interaction and set `in_review`; prose alone does not create a waiting path. Use `blockedByIssueIds` for issue dependencies. An agent may set an `unblockDescriptor` only for itself (`owner: { "agentId": "" }` plus `action`), not for the board/user or another agent. -- Respect budget, pause/cancel, approval gates, and company boundaries. - -Do not let work sit here. You must always update your task with a comment. +You are an agent in a Paperclip company. diff --git a/server/src/routes/agents.ts b/server/src/routes/agents.ts index 094a02a656e..11fdc1ab3c3 100644 --- a/server/src/routes/agents.ts +++ b/server/src/routes/agents.ts @@ -2759,8 +2759,8 @@ export function agentRoutes( // first-task/chief-of-staff/AGENTS.md, placeholders filled) over the agent's // entry file instead of the generic default. Honored only for board-authored // requests — the onboarding wizard runs as the board — so a client marker - // alone cannot swap another actor's instructions. The generic execution - // contract (default/AGENTS.md) is still appended on every run, unchanged. + // alone cannot swap another actor's instructions. Runtime coordination is + // supplied separately by the harness. async function resolveOnboardingFirstAgentBundle(params: { onboardingFirstAgent: unknown; actorType: string; diff --git a/server/src/services/onboarding-first-task-assets.ts b/server/src/services/onboarding-first-task-assets.ts index 2030faa519c..67e6aff2ae1 100644 --- a/server/src/services/onboarding-first-task-assets.ts +++ b/server/src/services/onboarding-first-task-assets.ts @@ -139,8 +139,8 @@ export async function renderChiefOfStaffPersona( } // The instruction bundle for the onboarding first agent: the chief-of-staff -// persona as the entry AGENTS.md. The generic execution contract -// (default/AGENTS.md) is still appended on every run by the runner, unchanged. +// persona as the entry AGENTS.md. Runtime coordination is supplied separately +// by the harness. export async function buildOnboardingFirstAgentInstructionsBundle( placeholders: OnboardingFirstTaskPlaceholders, ): Promise<{ files: Record; entryFile: string }> { diff --git a/skills/paperclip/SKILL.md b/skills/paperclip/SKILL.md index 695f91abf43..69a13988379 100644 --- a/skills/paperclip/SKILL.md +++ b/skills/paperclip/SKILL.md @@ -1,10 +1,10 @@ --- name: paperclip description: > - Interact with the Paperclip control plane API for task coordination and - governance. Use when checking assignments, updating issue status, posting - comments, delegating work, managing routines, or calling Paperclip API - endpoints. + Use for Paperclip-managed tasks and heartbeats: reading task context, delivering + task documents or files, updating completion or blockers, coordinating or + delegating work, and following company governance. Includes control plane API + operations for assignments, comments, approvals, and routines. --- # Paperclip Skill @@ -15,6 +15,8 @@ You run in **heartbeats** — short execution windows triggered by Paperclip. Ea In Paperclip, **task** and **issue** refer to the same work item. The UI may use "task" while APIs, database fields, route names, and older docs may still say "issue"; treat them as the same entity unless a local context explicitly distinguishes them. +**Task documents (API runtimes).** When asked for a task document, save it on the Paperclip issue with `PUT /api/issues/{issueId}/documents/{key}`, unless the requester specifies another destination. Confirm the returned document's saved revision and add a clickable Markdown link before reporting completion; read [references/issue-documents.md](references/issue-documents.md) for the payload and revision-safe updates. Downloadable files follow [Generated Artifacts and Work Products](#generated-artifacts-and-work-products). + ## Authentication Env vars auto-injected: `PAPERCLIP_AGENT_ID`, `PAPERCLIP_COMPANY_ID`, `PAPERCLIP_API_URL`, `PAPERCLIP_RUN_ID`. Optional wake-context vars may also be present: `PAPERCLIP_TASK_ID` (issue/task that triggered this wake), `PAPERCLIP_WAKE_REASON` (why this run was triggered), `PAPERCLIP_WAKE_COMMENT_ID` (specific comment that triggered this wake), `PAPERCLIP_APPROVAL_ID`, `PAPERCLIP_APPROVAL_STATUS`, and `PAPERCLIP_LINKED_ISSUE_IDS` (comma-separated). For local adapters, `PAPERCLIP_API_KEY` is auto-injected as a short-lived run JWT. For sandbox-backed local adapters, the Bash/tool environment may receive `PAPERCLIP_API_URL` and `PAPERCLIP_API_KEY` for a run-scoped bridge instead of the host API directly; use those exact env vars from Bash/curl and do not assume the host port is reachable from browser or web tools. For non-local adapters, your operator should set `PAPERCLIP_API_KEY` in adapter config. All requests use `Authorization: Bearer $PAPERCLIP_API_KEY`. All endpoints are under `/api`. Use JSON except for multipart attachment uploads and binary content downloads. Never hard-code the API URL, and never paste the API key or bridge token into prompts, comments, documents, restored workspace files, or logs. diff --git a/skills/paperclip/references/issue-documents.md b/skills/paperclip/references/issue-documents.md new file mode 100644 index 00000000000..4033dcc9422 --- /dev/null +++ b/skills/paperclip/references/issue-documents.md @@ -0,0 +1,58 @@ +# Issue documents through the API + +Use this procedure for requested Paperclip task documents in API-based runtimes. +Honor a destination explicitly chosen by the requester. Native Runner runtimes +use their document tool and its contract. Downloadable files follow +[artifacts.md](artifacts.md). + +## Create a document + +Use the injected `PAPERCLIP_API_URL` and bearer `PAPERCLIP_API_KEY`. Include +`X-Paperclip-Run-Id: $PAPERCLIP_RUN_ID` on the write and send JSON with +`Content-Type: application/json`. + +Choose a descriptive document key such as `report`; keys use lowercase letters, +numbers, underscores, or hyphens and are at most 64 characters. Plans use `plan`. + +```text +PUT /api/issues/{issueId}/documents/{key} +``` + +```json +{ + "title": "Task report", + "format": "markdown", + "body": "# Task report\n\nThe requested findings.", + "baseRevisionId": null +} +``` + +A successful write returns HTTP `201` for creation or `200` for an update, +with the saved document JSON. Check its `key`, `body`, and `latestRevisionId` +before claiming delivery. This receipt is sufficient; another GET is unnecessary +when it confirms the intended content and a saved revision. + +Construct a clickable Markdown link from the current issue and successful write +receipt, then add it to the completion comment or response: + +```javascript +const issuePath = issue.identifier + ? `/${issue.identifier.split("-")[0]}/issues/${issue.identifier}` + : `/issues/${issue.id}`; +const documentUrl = `${issuePath}#document-${saved.key}`; +const comment = `Saved [Task report](${documentUrl}).`; +``` + +Use the returned key: a locked document can redirect an agent's write to a new +document. Unnumbered issues use their ID; the board resolves their company. + +## Update or resolve an unclear write + +For an existing document, first `GET /api/issues/{issueId}/documents/{key}` +and read its body and `latestRevisionId`. Send the updated body with +`baseRevisionId` set to that revision. On `409`, fetch the latest document and +reconcile changes before retrying; preserve the skill's bounded-write retry rule. + +If a write's status or receipt is unclear, GET the document to check the saved +content and revision before retrying. If delivery cannot be confirmed, report +that limitation instead of claiming the requested document was saved. diff --git a/tests/runner-e2e/FIXTURES.md b/tests/runner-e2e/FIXTURES.md index 2ff1eace509..d65997921c0 100644 --- a/tests/runner-e2e/FIXTURES.md +++ b/tests/runner-e2e/FIXTURES.md @@ -17,6 +17,17 @@ profile, model qualification, environment, task, or ranking-snapshot change must change that fingerprint automatically so the dashboard can annotate the boundary instead of silently joining unlike totals. +The explicit [stock-harness suite](STOCK-HARNESS.md) wraps existing profiles with +`productionDefaultHireProfile`: omit only `instructionsBundle` so the public +hire route loads the shipped default, while preserving runtime, permissions, +auth, skills, and managed secret references. Do not replace this with a fixture +copy of the default manual. Public receipts check the exact independently +specified bundle before provider execution and again during cleanup, along with +both budget hard stops and actual legacy invocation prompts. Missing evidence +fails closed. The definition fingerprint includes the helper, graders, journey +sources, live fixture, and execution integration; editing those sources changes +the suite revision automatically. + ## Agent profiles Add `RunnerProfileFixture` entries in `catalog.ts`. A profile declares: diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 0da021312dc..b652b810ab7 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -444,6 +444,14 @@ pnpm test:e2e:runner -- --list --suite context-integrity pnpm test:e2e:runner -- --id context-integrity.runner-codex.local.ordered-comment-continuation ``` +`stock-harness` reuses ordered continuation, assigned-skill invocation, and chat +restart journeys with production-default hires instead of the custom QA manual. +Its 24 explicit local cells cover eight legacy/native profiles and are excluded +from `--all`. Run `pnpm test:e2e:runner:stock-harness` for the credential-free +instruction-layering, hire, and shared-prompt prerequisites. The +[suite contract](STOCK-HARNESS.md) maps each change to its graders, budgets, +evidence, and remaining qualification limits. + Each hardening oracle has positive and plausible-negative calibration tests. The review grader parses the worker's saved JSON and compares both source values and the consistency verdict. Hiring requires one identity, correct reporting diff --git a/tests/runner-e2e/STOCK-HARNESS.md b/tests/runner-e2e/STOCK-HARNESS.md new file mode 100644 index 00000000000..88e06c2cfb3 --- /dev/null +++ b/tests/runner-e2e/STOCK-HARNESS.md @@ -0,0 +1,248 @@ +# Stock harness with Paperclip + +This suite covers the instruction reductions tracked in the +[working checklist](../../doc/plans/2026-10-02-stock-harness-paperclip-checklist.md). +Existing context-integrity and chat evals supplied a custom QA instruction +bundle. Their passing results therefore did not qualify a hire with the tiny +production default. The new suite omits that fixture bundle and lets the public +agent-creation route materialize the shipped default. + +The 2026-10-03 legacy ACP Claude repair compares only the original +`assigned-skill-explicit-invocation` cell on matched current-master sources. +Both variants carry identical bounded skill-description/staged-path delivery +and env-free session persistence. Only the historical versus reduced default +manual/shared prompts and their declared structural unit expectations differ. +The public bundle/procedure observations derive from exact admitted source +bytes (eight-word identity or the independently pinned 4,249-byte historical +manual), never a branch/environment label. Candidate absence assertions stay +intact. Independent document/task and exact-credential guards are unchanged. +All stock tasks enforce `single_attempt` in the hosted launcher; automatic +product disposition recovery still counts as actual usage. This is not a new +24-cell campaign or qualification of the frozen native-completion context. + +## Coverage contract + +| Change | Required deterministic evidence | Product E2E evidence | +| --- | --- | --- | +| SH-1: native Codex preserves vendor base instructions | Serialized start/resume requests from the TypeScript driver, recovery paths, runnerd transport, Runner Lab/live sessions, and Rust provider use additive developer instructions. | Real new native Codex hires execute the three journeys below. Task success alone cannot prove vendor base preservation. | +| SH-2: identity-only default hire manual | Existing public agent-creation and onboarding-asset tests cover default, custom, and CEO exceptions. | The public bundle is exactly one `AGENTS.md` containing the eight-word shipped identity, checked before provider execution and again during cleanup. | +| SH-3: reduced shared legacy task/chat defaults and resume delta | Shared prompt tests and ACPX, Codex, OpenCode, Pi, Hermes, and Cursor Cloud adapter regressions retain runtime context and exclude removed generic procedures. | Actual legacy `adapter.invoke` prompts retain fresh identity/connection guidance, omit the removed procedures, and have complete receipts for every observed run. | + +`pnpm test:e2e:runner:stock-harness` runs these credential-free prerequisites, +including the Rust test. It writes JSON reports, the source SHA, a working-source +fingerprint, selected checks, test counts, and `providerCalls: 0` under +`results/stock-harness-preflight-/`. Missing reports or required skipped +assertions fail. Unrelated native tests filtered by the name selector remain +explicitly skipped; they are not counted as executed coverage. The live launcher +runs the prerequisites automatically before loading local credentials or starting +an isolated provider instance. Its subprocess receives only allowlisted toolchain +and operating-system variables. Disposable GitHub runners may resolve Cargo +dependencies; local prerequisites retain offline Cargo execution. +The ordinary server SDK dependency builder prepares the shared/SDK outputs +needed by route tests on a cold install with lifecycle scripts disabled. + +The direct Playwright path also requires the retained prerequisite receipt. It +verifies the exact checkout SHA, evaluated-source fingerprint, all requested +Vitest reports and required assertions, and the Rust result before creating a +company. Missing, stale, partial, or failed prerequisites cannot qualify a cell. +Oracle/admission calibration is itself included in the prerequisite gate. + +## Live matrix + +There are 24 explicit local cells: these eight existing profiles each run three +existing journeys with independently calibrated graders. + +- Legacy: `legacy-codex`, `legacy-claude`, `legacy-opencode`, + `legacy-acp-codex`, `legacy-acp-claude`. +- Native: `runner-codex`, `runner-acpx-claude`, `runner-opencode`. + +| Journey | Expected provider turns | Observable outcome | Cell deadline | +| --- | --- | --- | --- | +| `assigned-skill-explicit-invocation` | 1 | Public skill creation/pinning, an explicit skill request, and saved output containing the marker available only in the skill body. | 12 minutes | +| `ordered-comment-continuation` | 2 | Initial report followed by three ordered public comments, including repeated wording and a changed scope; final saved report preserves the ledger and requested scope. | 12 minutes | +| `continuity-restart` | 3 | Task-backed chat retains the requested context across a server restart and subsequent replies. | 15 minutes | + +A full matrix expects 48 provider turns. Models, credentials, effort, +permissions, assigned skills, environment, and managed secret references are +inherited from the existing profile; only its QA manual is omitted. Both company +and agent receive a 1,000-cent monthly hard stop before provider execution, +verified through public records. These limits do not predict final spend: +attempts, retries, partial runs, unknown billing, and cleanup remain in the +existing campaign accounting and qualification rules. + +`stock-harness` is excluded from `--all`; select it explicitly. It has no Daytona +cells. Pending or unrepresented harnesses are not live-qualified by this matrix. +Pi/Hermes/Cursor Cloud rendering coverage is deterministic here. The separate +Codex-through-ACP base-instruction patch is still an open checklist item: its +legacy cells qualify the common prompt reduction, not vendor base preservation. + +## Run and inspect + +```sh +# No credentials or paid providers: +pnpm test:e2e:runner:stock-harness --list +pnpm test:e2e:runner:stock-harness +pnpm test:e2e:runner:typecheck +pnpm test:e2e:runner:unit +pnpm test:e2e:runner -- --list --suite stock-harness + +# With authorization and the selected profile's required provider credential: +pnpm test:e2e:runner -- --id stock-harness.runner-codex.local.assigned-skill-explicit-invocation +``` + +Use an exact cell first, then expand profile/journey selections when its evidence +is understood. The ordinary launcher, isolated instance, fixture cleanup, +screenshots, attempt history, usage/cost reporting, and dashboard publisher are +unchanged. `snapshots/stock-harness-hire.json` captures the public bundle and +budgets before paid execution. `snapshots/stock-harness.json` captures the final +bundle, budgets, run IDs, actual legacy prompts, and matcher verdicts. Evidence +API failures retain an error snapshot and cannot pass. All snapshots use the +existing secret sanitizer; screenshots remain the original captured pixels. + +The oracle's identity is independent of the implementation constant, so changing +the shipped manual cannot silently change the expected result. The suite +definition digest incorporates the evaluated default manual, shared prompt +implementation and connection guidance, fixture, journeys, grader, prerequisite, and execution integration +sources. Positive and plausible-negative support tests exercise +missing/malformed receipts, manual regrowth, removed startup/resume procedures, +absent connection guidance, and budget drift. Existing calibrated lifecycle +graders still own task/chat success. + +The skill oracle proves the requested pinned skill's output marker reached the +saved result without appearing in the task request or follow-up comments. Skill +tool/read event detection is retained as supporting evidence; it is not a +required cross-provider tool-trace assertion. + +## Qualification status + +Setup was validated locally on 2026-10-02 with the deterministic prerequisites, +Product E2E support tests, typecheck, and discovery: 478 executed prerequisite +tests passed (477 TypeScript plus one Rust), along with 860 support tests and +discovery of all 24 cells. The 313 unrelated native tests filtered by the gate +are not counted as passing coverage. Local evidence is retained under +`results/stock-harness-preflight-2026-10-02T16-15-39.064Z/preflight.json`. +Earlier interrupted, discovery-failure, and setup/test-timeout attempts remain +retained; the final unchanged-assertion retry passed. No live cells or paid +providers were run during that initial setup. This establishes executable coverage, not a live reliability +result or improved coding quality. A quality claim needs comparable tasks, +models, effort, tools, and independently graded before/after results. + +The suite exercises fresh isolated hires and their continuations. Existing saved +manuals and old Codex sessions are not automatically migrated. The latter still +need a provider-session reset to restore a previously replaced vendor base. +Private Runner protocol definitions remain separate: they use mock control-plane +operations and cannot substitute for this public hiring and assembled-prompt +coverage. + +## GitHub qualification and matched comparison + +The instruction reductions and coverage are under review in +[PR #14948](https://github.com/paperclipai/paperclip/pull/14948). +The first diagnostic native Codex skill cell +[passed on GitHub](https://github.com/paperclipai/paperclip/actions/runs/37034213743) +at `a63437069de58d22ee5adbcb6a6202c007dcf037`. All seven independent skill/task +checks passed; the provider run lasted 34.745 seconds and the cell 56.660 seconds. +Its tiny public hire bundle and budget receipts passed, and cleanup passed. +This diagnostic predates enforced prerequisite admission and the expanded source +digest, so it is retained separately from the final qualification matrix. + +The first full candidate attempt at `36e987246b649927e96ce1184cd616c4e490106e` +[was cancelled during prerequisites](https://github.com/paperclipai/paperclip/actions/runs/37037105491). +Cold protected installs disable lifecycle scripts, leaving the plugin SDK unbuilt; +SH-2/SH-3 could not import it. Provider admission was not reached. This setup +failure is retained separately from behavioral results. The prerequisite now runs +the ordinary server dependency builder first, retains its output and exit status, +and refuses admission when setup fails. A cold legacy pilot must pass before the +full matrix retry. + +The [fixed legacy pilot](https://github.com/paperclipai/paperclip/actions/runs/37039240025) +at `4163dbfd0fd4d145bfa52b4d7f80eb59a362ee36` passed SDK setup, hire/shared +prompt/oracle checks and the Rust additive test. It stopped before providers: +the real daemon-frame test lacked the cold `paperclip-runnerd` binary. Setup now +builds that daemon from the locked Rust source before TypeScript gates, retains +`runnerd-build.txt`, and requires both setup exits in the admission receipt. +The required daemon-frame test selects that built debug binary explicitly, +without replacing staged product binaries. The receipt records its SHA-256; +verification rejects a changed binary before provider admission. +The [next cold pilot](https://github.com/paperclipai/paperclip/actions/runs/37040493183) +at `ac6ddefb589c18fa9c30946db40e58499e9fb0e0` then found the required fake Codex +protocol fixture absent. It also stopped before paid providers. Setup uses the +package's ordinary locked workspace `--bins` build, covering both the daemon +and its fixture; verification binds both binaries to the retained receipt. + +That final cold pilot passed all 537 prerequisites on GitHub and reached Claude +Sonnet 4.6. It saved a workspace `task-output.md` and completed the issue, while +the independent durable-document oracle found no Paperclip issue document. +The pinned skill's phrase "task document" does not explicitly name storage in +Paperclip; matched historical results must precede any regression attribution. +The task, instruction delivery, budget, and cleanup receipts remain retained. + +Its trusted merged report rejected the prerequisite folder alongside the campaign +root and synthesized a missing-result infrastructure error. Raw packaged cell +results were uploaded and remain inspectable. Prerequisites now live under the +exact campaign root; the unchanged trusted selector contract is calibrated in +the mandatory gate. The initial full matched campaigns at `f02d8d0df` and +`12c5433c6` retain their original raw results and publication outcomes. Any +reconstructed comparison must declare directory-layout recovery and preserve +every result, hash, original failure, and assertion. + +Dispatch the trusted workflow from `master`, with `target_branch` naming the +same-repository candidate and an exact cell selector first. The workflow resolves +that target once to an immutable SHA. Never dispatch target-controlled workflow +definitions with protected credentials. Protected environments, scoped provider +keys, frozen target dependencies, report sanitization, publication, and existing +bounded retry/cleanup policies retain their existing owners. + +A temporary `codex/stock-harness-previous-instructions` branch compares the same +24 cells with the previous default manual and shared startup/resume prompts. +It holds merged native Codex fix #14920 constant. Only those two production +instruction sources differ. Its explicit historical structural oracle expects +the old manual and records old generic procedures; the candidate's reduction +assertions remain mandatory. Historical tests verify that prior contract. +The independent skill/context/chat journeys, behavioral graders, fixtures, +models, effort, tools, permissions, and credentials are identical. The retained +comparison manifest records their hashes and the restored instruction revision. + +One full campaign per variant initially expects 48 provider turns, plus any +existing bounded automatic retries; the earlier one-cell diagnostic remains +separate. Compare behavioral results by profile and journey, with missing +evidence unqualified. Keep structural instruction differences separate from task +success. Report every attempt, failure attribution, provider timing, token usage, +reported costs and unknown spend. A reported zero subtotal is not proof of zero +provider spending. This small single-trial matrix cannot establish general coding +quality or broad performance equivalence, and does not compare #14920 before/after. + +## Measured comparison and unresolved delivery + +The [dated live report](../../doc/plans/2026-10-02-stock-harness-live-comparison.md) +and its safe JSON projection contain the per-profile/case comparison, exact +source hashes, all campaign and recovery links, timing/usage, cost coverage, +security failures, clipping limits and publication-layout recovery. Candidate +`f02d8d0df` has 24 retained results: 15 pass and nine fail. Historical +`12c5433c6` also has 24 retained results: 15 pass and nine fail. Its single +AWS-runner recovery timed out; original missing evidence remains recorded. Classic Claude/OpenCode skill runs save no Paperclip task document +where historical runs save one; the original oracle remains failed. + +The merged native Codex change is held constant. Same aggregate success counts +would not establish equivalence: case outcomes differ, fixture storage wording +is ambiguous, ACP credential guards fail, and some public prompt receipts are +clipped. Native finish/block tool guidance belongs to native runners; legacy +document-delivery guidance must use the actual Paperclip skill/API path. No +production or skill instructions have been changed to turn the measured failures +into passes. PR #14948 remains draft. + +The corrected packaging pilot at `1eb5ba420` passes on GitHub with valid +evidence and cleanup. It validates prerequisite nesting under the exact campaign +root using the unchanged trusted selector. Its current prerequisite runs 557 +checks (556 TypeScript plus one Rust); all 895 E2E support tests pass. The 313 +filtered native tests are not counted. Original failed/partial attempts and +publication failures remain retained; local reconstructed copies move folders +without editing results, graders or usage. Never dispatch a second development +campaign for an active target branch: workflow concurrency supersedes the older +run. Use a separate frozen-source branch when independent campaigns must overlap. + +## Focused legacy delivery repair + +The original 24 cells and original assigned-skill request/procedure remain unchanged. Two added explicit Paperclip-document cells apply only to classic Claude and OpenCode, making 26 catalog cells (50 expected turns if every cell is selected). The repair campaign selects only the original skill case and new document case for those two profiles: four cells per variant, eight expected turns total. No full matrix rerun is planned. + +Both skill sources are recorded in definition/admission digests. The pre-fix baseline restores the old SKILL.md and records the new reference as absent, without copying the new recipe into that baseline. It holds the tiny manual/shared prompts and all fixture/model/auth/effort inputs fixed. The explicit document oracle checks actual public content/revision and an exact same-app document link, with plausible-negative calibrations. diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index 49748c8a6f6..735e90e06f8 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -136,10 +136,10 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 50, 6, 20, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, + 6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 50, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, ]); - expect(validateRunnerCatalog()).toHaveLength(444); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(444); + expect(validateRunnerCatalog()).toHaveLength(470); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(470); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(48); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index 89301b7f111..8aa562705ed 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -8,7 +8,8 @@ import { taskTitleTasks, taskTitleDefinitionDigest, TASK_TITLE_BUDGET_CENTS } fr import { blockerTasks, blockerProfile } from "./blocker-cases.js"; import { accountingTasks } from "./accounting-cases.js"; import { continuationTasks } from "./continuation-cases.js"; -import { contextIntegrityTasks } from "./context-integrity-cases.js"; +import { contextIntegrityTasks, paperclipDocumentTask } from "./context-integrity-cases.js"; +import { productionDefaultHireProfile, stockHarnessSourceDigest, stockHarnessSkillSources } from "./stock-harness.js"; import { lifecycleLiveTasks, lifecycleLiveDefinitionDigest } from "./lifecycle-live-cases.js"; import { everydayTasks, productionStoryProfile } from "./everyday-cases.js"; @@ -54,6 +55,7 @@ const SELECTABLE_GROUPS = [ "chat", "onboarding", "context-integrity", + "stock-harness", ] as const; const SAMPLE_UUID = "11111111-1111-4111-8111-111111111111"; @@ -1224,6 +1226,30 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ prerequisiteGate: PENDING_PROFILE_PREREQUISITES, }, }, + { + id: "stock-harness", + label: "Stock harness with Paperclip", + manualOnly: true, + description: "Production-default hires, reduced shared prompts, assigned skills, continuation, and persistent chat.", + groups: ["stock-harness", "native", "legacy"], + profiles: [...contextIntegrityProfiles.filter(profile => !pendingContextIntegrityProfiles.includes(profile)), + runnerProfiles.find(profile => profile.id === "legacy-opencode")!].map(productionDefaultHireProfile), + environments: [localEnvironment], + tasks: [...contextIntegrityTasks, chatTasks.find(task => task.id === "continuity-restart")!, paperclipDocumentTask] + .map(task => ({ ...task, automaticRetryPolicy: "single_attempt" as const })), + expectedMatrixSize: 26, + excludedExecutionIds: ["legacy-codex", "legacy-acp-codex", "legacy-acp-claude", "runner-codex", "runner-acpx-claude", "runner-opencode"] + .map(profile => `stock-harness.${profile}.local.${paperclipDocumentTask.id}`), + definitionMetadata: { + version: 2, instructions: "production-default-hire", scheduling: "explicit-only", + automaticRetryPolicy: "single_attempt", maximumAttemptsPerCell: 1, + sourceDigest: stockHarnessSourceDigest(), + operationalSkillSources: stockHarnessSkillSources(), + grading: "public-default-bundle-and-delivered-prompts-plus-independent-lifecycle-oracles", + vendorBaseEvidence: "required deterministic Codex driver/runnerd/Rust gate; task success is not vendor-base proof", + paidCalls: "one skill turn, two ordered-comment turns, three chat turns per profile", + }, + }, { id: "first-task", label: "First-task onboarding", description: "Production onboarding, first replies, approval, and durable task execution.", diff --git a/tests/runner-e2e/context-integrity-cases.ts b/tests/runner-e2e/context-integrity-cases.ts index ac28da1cad9..388b87f8597 100644 --- a/tests/runner-e2e/context-integrity-cases.ts +++ b/tests/runner-e2e/context-integrity-cases.ts @@ -5,10 +5,14 @@ export const CONTEXT_INTEGRITY_CASES = [ "assigned-skill-explicit-invocation", ] as const; -export type ContextIntegrityCase = (typeof CONTEXT_INTEGRITY_CASES)[number]; +export const PAPERCLIP_DOCUMENT_CASE = "assigned-skill-paperclip-document" as const; +export type ContextIntegrityCase = (typeof CONTEXT_INTEGRITY_CASES)[number] | typeof PAPERCLIP_DOCUMENT_CASE; +export function isAssignedSkillContext(id: string): boolean { + return id === "assigned-skill-explicit-invocation" || id === PAPERCLIP_DOCUMENT_CASE; +} export function contextIntegrityScenario(id: string, nonce: string) { - if (!CONTEXT_INTEGRITY_CASES.includes(id as ContextIntegrityCase)) + if (!CONTEXT_INTEGRITY_CASES.includes(id as (typeof CONTEXT_INTEGRITY_CASES)[number]) && id !== PAPERCLIP_DOCUMENT_CASE) throw new Error(`Unknown context-integrity case: ${id}`); const marker = `CONTEXT_OK_${nonce.replace(/[^a-z0-9]/gi, "")}`; const repeated = "Append an entry for every request, even identical wording."; @@ -23,15 +27,17 @@ export function contextIntegrityScenario(id: string, nonce: string) { prompt: id === "ordered-comment-continuation" ? `Maintain a packing list report document. Start with the initial scope: passport and charger. Save the initial report and leave this task waiting for follow-up comments. When follow-up comments arrive, copy each user's exact wording verbatim into one ordered ledger, including identical wording repeated more than once. After the follow-up batch, save one report containing both initial items and every verbatim request in arrival order. Add a separate section headed "## Final scope" outside the quoted ledger. Put only the final requested scope name as plain text in that section. Then finish. Do not invent extra comments, child tasks, or deliverables.` - : "Use the assigned Context integrity output skill for this task, then follow its instructions and finish the task. Do not create child tasks or unrelated deliverables.", + : id === PAPERCLIP_DOCUMENT_CASE + ? "Use the assigned Context integrity output skill. Save its requested document as a Paperclip document on this task, check the saved revision and content, and post a completion comment with a clickable Markdown link to the saved document in the Paperclip task UI, including the company prefix. A workspace file alone does not deliver this requested Paperclip document. Finish after delivery. Do not create child tasks or unrelated deliverables." + : "Use the assigned Context integrity output skill for this task, then follow its instructions and finish the task. Do not create child tasks or unrelated deliverables.", comments: [repeated, repeated, changed] as const, }; } -export const contextIntegrityTasks: readonly RunnerTaskFixture[] = - CONTEXT_INTEGRITY_CASES.map((id) => ({ +function contextIntegrityTask(id: ContextIntegrityCase): RunnerTaskFixture { + return { id, - label: id === "ordered-comment-continuation" ? "Ordered comment continuation" : "Assigned skill invocation", + label: id === "ordered-comment-continuation" ? "Ordered comment continuation" : id === PAPERCLIP_DOCUMENT_CASE ? "Explicit Paperclip document delivery" : "Assigned skill invocation", groups: ["context-integrity"], workMode: "standard", flow: "context_integrity", @@ -42,4 +48,8 @@ export const contextIntegrityTasks: readonly RunnerTaskFixture[] = buildPrompt: (nonce) => contextIntegrityScenario(id, nonce).prompt, buildVisibleMarker: (nonce) => contextIntegrityScenario(id, nonce).marker, buildMatchers: () => [], - })); + }; +} + +export const contextIntegrityTasks: readonly RunnerTaskFixture[] = CONTEXT_INTEGRITY_CASES.map(contextIntegrityTask); +export const paperclipDocumentTask = contextIntegrityTask(PAPERCLIP_DOCUMENT_CASE); diff --git a/tests/runner-e2e/context-integrity-flow.ts b/tests/runner-e2e/context-integrity-flow.ts index b1a2dabb943..ad1e240111c 100644 --- a/tests/runner-e2e/context-integrity-flow.ts +++ b/tests/runner-e2e/context-integrity-flow.ts @@ -1,6 +1,6 @@ import type { Page } from "@playwright/test"; import { randomUUID } from "node:crypto"; -import { contextIntegrityScenario } from "./context-integrity-cases.js"; +import { contextIntegrityScenario, isAssignedSkillContext } from "./context-integrity-cases.js"; import { gradeContextIntegrity, type ContextIntegrityCheckpoint } from "./context-integrity-scoring.js"; import { pollUntil, type RunnerApi } from "./api.js"; import { createTaskThroughUi } from "./user-actions.js"; @@ -102,7 +102,7 @@ export async function runContextIntegrityFlow(input: { let assignedVersionId: string | null = null; const checkpoints: ContextIntegrityCheckpoint[] = []; - if (scenario.id === "assigned-skill-explicit-invocation") { + if (isAssignedSkillContext(scenario.id)) { await api.patch("/api/instance/settings/experimental", { enableBetaSkills: true }); const markdown = `---\nname: ${scenario.skillKey}\ndescription: Context integrity output procedure.\n---\n\n# Context integrity output procedure\n\nWrite exactly one task document whose body contains the marker ${scenario.marker}. Finish the task after saving that document.`; if (!markdown.startsWith(`---\nname: ${scenario.skillKey}\ndescription:`) || markdown.includes("\\n")) { @@ -141,18 +141,18 @@ export async function runContextIntegrityFlow(input: { api.get(`/api/issues/${issue!.id}/documents`), api.get(`${companyPath}/skills`), api.get(`/api/issues/${issue!.id}/queued-comments`), - scenario.id === "assigned-skill-explicit-invocation" ? api.get(`/api/agents/${fixtures.agent.id}/skills?companyId=${fixtures.company.id}`) : Promise.resolve({} as Row), + isAssignedSkillContext(scenario.id) ? api.get(`/api/agents/${fixtures.agent.id}/skills?companyId=${fixtures.company.id}`) : Promise.resolve({} as Row), ]); const detailedDocuments = await Promise.all(documents.map((document) => api.get(`/api/issues/${issue!.id}/documents/${encodeURIComponent(String(document.key))}`))); - const assignedSkill = scenario.id === "assigned-skill-explicit-invocation" + const assignedSkill = isAssignedSkillContext(scenario.id) ? skillRows.find((skill) => skill.key === scenario.assignedSkill?.key || skill.slug === scenario.assignedSkill?.key) : undefined; const desiredSkill = (assignedState.desiredSkillEntries as Array | undefined)?.find((entry) => entry.key === scenario.assignedSkill?.key); const runEvents = await Promise.all(runs.map((run) => api.get(`/api/heartbeat-runs/${run.id}/events?limit=1000`))); const runLogs = await Promise.all(runs.map((run) => readContextIntegrityRunLog(api, run.id))); - const skillInvocationEvidence = scenario.id === "assigned-skill-explicit-invocation" && runEvents.some((events) => events.some((event) => containsExplicitSkillInput(event, String(assignedSkill?.slug ?? scenario.skillKey)))); - skillRequestText = scenario.id === "assigned-skill-explicit-invocation" ? String(issue?.description ?? "") : ""; - checkpoints.push({ phase, issue: { id: issue!.id, status: String(issue!.status) }, comments, queuedComments, documents: detailedDocuments as Array<{ key: string; body?: string | null }>, runs, runEvents: runEvents.flat(), runLogs, assignedSkill: assignedSkill ? { key: String(desiredSkill?.key ?? assignedSkill.key ?? assignedSkill.slug), runtimeName: String(assignedSkill.slug ?? ""), versionId: desiredSkill?.versionId === assignedVersionId ? assignedVersionId : null, markdown: String(assignedSkill.markdown ?? scenario.assignedSkill?.markdown ?? "") } : undefined, skillRequestText: scenario.id === "assigned-skill-explicit-invocation" ? skillRequestText : undefined, skillInvocationEvidence }); + const skillInvocationEvidence = isAssignedSkillContext(scenario.id) && runEvents.some((events) => events.some((event) => containsExplicitSkillInput(event, String(assignedSkill?.slug ?? scenario.skillKey)))); + skillRequestText = isAssignedSkillContext(scenario.id) ? String(issue?.description ?? "") : ""; + checkpoints.push({ phase, issue: { id: issue!.id, status: String(issue!.status), identifier: String(issue!.identifier ?? ""), issuePrefix: String(fixtures.company.issuePrefix ?? ""), appOrigin: new URL(page.url()).origin }, comments, queuedComments, documents: detailedDocuments as Array<{ key: string; body?: string | null }>, runs, runEvents: runEvents.flat(), runLogs, assignedSkill: assignedSkill ? { key: String(desiredSkill?.key ?? assignedSkill.key ?? assignedSkill.slug), runtimeName: String(assignedSkill.slug ?? ""), versionId: desiredSkill?.versionId === assignedVersionId ? assignedVersionId : null, markdown: String(assignedSkill.markdown ?? scenario.assignedSkill?.markdown ?? "") } : undefined, skillRequestText: isAssignedSkillContext(scenario.id) ? skillRequestText : undefined, skillInvocationEvidence }); const checks = gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints }); input.observe(issue!, runs, checks); await input.evidence("context-integrity.json", { schema: "paperclip.context-integrity.v1", scenario, budgetGuard, checkpoints, checks }); @@ -175,7 +175,7 @@ export async function runContextIntegrityFlow(input: { await api.patch(`/api/agents/${fixtures.agent.id}/budgets`, { budgetMonthlyCents: budgetGuard.agentMonthlyCents, }); - const taskPrompt = scenario.id === "assigned-skill-explicit-invocation" + const taskPrompt = isAssignedSkillContext(scenario.id) ? `${scenario.prompt}\n\nUse /${scenario.skillKey} for this request.` : scenario.prompt; skillRequestText = taskPrompt; diff --git a/tests/runner-e2e/context-integrity-scoring.ts b/tests/runner-e2e/context-integrity-scoring.ts index e307a66bc04..c1aae9c2668 100644 --- a/tests/runner-e2e/context-integrity-scoring.ts +++ b/tests/runner-e2e/context-integrity-scoring.ts @@ -1,11 +1,11 @@ -import type { ContextIntegrityCase } from "./context-integrity-cases.js"; +import { isAssignedSkillContext, PAPERCLIP_DOCUMENT_CASE, type ContextIntegrityCase } from "./context-integrity-cases.js"; export interface ContextIntegrityCheckpoint { phase: "initial" | "comment-1" | "comment-2" | "comment-3" | "final"; - issue: { id: string; status: string }; + issue: { id: string; status: string; identifier?: string; issuePrefix?: string; appOrigin?: string }; comments: Array>; queuedComments?: Record; - documents: Array<{ key: string; body?: string | null }>; + documents: Array<{ key: string; body?: string | null; latestRevisionId?: string | null; latestRevisionNumber?: number | null }>; runs: Array>; assignedSkill?: { key: string; runtimeName?: string; versionId?: string | null; markdown?: string }; skillRequestText?: string; @@ -117,7 +117,7 @@ export function gradeContextIntegrity(input: { const expectedCommentIds = humanRows.slice(0, input.comments.length).map((comment) => String(comment.id ?? "")); check("continuation-wake-comment-ids", Boolean(continuation) && JSON.stringify(wakeCommentIds(continuation)) === JSON.stringify(expectedCommentIds), "The deferred continuation wake must carry all public comment IDs in arrival order."); } - if (input.id === "assigned-skill-explicit-invocation") { + if (isAssignedSkillContext(input.id)) { const runtimeName = String(initial?.assignedSkill?.runtimeName ?? initial?.assignedSkill?.key ?? ""); const requestText = String(initial?.skillRequestText ?? ""); const references = requestText.split(/\s+/).map((word) => word.replace(/[.,]$/, "")); @@ -126,8 +126,27 @@ export function gradeContextIntegrity(input: { check("skill-source-marker", Boolean(initial?.assignedSkill?.markdown?.includes(input.marker)), "The assigned pinned skill source must contain the output marker."); check("marker-not-in-request", !requestText.includes(input.marker) && !finalComments.some((comment) => comment.includes(input.marker)), "The output marker must originate from the assigned skill, not the task request or comments."); } - const output = final ? oneOutput(final, input.marker, input.id === "assigned-skill-explicit-invocation") : undefined; - check("single-durable-output", Boolean(output) && final!.documents.length === 1, input.id === "assigned-skill-explicit-invocation" ? "Exactly one durable task document must contain the skill's marker." : "Exactly one durable packing report document must be saved."); + const output = final ? oneOutput(final, input.marker, isAssignedSkillContext(input.id)) : undefined; + check("single-durable-output", Boolean(output) && final!.documents.length === 1, isAssignedSkillContext(input.id) ? "Exactly one durable task document must contain the skill's marker." : "Exactly one durable packing report document must be saved."); + if (input.id === PAPERCLIP_DOCUMENT_CASE) { + check("saved-document-revision", Boolean(output?.latestRevisionId) && Number.isInteger(output?.latestRevisionNumber) && Number(output?.latestRevisionNumber) > 0, + "The public saved document must have a persisted revision and the requested content."); + const route = output && final?.issue.identifier && final.issue.issuePrefix + ? `/${final.issue.issuePrefix}/issues/${final.issue.identifier}#document-${output.key}` : undefined; + const agentComments = (final?.comments ?? []).filter(comment => comment.authorType === "agent" || Boolean(comment.authorAgentId)); + const origin = final?.issue.appOrigin; + const usableLink = Boolean(route && origin && agentComments.some(comment => { + const links = String(comment.body ?? "").matchAll(/\[[^\]]*\]\((?:<([^>]+)>|([^\s)]+))(?:\s+["'][^"']*["'])?\)/g); + return [...links].some(link => { + try { + const url = new URL(link[1] ?? link[2]!, origin); + return url.origin === origin && `${url.pathname}${url.hash}` === route && !url.search; + } catch { return false; } + }); + })); + check("saved-document-link", usableLink, + "An agent completion comment must link the exact saved document on this task; a local path or claimed URL does not suffice."); + } check("completed-task", final?.issue.status === "done", `Final task status: ${final?.issue.status ?? "missing"}.`); check("successful-runs", Boolean(final?.runs.length) && final!.runs.every((run) => run.status === "succeeded"), "All recorded context-integrity runs must succeed."); return checks; diff --git a/tests/runner-e2e/fixtures/stock-harness/historical-default-agents.md b/tests/runner-e2e/fixtures/stock-harness/historical-default-agents.md new file mode 100644 index 00000000000..14724839144 --- /dev/null +++ b/tests/runner-e2e/fixtures/stock-harness/historical-default-agents.md @@ -0,0 +1,24 @@ +You are an agent at Paperclip company. + +## Execution Contract + +- Start actionable work in the same heartbeat. Do not stop at a plan unless the issue explicitly asks for planning. +- Keep the work moving until it is done. If you need QA to review it, ask them. If you need your boss to review it, ask them. +- Leave durable progress in task comments, documents, or work products, then update the issue to a clear final disposition before you exit. +- When your work produces a user-inspectable deliverable file, follow the Paperclip skill's "Generated Artifacts and Work Products" workflow before final disposition. Use `skills/paperclip/scripts/paperclip-upload-artifact.sh` when working in this repo, create/update an artifact work product when the file is the deliverable, and link the uploaded attachment in the final comment. Do not rely on local filesystem paths as the only access path. If an important file intentionally remains workspace-only, create/update a work product with `metadata.resourceRef.kind: "workspace_file"` and a workspace-relative path, then name that work product and path in the final comment. Treat browse/search as a fallback for recovering workspace files, not the preferred deliverable path. +- When your work produces or updates an operator-facing engineering output, create/update the matching work product: `pull_request` for opened PRs, `preview_url` for published previews, `runtime_service` for managed preview/dev services, `commit` for notable pushed commits, and `branch` when the branch itself is the handoff. A comment is not a substitute for the work product access path. +- Comments, documents, screenshots, work products, and `Remaining` bullets are evidence, not valid liveness paths by themselves. +- Final disposition checklist: mark `done` when complete and verified; use `in_review` only with a real reviewer, approval, interaction, or monitor path; use `blocked` only with first-class blockers or a named unblock owner/action; create delegated follow-up issues with blockers when another agent owns the next step; keep `in_progress` only when a live continuation path exists. +- Use child issues for parallel or long delegated work instead of polling agents, sessions, or processes. +- Create child issues directly when you know what needs to be done. If the board/user needs to choose suggested tasks, answer structured questions, or confirm a proposal first, create an issue-thread interaction on the current issue with `POST /api/issues/{issueId}/interactions` using `kind: "suggest_tasks"`, `kind: "ask_user_questions"`, or `kind: "request_confirmation"`. +- Use `request_confirmation` instead of asking for yes/no decisions in markdown. Before presenting a plan for review, you MUST complete this publish contract: + 1. `PUT /issues/{id}/documents/plan` with `{ format: 'markdown', body, changeSummary }`. + 2. Re-`GET /documents/plan`, assert it returns `200`, and capture its `latestRevisionId`. + 3. Only then create `request_confirmation` with `target={ type: 'issue_document', key: 'plan', revisionId: latestRevisionId }` and `idempotencyKey=confirmation:{issueId}:plan:{revisionId}`. + 4. Wait for acceptance before creating implementation subtasks. + Never present a plan only in a thread comment or through `ask_user_questions`; comments are supporting context and questions are for gathering input, not plan review. +- `ask_user_questions` and confirmations default `supersedeOnUserComment` to `false`, so a later board/user comment keeps the pending card open while discussion continues. Set it to `true` when a new comment should replace the pending request. If you wake up from a superseding comment, revise the artifact, question set, or proposal and create a fresh interaction if input is still needed. +- For human input, save a pending question/confirmation interaction and set `in_review`; prose alone does not create a waiting path. Use `blockedByIssueIds` for issue dependencies. An agent may set an `unblockDescriptor` only for itself (`owner: { "agentId": "" }` plus `action`), not for the board/user or another agent. +- Respect budget, pause/cancel, approval gates, and company boundaries. + +Do not let work sit here. You must always update your task with a comment. diff --git a/tests/runner-e2e/launch.ts b/tests/runner-e2e/launch.ts index 7cafc2c8f09..2e28e2a975f 100644 --- a/tests/runner-e2e/launch.ts +++ b/tests/runner-e2e/launch.ts @@ -52,6 +52,8 @@ import { } from "./types.js"; import { assertRunnerE2EPrerequisites } from "./prerequisites.js"; import { assertNativeCompletionSelection, prepareNativeCompletionPreflight, NATIVE_COMPLETION_PREFLIGHT_ENV } from "./native-completion-admission.js"; +import { prepareStockHarnessPreflight, STOCK_PREFLIGHT_ENV } from "./stock-harness-admission.js"; + import { reapNewDetachedDarwinSharedMemory, snapshotDarwinSharedMemory, @@ -1084,13 +1086,17 @@ async function main() { // profiles remain discoverable, but cannot reach a provider. assertRunnerE2EPrerequisites(executions); assertNativeCompletionSelection(executions); - let nativeCampaignId: string | undefined; + const campaignId = cleanId( + process.env.PAPERCLIP_E2E_CAMPAIGN_ID ?? + `local-${new Date().toISOString().replace(/[:.]/g, "-")}`, + ); + const summaryDir = path.join(resultsRoot, campaignId); + await mkdir(summaryDir, { recursive: true }); if (executions.some(execution => execution.suite.id === "native-completion")) { - nativeCampaignId = cleanId(process.env.PAPERCLIP_E2E_CAMPAIGN_ID ?? - `local-${new Date().toISOString().replace(/[:.]/g, "-")}`); - const nativeCampaignDirectory = path.join(resultsRoot, nativeCampaignId); - await mkdir(nativeCampaignDirectory, { recursive: true }); - process.env[NATIVE_COMPLETION_PREFLIGHT_ENV] = prepareNativeCompletionPreflight(nativeCampaignDirectory); + process.env[NATIVE_COMPLETION_PREFLIGHT_ENV] = prepareNativeCompletionPreflight(summaryDir); + } + if (executions.some(execution => execution.suite.id === "stock-harness")) { + process.env[STOCK_PREFLIGHT_ENV] = prepareStockHarnessPreflight(summaryDir); } await loadLocalEnvironment(process.env); @@ -1113,12 +1119,7 @@ async function main() { ); } - const campaignId = nativeCampaignId ?? cleanId( - process.env.PAPERCLIP_E2E_CAMPAIGN_ID ?? - `local-${new Date().toISOString().replace(/[:.]/g, "-")}`, - ); - const summaryDir = path.join(resultsRoot, campaignId); - await mkdir(summaryDir, { recursive: true }); + await writeFile( path.join(summaryDir, "invocation-policy.json"), `${JSON.stringify( diff --git a/tests/runner-e2e/live-fixtures.test.ts b/tests/runner-e2e/live-fixtures.test.ts index cf99914f16b..04be71d847f 100644 --- a/tests/runner-e2e/live-fixtures.test.ts +++ b/tests/runner-e2e/live-fixtures.test.ts @@ -4,6 +4,30 @@ import { runnerMatrix } from "./catalog.js"; import { setupLiveFixtures } from "./live-fixtures.js"; describe("live runner fixtures", () => { + it.each(["runner-codex", "legacy-codex", "runner-acpx-claude", "legacy-opencode"])( + "creates a production-default %s hire with company and agent budget stops", async (profile) => { + const execution = runnerMatrix.find(row => row.suite.id === "stock-harness" && row.profile.id === profile)!; + let agentBody: any; + const api = { + async get() { return [{ id: "local", driver: "local" }]; }, + async postSensitive() { return { id: "secret" }; }, + async post(url: string, data: any) { + if (url === "/api/companies") { + expect(data.budgetMonthlyCents).toBe(1_000); + return { id: "company", name: "Fixture" }; + } + if (url.endsWith("/agents")) { agentBody = data; return { id: "agent", ...data }; } + throw new Error(`Unexpected POST ${url}`); + }, + } as unknown as RunnerApi; + const fixtures = await setupLiveFixtures({ api, execution, executionNonce: "nonce", workspacePath: "/tmp/fixture", + credentials: { [execution.profile.credential]: "fixture-key" } }); + expect(agentBody).not.toHaveProperty("instructionsBundle"); + expect(agentBody.budgetMonthlyCents).toBe(1_000); + expect(agentBody.adapterConfig.env[execution.profile.credential]).toEqual({ type: "secret_ref", secretId: "secret", version: "latest" }); + await fixtures.teardown(); + }, + ); it("anchors extended file validation to a public project workspace for local and remote copy-back", async () => { const execution = runnerMatrix.find(e => e.id === "extended-harnesses.runner-acpx-pi.local.file-edit-validate")!; let projectBody: any; diff --git a/tests/runner-e2e/live-fixtures.ts b/tests/runner-e2e/live-fixtures.ts index da0aa8e19c9..391bdc6448f 100644 --- a/tests/runner-e2e/live-fixtures.ts +++ b/tests/runner-e2e/live-fixtures.ts @@ -136,7 +136,9 @@ export async function setupLiveFixtures(input: { return api.post("/api/companies", { name: `Runner E2E ${execution.id} ${input.executionNonce}`, description: "Ephemeral paid full-stack runner acceptance fixture", - budgetMonthlyCents: execution.suite.id === "native-completion" ? NATIVE_COMPLETION_BUDGET_CENTS : execution.suite.id === "task-titles" ? TASK_TITLE_BUDGET_CENTS : 0, + budgetMonthlyCents: execution.suite.id === "native-completion" ? NATIVE_COMPLETION_BUDGET_CENTS + : execution.suite.id === "task-titles" ? TASK_TITLE_BUDGET_CENTS + : execution.suite.id === "stock-harness" ? 1_000 : 0, }); }, async teardown() { @@ -288,6 +290,7 @@ export async function setupLiveFixtures(input: { secretRefs, executionId: input.executionNonce, }); + if (execution.suite.id === "stock-harness") agent.budgetMonthlyCents = 1_000; if (managedHiring) { const account = value(resolved, "ai-connection"); const config = agent.adapterConfig as Record; diff --git a/tests/runner-e2e/native-completion-defaults.test.ts b/tests/runner-e2e/native-completion-defaults.test.ts index 21c5c6dc082..8777a2cb39b 100644 --- a/tests/runner-e2e/native-completion-defaults.test.ts +++ b/tests/runner-e2e/native-completion-defaults.test.ts @@ -1,5 +1,6 @@ import { describe, expect, it } from "vitest"; import { readFileSync } from "node:fs"; +import { createHash } from "node:crypto"; import { captureNativeDefault, gradeNativeDefault, nativeCompletionProfile, nativeCompletionWorkspaceDigest, NATIVE_MASTER_DEFAULT_SHA256, type NativeDefaultReceipt } from "./native-completion-defaults.js"; import { mkdtemp, mkdir, writeFile, symlink, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; @@ -58,7 +59,7 @@ describe("native production defaults", () => { expect(original.instructionsBundle).toBeDefined(); expect(() => nativeCompletionProfile({ ...profile, generation: "legacy" })).toThrow(); }); - it("hashes actual public bundle bytes before execution without publishing the body", async () => { + it("captures current default configuration independently of the frozen native qualification contract", async () => { const content = readFileSync(new URL("../../server/src/onboarding-assets/default/AGENTS.md", import.meta.url), "utf8"); const called: string[] = []; const api = { async get(path: string): Promise { @@ -67,7 +68,12 @@ describe("native production defaults", () => { : path.includes("instructions-bundle/file") ? { content } : { budgetMonthlyCents: 1000 }) as T; } }; const actual = await captureNativeDefault({ api, agentId: "agent", companyId: "company" }); - expect(gradeNativeDefault(actual).passed).toBe(true); + const currentSha256 = createHash("sha256").update(content).digest("hex"); + expect(actual.files).toEqual([{ path: "AGENTS.md", sha256: currentSha256, bytes: Buffer.byteLength(content) }]); + // The archived native comparison intentionally pins its original master + // context. Accurate capture of a later configuration does not qualify it + // against those historical live results. + expect(gradeNativeDefault(actual).passed).toBe(currentSha256 === NATIVE_MASTER_DEFAULT_SHA256); expect(called).toContain("/api/agents/agent/instructions-bundle/file?path=AGENTS.md"); expect(JSON.stringify(actual)).not.toContain(content); }); diff --git a/tests/runner-e2e/paperclip-document.test.ts b/tests/runner-e2e/paperclip-document.test.ts new file mode 100644 index 00000000000..4fd62239c0d --- /dev/null +++ b/tests/runner-e2e/paperclip-document.test.ts @@ -0,0 +1,89 @@ +import { readFileSync } from "node:fs"; +import { runInNewContext } from "node:vm"; +import { describe, expect, it } from "vitest"; +import { contextIntegrityScenario, PAPERCLIP_DOCUMENT_CASE } from "./context-integrity-cases.js"; +import { gradeContextIntegrity, type ContextIntegrityCheckpoint } from "./context-integrity-scoring.js"; + +function recording() { + const scenario = contextIntegrityScenario(PAPERCLIP_DOCUMENT_CASE, "probe"); + const initial: ContextIntegrityCheckpoint = { + phase: "initial", issue: { id: "issue", identifier: "DOC-1", issuePrefix: "DOC", appOrigin: "https://paperclip.example", status: "in_progress" }, + documents: [], comments: [], runs: [{ id: "run", status: "running" }], + assignedSkill: { key: scenario.skillKey, runtimeName: scenario.skillKey, versionId: "version", markdown: `Write ${scenario.marker}` }, + skillRequestText: `${scenario.prompt}\nUse /${scenario.skillKey}`, + }; + const final: ContextIntegrityCheckpoint = { + ...initial, phase: "final", issue: { ...initial.issue, status: "done" }, + documents: [{ key: "report", body: `Requested ${scenario.marker}`, latestRevisionId: "revision", latestRevisionNumber: 1 }], + comments: [{ authorAgentId: "agent", body: "Saved [report](/DOC/issues/DOC-1#document-report)." }], + runs: [{ id: "run", status: "succeeded" }], + }; + return { scenario, initial, final }; +} + +describe("explicit Paperclip document delivery", () => { + it("checks the actual saved content, revision and exact document link", () => { + const { scenario, initial, final } = recording(); + expect(gradeContextIntegrity({ ...scenario, checkpoints: [initial, final] }).every(row => row.passed)).toBe(true); + expect(scenario.prompt).toContain("Paperclip document"); + expect(scenario.prompt).toContain("clickable Markdown link"); + expect(scenario.prompt).toContain("Paperclip task UI"); + expect(scenario.prompt).not.toContain(scenario.marker); + }); + it.each(["https://paperclip.example/DOC/issues/DOC-1#document-report", ""])("accepts the same-app document link %s", target => { + const { scenario, initial, final } = recording(); + final.comments[0]!.body = `Saved [report](${target}).`; + expect(gradeContextIntegrity({ ...scenario, checkpoints: [initial, final] }).every(row => row.passed)).toBe(true); + }); + it.each(["local-only", "missing-revision", "wrong-content", "local-link", "wrong-document-link", "wrong-company-link", "other-app-link", "user-link-only", "bare-path", "code-formatted-path"])( + "rejects plausible %s delivery", variant => { + const { scenario, initial, final } = recording(); + if (variant === "local-only") { final.documents = []; final.comments[0]!.body = "Saved report.md in the workspace."; } + if (variant === "missing-revision") final.documents[0]!.latestRevisionId = null; + if (variant === "wrong-content") final.documents[0]!.body = "A different report"; + if (variant === "bare-path") final.comments[0]!.body = "Saved /DOC/issues/DOC-1#document-report."; + if (variant === "code-formatted-path") final.comments[0]!.body = "Saved `/DOC/issues/DOC-1#document-report`."; + if (variant === "local-link") final.comments[0]!.body = "Saved [report](./report.md)."; + if (variant === "wrong-document-link") final.comments[0]!.body = "Saved [report](/DOC/issues/DOC-1#document-other)."; + if (variant === "wrong-company-link") final.comments[0]!.body = "Saved [report](/PAP/issues/DOC-1#document-report)."; + if (variant === "other-app-link") final.comments[0]!.body = "Saved [report](https://other.example/DOC/issues/DOC-1#document-report)."; + if (variant === "user-link-only") { delete final.comments[0]!.authorAgentId; final.comments[0]!.authorUserId = "user"; } + expect(gradeContextIntegrity({ ...scenario, checkpoints: [initial, final] }).some(row => !row.passed)).toBe(true); + }, + ); + it.each([ + { identifier: "ACME-17", issuePrefix: "ACME", key: "report" }, + { identifier: "TEAM-902", issuePrefix: "TEAM", key: "report-agent-redirected" }, + ])("the shipped recipe delivers the current $identifier and returned $key", ({ identifier, issuePrefix, key }) => { + const reference = readFileSync(new URL("../../skills/paperclip/references/issue-documents.md", import.meta.url), "utf8"); + const recipe = reference.match(/```javascript\n([\s\S]*?)```/)?.[1]; + expect(recipe).toBeDefined(); + const { scenario, initial, final } = recording(); + initial.issue = { ...initial.issue, identifier, issuePrefix }; + final.issue = { ...final.issue, identifier, issuePrefix }; + final.documents[0]!.key = key; + final.comments[0]!.body = runInNewContext(`${recipe}\ncomment`, { + issue: { identifier }, saved: { key }, + }) as string; + expect(gradeContextIntegrity({ ...scenario, checkpoints: [initial, final] }).every(row => row.passed)).toBe(true); + }); + it.each([null, undefined])("the shipped recipe supports an unnumbered issue with identifier %s", identifier => { + const reference = readFileSync(new URL("../../skills/paperclip/references/issue-documents.md", import.meta.url), "utf8"); + const recipe = reference.match(/```javascript\n([\s\S]*?)```/)?.[1]; + expect(recipe).toBeDefined(); + const comment = runInNewContext(`${recipe}\ncomment`, { + issue: { id: "849c6474-ab4e-4428-981a-0b9a0207fa9b", identifier }, + saved: { key: "report-agent-redirected" }, + }) as string; + const target = comment.match(/\]\(([^)]+)\)/)?.[1]; + expect(target).toBeDefined(); + const url = new URL(target!, "https://paperclip.example"); + expect(url.pathname).toBe("/issues/849c6474-ab4e-4428-981a-0b9a0207fa9b"); + expect(url.hash).toBe("#document-report-agent-redirected"); + }); + it("preserves the original ambiguous request and assigned output procedure", () => { + expect(contextIntegrityScenario("assigned-skill-explicit-invocation", "probe").prompt).toBe( + "Use the assigned Context integrity output skill for this task, then follow its instructions and finish the task. Do not create child tasks or unrelated deliverables.", + ); + }); +}); diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index 46cc8e96aaa..f94562fbb4c 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -18,6 +18,8 @@ import { runContinuationFlow } from "./continuation-flow.js"; import { runEverydayFlow } from "./everyday-flow.js"; import { gradeTaskTitle } from "./task-titles.js"; import { runContextIntegrityFlow } from "./context-integrity-flow.js"; +import { captureStockHarness, gradeStockHarness, gradeStockHire } from "./stock-harness.js"; +import { verifyStockHarnessPreflight, STOCK_PREFLIGHT_ENV } from "./stock-harness-admission.js"; import { createTaskThroughUi, submitTaskReply } from "./user-actions.js"; import { runFirstTaskFlow, setupFirstTaskFixtures } from "./first-task-flow.js"; @@ -825,6 +827,10 @@ for (const execution of executions) { }); try { + if (execution.suite.id === "stock-harness") { + const receipt = verifyStockHarnessPreflight(process.env[STOCK_PREFLIGHT_ENV]); + await writeSanitizedJson(snapshotsDir, "stock-harness-preflight.json", receipt, secrets); + } const experimental = await api.patch<{ enableNativeRunner: boolean; }>("/api/instance/settings/experimental", { @@ -865,6 +871,18 @@ for (const execution of executions) { nativeInitial = { issueIds: issues.map(value => value.id), agentIds: agents.map(value => value.id), workspaceDigest }; } + if (execution.suite.id === "stock-harness") { + const hire = await captureStockHarness({ api, companyId: fixtures.company.id, + agentId: fixtures.agent.id, generation: execution.profile.generation, runIds: [] }); + const checks = gradeStockHire(hire); + await writeSanitizedJson(snapshotsDir, "stock-harness-hire.json", { + capturePhase: "before-provider", ...hire, checks, + }, secrets); + const failed = checks.filter(check => !check.passed); + if (failed.length) throw new Error(`Stock hire matcher failures: ${failed.map(check => check.id).join(", ")}`); + + } + if (execution.suite.id === "api-response-reading") { const source = await api.post<{ id: string }>(`/api/companies/${fixtures.company.id}/issues`, { title: `Synthetic diagnostic evidence ${nonce}`, @@ -2764,6 +2782,24 @@ for (const execution of executions) { selectedRuns = await Promise.all(companyRuns.map(run => api.get(`/api/heartbeat-runs/${run.id}`))); await writeSanitizedJson(snapshotsDir, execution.task.flow === "first_task" ? "first-task-final-run-ledger.json" : "chat-final-run-ledger.json", selectedRuns, secrets); } + if (execution.suite.id === "stock-harness") { + try { + const stock = await captureStockHarness({ api, companyId: fixtures.company.id, + agentId: fixtures.agent.id, generation: execution.profile.generation, + runIds: selectedRuns.map(run => run.id) }); + const checks = gradeStockHarness(stock); + matcherResults.push(...checks.map(check => ({ matcher: { kind: "json_path" as const, + path: `stockHarness.${check.id}`, expected: true }, passed: check.passed, detail: check.detail }))); + await writeSanitizedJson(snapshotsDir, "stock-harness.json", { ...stock, checks }, secrets); + const failures = checks.filter(check => !check.passed); + if (failures.length && !primaryError) primaryError = new Error(`Stock harness matcher failures: ${failures.map(check => check.id).join(", ")}`); + } catch (error) { + await writeSanitizedJson(snapshotsDir, "stock-harness-evidence-error.json", { + error: error instanceof Error ? error.message : String(error), + }, secrets); + if (!primaryError) primaryError = error; + } + } await fixtures.teardown(); cleanup = "passed"; } catch (error) { diff --git a/tests/runner-e2e/select-rerun-artifacts.test.ts b/tests/runner-e2e/select-rerun-artifacts.test.ts index b6793b2885b..657fc2cea17 100644 --- a/tests/runner-e2e/select-rerun-artifacts.test.ts +++ b/tests/runner-e2e/select-rerun-artifacts.test.ts @@ -160,6 +160,19 @@ function singletonSelectionInput(paths: Awaited>) { } describe("runner E2E workflow rerun artifact selection", () => { + it("retains credential-free prerequisites inside the exact campaign root", async () => { + const paths = await fixture(); + const { artifactName, campaignName } = await addArtifact({ root: paths.artifactRoot, + executionId: RERUN, workflowAttempt: 2, status: "passed" }); + const prerequisite = path.join(paths.artifactRoot, artifactName, campaignName, + "stock-harness-prerequisites", "fixture", "preflight.json"); + await mkdir(path.dirname(prerequisite), { recursive: true }); + await writeFile(prerequisite, JSON.stringify({ passed: true, providerCalls: 0 })); + await selectRerunArtifacts(singletonSelectionInput(paths)); + expect(await readFile(path.join(paths.selectedRoot, artifactName, campaignName, + "stock-harness-prerequisites", "fixture", "preflight.json"), "utf8")) + .toContain('"providerCalls":0'); + }); it.each(["before", "after"])("ignores a queued placeholder %s a started replacement without hiding its failure", async order => { const paths = await fixture(); diff --git a/tests/runner-e2e/stock-harness-admission.test.ts b/tests/runner-e2e/stock-harness-admission.test.ts new file mode 100644 index 00000000000..20a07639ae4 --- /dev/null +++ b/tests/runner-e2e/stock-harness-admission.test.ts @@ -0,0 +1,44 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { spawnSync } from "node:child_process"; +import { prepareStockHarnessPreflight, stockPreflightEnvironment, verifyStockHarnessPreflight } from "./stock-harness-admission.js"; +import { CREDENTIAL_NAMES } from "./types.js"; + +vi.mock("node:child_process", () => ({ spawnSync: vi.fn() })); +const spawn = vi.mocked(spawnSync); +const campaignDirectory = "/fixture/results/gha-1-1-stock-harness.fixture"; +beforeEach(() => { vi.resetAllMocks(); vi.unstubAllEnvs(); }); + +describe("stock harness credential-free admission", () => { + it("allows toolchain paths and excludes every present or future credential", () => { + const source = { PATH: "/bin", HOME: "/home/fixture", CARGO_HOME: "/cargo", CI: "true", + GH_TOKEN: "secret", FUTURE_PROVIDER_API_KEY: "secret", ...Object.fromEntries(CREDENTIAL_NAMES.map(name => [name, "secret"])) }; + expect(stockPreflightEnvironment(source)).toEqual({ PATH: "/bin", HOME: "/home/fixture", CARGO_HOME: "/cargo", CI: "true" }); + }); + it("rejects missing receipts without invoking a process", () => { + expect(() => verifyStockHarnessPreflight(undefined)).toThrow("no prerequisite receipt"); + expect(spawn).not.toHaveBeenCalled(); + }); + it("fails before provider execution when the prerequisite subprocess fails", () => { + spawn.mockReturnValue({ status: 1 } as ReturnType); + expect(() => prepareStockHarnessPreflight(campaignDirectory)).toThrow("before provider execution"); + expect(spawn).toHaveBeenCalledTimes(1); + }); + it("verifies retained evidence after a passing prerequisite subprocess", () => { + spawn.mockReturnValueOnce({ status: 0 } as ReturnType); + spawn.mockReturnValueOnce({ status: 0, stdout: '{"passed":true}' } as ReturnType); + const receipt = prepareStockHarnessPreflight(campaignDirectory); + expect(receipt).toMatch(/^\/fixture\/results\/gha-1-1-stock-harness\.fixture\/stock-harness-prerequisites\/[^/]+\/preflight.json$/); + expect(spawn.mock.calls[1]?.[1]).toContain(`--verify=${receipt}`); + }); + it("allows Cargo dependency resolution only on the disposable GitHub runner", () => { + vi.stubEnv("GITHUB_ACTIONS", "true"); + spawn.mockReturnValueOnce({ status: 0 } as ReturnType); + spawn.mockReturnValueOnce({ status: 0, stdout: '{}' } as ReturnType); + prepareStockHarnessPreflight(campaignDirectory); + expect(spawn.mock.calls[0]?.[1]).toContain("--allow-rust-network"); + }); + it("refuses failed receipt verification", () => { + spawn.mockReturnValue({ status: 1, stderr: "stale source" } as ReturnType); + expect(() => verifyStockHarnessPreflight("/fixture/preflight.json")).toThrow("stale source"); + }); +}); diff --git a/tests/runner-e2e/stock-harness-admission.ts b/tests/runner-e2e/stock-harness-admission.ts new file mode 100644 index 00000000000..c1a0e40761f --- /dev/null +++ b/tests/runner-e2e/stock-harness-admission.ts @@ -0,0 +1,38 @@ +import { spawnSync } from "node:child_process"; +import { randomUUID } from "node:crypto"; +import path from "node:path"; + +const root = path.resolve(import.meta.dirname, "../.."); +export const STOCK_PREFLIGHT_ENV = "PAPERCLIP_RUNNER_E2E_STOCK_PREFLIGHT"; + +// An allowlist keeps every provider credential, ambient auth override, and GH +// token out of the prerequisite subprocess, including secrets added later. +export function stockPreflightEnvironment(source: NodeJS.ProcessEnv): NodeJS.ProcessEnv { + return Object.fromEntries([ + "PATH", "HOME", "TMPDIR", "TMP", "TEMP", "SYSTEMROOT", "LANG", "LC_ALL", + "CARGO_HOME", "RUSTUP_HOME", "CI", "GITHUB_ACTIONS", + ].flatMap(name => source[name] === undefined ? [] : [[name, source[name]]])); +} + +export function verifyStockHarnessPreflight(receiptPath: string | undefined) { + if (!receiptPath) throw new Error("Stock harness has no prerequisite receipt; use the live launcher."); + const run = spawnSync(process.execPath, [path.join(root, "tests/runner-e2e/stock-harness-checks.mjs"), `--verify=${receiptPath}`], { + cwd: root, env: stockPreflightEnvironment(process.env), encoding: "utf8", timeout: 30_000, + }); + if (run.status !== 0) throw new Error(`Stock harness prerequisite verification failed: ${run.stderr || run.error?.message || run.status}`); + return JSON.parse(run.stdout) as Record; +} + +export function prepareStockHarnessPreflight(campaignDirectory: string) { + // The trusted artifact selector admits exactly one campaign root. Keep all + // prerequisite evidence inside that root, including pre-provider failures. + const output = path.join(campaignDirectory, "stock-harness-prerequisites", randomUUID()); + const receipt = path.join(output, "preflight.json"); + const run = spawnSync(process.execPath, [path.join(root, "tests/runner-e2e/stock-harness-checks.mjs"), + `--output-dir=${output}`, ...(process.env.GITHUB_ACTIONS === "true" ? ["--allow-rust-network"] : [])], { + cwd: root, env: stockPreflightEnvironment(process.env), stdio: "inherit", timeout: 50 * 60_000, + }); + if (run.status !== 0) throw new Error(`Stock harness prerequisites failed before provider execution. Retained receipt: ${receipt}`); + verifyStockHarnessPreflight(receipt); + return receipt; +} diff --git a/tests/runner-e2e/stock-harness-checks.mjs b/tests/runner-e2e/stock-harness-checks.mjs new file mode 100644 index 00000000000..949593873f2 --- /dev/null +++ b/tests/runner-e2e/stock-harness-checks.mjs @@ -0,0 +1,246 @@ +// Deterministic prerequisite for the stock-harness Product E2E suite. No model +// calls or credentials. Reuse the existing protocol assertions, not model +// self-reports about what its system instructions contain. +import { spawnSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { join, resolve } from "node:path"; +import { readStockInstructionVariant } from "./stock-harness-instruction-variant.mjs"; + +const root = resolve(import.meta.dirname, "../.."); +export const stockHarnessGates = [ + { id: "SH-1", name: "Native Codex additive instructions", cwd: "packages/paperclip-runner", files: [ + "src/drivers/codex/codex-app-server-driver.test.ts", + "src/drivers/codex/codex-app-server-driver.lifecycle.test.ts", + "src/live/runnerd-codex-transport.test.ts", + "src/live/live-session.test.ts", + "src/cli/eval-provider-runtime.test.ts", + ], testPattern: "adds Paperclip developer instructions|preserves stock Codex instructions|captures exact provider frames and correlates|direct eval provider runtime|exposes native completion consistently on fresh and resumed sessions|passes caller-supplied native system instructions", + required: ["adds Paperclip developer instructions", "preserves stock Codex instructions on task recovery", + "preserves stock Codex instructions on prepared recovery", "preserves stock Codex instructions on direct recovery", + "captures exact provider frames and correlates", "exposes native completion consistently on fresh and resumed sessions", + "passes caller-supplied native system instructions"] }, + { id: "SH-2", name: "Production-default hire bundle", cwd: ".", files: [ + "server/src/__tests__/agent-skills-routes.test.ts", + "server/src/services/onboarding-first-task-assets.test.ts", + ], required: ["materializes minimal default instructions for non-CEO agents with no prompt template"] }, + { id: "SH-3", name: "Shared legacy startup and continuation", cwd: ".", files: [ + "packages/adapter-utils/src/server-utils.test.ts", + "packages/adapter-utils/src/prompt-sections.test.ts", + "packages/adapter-utils/src/acpx-engine/execute.test.ts", + "packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.test.ts", + "packages/adapters/pi-local/src/server/execute.remote.test.ts", + "packages/adapters/opencode-local/src/server/execute.test.ts", + "packages/adapters/cursor-cloud/src/server/execute.test.ts", + "server/src/__tests__/codex-local-execute.test.ts", + ], required: ["keeps task and chat defaults to identity and connection guidance", + "does not restore generic procedures on resume or with the legacy opt-in", + "integrates the env-free store", "advertises folded routing metadata", "bounds routing descriptions", + "loads an old env-bearing file with rotated credentials", "drops a skill that fails to materialize"] }, + // Hermes is not in the root Vitest project list. Run its package config so + // the requested file cannot silently disappear from discovery. + { id: "SH-3-hermes", name: "Hermes shared-prompt delivery", cwd: "packages/adapters/hermes", + files: ["src/server/prompt-rendering.test.ts"], + required: ["renders standard assignment wake with task authority", "renders scoped planning wake authority"] }, + { id: "SH-eval", name: "Independent oracle and qualification admission", cwd: ".", + config: "tests/runner-e2e/vitest.config.ts", + files: ["tests/runner-e2e/stock-harness-manifest.test.ts", "tests/runner-e2e/paperclip-document.test.ts", "tests/runner-e2e/stock-harness.test.ts", "tests/runner-e2e/stock-harness-checks.test.mjs", + "tests/runner-e2e/stock-harness-admission.test.ts", "tests/runner-e2e/stock-harness-digest.test.ts", + "tests/runner-e2e/select-rerun-artifacts.test.ts", "tests/runner-e2e/stock-harness-instruction-variant.test.mjs", "tests/runner-e2e/automatic-retry.test.ts"], + required: ["requires generated capability manifests for the current skill sources", "the shipped recipe delivers the current", "the shipped recipe supports an unnumbered issue", "rejects old SHA before providers", "allows toolchain paths and excludes every present or future credential", + "changes when the evaluated server/src/onboarding-assets/default/AGENTS.md changes", + "changes when the evaluated packages/adapter-utils/src/server-utils.ts changes", + "changes when the evaluated packages/shared/src/connection-intent-guidance.ts changes", + "retains credential-free prerequisites inside the exact campaign root"] }, +]; + +export function stockHarnessGatesForVariant(variant) { + if (variant !== "reduced" && variant !== "historical") throw new Error("Unknown stock instruction variant."); + return stockHarnessGates.map(gate => variant !== "historical" ? gate : { + ...gate, + required: gate.required.map(name => name === "materializes minimal default instructions for non-CEO agents with no prompt template" + ? "materializes the bundled default instruction set for non-CEO agents with no prompt template" + : name === "keeps task and chat defaults to identity and connection guidance" + ? "keeps the default local-agent prompt action-oriented" + : name === "does not restore generic procedures on resume or with the legacy opt-in" + ? "adds the execution contract to resume delta prompts and opted-in fresh prompts" : name), + }); +} + +export function gradeGate(gate, report, exitCode) { + const assertions = (report?.testResults ?? []).flatMap(file => file.assertionResults ?? []); + const requirements = (gate.required ?? []).map(name => ({ name, + passed: assertions.some(assertion => assertion.fullName?.includes(name) && assertion.status === "passed") })); + const files = gate.files.map(file => ({ file, + passed: (report?.testResults ?? []).some(result => result.name?.endsWith(file) && result.status === "passed" && + result.assertionResults?.some(assertion => assertion.status === "passed") && + result.assertionResults.every(assertion => assertion.status === "passed" || + (gate.testPattern && assertion.status === "skipped" && !(gate.required ?? []).some(name => assertion.fullName?.includes(name))))) })); + return { id: gate.id, name: gate.name, passed: exitCode === 0 && requirements.every(row => row.passed) && files.every(row => row.passed), + exitCode, files, requirements, total: report?.numTotalTests ?? 0, passedTests: report?.numPassedTests ?? 0, + failedTests: report?.numFailedTests ?? 0, pendingTests: report?.numPendingTests ?? 0 }; +} + +export function sourceFingerprint() { + const hash = createHash("sha256"); + const sources = new Set([ + ...stockHarnessGates.flatMap(gate => gate.files.map(file => join(gate.cwd, file))), + "tests/runner-e2e/stock-harness.ts", "tests/runner-e2e/stock-harness-checks.mjs", "tests/runner-e2e/catalog.ts", + "tests/runner-e2e/stock-harness-admission.ts", "tests/runner-e2e/launch.ts", "tests/runner-e2e/runner.spec.ts", + "tests/runner-e2e/stock-harness-instruction-variant.mjs", "tests/runner-e2e/stock-harness-instruction-variant.d.mts", "tests/runner-e2e/fixtures/stock-harness/historical-default-agents.md", + "tests/runner-e2e/automatic-retry.ts", "tests/runner-e2e/types.ts", + "packages/adapter-utils/src/acpx-engine/execute.ts", "packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.ts", + "tests/runner-e2e/context-integrity-cases.ts", "tests/runner-e2e/context-integrity-flow.ts", "tests/runner-e2e/context-integrity-scoring.ts", + "tests/runner-e2e/stock-harness-manifest.ts", "packages/paperclip-runner/scripts/generate-capability-contract.mjs", + "packages/paperclip-runner/spec/capability/source-contract.json", + "packages/paperclip-runner/scripts/check-capability-inventory.mjs", "packages/paperclip-runner/scripts/lib/capability-inventory.mjs", + "packages/paperclip-runner/spec/capability/capabilities.yaml", "packages/paperclip-runner/spec/capability/eval-traceability.yaml", + "packages/paperclip-runner/spec/capability/mcp-tool-map.yaml", "packages/paperclip-runner/spec/capability/inventory.schema.json", + "packages/paperclip-runner/src/generated/capability-contract.ts", "packages/paperclip-runner/docs/capability-contract.md", + ...["capabilities.yaml", "mcp-tool-map.yaml", "eval-traceability.yaml", "capability-contract.md", "downstream-handoff.md"] + .map(file => `packages/paperclip-runner/generated/capability/${file}`), + "packages/adapter-utils/src/server-utils.ts", "packages/shared/src/connection-intent-guidance.ts", + "server/src/onboarding-assets/default/AGENTS.md", "server/src/routes/agents.ts", "scripts/ensure-plugin-build-deps.mjs", + "packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts", + "packages/paperclip-runner/src/live/runnerd-codex-transport.ts", + "packages/paperclip-runner/src/live/live-session.ts", + "packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs", + "packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs", + "packages/paperclip-runner/runner/Cargo.toml", "packages/paperclip-runner/runner/Cargo.lock", + "packages/paperclip-runner/runner/crates/runner-core/Cargo.toml", + "packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs", + ]); + const sourceErrors = []; + sources.add("skills/paperclip/SKILL.md"); + sources.add("skills/paperclip/references/issue-documents.md"); + for (const source of [...sources].sort()) { + hash.update(source); + try { hash.update("present\0").update(readFileSync(join(root, source))); } + catch (error) { + if (source === "skills/paperclip/references/issue-documents.md" && error.code === "ENOENT") hash.update("absent\0"); + else sourceErrors.push(source); + } + } + return { fingerprint: hash.digest("hex"), sourceErrors }; +} + +export function assertPreflightReceipt(report, current) { + const instructionVariant = readStockInstructionVariant(); + const expected = [...stockHarnessGates.map(gate => gate.id), "SH-1-rust"]; + if (report?.schema !== "paperclip.stock-harness-preflight.v3" || report.passed !== true || + report.setup?.passed !== true || report.setup?.exitCode !== 0 || + report.setup?.sdkExitCode !== 0 || report.setup?.runnerdExitCode !== 0 || + report.setup?.runnerdSha256 !== current.runnerdSha256 || + report.setup?.fakeCodexSha256 !== current.fakeCodexSha256 || + report.providerCalls !== 0 || report.sourceSha !== current.sha || + report.sourceFingerprint !== current.fingerprint || report.sourceErrors?.length !== 0 || + report.instructionVariant?.variant !== instructionVariant.variant || report.instructionVariant?.sha256 !== instructionVariant.sha256 || + !Array.isArray(report.gates) || report.gates.length !== expected.length || + expected.some(id => report.gates.filter(gate => gate.id === id && gate.passed === true && gate.exitCode === 0).length !== 1)) { + throw new Error("Stock harness requires passing prerequisites for this exact source SHA and fingerprint."); + } + return report; +} + +export function main(args = process.argv.slice(2)) { + const { variant, sha256 } = readStockInstructionVariant(); + const instructionVariant = { variant, sha256 }; + const gates = stockHarnessGatesForVariant(variant); + if (args.includes("--list")) { + console.log(JSON.stringify({ gates, instructionVariant, rust: "runtime_instructions_are_additive_for_codex_on_start_and_resume", live: "not invoked" }, null, 2)); + return; + } + if (args.some(arg => !arg.startsWith("--output-dir=") && !arg.startsWith("--verify=") && arg !== "--allow-rust-network")) + throw new Error("Use --list, --output-dir=, --verify=, or --allow-rust-network; never paid providers."); + const git = spawnSync("git", ["rev-parse", "HEAD"], { cwd: root, encoding: "utf8" }); + const verify = args.find(arg => arg.startsWith("--verify="))?.slice("--verify=".length); + if (verify) { + const source = sourceFingerprint(); + if (git.status !== 0 || source.sourceErrors.length) throw new Error("Cannot verify stock harness source provenance."); + const report = assertPreflightReceipt(JSON.parse(readFileSync(verify, "utf8")), { + sha: git.stdout.trim(), fingerprint: source.fingerprint, + runnerdSha256: createHash("sha256").update(readFileSync(join(root, + "packages/paperclip-runner/runner/target/debug", `paperclip-runnerd${process.platform === "win32" ? ".exe" : ""}`))).digest("hex"), + fakeCodexSha256: createHash("sha256").update(readFileSync(join(root, + "packages/paperclip-runner/runner/target/debug/fake-codex-app-server"))).digest("hex"), + }); + const output = resolve(verify, ".."); + for (const gate of gates) { + const grade = gradeGate(gate, JSON.parse(readFileSync(join(output, `${gate.id}.json`), "utf8")), 0); + if (!grade.passed) throw new Error(`Missing or failed retained prerequisite assertions: ${gate.id}`); + } + if (!/test runtime_instructions_are_additive_for_codex_on_start_and_resume \.\.\. ok/.test(readFileSync(join(output, "rust.txt"), "utf8"))) + throw new Error("Missing passing retained Rust prerequisite."); + console.log(JSON.stringify(report)); + return report; + } + const output = args.find(arg => arg.startsWith("--output-dir="))?.slice("--output-dir=".length) ?? + join(root, "tests/runner-e2e/results", `stock-harness-preflight-${new Date().toISOString().replaceAll(":", "-")}`); + mkdirSync(output, { recursive: true }); + const env = Object.fromEntries([ + "PATH", "HOME", "TMPDIR", "TMP", "TEMP", "SYSTEMROOT", "LANG", "LC_ALL", + "CARGO_HOME", "RUSTUP_HOME", "CI", "GITHUB_ACTIONS", + ].flatMap(name => process.env[name] === undefined ? [] : [[name, process.env[name]]])); + // A protected cold install disables lifecycle scripts. Use the same ordinary + // SDK dependency builder as server startup, before importing server tests. + const setupRun = spawnSync(process.execPath, [join(root, "scripts/ensure-plugin-build-deps.mjs")], { + cwd: root, env, encoding: "utf8", timeout: 5 * 60_000, + }); + writeFileSync(join(output, "setup.txt"), `${setupRun.stdout ?? ""}\n${setupRun.stderr ?? ""}`); + // The exact daemon-frame test uses the real local Rust daemon. Legacy cold + // cells do not download native artifacts, so compile it before TS discovery. + const runnerd = setupRun.status === 0 ? spawnSync("cargo", ["build", "--locked", + ...(args.includes("--allow-rust-network") ? [] : ["--offline"]), + "--workspace", "--bins"], { + cwd: join(root, "packages/paperclip-runner/runner"), env, encoding: "utf8", timeout: 10 * 60_000, + }) : null; + writeFileSync(join(output, "runnerd-build.txt"), `${runnerd?.stdout ?? ""}\n${runnerd?.stderr ?? ""}`); + const setup = { passed: setupRun.status === 0 && runnerd?.status === 0, + exitCode: setupRun.status !== 0 ? setupRun.status : runnerd?.status ?? null, + sdkExitCode: setupRun.status, runnerdExitCode: runnerd?.status ?? null }; + if (!setup.passed) { + const source = sourceFingerprint(); + writeFileSync(join(output, "preflight.json"), JSON.stringify({ + schema: "paperclip.stock-harness-preflight.v3", instructionVariant, sourceSha: git.stdout?.trim() || null, + sourceFingerprint: source.fingerprint, measuredAt: new Date().toISOString(), providerCalls: 0, + live: "not_run", passed: false, sourceErrors: source.sourceErrors, setup, gates: [], + }, null, 2) + "\n"); + process.exitCode = 1; + return; + } + const runnerdBinary = join(root, "packages/paperclip-runner/runner/target/debug", + `paperclip-runnerd${process.platform === "win32" ? ".exe" : ""}`); + setup.runnerdSha256 = createHash("sha256").update(readFileSync(runnerdBinary)).digest("hex"); + setup.fakeCodexSha256 = createHash("sha256").update(readFileSync(join(root, + "packages/paperclip-runner/runner/target/debug/fake-codex-app-server"))).digest("hex"); + const results = []; + for (const gate of gates) { + console.log(`Checking ${gate.id}: ${gate.name}`); + const file = join(output, `${gate.id}.json`); + const run = spawnSync(process.execPath, [join(root, "node_modules/vitest/vitest.mjs"), "run", ...gate.files, + ...(gate.config ? ["--config", gate.config] : []), + ...(gate.testPattern ? ["--testNamePattern", gate.testPattern] : []), + "--reporter=default", "--reporter=json", `--outputFile.json=${file}`], + { cwd: resolve(root, gate.cwd), + env: gate.id === "SH-1" ? { ...env, PAPERCLIP_STOCK_PREFLIGHT_RUNNERD: runnerdBinary } : env, + stdio: "inherit", timeout: 10 * 60_000 }); + let report; + try { report = JSON.parse(readFileSync(file, "utf8")); } catch { /* missing evidence fails closed below */ } + results.push(gradeGate(gate, report, run.status)); + } + const rust = spawnSync("cargo", ["test", ...(args.includes("--allow-rust-network") ? [] : ["--offline"]), "-p", "paperclip-runner-core", "--test", "codex_provider", + "runtime_instructions_are_additive_for_codex_on_start_and_resume", "--", "--exact"], + { cwd: join(root, "packages/paperclip-runner/runner"), env, encoding: "utf8", timeout: 10 * 60_000 }); + const rustOutput = `${rust.stdout ?? ""}\n${rust.stderr ?? ""}`; + writeFileSync(join(output, "rust.txt"), rustOutput); + results.push({ id: "SH-1-rust", passed: rust.status === 0 && /test runtime_instructions_are_additive_for_codex_on_start_and_resume \.\.\. ok/.test(rustOutput), exitCode: rust.status }); + const { fingerprint, sourceErrors } = sourceFingerprint(); + const report = { schema: "paperclip.stock-harness-preflight.v3", instructionVariant, sourceSha: git.stdout?.trim() || null, + sourceFingerprint: fingerprint, measuredAt: new Date().toISOString(), providerCalls: 0, + live: "not_run", passed: git.status === 0 && sourceErrors.length === 0 && results.every(row => row.passed), sourceErrors, setup, gates: results }; + writeFileSync(join(output, "preflight.json"), JSON.stringify(report, null, 2) + "\n"); + console.log(`Stock harness prerequisite evidence: ${output}/preflight.json`); + if (!report.passed) process.exitCode = 1; +} + +if (process.argv[1] && resolve(process.argv[1]) === resolve(import.meta.filename)) main(); diff --git a/tests/runner-e2e/stock-harness-checks.test.mjs b/tests/runner-e2e/stock-harness-checks.test.mjs new file mode 100644 index 00000000000..e4a41394555 --- /dev/null +++ b/tests/runner-e2e/stock-harness-checks.test.mjs @@ -0,0 +1,79 @@ +import { describe, expect, it } from "vitest"; +import { assertPreflightReceipt, gradeGate, stockHarnessGates, stockHarnessGatesForVariant } from "./stock-harness-checks.mjs"; +import { readStockInstructionVariant } from "./stock-harness-instruction-variant.mjs"; + +const gate = { id: "SH-test", name: "Boundary", files: ["boundary.test.ts"], required: ["must preserve instructions"] }; +const report = () => ({ testResults: [{ name: "/repo/boundary.test.ts", status: "passed", + assertionResults: [{ fullName: "Boundary must preserve instructions", status: "passed" }] }], +numTotalTests: 1, numPassedTests: 1, numFailedTests: 0, numPendingTests: 0 }); +describe("stock harness prerequisite coverage", () => { + it("maps every implemented change to an executable gate", () => { + expect(stockHarnessGates.map(gate => gate.id)).toEqual(["SH-1", "SH-2", "SH-3", "SH-3-hermes", "SH-eval"]); + expect(stockHarnessGates.every(gate => gate.files.length > 0 && gate.required.length > 0)).toBe(true); + }); + it("accepts an executed passing boundary", () => expect(gradeGate(gate, report(), 0).passed).toBe(true)); + it("preserves candidate assertions and admits only declared historical structural alternatives", () => { + expect(stockHarnessGatesForVariant("reduced")).toEqual(stockHarnessGates); + const historical = stockHarnessGatesForVariant("historical"); + expect(historical.map(value => value.files)).toEqual(stockHarnessGates.map(value => value.files)); + expect(historical.find(value => value.id === "SH-2").required).toContain("materializes the bundled default instruction set for non-CEO agents with no prompt template"); + expect(historical.find(value => value.id === "SH-3").required).toContain("adds the execution contract to resume delta prompts and opted-in fresh prompts"); + expect(historical.find(value => value.id === "SH-3").required).toContain("integrates the env-free store"); + expect(() => stockHarnessGatesForVariant("other")).toThrow("Unknown"); + }); + it.each([null, undefined, { testResults: [] }])("rejects unavailable evidence %s", (report) => { + expect(gradeGate(gate, report, 0).passed).toBe(false); + }); + it.each(["pending", "skipped", "failed", "todo"])("rejects a %s required assertion", (status) => { + const observation = report(); + observation.testResults[0].assertionResults[0].status = status; + expect(gradeGate(gate, observation, 0).passed).toBe(false); + }); + it("rejects a passing different assertion and a nonzero launcher exit", () => { + const different = report(); + different.testResults[0].assertionResults[0].fullName = "Unrelated pass"; + expect(gradeGate(gate, different, 0).passed).toBe(false); + expect(gradeGate(gate, report(), 1).passed).toBe(false); + }); + it("allows explicitly filtered unrelated tests while rejecting a skipped required test", () => { + const filtered = { ...gate, testPattern: "must preserve instructions" }; + const observation = report(); + observation.testResults[0].assertionResults.push({ fullName: "Unrelated", status: "skipped" }); + expect(gradeGate(filtered, observation, 0).passed).toBe(true); + observation.testResults[0].assertionResults[0].status = "skipped"; + expect(gradeGate(filtered, observation, 0).passed).toBe(false); + }); + it("rejects a requested file absent from Vitest discovery even when required names pass", () => { + expect(gradeGate({ ...gate, files: [...gate.files, "undiscovered.test.ts"] }, report(), 0).passed).toBe(false); + expect(stockHarnessGates.find(gate => gate.id === "SH-3-hermes").cwd).toBe("packages/adapters/hermes"); + }); +}); + +describe("stock harness prerequisite admission", () => { + const current = { sha: "a".repeat(40), fingerprint: "b".repeat(64), runnerdSha256: "c".repeat(64), fakeCodexSha256: "e".repeat(64) }; + const receipt = () => ({ schema: "paperclip.stock-harness-preflight.v3", passed: true, + instructionVariant: (({ variant, sha256 }) => ({ variant, sha256 }))(readStockInstructionVariant()), + setup: { passed: true, exitCode: 0, sdkExitCode: 0, runnerdExitCode: 0, runnerdSha256: current.runnerdSha256, + fakeCodexSha256: current.fakeCodexSha256 }, + providerCalls: 0, sourceSha: current.sha, sourceFingerprint: current.fingerprint, + sourceErrors: [], gates: [...stockHarnessGates.map(g => g.id), "SH-1-rust"].map(id => ({ id, passed: true, exitCode: 0 })) }); + it("admits the same passing source revision", () => expect(assertPreflightReceipt(receipt(), current).passed).toBe(true)); + it.each([ + ["old SHA", r => { r.sourceSha = "c".repeat(40); }], + ["changed source", r => { r.sourceFingerprint = "d".repeat(64); }], + ["wrong instruction variant", r => { r.instructionVariant.variant = "other"; }], + ["wrong manual hash", r => { r.instructionVariant.sha256 = "f".repeat(64); }], + ["failed boundary", r => { r.gates[0].passed = false; }], + ["nonzero exit", r => { r.gates[0].exitCode = 1; }], + ["missing Rust", r => { r.gates.pop(); }], + ["duplicate boundary", r => { r.gates[1] = r.gates[0]; }], + ["provider calls", r => { r.providerCalls = 1; }], + ["missing source", r => { r.sourceErrors.push("missing.ts"); }], + ["failed cold setup", r => { r.setup.passed = false; r.setup.exitCode = 1; }], + ["missing daemon build", r => { delete r.setup.runnerdExitCode; }], + ["changed daemon binary", r => { r.setup.runnerdSha256 = "d".repeat(64); }], + ["changed protocol fixture", r => { r.setup.fakeCodexSha256 = "d".repeat(64); }], + ])("rejects %s before providers", (_name, mutate) => { + const r = receipt(); mutate(r); expect(() => assertPreflightReceipt(r, current)).toThrow("exact source SHA and fingerprint"); + }); +}); diff --git a/tests/runner-e2e/stock-harness-digest.test.ts b/tests/runner-e2e/stock-harness-digest.test.ts new file mode 100644 index 00000000000..6fbfd3597ab --- /dev/null +++ b/tests/runner-e2e/stock-harness-digest.test.ts @@ -0,0 +1,27 @@ +import { describe, expect, it, vi } from "vitest"; +import { readFileSync } from "node:fs"; +import { stockHarnessSourceDigest, stockHarnessSkillSources } from "./stock-harness.js"; + +vi.mock("node:fs", async importOriginal => ({ ...await importOriginal(), readFileSync: vi.fn() })); + +describe("stock harness instruction revision", () => { + it.each(["server/src/onboarding-assets/default/AGENTS.md", "packages/adapter-utils/src/server-utils.ts", "packages/shared/src/connection-intent-guidance.ts", "skills/paperclip/SKILL.md", "skills/paperclip/references/issue-documents.md", "packages/paperclip-runner/generated/capability/capabilities.yaml", "packages/paperclip-runner/spec/capability/capabilities.yaml", "tests/runner-e2e/stock-harness-manifest.ts", "packages/adapter-utils/src/acpx-engine/execute.ts", "packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.ts", "tests/runner-e2e/stock-harness-instruction-variant.mjs", "tests/runner-e2e/automatic-retry.ts", "tests/runner-e2e/catalog.ts"])( + "changes when the evaluated %s changes", source => { + vi.mocked(readFileSync).mockImplementation(() => Buffer.from("unchanged")); + const original = stockHarnessSourceDigest(); + vi.mocked(readFileSync).mockImplementation(file => Buffer.from(String(file).endsWith(source) ? "changed instructions" : "unchanged")); + expect(stockHarnessSourceDigest()).not.toBe(original); + }); + it("records an absent historical recipe without introducing its content or hiding other read errors", () => { + vi.mocked(readFileSync).mockImplementation(() => Buffer.from("unchanged")); + const present = stockHarnessSourceDigest(); + vi.mocked(readFileSync).mockImplementation(file => { + if (String(file).endsWith("references/issue-documents.md")) throw Object.assign(new Error("absent"), { code: "ENOENT" }); + return Buffer.from("unchanged"); + }); + expect(stockHarnessSkillSources()[1]).toEqual({ path: "skills/paperclip/references/issue-documents.md", present: false, sha256: null }); + expect(stockHarnessSourceDigest()).not.toBe(present); + vi.mocked(readFileSync).mockImplementation(() => { throw Object.assign(new Error("unreadable"), { code: "EACCES" }); }); + expect(stockHarnessSourceDigest).toThrow("unreadable"); + }); +}); diff --git a/tests/runner-e2e/stock-harness-instruction-variant.d.mts b/tests/runner-e2e/stock-harness-instruction-variant.d.mts new file mode 100644 index 00000000000..3c8a570dfbb --- /dev/null +++ b/tests/runner-e2e/stock-harness-instruction-variant.d.mts @@ -0,0 +1,7 @@ +export interface StockInstructionVariant { + variant: "reduced" | "historical"; + sha256: string; + content: string; +} +export function classifyStockInstructionManual(content: unknown): StockInstructionVariant; +export function readStockInstructionVariant(): StockInstructionVariant; diff --git a/tests/runner-e2e/stock-harness-instruction-variant.mjs b/tests/runner-e2e/stock-harness-instruction-variant.mjs new file mode 100644 index 00000000000..ec2bed70724 --- /dev/null +++ b/tests/runner-e2e/stock-harness-instruction-variant.mjs @@ -0,0 +1,24 @@ +import { createHash } from "node:crypto"; +import { readFileSync } from "node:fs"; + +const reduced = "You are an agent in a Paperclip company.\n"; +const historicalSha256 = "e4d2375d722602cd744403292d99f6e6c9b7e8014d9cdfa6f9811ace03428e5f"; + +/** + * @param {unknown} content + * @returns {{ variant: 'reduced' | 'historical', sha256: string, content: string }} + */ +export function classifyStockInstructionManual(content) { + if (typeof content !== "string") throw new Error("Stock harness instruction manual must be exact UTF-8 text."); + const sha256 = createHash("sha256").update(content, "utf8").digest("hex"); + if (content === reduced) return { variant: "reduced", sha256, content }; + if (Buffer.byteLength(content, "utf8") === 4249 && sha256 === historicalSha256) + return { variant: "historical", sha256, content }; + throw new Error("Stock harness instruction manual is not an admitted reduced or historical source."); +} + +export function readStockInstructionVariant() { + return classifyStockInstructionManual(readFileSync(new URL( + "../../server/src/onboarding-assets/default/AGENTS.md", import.meta.url, + ), "utf8")); +} diff --git a/tests/runner-e2e/stock-harness-instruction-variant.test.mjs b/tests/runner-e2e/stock-harness-instruction-variant.test.mjs new file mode 100644 index 00000000000..10f027d8fff --- /dev/null +++ b/tests/runner-e2e/stock-harness-instruction-variant.test.mjs @@ -0,0 +1,48 @@ +import { createHash } from "node:crypto"; +import { readFileSync } from "node:fs"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { classifyStockInstructionManual, readStockInstructionVariant } from "./stock-harness-instruction-variant.mjs"; + +const reduced = "You are an agent in a Paperclip company.\n"; +const historicalBytes = readFileSync(new URL("./fixtures/stock-harness/historical-default-agents.md", import.meta.url)); +const historical = historicalBytes.toString("utf8"); +const historicalSha256 = "e4d2375d722602cd744403292d99f6e6c9b7e8014d9cdfa6f9811ace03428e5f"; + +afterEach(() => vi.unstubAllEnvs()); +describe("closed stock harness instruction variants", () => { + it("admits only the exact reduced manual text", () => { + expect(classifyStockInstructionManual(reduced)).toEqual({ variant: "reduced", content: reduced, + sha256: createHash("sha256").update(reduced).digest("hex") }); + }); + it("admits the independently pinned 4249-byte historical manual", () => { + expect(historicalBytes.length).toBe(4249); + expect(createHash("sha256").update(historicalBytes).digest("hex")).toBe(historicalSha256); + expect(classifyStockInstructionManual(historical)).toEqual({ variant: "historical", sha256: historicalSha256, content: historical }); + }); + it.each([ + ["missing newline", reduced.trimEnd()], + ["appended reduced instructions", `${reduced}Follow additional procedures.\n`], + ["altered reduced wording", reduced.replace("agent", "coder")], + ["CRLF reduced manual", reduced.replace("\n", "\r\n")], + ["leading byte order mark", `\uFEFF${reduced}`], + ["appended historical instructions", `${historical}Extra instructions.\n`], + ["same-size altered historical content", `X${historical.slice(1)}`], + ["unknown manual", "Historical instructions"], + ["empty text", ""], + ])("rejects %s", (_name, content) => { + expect(() => classifyStockInstructionManual(content)).toThrow("not an admitted"); + }); + it.each([undefined, null, 42, {}, [], Buffer.from(reduced)])("rejects malformed non-string content %#", content => { + expect(() => classifyStockInstructionManual(content)).toThrow("must be exact UTF-8 text"); + }); + it("reads and classifies the exact current repository source without selecting by environment or branch labels", () => { + const source = readFileSync(new URL("../../server/src/onboarding-assets/default/AGENTS.md", import.meta.url), "utf8"); + const expected = classifyStockInstructionManual(source); + for (const label of ["historical", "reduced", "unknown"]) { + vi.stubEnv("PAPERCLIP_STOCK_HARNESS_VARIANT", label); + vi.stubEnv("PAPERCLIP_RUNNER_E2E_SOURCE_REF", `refs/heads/${label}`); + vi.stubEnv("GITHUB_HEAD_REF", label); + expect(readStockInstructionVariant()).toEqual(expected); + } + }); +}); diff --git a/tests/runner-e2e/stock-harness-manifest.test.ts b/tests/runner-e2e/stock-harness-manifest.test.ts new file mode 100644 index 00000000000..e8ce32b645d --- /dev/null +++ b/tests/runner-e2e/stock-harness-manifest.test.ts @@ -0,0 +1,21 @@ +import { describe, expect, it } from "vitest"; +import { readFileSync } from "node:fs"; +import { verifyCapabilityManifest } from "./stock-harness-manifest.js"; + +describe("stock harness generated instruction metadata", () => { + it("requires generated capability manifests for the current skill sources", async () => { + expect(await verifyCapabilityManifest()).toBe(true); + }); + it("rejects a stale manifest before provider admission", async () => { + await expect(verifyCapabilityManifest(path => path.endsWith("capabilities.yaml") + ? "stale skill anchors" : readFileSync(path, "utf8"))).rejects.toThrow("manifest drift"); + }); + it("rejects unavailable generated evidence", async () => { + await expect(verifyCapabilityManifest(() => { throw new Error("missing generated evidence"); })) + .rejects.toThrow("missing generated evidence"); + }); + it("rejects a stale source inventory before provider admission", async () => { + await expect(verifyCapabilityManifest(undefined, () => { throw new Error("source inventory drift"); })) + .rejects.toThrow("source inventory drift"); + }); +}); diff --git a/tests/runner-e2e/stock-harness-manifest.ts b/tests/runner-e2e/stock-harness-manifest.ts new file mode 100644 index 00000000000..dbc260923b2 --- /dev/null +++ b/tests/runner-e2e/stock-harness-manifest.ts @@ -0,0 +1,19 @@ +import { readFileSync } from "node:fs"; +import { execFileSync } from "node:child_process"; +import { fileURLToPath } from "node:url"; +import { buildContract } from "../../packages/paperclip-runner/scripts/generate-capability-contract.mjs"; + +/** Canonical derived metadata must match the instruction sources before providers. */ +export async function verifyCapabilityManifest( + read = (path: string) => readFileSync(path, "utf8"), + checkInventory = () => execFileSync(process.execPath, [fileURLToPath(new URL( + "../../packages/paperclip-runner/scripts/check-capability-inventory.mjs", import.meta.url, + ))], { stdio: "pipe" }), +) { + const expected = await buildContract(); + for (const [path, content] of Object.entries(expected)) { + if (read(path) !== content) throw new Error(`Generated capability manifest drift: ${path}`); + } + checkInventory(); + return true; +} diff --git a/tests/runner-e2e/stock-harness.test.ts b/tests/runner-e2e/stock-harness.test.ts new file mode 100644 index 00000000000..b8ef4087c01 --- /dev/null +++ b/tests/runner-e2e/stock-harness.test.ts @@ -0,0 +1,189 @@ +import { describe, expect, it } from "vitest"; +import { runnerMatrix, runnerProfiles, suiteDefinitionHash } from "./catalog.js"; +import { captureStockHarness, gradeStockHarness as gradeSourceHarness, gradeStockHire as gradeSourceHire, STOCK_HIRE_IDENTITY, type StockHarnessEvidence } from "./stock-harness.js"; +import { readFileSync } from "node:fs"; +import { classifyStockInstructionManual, readStockInstructionVariant } from "./stock-harness-instruction-variant.mjs"; + +const reduced = classifyStockInstructionManual(STOCK_HIRE_IDENTITY); +const gradeStockHire = (evidence: Parameters[0]) => gradeSourceHire(evidence, reduced); +const gradeStockHarness = (evidence: StockHarnessEvidence) => gradeSourceHarness(evidence, reduced); + +function recording(generation: "legacy" | "native" = "legacy"): StockHarnessEvidence { + return { + schema: "paperclip.stock-harness.v1", generation, agentId: "agent", + budgets: { companyMonthlyCents: 1_000, agentMonthlyCents: 1_000 }, + bundle: { entryFile: "AGENTS.md", files: [{ path: "AGENTS.md", content: STOCK_HIRE_IDENTITY }] }, + runIds: ["fresh", "resumed"], + invocations: generation === "native" ? [] : [ + { runId: "fresh", conversationMode: false, prompt: "You are agent agent (QA).\nConnection tools:\nUse connections_search.\nCurrent assignment.", promptMetrics: { heartbeatPromptChars: 113 } }, + { runId: "resumed", conversationMode: false, prompt: "## Paperclip Resume Delta\nCurrent ordered comments.", promptMetrics: { heartbeatPromptChars: 0 } }, + ], + }; +} + +describe("stock harness Product E2E", () => { + it("registers 26 explicit local cells with the original cases and two focused delivery repair cells", () => { + const cells = runnerMatrix.filter(row => row.suite.id === "stock-harness"); + expect(cells).toHaveLength(26); + expect(new Set(cells.map(row => row.profile.id))).toEqual(new Set([ + "legacy-codex", "legacy-claude", "legacy-opencode", "legacy-acp-codex", "legacy-acp-claude", + "runner-codex", "runner-acpx-claude", "runner-opencode", + ])); + expect(new Set(cells.map(row => row.task.id))).toEqual(new Set([ + "ordered-comment-continuation", "assigned-skill-explicit-invocation", "continuity-restart", + "assigned-skill-paperclip-document", + ])); + expect(cells.every(row => row.suite.manualOnly && row.environment.id === "local")).toBe(true); + expect(cells.every(row => row.task.automaticRetryPolicy === "single_attempt")).toBe(true); + expect(cells[0]!.suite.definitionMetadata).toMatchObject({ automaticRetryPolicy: "single_attempt", maximumAttemptsPerCell: 1 }); + expect(cells.reduce((turns, row) => turns + row.task.expectedRunCount, 0)).toBe(50); + expect(cells.filter(row => row.task.id === "assigned-skill-paperclip-document").map(row => row.profile.id).sort()).toEqual(["legacy-claude", "legacy-opencode"]); + expect(cells[0]!.suite.definitionMetadata?.sourceDigest).toMatch(/^[a-f0-9]{64}$/); + }); + + it.each(["legacy-codex", "legacy-claude", "legacy-opencode", "runner-codex", "runner-acpx-claude", "runner-opencode"])( + "omits the custom QA bundle for %s while preserving runtime, auth and skills configuration", (profileId) => { + const profile = runnerMatrix.find(row => row.suite.id === "stock-harness" && row.profile.id === profileId)!.profile; + const original = runnerProfiles.find(row => row.id === profileId)!; + const input = { executionId: "test", workspacePath: "/workspace", environmentId: "environment", environmentFixtureId: "local" as const, + secretRefs: { [profile.credential]: { type: "secret_ref" as const, secretId: "secret", version: "latest" as const } } }; + const expected = original.buildAgent(input); + const { instructionsBundle: _manual, ...withoutManual } = expected; + expect(profile.buildAgent(input)).toEqual(withoutManual); + expect(profile.buildAgent(input)).not.toHaveProperty("instructionsBundle"); + }, + ); + + it.each(["legacy", "native"] as const)("accepts complete %s evidence without interpreting task success as vendor-base proof", (generation) => { + expect(gradeStockHarness(recording(generation)).every(check => check.passed)).toBe(true); + expect(gradeStockHarness(recording(generation)).some(check => /vendor|base/.test(check.id))).toBe(false); + }); + + it("checks the hire before paid execution without treating that receipt as provider evidence", () => { + const hire = recording(); + hire.runIds = []; + hire.invocations = []; + expect(gradeStockHire(hire).every(check => check.passed)).toBe(true); + expect(gradeStockHarness(hire)).toContainEqual(expect.objectContaining({ id: "provider-runs-present", passed: false })); + hire.bundle.files[0]!.content += "Old operating manual."; + expect(gradeStockHire(hire)).toContainEqual(expect.objectContaining({ id: "default-hire-bundle", passed: false })); + }); + + it("keeps historical instruction observations separate from the unchanged behavioral oracle", () => { + const historical = classifyStockInstructionManual(readFileSync(new URL("fixtures/stock-harness/historical-default-agents.md", import.meta.url), "utf8")); + const evidence = recording(); + evidence.bundle.files[0]!.content = historical.content; + evidence.invocations[0]!.prompt += "\nExecution contract:\nFinal disposition checklist:"; + evidence.invocations[1]!.prompt += "\nExecution contract: take concrete action\na successful process exit or final response is not sufficient"; + expect(gradeSourceHarness(evidence, historical).every(check => check.passed)).toBe(true); + expect(gradeSourceHarness(evidence, historical)).toContainEqual(expect.objectContaining({ id: "historical-continuation-contract", passed: true })); + expect(gradeSourceHarness(evidence, reduced)).toContainEqual(expect.objectContaining({ id: "default-hire-bundle", passed: false })); + evidence.invocations[0]!.prompt = "You are agent agent. Connection tools: connections_search"; + expect(gradeSourceHarness(evidence, historical)).toContainEqual(expect.objectContaining({ id: "historical-startup-contract", passed: false })); + const current = readStockInstructionVariant(); + evidence.bundle.files[0]!.content = current.content; + expect(gradeSourceHire(evidence).every(check => check.passed)).toBe(true); + }); + + it("cannot use valid historical startup to conceal a missing task continuation or misclassified mode", () => { + const historical = classifyStockInstructionManual(readFileSync(new URL("fixtures/stock-harness/historical-default-agents.md", import.meta.url), "utf8")); + const evidence = recording(); + evidence.bundle.files[0]!.content = historical.content; + evidence.invocations[0]!.prompt += "\nExecution contract:\nFinal disposition checklist:"; + expect(gradeSourceHarness(evidence, historical)).toContainEqual(expect.objectContaining({ id: "historical-startup-contract", passed: true })); + expect(gradeSourceHarness(evidence, historical)).toContainEqual(expect.objectContaining({ id: "historical-continuation-contract", passed: false })); + evidence.invocations[1]!.prompt += "\nExecution contract: take concrete action\na successful process exit or final response is not sufficient"; + delete evidence.invocations[1]!.conversationMode; + expect(gradeSourceHarness(evidence, historical)).toContainEqual(expect.objectContaining({ id: "historical-invocation-classification", passed: false })); + }); + + it("checks historical conversation startup and resume separately without injecting the task contract", () => { + const historical = classifyStockInstructionManual(readFileSync(new URL("fixtures/stock-harness/historical-default-agents.md", import.meta.url), "utf8")); + const evidence = recording(); + evidence.bundle.files[0]!.content = historical.content; + for (const row of evidence.invocations) row.conversationMode = true; + evidence.invocations[0]!.prompt += "\nContinue your Paperclip conversation using the supplied chat mode directive.\nAfter 2 consecutive failures of the same control-plane write"; + expect(gradeSourceHarness(evidence, historical).every(check => check.passed)).toBe(true); + evidence.invocations[1]!.prompt += "\nExecution contract:"; + expect(gradeSourceHarness(evidence, historical)).toContainEqual(expect.objectContaining({ id: "historical-continuation-contract", passed: false })); + }); + + it.each([ + ["manual-regrew", (e: StockHarnessEvidence) => { e.bundle.files[0]!.content += "Always comment and delegate."; }, "default-hire-bundle"], + ["extra-file", (e: StockHarnessEvidence) => { e.bundle.files.push({ path: "HEARTBEAT.md", content: "Do everything again." }); }, "default-hire-bundle"], + ["wrong-entry", (e: StockHarnessEvidence) => { e.bundle.entryFile = "OTHER.md"; }, "default-hire-bundle"], + ["missing-runs", (e: StockHarnessEvidence) => { e.runIds = []; }, "provider-runs-present"], + ["missing-invocation", (e: StockHarnessEvidence) => { e.invocations.pop(); }, "invocation-evidence-complete"], + ["empty-prompt", (e: StockHarnessEvidence) => { e.invocations[1]!.prompt = ""; }, "invocation-evidence-complete"], + ["malformed-prompt", (e: StockHarnessEvidence) => { e.invocations[1]!.prompt = { text: "hidden" }; }, "invocation-evidence-complete"], + ["old-startup", (e: StockHarnessEvidence) => { e.invocations[0]!.prompt += "\nExecution contract:"; }, "generic-procedures-absent"], + ["old-resume", (e: StockHarnessEvidence) => { e.invocations[1]!.prompt += "\na successful process exit or final response is not sufficient"; }, "generic-procedures-absent"], + ["missing-connections", (e: StockHarnessEvidence) => { e.invocations[0]!.prompt = "You are agent agent."; }, "fresh-default-delivered"], + ["missing-fresh-evidence", (e: StockHarnessEvidence) => { e.invocations[0]!.promptMetrics = {}; }, "fresh-default-delivered"], + ["wrong-budget", (e: StockHarnessEvidence) => { e.budgets.agentMonthlyCents = 0; }, "budget-hard-stops"], + ] as const)("fails closed for %s", (_name, mutate, failedId) => { + const evidence = recording(); + mutate(evidence); + expect(gradeStockHarness(evidence)).toContainEqual(expect.objectContaining({ id: failedId, passed: false })); + }); + + it("captures actual public bundle and invocation receipts, ignoring unrelated events", async () => { + const paths: string[] = []; + const api = { async get(url: string): Promise { + paths.push(url); + if (url.endsWith("/instructions-bundle")) return { entryFile: "AGENTS.md", files: [{ path: "AGENTS.md" }] } as T; + if (url.includes("/file?")) return { content: STOCK_HIRE_IDENTITY } as T; + if (url.includes("/events?")) return [ + { eventType: "adapter.invoke", payload: { prompt: "Actual prompt", promptMetrics: { heartbeatPromptChars: 10 } } }, + { eventType: "assistant", payload: { prompt: "Model claims about instructions" } }, + ] as T; + if (url.endsWith("/heartbeat-runs/run")) return { id: "run", companyId: "company", agentId: "agent", contextSnapshot: { conversationMode: false } } as T; + return { budgetMonthlyCents: 1_000 } as T; + } }; + const evidence = await captureStockHarness({ api, companyId: "company", agentId: "agent", generation: "legacy", runIds: ["run"] }); + expect(evidence.bundle.files[0]?.content).toBe(STOCK_HIRE_IDENTITY); + expect(evidence.invocations).toHaveLength(1); + expect(evidence.invocations[0]).toMatchObject({ runId: "run", prompt: "Actual prompt" }); + expect(paths).toContain("/api/heartbeat-runs/run/events?limit=1000"); + }); + + it("does not hide evidence API failures", async () => { + const api = { async get(): Promise { throw new Error("Public API unavailable"); } }; + await expect(captureStockHarness({ api, companyId: "company", agentId: "agent", generation: "legacy", runIds: ["run"] })) + .rejects.toThrow("Public API unavailable"); + }); + + it("normalizes an omitted conversationMode only from a complete scoped task snapshot", async () => { + const api = { async get(url: string): Promise { + return (url.endsWith("/instructions-bundle") ? { entryFile: "AGENTS.md", files: [] } + : url.endsWith("/heartbeat-runs/run") ? { id: "run", companyId: "company", agentId: "agent", contextSnapshot: { taskId: "task", issueId: "task" } } + : url.includes("/events?") ? [{ eventType: "adapter.invoke", payload: { prompt: "Task context", promptMetrics: { heartbeatPromptChars: 1 } } }] + : { budgetMonthlyCents: 1000 }) as T; + } }; + const result = await captureStockHarness({ api, companyId: "company", agentId: "agent", generation: "legacy", runIds: ["run"] }); + expect(result.invocations[0]?.conversationMode).toBe(false); + }); + + it.each([ + { id: "other", companyId: "company", agentId: "agent", contextSnapshot: {} }, + { id: "run", companyId: "other", agentId: "agent", contextSnapshot: {} }, + { id: "run", companyId: "company", agentId: "other", contextSnapshot: {} }, + { id: "run", companyId: "company", agentId: "agent" }, + { id: "run", companyId: "company", agentId: "agent", contextSnapshot: "invalid" }, + { id: "run", companyId: "company", agentId: "agent", contextSnapshot: [] }, + { id: "run", companyId: "company", agentId: "agent", contextSnapshot: { conversationMode: "unknown" } }, + ])("rejects missing or mismatched public invocation mode attribution %#", async run => { + const api = { async get(url: string): Promise { + return (url.endsWith("/instructions-bundle") ? { entryFile: "AGENTS.md", files: [] } + : url.endsWith("/heartbeat-runs/run") ? run : url.includes("/events?") ? [] : { budgetMonthlyCents: 1000 }) as T; + } }; + await expect(captureStockHarness({ api, companyId: "company", agentId: "agent", generation: "legacy", runIds: ["run"] })) + .rejects.toThrow("mode receipt"); + }); + + it("changes the definition hash when the grader digest changes", () => { + const suite = runnerMatrix.find(row => row.suite.id === "stock-harness")!.suite; + expect(suiteDefinitionHash({ ...suite, definitionMetadata: { ...suite.definitionMetadata, sourceDigest: "changed" } })) + .not.toBe(suiteDefinitionHash(suite)); + }); +}); diff --git a/tests/runner-e2e/stock-harness.ts b/tests/runner-e2e/stock-harness.ts new file mode 100644 index 00000000000..7e7fd1dac1d --- /dev/null +++ b/tests/runner-e2e/stock-harness.ts @@ -0,0 +1,184 @@ +import { createHash } from "node:crypto"; +import { readFileSync } from "node:fs"; +import type { RunnerProfileFixture } from "./types.js"; +import { readStockInstructionVariant } from "./stock-harness-instruction-variant.mjs"; + +// These are an independent product contract, not imports of the implementation +// constants. Otherwise a larger shipped manual could silently update the oracle. +export const STOCK_HIRE_IDENTITY = "You are an agent in a Paperclip company.\n"; +export const STOCK_TEMPLATE_IDENTITY = "You are agent"; +const SKILL_SOURCES = ["../../skills/paperclip/SKILL.md", "../../skills/paperclip/references/issue-documents.md"]; +export function stockHarnessSkillSources() { + return SKILL_SOURCES.map(source => { + try { + return { path: source.replace(/^\.\.\/\.\.\//, ""), present: true, + sha256: createHash("sha256").update(readFileSync(new URL(source, import.meta.url))).digest("hex") }; + } catch (error) { + if (source.endsWith("/references/issue-documents.md") && (error as NodeJS.ErrnoException).code === "ENOENT") + return { path: source.replace(/^\.\.\/\.\.\//, ""), present: false, sha256: null }; + throw error; + } + }); +} +const REMOVED_PROCEDURES = [ + "Execution contract:", + "Start actionable work in this heartbeat", + "clear final disposition", + "After 2 consecutive failures of the same control-plane write", + "Use child issues for parallel or long delegated work", + "a successful process exit or final response is not sufficient", +]; + +export function productionDefaultHireProfile(profile: RunnerProfileFixture): RunnerProfileFixture { + return { + ...profile, + buildAgent(input) { + const { instructionsBundle: _fixtureManual, ...agent } = profile.buildAgent(input); + return agent; + }, + }; +} + +export interface StockHarnessEvidence { + schema: "paperclip.stock-harness.v1"; + generation: "legacy" | "native"; + agentId: string; + budgets: { companyMonthlyCents: unknown; agentMonthlyCents: unknown }; + bundle: { entryFile?: string; files: Array<{ path: string; content: string }> }; + invocations: Array<{ runId: string; prompt: unknown; promptMetrics?: Record; conversationMode?: boolean }>; + runIds: string[]; +} + +export function gradeStockHire(evidence: Pick, + instructionVariant = readStockInstructionVariant()) { + const checks: Array<{ id: string; passed: boolean; detail: string }> = []; + const check = (id: string, passed: boolean, detail: string) => checks.push({ id, passed, detail }); + check("budget-hard-stops", evidence.budgets.companyMonthlyCents === 1_000 && evidence.budgets.agentMonthlyCents === 1_000, + "Public company and agent records must both retain the 1,000-cent monthly hard stops."); + check("default-hire-bundle", evidence.bundle.entryFile === "AGENTS.md" && + evidence.bundle.files.length === 1 && evidence.bundle.files[0]?.path === "AGENTS.md" && + evidence.bundle.files[0]?.content === instructionVariant.content, + instructionVariant.variant === "reduced" + ? "The public managed bundle must contain only the eight-word shipped identity, without an injected QA manual." + : "The comparison bundle must exactly match the independently pinned historical manual; this is structural evidence, not a task outcome."); + return checks; +} + +export function gradeStockHarness(evidence: StockHarnessEvidence, instructionVariant = readStockInstructionVariant()) { + const checks = gradeStockHire(evidence, instructionVariant); + const check = (id: string, passed: boolean, detail: string) => checks.push({ id, passed, detail }); + check("provider-runs-present", evidence.runIds.length > 0 && evidence.runIds.every(Boolean), + "The lifecycle oracle must reach actual provider runs; missing runs cannot pass instruction delivery."); + if (evidence.generation === "legacy") { + const prompts = evidence.invocations.filter(row => typeof row.prompt === "string" && row.prompt.length > 0); + check("invocation-evidence-complete", evidence.runIds.length > 0 && evidence.runIds.every(runId => + prompts.some(row => row.runId === runId)), + "Every legacy run must have a public adapter.invoke event with its actual nonempty prompt."); + if (instructionVariant.variant === "reduced") { + check("generic-procedures-absent", prompts.length > 0 && prompts.every(row => + REMOVED_PROCEDURES.every(procedure => !String(row.prompt).includes(procedure))), + "Neither startup nor continuation may reintroduce the removed generic manual."); + } else { + const classified = prompts.every(row => typeof row.conversationMode === "boolean" && + typeof row.promptMetrics?.heartbeatPromptChars === "number" && + Number.isFinite(row.promptMetrics.heartbeatPromptChars) && row.promptMetrics.heartbeatPromptChars >= 0); + const fresh = prompts.filter(row => Number(row.promptMetrics?.heartbeatPromptChars) > 0); + const resumed = prompts.filter(row => row.promptMetrics?.heartbeatPromptChars === 0); + check("historical-invocation-classification", prompts.length > 0 && classified, + "Every invocation needs public run-scoped conversation mode and explicit template-delivery metrics."); + check("historical-startup-contract", classified && fresh.length > 0 && fresh.every(row => row.conversationMode + ? String(row.prompt).includes("Continue your Paperclip conversation using the supplied chat mode directive.") && + String(row.prompt).includes("After 2 consecutive failures of the same control-plane write") + : String(row.prompt).includes("Execution contract:") && String(row.prompt).includes("Final disposition checklist:")), + `Check each of ${fresh.length} fresh invocations against its historical task or conversation template.`); + check("historical-continuation-contract", classified && resumed.every(row => + String(row.prompt).includes("## Paperclip Resume Delta") && (row.conversationMode + ? !String(row.prompt).includes("Execution contract:") + : String(row.prompt).includes("Execution contract: take concrete action") && + String(row.prompt).includes("a successful process exit or final response is not sufficient"))), + `Check all ${resumed.length} observed continuation invocations separately; zero observations do not claim live continuation coverage.`); + } + const fresh = prompts.filter(row => Number(row.promptMetrics?.heartbeatPromptChars) > 0); + check("fresh-default-delivered", fresh.length > 0 && fresh.every(row => + String(row.prompt).includes(STOCK_TEMPLATE_IDENTITY) && String(row.prompt).includes("Connection tools:") && + String(row.prompt).includes("connections_search")), + "At least one fresh invocation must carry default identity and connection guidance."); + } + return checks; +} + +/** Public API receipts only; no provider-memory claim or private runner hook. */ +export async function captureStockHarness(input: { + api: { get(path: string): Promise }; + agentId: string; + companyId: string; + generation: "legacy" | "native"; + runIds: string[]; +}): Promise { + const bundle = await input.api.get<{ entryFile: string; files: Array<{ path: string }> }>( + `/api/agents/${input.agentId}/instructions-bundle`, + ); + const [company, agent] = await Promise.all([ + input.api.get<{ budgetMonthlyCents: unknown }>(`/api/companies/${input.companyId}`), + input.api.get<{ budgetMonthlyCents: unknown }>(`/api/agents/${input.agentId}`), + ]); + const files = await Promise.all(bundle.files.map(async file => { + const detail = await input.api.get<{ content: string }>( + `/api/agents/${input.agentId}/instructions-bundle/file?path=${encodeURIComponent(file.path)}`, + ); + return { path: file.path, content: detail.content }; + })); + const events = await Promise.all(input.runIds.map(async runId => ({ + runId, + run: await input.api.get<{ id: string; companyId: string; agentId: string; contextSnapshot?: { conversationMode?: boolean } }>( + `/api/heartbeat-runs/${runId}`, + ), + rows: await input.api.get }>>( + `/api/heartbeat-runs/${runId}/events?limit=1000`, + ), + }))); + if (events.some(({ runId, run }) => run.id !== runId || run.companyId !== input.companyId || run.agentId !== input.agentId || + !run.contextSnapshot || typeof run.contextSnapshot !== "object" || Array.isArray(run.contextSnapshot) || + (run.contextSnapshot.conversationMode !== undefined && typeof run.contextSnapshot.conversationMode !== "boolean"))) { + throw new Error("Stock invocation mode receipt is missing or bound to a different run/company/agent."); + } + return { + schema: "paperclip.stock-harness.v1", generation: input.generation, + budgets: { companyMonthlyCents: company.budgetMonthlyCents, agentMonthlyCents: agent.budgetMonthlyCents }, + agentId: input.agentId, bundle: { entryFile: bundle.entryFile, files }, runIds: input.runIds, + invocations: events.flatMap(({ runId, rows, run }) => rows.filter(row => row.eventType === "adapter.invoke") + .map(row => ({ runId, prompt: row.payload?.prompt, promptMetrics: row.payload?.promptMetrics as Record | undefined, + conversationMode: run.contextSnapshot?.conversationMode === true }))), + }; +} + +export function stockHarnessSourceDigest() { + const hash = createHash("sha256"); + for (const source of [ + "stock-harness.ts", "context-integrity-cases.ts", "context-integrity-scoring.ts", + "context-integrity-flow.ts", "chat-cases.ts", "chat-flow.ts", "live-fixtures.ts", "runner.spec.ts", + "stock-harness-checks.mjs", "stock-harness-admission.ts", "stock-harness-manifest.ts", + "stock-harness-instruction-variant.mjs", "stock-harness-instruction-variant.d.mts", "stock-harness-instruction-variant.test.mjs", + "fixtures/stock-harness/historical-default-agents.md", "automatic-retry.ts", "automatic-retry.test.ts", "types.ts", "catalog.ts", "launch.ts", + "../../packages/adapter-utils/src/acpx-engine/execute.ts", + "../../packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.ts", + "../../packages/adapter-utils/src/acpx-engine/ephemeral-session-environment.test.ts", + "../../packages/paperclip-runner/scripts/generate-capability-contract.mjs", + "../../packages/paperclip-runner/scripts/check-capability-inventory.mjs", + "../../packages/paperclip-runner/scripts/lib/capability-inventory.mjs", + "../../packages/paperclip-runner/spec/capability/source-contract.json", + "../../packages/paperclip-runner/spec/capability/capabilities.yaml", + "../../packages/paperclip-runner/spec/capability/eval-traceability.yaml", + "../../packages/paperclip-runner/spec/capability/mcp-tool-map.yaml", + "../../packages/paperclip-runner/spec/capability/inventory.schema.json", + "../../packages/paperclip-runner/src/generated/capability-contract.ts", + "../../packages/paperclip-runner/docs/capability-contract.md", + ...["capabilities.yaml", "mcp-tool-map.yaml", "eval-traceability.yaml", "capability-contract.md", "downstream-handoff.md"] + .map(file => `../../packages/paperclip-runner/generated/capability/${file}`), + "../../server/src/onboarding-assets/default/AGENTS.md", + "../../packages/adapter-utils/src/server-utils.ts", + "../../packages/shared/src/connection-intent-guidance.ts", + ]) hash.update(source).update(readFileSync(new URL(source, import.meta.url))); + hash.update(JSON.stringify(stockHarnessSkillSources())); + return hash.digest("hex"); +} diff --git a/tests/runner-e2e/vitest.config.ts b/tests/runner-e2e/vitest.config.ts index 38dcf1b369a..fd13ba8fd34 100644 --- a/tests/runner-e2e/vitest.config.ts +++ b/tests/runner-e2e/vitest.config.ts @@ -4,7 +4,7 @@ export default defineConfig({ root: import.meta.dirname, test: { name: "runner-e2e-support", - include: ["**/*.test.ts"], + include: ["**/*.test.ts", "**/*.test.mjs"], testTimeout: 30_000, }, }); From eb049aebf2a7867f1865e93bca33119f92d019fa Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 3 Oct 2026 12:37:55 -0500 Subject: [PATCH 10/37] feat(skills): let agents update company skills safely (#15049) ## Thinking Path > - Paperclip is an open source control plane for AI-agent companies. > - Company skills give agents reusable work instructions. > - Skill Studio can edit skill files and save version history. > - Agents can create a skill, but they do not have a first-class update tool. > - An agent update needs a version check and safe retry behavior to prevent lost edits. > - This pull request adds `update_skill` through the existing company skill file API. > - The change keeps company policy, version history, and audit records in one path. ## Linked Issues or Issue Description **Subsystem affected** Cross-cutting: the server API, shared validation, and runner tool catalog. **Problem or motivation** An agent can create a company skill but cannot update its `SKILL.md` through a first-class tool. An unguarded retry can also create duplicate versions or overwrite a newer edit. **Proposed solution** Add `update_skill` with a required current version ID and a retry key. Route it through the existing skill file API. Reject stale versions and changed-input retries. Save the version and audit event together. **Alternatives considered** A separate write endpoint would duplicate the Skill Studio mutation path and policy checks. This PR reuses that path instead. **Roadmap alignment** This work extends Skills Manager and Skill Studio, which are listed in `ROADMAP.md`. **Additional context** The tool accepts a complete `SKILL.md`, not a partial patch. Callers must read the current version before they edit it. ## What Changed - Add optional version and retry fields to the existing skill file update contract. - Add a guarded API update with a stable retry receipt and attributed audit event. - Add `update_skill` to native and semantic runner tool catalogs, with mode and policy gates. - Add unit, integration, protocol, and semantic-tool regression coverage. - Document agent use and extend the OpenAPI request contract. ## Verification - Focused tests and direct server and runner TypeScript checks passed before this PR. - `git diff --check` passed after the rebase onto `master`. - CI passed on the latest PR head, including the full test matrix, typecheck, and build. Local full typecheck and build stopped because `cargo` is not installed. The local full test run ended without a verdict. - No dedicated end-to-end eval scenario was added or run. The protocol coverage and semantic-tool test cover the new action deterministically. - Reviewers can read a skill version, call `update_skill`, repeat the same key, then try a stale version and a changed-input key. Only the first edit must create a new version. ## Risks - File writes and database transactions must stay in sync when a write fails. The integration tests cover failed writes and retry behavior, but CI must verify them on the PR head. - Existing Skill Studio callers do not send the new optional coordination fields. Their request shape remains valid. ## Model Used - OpenAI Codex CLI assisted with this change. The runner did not expose the exact model ID or context window. The agent used code execution and repository tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details; exact model ID and context window were not exposed) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused tests; full suite is pending CI) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green (53 pass, 4 skip on the latest head) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/runner-created-skills.md | 21 ++++ .../docs/capability-semantic-tools.md | 6 +- .../capability/semantic-tool-contracts.json | 2 +- .../evals/native-execution-seeded.json | 2 +- .../paperclip-runner/protocol/manifest.json | 2 +- .../spec/capability/protocol-coverage.json | 22 +++- .../spec/operation-groups/source.json | 6 +- .../spec/paperclip-agent-operation-groups.md | 15 +-- .../src/catalog/catalog-docs.test.ts | 2 +- .../src/catalog/reconciliation.test.ts | 14 +-- .../src/catalog/semantic-action-types.ts | 1 + .../src/eval/workflow-evals.test.ts | 6 +- .../capability-control-plane-types.ts | 2 + .../capability-mock-control-plane-adapter.ts | 15 +++ .../src/protocol-actions/index.ts | 2 + .../src/protocol-actions/update-skill.ts | 54 ++++++++ .../src/semantic-tools/discovery.ts | 2 +- .../src/semantic-tools/dispatcher.ts | 11 ++ .../src/semantic-tools/paperclip-discovery.ts | 2 +- .../src/semantic-tools/semantic-tools.test.ts | 21 +++- .../src/semantic-tools/types.ts | 1 + .../tools/capability-semantic-tool-runtime.ts | 3 + .../shared/src/validators/company-skill.ts | 2 + server/src/routes/company-skills.ts | 57 ++++++++- server/src/routes/openapi.ts | 4 +- server/src/services/company-skills.ts | 91 +++++++++----- .../paperclip-runner-tool-authority.test.ts | 3 +- .../paperclip-runner-tool-authority.ts | 10 +- .../native-runtime/runner-api-catalog.ts | 1 + .../update-skill.integration.test.ts | 116 ++++++++++++++++++ server/src/services/skill-tools.test.ts | 28 ++++- server/src/services/skill-tools.ts | 29 +++++ 32 files changed, 481 insertions(+), 72 deletions(-) create mode 100644 packages/paperclip-runner/src/protocol-actions/update-skill.ts create mode 100644 server/src/services/native-runtime/update-skill.integration.test.ts diff --git a/doc/runner-created-skills.md b/doc/runner-created-skills.md index cf3c49c3648..9d9ad472b54 100644 --- a/doc/runner-created-skills.md +++ b/doc/runner-created-skills.md @@ -4,6 +4,27 @@ The native Runner exposes `create_skill` in Auto and skill-test work. An agent can save a reusable, single-file skill directly in the company's skill library. The tool is unavailable in Ask and pre-acceptance Plan modes. +The native Runner also exposes `update_skill` in the same modes. Read the +existing skill and its `currentVersionId` through the company skill API before +editing. Supply the **complete** primary file, not a fragment: + +```json +{ + "skillId": "", + "expectedVersionId": "", + "markdown": "---\nname: release-review\ndescription: Review release notes.\n---\n\n# Review\nCheck each release note and its tests.\n", + "idempotencyKey": "review-update-1" +} +``` + +The tool uses the existing Skill Studio file endpoint and company `skills.edit` +policy. It returns a skill ID, path, resulting version ID, and Skill Studio +path. Retry a lost response with **exactly** the same key and inputs; that +returns the original receipt without another version or activity record. +Reusing the key with different inputs, or updating after the version changes, +returns a conflict. After a version conflict, reread the skill and choose a +new key for the revised edit. Policy is checked again even on exact retries. + ```json { "name": "release-review", diff --git a/packages/paperclip-runner/docs/capability-semantic-tools.md b/packages/paperclip-runner/docs/capability-semantic-tools.md index a09753d159d..e58ca83559a 100644 --- a/packages/paperclip-runner/docs/capability-semantic-tools.md +++ b/packages/paperclip-runner/docs/capability-semantic-tools.md @@ -26,7 +26,7 @@ list, not a disposition — they are never exposed as tools. See [capability disposition](capability-disposition.md) and [authorization and exposure](capability-authorization-and-exposure.md). -The catalog holds **39 tools**: 14 always-agent tools and 25 optional tools +The catalog holds **40 tools**: 14 always-agent tools and 26 optional tools across 10 groups. `finish_task`, `block_task`, `request_review`, `write_document`, `request_human_input`, `register_deliverable`, `inspect_operation_result`. -### Optional tools (25), by group +### Optional tools (26), by group | Group | Tools | | --- | --- | @@ -49,7 +49,7 @@ by src/catalog/catalog-docs.test.ts; update the catalog, not the numbers. --> | cases | `list_cases`, `upsert_case` | | workspace_runtime | `get_workspace_runtime`, `control_workspace_service` | | routines | `list_routines`, `manage_routine` | -| company_skills | `create_skill`, `list_company_skills`, `sync_company_skills` | +| company_skills | `create_skill`, `update_skill`, `list_company_skills`, `sync_company_skills` | | secrets | `list_secret_metadata`, `read_secret_value` | | portability_admin | `export_company`, `administer_company` | | test_escape_hatch | `generic_api_request` | diff --git a/packages/paperclip-runner/generated/capability/semantic-tool-contracts.json b/packages/paperclip-runner/generated/capability/semantic-tool-contracts.json index 4943b0a00a7..f76c3ec6371 100644 --- a/packages/paperclip-runner/generated/capability/semantic-tool-contracts.json +++ b/packages/paperclip-runner/generated/capability/semantic-tool-contracts.json @@ -1 +1 @@ -[{"annotations":{"exposure":"always","operationId":"get_task_context","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the active task and actor, including the exact approved Markdown revision when this issue has an accepted plan.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"get_task_context","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"get_task_history","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read bounded comments on the active task.","inputSchema":{"additionalProperties":false,"properties":{"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"}},"required":[],"type":"object"},"name":"get_task_history","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"list_documents","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List revisioned documents on the active task.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_documents","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"read_document","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the current revision of one active-task document.","inputSchema":{"additionalProperties":false,"properties":{"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"}},"required":["key"],"type":"object"},"name":"read_document","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"list_document_revisions","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read bounded revision history for one active-task document.","inputSchema":{"additionalProperties":false,"properties":{"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"},"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"}},"required":["key"],"type":"object"},"name":"list_document_revisions","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"report_progress","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append a durable progress comment to the active task.","inputSchema":{"additionalProperties":false,"properties":{"body":{"description":"Multiline progress update.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","body"],"type":"object"},"name":"report_progress","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"answer_status_question","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append the answer to a status-only wake without changing task disposition.","inputSchema":{"additionalProperties":false,"properties":{"body":{"description":"Concise status answer.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","body"],"type":"object"},"name":"answer_status_question","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"write_document","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create or update an active-task document with optimistic revision safety.","inputSchema":{"additionalProperties":false,"properties":{"baseRevisionId":{"description":"Current revision id, or null when creating.","maxLength":20000,"type":["string","null"]},"body":{"description":"Markdown document body.","maxLength":200000,"minLength":1,"type":"string"},"changeSummary":{"description":"Optional revision summary.","maxLength":20000,"type":["string","null"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"},"title":{"description":"Document title.","maxLength":300,"minLength":1,"type":"string"}},"required":["idempotencyKey","key","title","body","baseRevisionId"],"type":"object"},"name":"write_document","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"request_human_input","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a durable human question or approval card on the current Paperclip task bound to this run; Paperclip renders it and authenticates the response. Use questions with continuationPolicy='wake_assignee' when an answer is needed, including otherwise tool-free chat turns. Supply a stable idempotencyKey and reuse it on retries. For one question at a time, ask only the next unanswered question and wait for its real answer. Never fabricate answers or treat ambiguous clarification as approval. For an ordinary confirmation or checkbox card, record a clear user chat answer with call_api POST /api/issues/{id}/interactions/{interactionId}/resolve-from-comment, using the source commentId and decision (accept/reject), plus explicit selectedOptionIds for checkbox acceptance. Existing resolver permissions still apply; governed tool, secret, and connection approvals are excluded. Question forms retain their dedicated answer workflow. Preserve existing review gates. Call this tool before claiming a question was asked; if creation fails, report the failure. Do not fabricate answer links or Markdown buttons, post duplicate cards, or use call_api to create the card. Use one complete payload.questionSet for text and choice questions. Paperclip generates compatibility questions; see the payload schema for formats.","inputSchema":{"additionalProperties":false,"allOf":[{"if":{"properties":{"interactionKind":{"const":"questions"}},"required":["interactionKind"]},"then":{"properties":{"payload":{"anyOf":[{"required":["questionSet"]},{"required":["questions"]}],"required":["version"]}},"required":["payload"]}}],"properties":{"continuationPolicy":{"enum":["none","wake_assignee","wake_assignee_on_accept"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"interactionKind":{"enum":["confirmation","checkbox","questions","suggest_tasks","item_verdicts"]},"payload":{"additionalProperties":true,"description":"Kind-specific interaction data. For questions, send version:1 and one complete questionSet containing every text and choice question. Paperclip generates compatibility questions. Each canonical question needs id, prompt, required, and answerMode: text, single_select, or multi_select. Text questions have no options or customAnswer. Choice questions need at least two meaningful options with id/label. Use customAnswer:{enabled:true} for an optional written answer to a choice question. Legacy questions remain supported; if both representations are supplied, they must describe the same complete form. Keep IDs stable across retries. For confirmation, payload may be {}.","properties":{"questionSet":{"additionalProperties":false,"properties":{"description":{"maxLength":100000,"type":"string"},"questions":{"items":{"additionalProperties":false,"allOf":[{"if":{"properties":{"answerMode":{"const":"text"}},"required":["answerMode"]},"then":{"not":{"required":["customAnswer"]},"properties":{"options":{"maxItems":0,"type":"array"}}}},{"if":{"properties":{"answerMode":{"enum":["single_select","multi_select"]}},"required":["answerMode"]},"then":{"properties":{"options":{"minItems":1,"type":"array"}},"required":["options"]}}],"properties":{"answerMode":{"enum":["single_select","multi_select","text"]},"customAnswer":{"additionalProperties":false,"properties":{"enabled":{"const":true},"label":{"maxLength":1000,"type":"string"},"placeholder":{"maxLength":1000,"type":"string"}},"required":["enabled"],"type":"object"},"header":{"maxLength":1000,"type":"string"},"helpText":{"maxLength":4000,"type":"string"},"id":{"maxLength":160,"minLength":1,"type":"string"},"options":{"items":{"additionalProperties":false,"properties":{"description":{"maxLength":4000,"type":"string"},"id":{"maxLength":160,"minLength":1,"type":"string"},"label":{"maxLength":1000,"minLength":1,"type":"string"},"recommended":{"type":"boolean"}},"required":["id","label"],"type":"object"},"maxItems":128,"type":"array"},"prompt":{"maxLength":4000,"minLength":1,"type":"string"},"required":{"type":"boolean"},"textValidation":{"additionalProperties":false,"properties":{"inputType":{"enum":["text","number","integer"]},"maxLength":{"maximum":100000,"minimum":0,"type":"integer"},"maximum":{"type":"number"},"minLength":{"maximum":100000,"minimum":0,"type":"integer"},"minimum":{"type":"number"},"pattern":{"maxLength":1000,"type":"string"}},"type":"object"}},"required":["id","prompt","required","answerMode"],"type":"object"},"maxItems":64,"minItems":1,"type":"array"},"schema":{"const":"paperclip.question_set.v1"},"submitLabel":{"maxLength":200,"type":"string"},"title":{"maxLength":1000,"type":"string"}},"required":["schema","questions"],"type":"object"},"questions":{"items":{"additionalProperties":true,"properties":{"id":{"maxLength":160,"minLength":1,"type":"string"},"options":{"items":{"additionalProperties":true,"properties":{"freeText":{"type":"boolean"},"id":{"maxLength":160,"minLength":1,"type":"string"},"label":{"maxLength":1000,"minLength":1,"type":"string"}},"required":["id","label"],"type":"object"},"maxItems":129,"minItems":1,"type":"array"},"prompt":{"maxLength":4000,"minLength":1,"type":"string"},"required":{"type":"boolean"},"selectionMode":{"enum":["single","multi"]}},"required":["id","prompt","selectionMode","options"],"type":"object"},"maxItems":64,"minItems":1,"type":"array"},"version":{"const":1}},"type":"object"},"prompt":{"description":"Question or decision prompt.","maxLength":10000,"minLength":1,"type":"string"},"targetRevisionId":{"description":"Optional bound document revision.","maxLength":20000,"type":["string","null"]},"title":{"description":"Interaction card title.","maxLength":300,"minLength":1,"type":"string"}},"required":["idempotencyKey","interactionKind","title","prompt","continuationPolicy"],"type":"object"},"name":"request_human_input","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"register_deliverable","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Register attachment metadata and its artifact work product without credentials or bytes in the tool result.","inputSchema":{"additionalProperties":false,"properties":{"byteSize":{"maximum":100000000,"minimum":0,"type":"integer"},"contentRef":{"description":"Opaque package-local content reference.","maxLength":2000,"minLength":1,"type":"string"},"contentType":{"description":"Media type.","maxLength":200,"minLength":1,"type":"string"},"filename":{"description":"Display filename.","maxLength":500,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"sha256":{"pattern":"^[a-fA-F0-9]{64}$","type":"string"},"title":{"description":"Work-product title.","maxLength":500,"minLength":1,"type":"string"}},"required":["idempotencyKey","filename","contentType","byteSize","sha256","contentRef","title"],"type":"object"},"name":"register_deliverable","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"finish_task","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Finish the active task with a durable summary.","inputSchema":{"additionalProperties":false,"properties":{"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"summary":{"description":"Completion summary.","maxLength":20000,"minLength":1,"type":"string"}},"required":["idempotencyKey","summary"],"type":"object"},"name":"finish_task","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"block_task","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Block the active task with a durable reason and optional first-class dependencies.","inputSchema":{"additionalProperties":false,"properties":{"blockedByTaskIds":{"description":"Internal task ids that block this task.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"reason":{"description":"Block reason.","maxLength":20000,"minLength":1,"type":"string"}},"required":["idempotencyKey","reason"],"type":"object"},"name":"block_task","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"request_review","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Move the active task to review with a durable summary.","inputSchema":{"additionalProperties":false,"properties":{"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"summary":{"description":"Review handoff summary.","maxLength":20000,"minLength":1,"type":"string"}},"required":["idempotencyKey","summary"],"type":"object"},"name":"request_review","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"list_agents","requiredClaims":["discovery:agents:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List redacted actor profiles.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_agents","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"get_agent","requiredClaims":["discovery:agents:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read one redacted actor profile.","inputSchema":{"additionalProperties":false,"properties":{"actorId":{"description":"Actor id.","maxLength":200,"minLength":1,"type":"string"}},"required":["actorId"],"type":"object"},"name":"get_agent","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"hire_agent","requiredClaims":["delegation:agents:create"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a persistent native Runner teammate with an identity and persona. The teammate reports to the caller and inherits the caller's native runtime; provider, adapter, environment, and credential settings are selected by Paperclip and are never caller-supplied here. Reuse a suitable teammate from list_agents when possible.","inputSchema":{"additionalProperties":false,"properties":{"capabilities":{"maxLength":2000,"type":["string","null"]},"instructions":{"maxLength":20000,"type":["string","null"]},"name":{"maxLength":200,"minLength":1,"type":"string"},"role":{"enum":["ceo","cto","cmo","cfo","security","engineer","designer","pm","qa","devops","researcher","general"],"type":"string"},"title":{"maxLength":300,"type":["string","null"]}},"required":["name"],"type":"object"},"name":"hire_agent","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"search_tasks","requiredClaims":["discovery:tasks:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Search tasks by text and status within the run company.","inputSchema":{"additionalProperties":false,"properties":{"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"},"query":{"maxLength":500,"type":"string"},"statuses":{"items":{"enum":["backlog","todo","in_progress","in_review","done","blocked","cancelled"]},"maxItems":7,"type":"array","uniqueItems":true}},"required":[],"type":"object"},"name":"search_tasks","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"list_approvals","requiredClaims":["governance:approvals:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List approvals in the run company.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_approvals","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"get_approval","requiredClaims":["governance:approvals:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read one approval without protected data.","inputSchema":{"additionalProperties":false,"properties":{"approvalId":{"description":"Approval id.","maxLength":200,"minLength":1,"type":"string"}},"required":["approvalId"],"type":"object"},"name":"get_approval","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"get_approval_context","requiredClaims":["governance:approvals:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read one approval, its comments, and linked tasks.","inputSchema":{"additionalProperties":false,"properties":{"approvalId":{"description":"Approval id.","maxLength":200,"minLength":1,"type":"string"}},"required":["approvalId"],"type":"object"},"name":"get_approval_context","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"get_workspace_runtime","requiredClaims":["workspace:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read active-task workspace services.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"get_workspace_runtime","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"control_workspace_service","requiredClaims":["workspace:control"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Start, stop, or fault one active-task workspace service.","inputSchema":{"additionalProperties":false,"properties":{"action":{"enum":["start","stop","fail"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"serviceId":{"description":"Workspace service id.","maxLength":200,"minLength":1,"type":"string"},"url":{"description":"Optional service URL.","maxLength":20000,"type":["string","null"]}},"required":["idempotencyKey","serviceId","action"],"type":"object"},"name":"control_workspace_service","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"set_dependencies","requiredClaims":["dependencies:write"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Replace the active task's first-class blocker set.","inputSchema":{"additionalProperties":false,"properties":{"blockedByTaskIds":{"description":"Replacement blocker task ids.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","blockedByTaskIds"],"type":"object"},"name":"set_dependencies","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"create_task","requiredClaims":["delegation:tasks:create"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a project task from a conversation, or a child from an ordinary task. Persist initialPlan before execution. Set status to backlog when the user wants to save or plan work without starting it; backlog tasks never wake an agent. Omitted status means todo, subject to blockers.","inputSchema":{"additionalProperties":false,"properties":{"assigneeActorId":{"description":"Optional agent assignee. Omit to assign the current agent.","maxLength":20000,"type":["string","null"]},"blockedByTaskIds":{"description":"Initial blocker task ids.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"description":{"description":"Child task description.","maxLength":20000,"type":["string","null"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"initialPlan":{"description":"Relevant markdown plan saved on the new task before execution starts.","maxLength":200000,"type":["string","null"]},"priority":{"enum":["critical","high","medium","low"]},"projectId":{"description":"Project ID for the task.","type":["string","null"]},"status":{"description":"Initial status. Use backlog to save work without executing it. Defaults to todo (blocked when dependencies are unresolved).","enum":["backlog","todo"]},"title":{"description":"Child task title.","maxLength":500,"minLength":1,"type":"string"}},"required":["idempotencyKey","title"],"type":"object"},"name":"create_task","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"},"task":{"additionalProperties":false,"properties":{"assigneeActorId":{"type":["string","null"]},"id":{"minLength":1,"type":"string"},"identifier":{"type":["string","null"]},"parentId":{"minLength":1,"type":["string","null"]},"projectId":{"minLength":1,"type":["string","null"]},"status":{"minLength":1,"type":"string"}},"required":["id","identifier","parentId","status","assigneeActorId"],"type":"object"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds","task"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"request_approval","requiredClaims":["governance:approvals:request"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a governed approval and waiting posture.","inputSchema":{"additionalProperties":false,"properties":{"approvalType":{"description":"Stable approval type.","maxLength":200,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"payload":{"additionalProperties":true,"type":"object"}},"required":["idempotencyKey","approvalType","payload"],"type":"object"},"name":"request_approval","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"decide_approval","requiredClaims":["governance:approvals:decide"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Decide an approval as an explicitly authorized approver.","inputSchema":{"additionalProperties":false,"properties":{"approvalId":{"description":"Approval id.","maxLength":200,"minLength":1,"type":"string"},"decision":{"enum":["approved","rejected","cancelled"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"note":{"description":"Decision note.","maxLength":20000,"minLength":1,"type":"string"}},"required":["idempotencyKey","approvalId","decision","note"],"type":"object"},"name":"decide_approval","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"comment_on_approval","requiredClaims":["governance:approvals:comment"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Add a durable comment to an approval.","inputSchema":{"additionalProperties":false,"properties":{"approvalId":{"description":"Approval id.","maxLength":200,"minLength":1,"type":"string"},"body":{"description":"Approval comment.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","approvalId","body"],"type":"object"},"name":"comment_on_approval","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"schedule_wake","requiredClaims":["control_plane:wakes"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Schedule a deterministic continuation wake.","inputSchema":{"additionalProperties":false,"properties":{"delayTicks":{"maximum":10000,"minimum":1,"type":"integer"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"payload":{"additionalProperties":true,"type":"object"},"reason":{"enum":["manual","issue_commented","interaction_resolved","approval_resolved","blockers_resolved","scheduled_retry","resume"]}},"required":["idempotencyKey","reason","delayTicks"],"type":"object"},"name":"schedule_wake","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"generic_api_request","requiredClaims":["test:generic_api_request"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Test-only escape hatch. Disabled unless the scenario and explicit claim both enable it.","inputSchema":{"additionalProperties":false,"properties":{"body":{"additionalProperties":true,"type":"object"},"method":{"enum":["GET","POST","PATCH"]},"path":{"maxLength":500,"pattern":"^/mock/","type":"string"}},"required":["method","path"],"type":"object"},"name":"generic_api_request","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"read_agent_instructions","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the current canonical instruction entry and its revision, or inspect a historical revision by supplying both entryFile and revisionId. Read before updating; do not edit shared instruction caches.","inputSchema":{"additionalProperties":false,"properties":{"entryFile":{"description":"Configured relative entry filename returned by read_agent_instructions; retain it with the revision.","maxLength":4096,"minLength":1,"type":"string"},"revisionId":{"description":"Historical revision to inspect; also provide its entryFile.","pattern":"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$","type":"string"},"targetAgentId":{"description":"Same-company target agent. Omit to use the calling agent.","pattern":"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$","type":"string"}},"required":[],"type":"object"},"name":"read_agent_instructions","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"search_api","requiredClaims":["api:discover"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Fallback only: discover Paperclip API operations when the available dedicated tools cannot express the task. Prefer dedicated tools for common operations; do not search before using them. For a persistent hire, first list_agents to reuse a suitable teammate. Search agent-hires for the hiring schema and agent-configurations for compatible adapter/runtime settings, then use call_api with the returned operationId. Supply role-specific instructionsBundle.files as a filename-to-content record. Use the returned agent.id for delegation and obey any pending approval. Provider helper threads are temporary workers, not Paperclip hires. If a hire response is uncertain, reconcile with list_agents before retrying.","inputSchema":{"additionalProperties":false,"properties":{"cursor":{"maxLength":200,"type":"string"},"limit":{"default":5,"maximum":10,"minimum":1,"type":"integer"},"query":{"maxLength":500,"minLength":1,"type":"string"}},"required":["query"],"type":"object"},"name":"search_api","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"update_agent_instructions","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Commit instruction content with the exact entryFile and baseRevisionId from a prior read. Requires the responsible user’s current target edit permission. On conflict, preserve your candidate and explicitly resolve it; never overwrite the newer head.","inputSchema":{"additionalProperties":false,"properties":{"baseRevisionId":{"description":"Revision read before editing; null only when no canonical entry exists. Never replace a stale base silently.","pattern":"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$","type":["string","null"]},"content":{"description":"Complete UTF-8 instruction content, at most 1 MiB; empty content is valid.","maxLength":1048576,"type":"string"},"entryFile":{"description":"Configured relative entry filename returned by read_agent_instructions; retain it with the revision.","maxLength":4096,"minLength":1,"type":"string"},"targetAgentId":{"description":"Same-company target agent. Omit to use the calling agent.","pattern":"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$","type":"string"}},"required":["entryFile","content","baseRevisionId"],"type":"object"},"name":"update_agent_instructions","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"call_api","requiredClaims":["api:call"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool. For large text responses, read the returned artifact with GET /api/assets/{assetId}/content and responseText; follow nextOffsetBytes until null.","inputSchema":{"additionalProperties":false,"properties":{"body":{"anyOf":[{"additionalProperties":true,"type":"object"},{"items":{},"type":"array"},{"type":"string"},{"type":"number"},{"type":"boolean"},{"type":"null"}],"description":"Request value matching the discovered schema. For JSON object or array requests, pass the object or array directly, never a JSON-encoded string. Strings are for text bodies or endpoints whose schema explicitly accepts a string."},"contentType":{"maxLength":120,"type":"string"},"files":{"items":{"additionalProperties":false,"oneOf":[{"properties":{"artifactId":{}},"required":["artifactId"]},{"properties":{"path":{}},"required":["path"]}],"properties":{"artifactId":{"type":"string"},"field":{"type":"string"},"path":{"description":"File relative to the active issue workspace. Remote files must first be uploaded as an artifact.","type":"string"}},"type":"object"},"maxItems":10,"type":"array"},"operationId":{"description":"Exact operationId returned by search_api, for example GET /api/projects/{id}. Do not guess identifiers.","maxLength":500,"minLength":1,"type":"string"},"pathParams":{"additionalProperties":{"type":"string"},"type":"object"},"query":{"additionalProperties":true,"type":"object"},"responseText":{"additionalProperties":false,"description":"GET only: return a bounded UTF-8 text window inline, including JSON as text, without saving another artifact. Offsets and limits are bytes. Use the returned nextOffsetBytes to continue; null means complete. Prefer reading a saved artifact for a stable snapshot. New live responses have a 1 GiB capture limit and a 4 GiB per-run capture budget. Existing larger assets remain readable in bounded pages. If a live response returns an artifact, continue on its content operation for a stable snapshot.","properties":{"limitBytes":{"default":24576,"maximum":24576,"minimum":4,"type":"integer"},"offsetBytes":{"default":0,"maximum":9007199254740991,"minimum":0,"type":"integer"}},"type":"object"}},"required":["operationId"],"type":"object"},"name":"call_api","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"get_agent_instruction_history","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List bounded canonical instruction revision metadata, including actor, source run, and restore origin. Use read_agent_instructions with entryFile and revisionId to inspect exact historical content.","inputSchema":{"additionalProperties":false,"properties":{"cursor":{"maxLength":2048,"minLength":1,"type":"string"},"entryFile":{"description":"Configured relative entry filename returned by read_agent_instructions; retain it with the revision.","maxLength":4096,"minLength":1,"type":"string"},"limit":{"maximum":100,"minimum":1,"type":"integer"},"targetAgentId":{"description":"Same-company target agent. Omit to use the calling agent.","pattern":"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$","type":"string"}},"required":["entryFile"],"type":"object"},"name":"get_agent_instruction_history","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"restore_agent_instructions","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append a historical instruction revision as the current content using the current baseRevisionId. Requires the responsible user’s current target edit permission. History remains intact; conflicts require an explicit resolution.","inputSchema":{"additionalProperties":false,"properties":{"baseRevisionId":{"description":"Current revision read before restoring. Never replace a stale base silently.","pattern":"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$","type":"string"},"entryFile":{"description":"Configured relative entry filename returned by read_agent_instructions; retain it with the revision.","maxLength":4096,"minLength":1,"type":"string"},"revisionId":{"description":"Historical revision whose exact content should be restored.","pattern":"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$","type":"string"},"targetAgentId":{"description":"Same-company target agent. Omit to use the calling agent.","pattern":"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$","type":"string"}},"required":["entryFile","revisionId","baseRevisionId"],"type":"object"},"name":"restore_agent_instructions","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"create_project","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a project after considering existing projects and available repositories. repositoryIds and repositoryUrls accept multiple existing repositories. Use HTTPS GitHub repositoryUrls when an accessible repo is not in the catalog; this registers project repositories, not remote GitHub repositories. Non-code projects may omit repositories. Cannot combine repositoryIds/repositoryUrls with workspace. Reuse the idempotency key on retries.","inputSchema":{"additionalProperties":false,"properties":{"archivedAt":{"description":"Archive timestamp.","maxLength":20000,"type":["string","null"]},"color":{"description":"Project color.","maxLength":20000,"type":["string","null"]},"description":{"description":"Project outcome and context.","maxLength":20000,"type":["string","null"]},"env":{"additionalProperties":true,"type":"object"},"executionWorkspacePolicy":{"additionalProperties":true,"type":"object"},"goalId":{"description":"Goal ID.","maxLength":20000,"type":["string","null"]},"goalIds":{"description":"Goal IDs.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"icon":{"description":"Project icon.","enum":["folder","rocket","code","terminal","database","globe","package","boxes","box","layers","briefcase","compass","target","flame","zap","star","bug","wrench","hammer","lightbulb","sparkles","shield","lock","search","cog","brain","cpu","git-branch","file-code","puzzle","gem","atom","heart","mail","message-square","crown","radar","telescope","hexagon",null],"type":["string","null"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"leadAgentId":{"description":"Lead agent ID.","maxLength":20000,"type":["string","null"]},"name":{"description":"Project name.","maxLength":500,"minLength":1,"type":"string"},"repositoryIds":{"description":"Authorized repository IDs from list_project_repositories; may contain multiple repositories.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"repositoryUrls":{"description":"Existing HTTPS GitHub repository URLs, including repos absent from the catalog.","items":{"maxLength":2000,"pattern":"^https://github\\.com/(?!\\.{1,2}/)[A-Za-z0-9_.-]+/(?!\\.{1,2}/?$)[A-Za-z0-9_.-]+/?$","type":"string"},"maxItems":100,"type":"array"},"status":{"enum":["backlog","planned","in_progress","completed","cancelled"]},"targetDate":{"description":"Target date.","maxLength":20000,"type":["string","null"]},"workspace":{"additionalProperties":true,"type":"object"}},"required":["idempotencyKey","name"],"type":"object"},"name":"create_project","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"list_project_repositories","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List authorized repositories with stable IDs and names. Consider appropriate repositories before creating a project; never invent IDs.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_project_repositories","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"list_projects","requiredClaims":["discovery:projects:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Inspect available company projects before selecting a project for new work.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_projects","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"create_skill","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a reusable single-file skill in the company library. Supply a complete SKILL.md whose name and description match the inputs. This saves the skill and shows a card; it does not assign the skill to any agent. Reuse idempotencyKey on retries.","inputSchema":{"additionalProperties":false,"properties":{"description":{"maxLength":2000,"minLength":1,"type":"string"},"idempotencyKey":{"maxLength":240,"minLength":1,"type":"string"},"markdown":{"description":"Complete SKILL.md including name and description frontmatter and substantive instructions.","maxLength":200000,"minLength":1,"type":"string"},"name":{"description":"Lowercase skill name, matching SKILL.md frontmatter.","maxLength":120,"minLength":1,"pattern":"^[a-z0-9]+(?:-[a-z0-9]+)*$","type":"string"},"slug":{"description":"Optional; must equal name.","maxLength":120,"minLength":1,"pattern":"^[a-z0-9]+(?:-[a-z0-9]+)*$","type":"string"}},"required":["idempotencyKey","name","description","markdown"],"type":"object"},"name":"create_skill","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"reassign_task","requiredClaims":["delegation:tasks:assign"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Reassign an existing task to another company agent. Read search_tasks first and supply its current assignee and statusVersion to prevent overwriting a concurrent change. Preserve the task, documents, dependencies, and blocked/backlog status. Active work is stopped before handoff. Use a stable idempotency key for retries. Cannot reassign this run’s own task, conversations, completed tasks, or pending reviews; use create_task for delegation from the current task.","inputSchema":{"additionalProperties":false,"properties":{"assigneeActorId":{"description":"New company agent ID from list_agents.","minLength":1,"type":"string"},"expectedAssigneeActorId":{"description":"Current assigneeAgentId from search_tasks; null means unassigned.","type":["string","null"]},"expectedStatusVersion":{"description":"Current statusVersion from search_tasks.","minimum":0,"type":"integer"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"reason":{"description":"Why the task should move and context for the new owner.","maxLength":20000,"minLength":1,"type":"string"},"taskId":{"description":"Existing task ID from search_tasks.","minLength":1,"type":"string"}},"required":["idempotencyKey","taskId","assigneeActorId","expectedAssigneeActorId","expectedStatusVersion","reason"],"type":"object"},"name":"reassign_task","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"set_task_title","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Set a concise, descriptive title for the active task. When its titleNeedsGeneration is true, call this early with onlyIfProvisional: true to replace the initial prompt slice without overwriting a user's title. Use false only for an intentional rename. This changes no task status, ownership, or description.","inputSchema":{"additionalProperties":false,"properties":{"idempotencyKey":{"description":"Reuse this key on retries.","maxLength":240,"minLength":1,"type":"string"},"onlyIfProvisional":{"description":"True for automatic initial naming; preserves any title already chosen by a user or agent.","type":"boolean"},"title":{"description":"Short title describing the requested outcome.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","title","onlyIfProvisional"],"type":"object"},"name":"set_task_title","outputSchema":{"additionalProperties":true,"type":"object"}}] +[{"annotations":{"exposure":"always","operationId":"get_task_context","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the active task and actor, including the exact approved Markdown revision when this issue has an accepted plan.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"get_task_context","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"get_task_history","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read bounded comments on the active task.","inputSchema":{"additionalProperties":false,"properties":{"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"}},"required":[],"type":"object"},"name":"get_task_history","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"list_documents","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List revisioned documents on the active task.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_documents","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"read_document","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the current revision of one active-task document.","inputSchema":{"additionalProperties":false,"properties":{"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"}},"required":["key"],"type":"object"},"name":"read_document","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"list_document_revisions","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read bounded revision history for one active-task document.","inputSchema":{"additionalProperties":false,"properties":{"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"},"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"}},"required":["key"],"type":"object"},"name":"list_document_revisions","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"report_progress","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append a durable progress comment to the active task.","inputSchema":{"additionalProperties":false,"properties":{"body":{"description":"Multiline progress update.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","body"],"type":"object"},"name":"report_progress","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"answer_status_question","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append the answer to a status-only wake without changing task disposition.","inputSchema":{"additionalProperties":false,"properties":{"body":{"description":"Concise status answer.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","body"],"type":"object"},"name":"answer_status_question","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"write_document","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create or update an active-task document with optimistic revision safety.","inputSchema":{"additionalProperties":false,"properties":{"baseRevisionId":{"description":"Current revision id, or null when creating.","maxLength":20000,"type":["string","null"]},"body":{"description":"Markdown document body.","maxLength":200000,"minLength":1,"type":"string"},"changeSummary":{"description":"Optional revision summary.","maxLength":20000,"type":["string","null"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"},"title":{"description":"Document title.","maxLength":300,"minLength":1,"type":"string"}},"required":["idempotencyKey","key","title","body","baseRevisionId"],"type":"object"},"name":"write_document","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"request_human_input","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a durable human question or approval card on the current Paperclip task bound to this run; Paperclip renders it and authenticates the response. Use questions with continuationPolicy='wake_assignee' when an answer is needed, including otherwise tool-free chat turns. Supply a stable idempotencyKey and reuse it on retries. For one question at a time, ask only the next unanswered question and wait for its real answer. Never fabricate answers or treat ambiguous clarification as approval. For an ordinary confirmation or checkbox card, record a clear user chat answer with call_api POST /api/issues/{id}/interactions/{interactionId}/resolve-from-comment, using the source commentId and decision (accept/reject), plus explicit selectedOptionIds for checkbox acceptance. Existing resolver permissions still apply; governed tool, secret, and connection approvals are excluded. Question forms retain their dedicated answer workflow. Preserve existing review gates. Call this tool before claiming a question was asked; if creation fails, report the failure. Do not fabricate answer links or Markdown buttons, post duplicate cards, or use call_api to create the card. Use one complete payload.questionSet for text and choice questions. Paperclip generates compatibility questions; see the payload schema for formats.","inputSchema":{"additionalProperties":false,"allOf":[{"if":{"properties":{"interactionKind":{"const":"questions"}},"required":["interactionKind"]},"then":{"properties":{"payload":{"anyOf":[{"required":["questionSet"]},{"required":["questions"]}],"required":["version"]}},"required":["payload"]}}],"properties":{"continuationPolicy":{"enum":["none","wake_assignee","wake_assignee_on_accept"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"interactionKind":{"enum":["confirmation","checkbox","questions","suggest_tasks","item_verdicts"]},"payload":{"additionalProperties":true,"description":"Kind-specific interaction data. For questions, send version:1 and one complete questionSet containing every text and choice question. Paperclip generates compatibility questions. Each canonical question needs id, prompt, required, and answerMode: text, single_select, or multi_select. Text questions have no options or customAnswer. Choice questions need at least two meaningful options with id/label. Use customAnswer:{enabled:true} for an optional written answer to a choice question. Legacy questions remain supported; if both representations are supplied, they must describe the same complete form. Keep IDs stable across retries. For confirmation, payload may be {}.","properties":{"questionSet":{"additionalProperties":false,"properties":{"description":{"maxLength":100000,"type":"string"},"questions":{"items":{"additionalProperties":false,"allOf":[{"if":{"properties":{"answerMode":{"const":"text"}},"required":["answerMode"]},"then":{"not":{"required":["customAnswer"]},"properties":{"options":{"maxItems":0,"type":"array"}}}},{"if":{"properties":{"answerMode":{"enum":["single_select","multi_select"]}},"required":["answerMode"]},"then":{"properties":{"options":{"minItems":1,"type":"array"}},"required":["options"]}}],"properties":{"answerMode":{"enum":["single_select","multi_select","text"]},"customAnswer":{"additionalProperties":false,"properties":{"enabled":{"const":true},"label":{"maxLength":1000,"type":"string"},"placeholder":{"maxLength":1000,"type":"string"}},"required":["enabled"],"type":"object"},"header":{"maxLength":1000,"type":"string"},"helpText":{"maxLength":4000,"type":"string"},"id":{"maxLength":160,"minLength":1,"type":"string"},"options":{"items":{"additionalProperties":false,"properties":{"description":{"maxLength":4000,"type":"string"},"id":{"maxLength":160,"minLength":1,"type":"string"},"label":{"maxLength":1000,"minLength":1,"type":"string"},"recommended":{"type":"boolean"}},"required":["id","label"],"type":"object"},"maxItems":128,"type":"array"},"prompt":{"maxLength":4000,"minLength":1,"type":"string"},"required":{"type":"boolean"},"textValidation":{"additionalProperties":false,"properties":{"inputType":{"enum":["text","number","integer"]},"maxLength":{"maximum":100000,"minimum":0,"type":"integer"},"maximum":{"type":"number"},"minLength":{"maximum":100000,"minimum":0,"type":"integer"},"minimum":{"type":"number"},"pattern":{"maxLength":1000,"type":"string"}},"type":"object"}},"required":["id","prompt","required","answerMode"],"type":"object"},"maxItems":64,"minItems":1,"type":"array"},"schema":{"const":"paperclip.question_set.v1"},"submitLabel":{"maxLength":200,"type":"string"},"title":{"maxLength":1000,"type":"string"}},"required":["schema","questions"],"type":"object"},"questions":{"items":{"additionalProperties":true,"properties":{"id":{"maxLength":160,"minLength":1,"type":"string"},"options":{"items":{"additionalProperties":true,"properties":{"freeText":{"type":"boolean"},"id":{"maxLength":160,"minLength":1,"type":"string"},"label":{"maxLength":1000,"minLength":1,"type":"string"}},"required":["id","label"],"type":"object"},"maxItems":129,"minItems":1,"type":"array"},"prompt":{"maxLength":4000,"minLength":1,"type":"string"},"required":{"type":"boolean"},"selectionMode":{"enum":["single","multi"]}},"required":["id","prompt","selectionMode","options"],"type":"object"},"maxItems":64,"minItems":1,"type":"array"},"version":{"const":1}},"type":"object"},"prompt":{"description":"Question or decision prompt.","maxLength":10000,"minLength":1,"type":"string"},"targetRevisionId":{"description":"Optional bound document revision.","maxLength":20000,"type":["string","null"]},"title":{"description":"Interaction card title.","maxLength":300,"minLength":1,"type":"string"}},"required":["idempotencyKey","interactionKind","title","prompt","continuationPolicy"],"type":"object"},"name":"request_human_input","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"register_deliverable","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Register attachment metadata and its artifact work product without credentials or bytes in the tool result.","inputSchema":{"additionalProperties":false,"properties":{"byteSize":{"maximum":100000000,"minimum":0,"type":"integer"},"contentRef":{"description":"Opaque package-local content reference.","maxLength":2000,"minLength":1,"type":"string"},"contentType":{"description":"Media type.","maxLength":200,"minLength":1,"type":"string"},"filename":{"description":"Display filename.","maxLength":500,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"sha256":{"pattern":"^[a-fA-F0-9]{64}$","type":"string"},"title":{"description":"Work-product title.","maxLength":500,"minLength":1,"type":"string"}},"required":["idempotencyKey","filename","contentType","byteSize","sha256","contentRef","title"],"type":"object"},"name":"register_deliverable","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"finish_task","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Finish the active task with a durable summary.","inputSchema":{"additionalProperties":false,"properties":{"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"summary":{"description":"Completion summary.","maxLength":20000,"minLength":1,"type":"string"}},"required":["idempotencyKey","summary"],"type":"object"},"name":"finish_task","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"block_task","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Block the active task with a durable reason and optional first-class dependencies.","inputSchema":{"additionalProperties":false,"properties":{"blockedByTaskIds":{"description":"Internal task ids that block this task.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"reason":{"description":"Block reason.","maxLength":20000,"minLength":1,"type":"string"}},"required":["idempotencyKey","reason"],"type":"object"},"name":"block_task","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"request_review","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Move the active task to review with a durable summary.","inputSchema":{"additionalProperties":false,"properties":{"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"summary":{"description":"Review handoff summary.","maxLength":20000,"minLength":1,"type":"string"}},"required":["idempotencyKey","summary"],"type":"object"},"name":"request_review","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"list_agents","requiredClaims":["discovery:agents:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List redacted actor profiles.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_agents","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"get_agent","requiredClaims":["discovery:agents:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read one redacted actor profile.","inputSchema":{"additionalProperties":false,"properties":{"actorId":{"description":"Actor id.","maxLength":200,"minLength":1,"type":"string"}},"required":["actorId"],"type":"object"},"name":"get_agent","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"hire_agent","requiredClaims":["delegation:agents:create"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a persistent native Runner teammate with an identity and persona. The teammate reports to the caller and inherits the caller's native runtime; provider, adapter, environment, and credential settings are selected by Paperclip and are never caller-supplied here. Reuse a suitable teammate from list_agents when possible.","inputSchema":{"additionalProperties":false,"properties":{"capabilities":{"maxLength":2000,"type":["string","null"]},"instructions":{"maxLength":20000,"type":["string","null"]},"name":{"maxLength":200,"minLength":1,"type":"string"},"role":{"enum":["ceo","cto","cmo","cfo","security","engineer","designer","pm","qa","devops","researcher","general"],"type":"string"},"title":{"maxLength":300,"type":["string","null"]}},"required":["name"],"type":"object"},"name":"hire_agent","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"search_tasks","requiredClaims":["discovery:tasks:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Search tasks by text and status within the run company.","inputSchema":{"additionalProperties":false,"properties":{"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"},"query":{"maxLength":500,"type":"string"},"statuses":{"items":{"enum":["backlog","todo","in_progress","in_review","done","blocked","cancelled"]},"maxItems":7,"type":"array","uniqueItems":true}},"required":[],"type":"object"},"name":"search_tasks","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"list_approvals","requiredClaims":["governance:approvals:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List approvals in the run company.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_approvals","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"get_approval","requiredClaims":["governance:approvals:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read one approval without protected data.","inputSchema":{"additionalProperties":false,"properties":{"approvalId":{"description":"Approval id.","maxLength":200,"minLength":1,"type":"string"}},"required":["approvalId"],"type":"object"},"name":"get_approval","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"get_approval_context","requiredClaims":["governance:approvals:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read one approval, its comments, and linked tasks.","inputSchema":{"additionalProperties":false,"properties":{"approvalId":{"description":"Approval id.","maxLength":200,"minLength":1,"type":"string"}},"required":["approvalId"],"type":"object"},"name":"get_approval_context","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"get_workspace_runtime","requiredClaims":["workspace:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read active-task workspace services.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"get_workspace_runtime","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"control_workspace_service","requiredClaims":["workspace:control"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Start, stop, or fault one active-task workspace service.","inputSchema":{"additionalProperties":false,"properties":{"action":{"enum":["start","stop","fail"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"serviceId":{"description":"Workspace service id.","maxLength":200,"minLength":1,"type":"string"},"url":{"description":"Optional service URL.","maxLength":20000,"type":["string","null"]}},"required":["idempotencyKey","serviceId","action"],"type":"object"},"name":"control_workspace_service","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"set_dependencies","requiredClaims":["dependencies:write"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Replace the active task's first-class blocker set.","inputSchema":{"additionalProperties":false,"properties":{"blockedByTaskIds":{"description":"Replacement blocker task ids.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","blockedByTaskIds"],"type":"object"},"name":"set_dependencies","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"create_task","requiredClaims":["delegation:tasks:create"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a project task from a conversation, or a child from an ordinary task. Persist initialPlan before execution. Set status to backlog when the user wants to save or plan work without starting it; backlog tasks never wake an agent. Omitted status means todo, subject to blockers.","inputSchema":{"additionalProperties":false,"properties":{"assigneeActorId":{"description":"Optional agent assignee. Omit to assign the current agent.","maxLength":20000,"type":["string","null"]},"blockedByTaskIds":{"description":"Initial blocker task ids.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"description":{"description":"Child task description.","maxLength":20000,"type":["string","null"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"initialPlan":{"description":"Relevant markdown plan saved on the new task before execution starts.","maxLength":200000,"type":["string","null"]},"priority":{"enum":["critical","high","medium","low"]},"projectId":{"description":"Project ID for the task.","type":["string","null"]},"status":{"description":"Initial status. Use backlog to save work without executing it. Defaults to todo (blocked when dependencies are unresolved).","enum":["backlog","todo"]},"title":{"description":"Child task title.","maxLength":500,"minLength":1,"type":"string"}},"required":["idempotencyKey","title"],"type":"object"},"name":"create_task","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"},"task":{"additionalProperties":false,"properties":{"assigneeActorId":{"type":["string","null"]},"id":{"minLength":1,"type":"string"},"identifier":{"type":["string","null"]},"parentId":{"minLength":1,"type":["string","null"]},"projectId":{"minLength":1,"type":["string","null"]},"status":{"minLength":1,"type":"string"}},"required":["id","identifier","parentId","status","assigneeActorId"],"type":"object"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds","task"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"request_approval","requiredClaims":["governance:approvals:request"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a governed approval and waiting posture.","inputSchema":{"additionalProperties":false,"properties":{"approvalType":{"description":"Stable approval type.","maxLength":200,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"payload":{"additionalProperties":true,"type":"object"}},"required":["idempotencyKey","approvalType","payload"],"type":"object"},"name":"request_approval","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"decide_approval","requiredClaims":["governance:approvals:decide"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Decide an approval as an explicitly authorized approver.","inputSchema":{"additionalProperties":false,"properties":{"approvalId":{"description":"Approval id.","maxLength":200,"minLength":1,"type":"string"},"decision":{"enum":["approved","rejected","cancelled"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"note":{"description":"Decision note.","maxLength":20000,"minLength":1,"type":"string"}},"required":["idempotencyKey","approvalId","decision","note"],"type":"object"},"name":"decide_approval","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"comment_on_approval","requiredClaims":["governance:approvals:comment"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Add a durable comment to an approval.","inputSchema":{"additionalProperties":false,"properties":{"approvalId":{"description":"Approval id.","maxLength":200,"minLength":1,"type":"string"},"body":{"description":"Approval comment.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","approvalId","body"],"type":"object"},"name":"comment_on_approval","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"schedule_wake","requiredClaims":["control_plane:wakes"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Schedule a deterministic continuation wake.","inputSchema":{"additionalProperties":false,"properties":{"delayTicks":{"maximum":10000,"minimum":1,"type":"integer"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"payload":{"additionalProperties":true,"type":"object"},"reason":{"enum":["manual","issue_commented","interaction_resolved","approval_resolved","blockers_resolved","scheduled_retry","resume"]}},"required":["idempotencyKey","reason","delayTicks"],"type":"object"},"name":"schedule_wake","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"generic_api_request","requiredClaims":["test:generic_api_request"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Test-only escape hatch. Disabled unless the scenario and explicit claim both enable it.","inputSchema":{"additionalProperties":false,"properties":{"body":{"additionalProperties":true,"type":"object"},"method":{"enum":["GET","POST","PATCH"]},"path":{"maxLength":500,"pattern":"^/mock/","type":"string"}},"required":["method","path"],"type":"object"},"name":"generic_api_request","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"read_agent_instructions","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the current canonical instruction entry and its revision, or inspect a historical revision by supplying both entryFile and revisionId. Read before updating; do not edit shared instruction caches.","inputSchema":{"additionalProperties":false,"properties":{"entryFile":{"description":"Configured relative entry filename returned by read_agent_instructions; retain it with the revision.","maxLength":4096,"minLength":1,"type":"string"},"revisionId":{"description":"Historical revision to inspect; also provide its entryFile.","pattern":"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$","type":"string"},"targetAgentId":{"description":"Same-company target agent. Omit to use the calling agent.","pattern":"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$","type":"string"}},"required":[],"type":"object"},"name":"read_agent_instructions","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"search_api","requiredClaims":["api:discover"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Fallback only: discover Paperclip API operations when the available dedicated tools cannot express the task. Prefer dedicated tools for common operations; do not search before using them. For a persistent hire, first list_agents to reuse a suitable teammate. Search agent-hires for the hiring schema and agent-configurations for compatible adapter/runtime settings, then use call_api with the returned operationId. Supply role-specific instructionsBundle.files as a filename-to-content record. Use the returned agent.id for delegation and obey any pending approval. Provider helper threads are temporary workers, not Paperclip hires. If a hire response is uncertain, reconcile with list_agents before retrying.","inputSchema":{"additionalProperties":false,"properties":{"cursor":{"maxLength":200,"type":"string"},"limit":{"default":5,"maximum":10,"minimum":1,"type":"integer"},"query":{"maxLength":500,"minLength":1,"type":"string"}},"required":["query"],"type":"object"},"name":"search_api","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"update_agent_instructions","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Commit instruction content with the exact entryFile and baseRevisionId from a prior read. Requires the responsible user’s current target edit permission. On conflict, preserve your candidate and explicitly resolve it; never overwrite the newer head.","inputSchema":{"additionalProperties":false,"properties":{"baseRevisionId":{"description":"Revision read before editing; null only when no canonical entry exists. Never replace a stale base silently.","pattern":"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$","type":["string","null"]},"content":{"description":"Complete UTF-8 instruction content, at most 1 MiB; empty content is valid.","maxLength":1048576,"type":"string"},"entryFile":{"description":"Configured relative entry filename returned by read_agent_instructions; retain it with the revision.","maxLength":4096,"minLength":1,"type":"string"},"targetAgentId":{"description":"Same-company target agent. Omit to use the calling agent.","pattern":"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$","type":"string"}},"required":["entryFile","content","baseRevisionId"],"type":"object"},"name":"update_agent_instructions","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"call_api","requiredClaims":["api:call"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool. For large text responses, read the returned artifact with GET /api/assets/{assetId}/content and responseText; follow nextOffsetBytes until null.","inputSchema":{"additionalProperties":false,"properties":{"body":{"anyOf":[{"additionalProperties":true,"type":"object"},{"items":{},"type":"array"},{"type":"string"},{"type":"number"},{"type":"boolean"},{"type":"null"}],"description":"Request value matching the discovered schema. For JSON object or array requests, pass the object or array directly, never a JSON-encoded string. Strings are for text bodies or endpoints whose schema explicitly accepts a string."},"contentType":{"maxLength":120,"type":"string"},"files":{"items":{"additionalProperties":false,"oneOf":[{"properties":{"artifactId":{}},"required":["artifactId"]},{"properties":{"path":{}},"required":["path"]}],"properties":{"artifactId":{"type":"string"},"field":{"type":"string"},"path":{"description":"File relative to the active issue workspace. Remote files must first be uploaded as an artifact.","type":"string"}},"type":"object"},"maxItems":10,"type":"array"},"operationId":{"description":"Exact operationId returned by search_api, for example GET /api/projects/{id}. Do not guess identifiers.","maxLength":500,"minLength":1,"type":"string"},"pathParams":{"additionalProperties":{"type":"string"},"type":"object"},"query":{"additionalProperties":true,"type":"object"},"responseText":{"additionalProperties":false,"description":"GET only: return a bounded UTF-8 text window inline, including JSON as text, without saving another artifact. Offsets and limits are bytes. Use the returned nextOffsetBytes to continue; null means complete. Prefer reading a saved artifact for a stable snapshot. New live responses have a 1 GiB capture limit and a 4 GiB per-run capture budget. Existing larger assets remain readable in bounded pages. If a live response returns an artifact, continue on its content operation for a stable snapshot.","properties":{"limitBytes":{"default":24576,"maximum":24576,"minimum":4,"type":"integer"},"offsetBytes":{"default":0,"maximum":9007199254740991,"minimum":0,"type":"integer"}},"type":"object"}},"required":["operationId"],"type":"object"},"name":"call_api","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"get_agent_instruction_history","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List bounded canonical instruction revision metadata, including actor, source run, and restore origin. Use read_agent_instructions with entryFile and revisionId to inspect exact historical content.","inputSchema":{"additionalProperties":false,"properties":{"cursor":{"maxLength":2048,"minLength":1,"type":"string"},"entryFile":{"description":"Configured relative entry filename returned by read_agent_instructions; retain it with the revision.","maxLength":4096,"minLength":1,"type":"string"},"limit":{"maximum":100,"minimum":1,"type":"integer"},"targetAgentId":{"description":"Same-company target agent. Omit to use the calling agent.","pattern":"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$","type":"string"}},"required":["entryFile"],"type":"object"},"name":"get_agent_instruction_history","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"restore_agent_instructions","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append a historical instruction revision as the current content using the current baseRevisionId. Requires the responsible user’s current target edit permission. History remains intact; conflicts require an explicit resolution.","inputSchema":{"additionalProperties":false,"properties":{"baseRevisionId":{"description":"Current revision read before restoring. Never replace a stale base silently.","pattern":"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$","type":"string"},"entryFile":{"description":"Configured relative entry filename returned by read_agent_instructions; retain it with the revision.","maxLength":4096,"minLength":1,"type":"string"},"revisionId":{"description":"Historical revision whose exact content should be restored.","pattern":"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$","type":"string"},"targetAgentId":{"description":"Same-company target agent. Omit to use the calling agent.","pattern":"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$","type":"string"}},"required":["entryFile","revisionId","baseRevisionId"],"type":"object"},"name":"restore_agent_instructions","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"create_project","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a project after considering existing projects and available repositories. repositoryIds and repositoryUrls accept multiple existing repositories. Use HTTPS GitHub repositoryUrls when an accessible repo is not in the catalog; this registers project repositories, not remote GitHub repositories. Non-code projects may omit repositories. Cannot combine repositoryIds/repositoryUrls with workspace. Reuse the idempotency key on retries.","inputSchema":{"additionalProperties":false,"properties":{"archivedAt":{"description":"Archive timestamp.","maxLength":20000,"type":["string","null"]},"color":{"description":"Project color.","maxLength":20000,"type":["string","null"]},"description":{"description":"Project outcome and context.","maxLength":20000,"type":["string","null"]},"env":{"additionalProperties":true,"type":"object"},"executionWorkspacePolicy":{"additionalProperties":true,"type":"object"},"goalId":{"description":"Goal ID.","maxLength":20000,"type":["string","null"]},"goalIds":{"description":"Goal IDs.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"icon":{"description":"Project icon.","enum":["folder","rocket","code","terminal","database","globe","package","boxes","box","layers","briefcase","compass","target","flame","zap","star","bug","wrench","hammer","lightbulb","sparkles","shield","lock","search","cog","brain","cpu","git-branch","file-code","puzzle","gem","atom","heart","mail","message-square","crown","radar","telescope","hexagon",null],"type":["string","null"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"leadAgentId":{"description":"Lead agent ID.","maxLength":20000,"type":["string","null"]},"name":{"description":"Project name.","maxLength":500,"minLength":1,"type":"string"},"repositoryIds":{"description":"Authorized repository IDs from list_project_repositories; may contain multiple repositories.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"repositoryUrls":{"description":"Existing HTTPS GitHub repository URLs, including repos absent from the catalog.","items":{"maxLength":2000,"pattern":"^https://github\\.com/(?!\\.{1,2}/)[A-Za-z0-9_.-]+/(?!\\.{1,2}/?$)[A-Za-z0-9_.-]+/?$","type":"string"},"maxItems":100,"type":"array"},"status":{"enum":["backlog","planned","in_progress","completed","cancelled"]},"targetDate":{"description":"Target date.","maxLength":20000,"type":["string","null"]},"workspace":{"additionalProperties":true,"type":"object"}},"required":["idempotencyKey","name"],"type":"object"},"name":"create_project","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"list_project_repositories","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List authorized repositories with stable IDs and names. Consider appropriate repositories before creating a project; never invent IDs.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_project_repositories","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"list_projects","requiredClaims":["discovery:projects:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Inspect available company projects before selecting a project for new work.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_projects","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"create_skill","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a reusable single-file skill in the company library. Supply a complete SKILL.md whose name and description match the inputs. This saves the skill and shows a card; it does not assign the skill to any agent. Reuse idempotencyKey on retries.","inputSchema":{"additionalProperties":false,"properties":{"description":{"maxLength":2000,"minLength":1,"type":"string"},"idempotencyKey":{"maxLength":240,"minLength":1,"type":"string"},"markdown":{"description":"Complete SKILL.md including name and description frontmatter and substantive instructions.","maxLength":200000,"minLength":1,"type":"string"},"name":{"description":"Lowercase skill name, matching SKILL.md frontmatter.","maxLength":120,"minLength":1,"pattern":"^[a-z0-9]+(?:-[a-z0-9]+)*$","type":"string"},"slug":{"description":"Optional; must equal name.","maxLength":120,"minLength":1,"pattern":"^[a-z0-9]+(?:-[a-z0-9]+)*$","type":"string"}},"required":["idempotencyKey","name","description","markdown"],"type":"object"},"name":"create_skill","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"reassign_task","requiredClaims":["delegation:tasks:assign"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Reassign an existing task to another company agent. Read search_tasks first and supply its current assignee and statusVersion to prevent overwriting a concurrent change. Preserve the task, documents, dependencies, and blocked/backlog status. Active work is stopped before handoff. Use a stable idempotency key for retries. Cannot reassign this run’s own task, conversations, completed tasks, or pending reviews; use create_task for delegation from the current task.","inputSchema":{"additionalProperties":false,"properties":{"assigneeActorId":{"description":"New company agent ID from list_agents.","minLength":1,"type":"string"},"expectedAssigneeActorId":{"description":"Current assigneeAgentId from search_tasks; null means unassigned.","type":["string","null"]},"expectedStatusVersion":{"description":"Current statusVersion from search_tasks.","minimum":0,"type":"integer"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"reason":{"description":"Why the task should move and context for the new owner.","maxLength":20000,"minLength":1,"type":"string"},"taskId":{"description":"Existing task ID from search_tasks.","minLength":1,"type":"string"}},"required":["idempotencyKey","taskId","assigneeActorId","expectedAssigneeActorId","expectedStatusVersion","reason"],"type":"object"},"name":"reassign_task","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"update_skill","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Replace an existing company skill's complete SKILL.md using the current version as a guard. Reuse idempotencyKey on lost-response retries; read again after a version conflict.","inputSchema":{"additionalProperties":false,"properties":{"expectedVersionId":{"minLength":1,"type":"string"},"idempotencyKey":{"maxLength":240,"minLength":1,"type":"string"},"markdown":{"maxLength":200000,"minLength":1,"type":"string"},"skillId":{"minLength":1,"type":"string"}},"required":["skillId","markdown","expectedVersionId","idempotencyKey"],"type":"object"},"name":"update_skill","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"set_task_title","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Set a concise, descriptive title for the active task. When its titleNeedsGeneration is true, call this early with onlyIfProvisional: true to replace the initial prompt slice without overwriting a user's title. Use false only for an intentional rename. This changes no task status, ownership, or description.","inputSchema":{"additionalProperties":false,"properties":{"idempotencyKey":{"description":"Reuse this key on retries.","maxLength":240,"minLength":1,"type":"string"},"onlyIfProvisional":{"description":"True for automatic initial naming; preserves any title already chosen by a user or agent.","type":"boolean"},"title":{"description":"Short title describing the requested outcome.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","title","onlyIfProvisional"],"type":"object"},"name":"set_task_title","outputSchema":{"additionalProperties":true,"type":"object"}}] diff --git a/packages/paperclip-runner/protocol/fixtures/evals/native-execution-seeded.json b/packages/paperclip-runner/protocol/fixtures/evals/native-execution-seeded.json index 24624d64ea1..3248198c2e3 100644 --- a/packages/paperclip-runner/protocol/fixtures/evals/native-execution-seeded.json +++ b/packages/paperclip-runner/protocol/fixtures/evals/native-execution-seeded.json @@ -24,7 +24,7 @@ "prpVersion": 1, "nativeExecutionVersion": 1, "catalogVersion": 1, - "catalogSha256": "sha256:81ed59a7fa4ceb6e0f28dffabf7a7ae7bdfa9e23e6bb643a14612368262abfe3", + "catalogSha256": "sha256:1dd4904ba9ae5c2b6bb0cbd4cfef65734d855bb5ed10ed05c70b603b8589bec3", "driverContractVersion": 1, "driverKind": "paperclip-deterministic", "driverVersion": "1.0.0" diff --git a/packages/paperclip-runner/protocol/manifest.json b/packages/paperclip-runner/protocol/manifest.json index f2162c99cad..68baeb1c950 100644 --- a/packages/paperclip-runner/protocol/manifest.json +++ b/packages/paperclip-runner/protocol/manifest.json @@ -160,7 +160,7 @@ }, { "path": "fixtures/evals/native-execution-seeded.json", - "sha256": "a5fde52c41fe77a8872ec1d1613ba9bb426b9231835e1464d943930ab799d933", + "sha256": "b4e488f922dd328a3967b2560951c2377f4c1411fb2c4c9eef7a44195a8f7514", "expectation": "accept", "compatibilityCase": "canonical" }, diff --git a/packages/paperclip-runner/spec/capability/protocol-coverage.json b/packages/paperclip-runner/spec/capability/protocol-coverage.json index 6f3a4f0b6b4..09f5e21f348 100644 --- a/packages/paperclip-runner/spec/capability/protocol-coverage.json +++ b/packages/paperclip-runner/spec/capability/protocol-coverage.json @@ -7,7 +7,7 @@ "src/scenarios/scenario-plan.ts" ], "counts": { - "actions": 53, + "actions": 54, "legacyRequirements": 106 }, "actions": [ @@ -126,6 +126,26 @@ "src/scenarios/scenario-explorer.test.ts::renders every scenario with exposure, control plane, authorization, diff, and parity" ] }, + { + "id": "update_skill", + "ownership": "optional_agent_tool", + "surfaces": [ + "scenario", + "live" + ], + "legacyAliases": [], + "contractCase": "protocol-action:update_skill", + "contractOwner": "src/catalog/protocol-action-contracts.test.ts::update_skill has a schema-valid canonical example and every declared projection", + "legacyBehavioralCases": [], + "deterministicCases": [ + "protocol-action:update_skill" + ], + "legacyRequirementCases": [], + "deterministicOwners": [ + "src/catalog/protocol-action-contracts.test.ts::update_skill has a schema-valid canonical example and every declared projection", + "src/scenarios/scenario-explorer.test.ts::renders every scenario with exposure, control plane, authorization, diff, and parity" + ] + }, { "id": "create_project", "ownership": "optional_agent_tool", diff --git a/packages/paperclip-runner/spec/operation-groups/source.json b/packages/paperclip-runner/spec/operation-groups/source.json index f2f46af8229..f4c58f830d6 100644 --- a/packages/paperclip-runner/spec/operation-groups/source.json +++ b/packages/paperclip-runner/spec/operation-groups/source.json @@ -87,8 +87,8 @@ }, { "id": "company_skills", - "description": "Create or inspect company skills and synchronize the current agent's skills.", - "operationIds": ["create_skill", "list_company_skills", "sync_company_skills"] + "description": "Create, update, or inspect company skills and synchronize the current agent's skills.", + "operationIds": ["create_skill", "update_skill", "list_company_skills", "sync_company_skills"] }, { "id": "secrets", @@ -264,7 +264,7 @@ "legacyGroup": 13, "name": "Reference files", "owner": "always instruction tools + optional domain tools + test-only escape hatch", - "operationIds": ["list_cases", "upsert_case", "list_routines", "manage_routine", "create_skill", "list_company_skills", "sync_company_skills", "list_secret_metadata", "read_secret_value", "export_company", "administer_company", "generic_api_request", "search_api", "call_api", "create_project", "read_agent_instructions", "update_agent_instructions", "get_agent_instruction_history", "restore_agent_instructions"], + "operationIds": ["list_cases", "upsert_case", "list_routines", "manage_routine", "create_skill", "update_skill", "list_company_skills", "sync_company_skills", "list_secret_metadata", "read_secret_value", "export_company", "administer_company", "generic_api_request", "search_api", "call_api", "create_project", "read_agent_instructions", "update_agent_instructions", "get_agent_instruction_history", "restore_agent_instructions"], "controlPlaneOperationIds": ["append_audit_record"], "realSurface": "agent instruction revisions, project, case, routine, company-skill, secret, portability, and administration services", "mockStateDomains": ["company", "cases", "routines", "skills", "secrets", "audit", "fault"], diff --git a/packages/paperclip-runner/spec/paperclip-agent-operation-groups.md b/packages/paperclip-runner/spec/paperclip-agent-operation-groups.md index 5a60bb0684a..98366d340c8 100644 --- a/packages/paperclip-runner/spec/paperclip-agent-operation-groups.md +++ b/packages/paperclip-runner/spec/paperclip-agent-operation-groups.md @@ -6,7 +6,7 @@ Status: canonical explanatory contract for the Paperclip runner V1 surface. This document keeps three independent meanings of **group** separate. PRP families describe wire evidence and controller commands; capability placement decides who owns an operation; behavioral eval groups organize the 106 scenario corpus. None of the three axes can be used as a substitute for another. -The generated totals are **105 PRP events in 31 event families**, **18 controller commands in 7 command families**, **10 control-plane operations**, **53 reconciled semantic operations** (18 always, 35 optional), and **106 scenarios in 16 behavior groups**. +The generated totals are **105 PRP events in 31 event families**, **18 controller commands in 7 command families**, **10 control-plane operations**, **54 reconciled semantic operations** (18 always, 36 optional), and **106 scenarios in 16 behavior groups**. ## Axis 1: PRP v1 event and command families @@ -87,7 +87,7 @@ Placement has exactly three outcomes: `answer_status_question`, `block_task`, `finish_task`, `get_agent_instruction_history`, `get_task_context`, `get_task_history`, `inspect_operation_result`, `list_document_revisions`, `list_documents`, `read_agent_instructions`, `read_document`, `register_deliverable`, `report_progress`, `request_human_input`, `request_review`, `restore_agent_instructions`, `update_agent_instructions`, `write_document`. -### Optional operations (35) and grant groups (13) +### Optional operations (36) and grant groups (13) Grant groups are documentation/exposure bundles, not additional authority. The operation descriptor's exact `requiredClaims` remains decisive. @@ -101,7 +101,7 @@ Grant groups are documentation/exposure bundles, not additional authority. The o | `workspace_runtime` | `get_workspace_runtime`
`control_workspace_service` | `workspace:control`
`workspace:read` | Inspect and control the active issue workspace runtime. | | `wake_scheduling` | `schedule_wake` | `control_plane:wakes` | Schedule a bounded continuation wake when the current task owns the future check. | | `routines` | `list_routines`
`manage_routine` | `routines:read`
`routines:write` | Inspect or manage company routines. | -| `company_skills` | `create_skill`
`list_company_skills`
`sync_company_skills` | `company_skills:read`
`company_skills:write` | Create or inspect company skills and synchronize the current agent's skills. | +| `company_skills` | `create_skill`
`update_skill`
`list_company_skills`
`sync_company_skills` | `company_skills:read`
`company_skills:write` | Create, update, or inspect company skills and synchronize the current agent's skills. | | `secrets` | `list_secret_metadata`
`read_secret_value` | `secrets:metadata:read`
`secrets:values:read` | Inspect secret metadata or use a brokered secret value without exposing plaintext evidence. | | `portability_admin` | `export_company`
`administer_company` | `company:admin`
`portability:export` | Export portable company state; broad administration remains deferred until split into governed operations. | | `test_escape_hatch` | `generic_api_request` | `test:generic_api_request` | Controlled skill-test transport only; never product coverage. | @@ -164,6 +164,7 @@ Grant groups are documentation/exposure bundles, not additional authority. The o | `set_task_title` | `optional_agent_tool` | none | `standard`
`ask`
`planning`
`skill_test` | `task_write` | `required` | no | inline/no mapping | `live`
`live_codex` | `setIssueTitle`
Run-bound title update and transactional activity record.
catalog PRP status: `bound` | | `sync_company_skills` | `optional_agent_tool` | `company_skills:write` | `standard`
`skill_test` | `admin` | `required` | no | `mock_extension:company_skills.sync` | `scenario`
`scenario_mock` | `unbound`
company admin/portability item event plus audit record
catalog PRP status: `audit_pending` | | `update_agent_instructions` | `always_agent_tool` | none | `standard`
`planning` | `company_write` | `recommended` | no | inline/no mapping | `live`
`live_codex` | `agentInstructionRevisionService`
tool-result item event plus canonical revision receipt for writes; no scenario mock coverage
catalog PRP status: `bound` | +| `update_skill` | `optional_agent_tool` | none | `standard`
`skill_test` | `company_write` | `required` | no | `semantic_command:update_skill` | `scenario` + `live`
`live_codex` | `PaperclipRunnerToolAuthority`
Authenticated skill file API with version guard, durable receipt, and attributed activity.
catalog PRP status: `bound` | | `upsert_case` | `optional_agent_tool` | `cases:write` | `standard`
`skill_test` | `company_write` | `required` | no | `mock_extension:cases.upsert` | `scenario`
`scenario_mock` | `unbound`
semantic-operation item event plus company-entity state diff and audit record
catalog PRP status: `audit_pending` | | `write_document` | `always_agent_tool` | none | `standard`
`planning`
`skill_test` | `task_write` | `required` | no | `semantic_command:write_document` | `scenario` + `live`
`live_codex` | `unbound`
semantic-operation item event plus active-task state diff, work-assessment, and issue-status-decision events
catalog PRP status: `audit_pending` | @@ -187,7 +188,7 @@ Behavior groups describe expected outcomes and trajectories. They do not grant t | [`ap` — Approvals](#behavior-group-ap-approvals) | optional governance tools + governed approver | `list_approvals`
`get_approval`
`get_approval_context`
`request_approval`
`decide_approval`
`comment_on_approval` | `route_wake`
`append_audit_record` | company approval, decision, issue-link, comment, and governed-action services | `company`
`task`
`approvals`
`actor`
`wake`
`audit`
`idempotency` | 6 | governed semantic receipts, audit references, and attention/continuation linkage | Production binding and additive governed-action receipts are unbound; board-only authority stays outside grants. | | [`ar` — Artifacts](#behavior-group-ar-artifacts) | always tools + artifact/work-product services | `register_deliverable` | `append_audit_record` | attachment upload and issue work-product routes | `task`
`artifacts`
`workProducts`
`workspace`
`audit`
`idempotency` | 4 | artifact/work-product reference and durable inspectability receipt; never binary bytes | Production upload/register composite and additive durable-reference receipt are unbound. | | [`er` — Errors and critical rules](#behavior-group-er-errors-and-critical-rules) | runner/control plane + optional workspace/wake tools | `get_workspace_runtime`
`control_workspace_service`
`schedule_wake`
`inspect_operation_result` | `release_task`
`enforce_budget`
`persist_run`
`replay_run`
`reconcile_run` | workspace runtime, monitor/recovery, budget, run persistence/replay, release, and terminal services | `workspace`
`budget`
`run`
`wake`
`audit`
`idempotency`
`fault` | 9 | runtime/workspace/attention/run lifecycle, typed denials, replay facts, and terminal causality | Budget stop reasons and semantic denial/conflict receipts require additive v1 envelopes; inspect_operation_result remains scenario-only. | -| [`rf` — Reference files](#behavior-group-rf-reference-files) | always instruction tools + optional domain tools + test-only escape hatch | `list_cases`
`upsert_case`
`list_routines`
`manage_routine`
`create_skill`
`list_company_skills`
`sync_company_skills`
`list_secret_metadata`
`read_secret_value`
`export_company`
`administer_company`
`generic_api_request`
`search_api`
`call_api`
`create_project`
`read_agent_instructions`
`update_agent_instructions`
`get_agent_instruction_history`
`restore_agent_instructions` | `append_audit_record` | agent instruction revisions, project, case, routine, company-skill, secret, portability, and administration services | `company`
`cases`
`routines`
`skills`
`secrets`
`audit`
`fault` | 22 | bounded domain projections, redacted broker receipts, company diffs, and audit references | Instruction read/update/history/restore require the live canonical revision service, current responsible-user permissions, and pinned base revisions for writes. Their service, authority, and product E2E tests are separate from the legacy mock scenarios. Project and skill creation and API tools require the live server authority; generic_api_request is test-only and the remaining domain operations are scenario-only. Broad administer_company is deferred and cannot claim product coverage. Production API and paired dedicated-tool regressions are recorded separately in paperclip-evals/evals/runner-api-tools; the legacy scenario count is not evidence of that coverage. | +| [`rf` — Reference files](#behavior-group-rf-reference-files) | always instruction tools + optional domain tools + test-only escape hatch | `list_cases`
`upsert_case`
`list_routines`
`manage_routine`
`create_skill`
`update_skill`
`list_company_skills`
`sync_company_skills`
`list_secret_metadata`
`read_secret_value`
`export_company`
`administer_company`
`generic_api_request`
`search_api`
`call_api`
`create_project`
`read_agent_instructions`
`update_agent_instructions`
`get_agent_instruction_history`
`restore_agent_instructions` | `append_audit_record` | agent instruction revisions, project, case, routine, company-skill, secret, portability, and administration services | `company`
`cases`
`routines`
`skills`
`secrets`
`audit`
`fault` | 22 | bounded domain projections, redacted broker receipts, company diffs, and audit references | Instruction read/update/history/restore require the live canonical revision service, current responsible-user permissions, and pinned base revisions for writes. Their service, authority, and product E2E tests are separate from the legacy mock scenarios. Project and skill creation and API tools require the live server authority; generic_api_request is test-only and the remaining domain operations are scenario-only. Broad administer_company is deferred and cannot claim product coverage. Production API and paired dedicated-tool regressions are recorded separately in paperclip-evals/evals/runner-api-tools; the legacy scenario count is not evidence of that coverage. | | [`mh` — Multi-hop](#behavior-group-mh-multi-hop) | composed semantic operations + control-plane continuation | `create_task`
`set_dependencies`
`request_human_input`
`request_approval`
`register_deliverable` | `route_wake`
`reconcile_run` | delegation, dependency, interaction, approval, artifact, and terminal orchestration services | `task`
`blockers`
`interactions`
`approvals`
`artifacts`
`wake`
`run`
`audit` | 4 | correlated operation receipts, state diffs, attention hops, work assessment, status decision, and terminal outcome | No generic transaction tool is allowed; shared mock/real conformance must prove each composed effect. | | [`rs` — Restraint and no-call](#behavior-group-rs-restraint-and-no-call) | policy/exposure layer | `answer_status_question`
`read_secret_value`
`generic_api_request` | `enforce_budget` | task-mode, secret-broker, test-scope, pause, and budget policy checks | `actor`
`task`
`budget`
`secrets`
`audit`
`fault` | 3 | absence of forbidden effects plus typed policy denial/redaction receipts when a call is attempted | Typed redaction/authorization receipts need additive v1 evidence; generic_api_request is never a product fallback. | | [`wk` — Wake situations](#behavior-group-wk-wake-situations) | control plane + always context/history tools | `get_task_context`
`get_task_history`
`schedule_wake` | `select_work`
`route_wake` | wakeup requests, heartbeat context, comment/interaction/approval/blocker wake routing, and scheduled wake services | `wake`
`task`
`comments`
`interactions`
`approvals`
`blockers`
`run` | 8 | attention request routing/resolution plus resumed session/run causality | Production scheduling binding is unbound; control-plane routing remains non-callable. | @@ -463,9 +464,9 @@ Current responsibility-based paths are normative. Numbered `phase-*` or mileston ### Catalog split and deliberate replacement -- Scenario/eval catalog: **39** operations. -- Live dispatcher catalog: **41** operations. -- Shared: **27**; union/canonical authority: **53**. +- Scenario/eval catalog: **40** operations. +- Live dispatcher catalog: **42** operations. +- Shared: **28**; union/canonical authority: **54**. - Scenario-only: `administer_company`, `export_company`, `inspect_operation_result`, `list_cases`, `list_company_skills`, `list_goals`, `list_routines`, `list_secret_metadata`, `manage_routine`, `read_secret_value`, `sync_company_skills`, `upsert_case`. - Live-only: `call_api`, `create_project`, `get_agent`, `get_agent_instruction_history`, `get_approval`, `get_approval_context`, `hire_agent`, `list_project_repositories`, `read_agent_instructions`, `restore_agent_instructions`, `schedule_wake`, `search_api`, `set_task_title`, `update_agent_instructions`. - The generated provider contract contains exactly the live catalog; the canonical union remains the migration authority until all scenario-only operations are either implemented, deferred, or removed by an explicit reconciliation decision. diff --git a/packages/paperclip-runner/src/catalog/catalog-docs.test.ts b/packages/paperclip-runner/src/catalog/catalog-docs.test.ts index 055503d1b7a..f418b1c48b6 100644 --- a/packages/paperclip-runner/src/catalog/catalog-docs.test.ts +++ b/packages/paperclip-runner/src/catalog/catalog-docs.test.ts @@ -43,7 +43,7 @@ describe("capability-semantic-tools.md stays in sync with the catalog", () => { expect(Number(alwaysHeader)).toBe(always.length); expect(Number(optionalHeader)).toBe(optional.length); // Guard against the historical 36-vs-37 drift the reconciliation plan flagged. - expect(SCENARIO_CATALOG.length).toBe(39); + expect(SCENARIO_CATALOG.length).toBe(40); }); it("lists exactly the catalog's operation ids in the counts region", () => { diff --git a/packages/paperclip-runner/src/catalog/reconciliation.test.ts b/packages/paperclip-runner/src/catalog/reconciliation.test.ts index 5c6cd961393..650cd4a9a02 100644 --- a/packages/paperclip-runner/src/catalog/reconciliation.test.ts +++ b/packages/paperclip-runner/src/catalog/reconciliation.test.ts @@ -19,10 +19,10 @@ const packageRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..", "..") describe("canonical semantic-catalog reconciliation authority", () => { it("pins the reconciled op-set relationship between the two catalogs", () => { const summary = capabilityCatalogReconciliation(); - expect(summary.scenarioCount).toBe(39); - expect(summary.liveCount).toBe(41); - expect(summary.sharedCount).toBe(27); - expect(summary.unionCount).toBe(53); + expect(summary.scenarioCount).toBe(40); + expect(summary.liveCount).toBe(42); + expect(summary.sharedCount).toBe(28); + expect(summary.unionCount).toBe(54); // Any operation added to or removed from either catalog without a // reconciliation decision changes these exact sets and fails the gate. expect(summary.liveOnly).toEqual([ @@ -58,7 +58,7 @@ describe("canonical semantic-catalog reconciliation authority", () => { }); it("is the single source both catalogs derive their operation set from", () => { - expect(CAPABILITY_CANONICAL_OPERATIONS).toHaveLength(53); + expect(CAPABILITY_CANONICAL_OPERATIONS).toHaveLength(54); const canonicalIds = new Set(CAPABILITY_CANONICAL_OPERATIONS.map((operation) => operation.operationId)); // Neither catalog may contain an operation absent from the canonical source. for (const tool of SCENARIO_CATALOG) expect(canonicalIds.has(tool.operationId)).toBe(true); @@ -92,7 +92,7 @@ describe("canonical semantic-catalog reconciliation authority", () => { }); it("names placement, claims, task modes, side-effect class, idempotency, redaction, mock mapping, real binding status, and PRP evidence for every operation", () => { - expect(CAPABILITY_CANONICAL_CATALOG).toHaveLength(53); + expect(CAPABILITY_CANONICAL_CATALOG).toHaveLength(54); for (const operation of CAPABILITY_CANONICAL_CATALOG) { expect(operation.placement).toMatch(/^(always|optional)_agent_tool$/); expect(Array.isArray(operation.requiredClaims)).toBe(true); @@ -118,7 +118,7 @@ describe("canonical semantic-catalog reconciliation authority", () => { it("classifies real binding status so generic_api_request is never product coverage", () => { const summary = capabilityCatalogReconciliation(); expect(summary.byRealBindingStatus).toEqual({ - live_codex: 40, + live_codex: 41, scenario_mock: 12, test_only: 1, }); diff --git a/packages/paperclip-runner/src/catalog/semantic-action-types.ts b/packages/paperclip-runner/src/catalog/semantic-action-types.ts index 0bc0d1d022f..0dd1f9c38ec 100644 --- a/packages/paperclip-runner/src/catalog/semantic-action-types.ts +++ b/packages/paperclip-runner/src/catalog/semantic-action-types.ts @@ -27,6 +27,7 @@ export type PaperclipSemanticActionId = | "reassign_task" | "set_dependencies" | "create_skill" + | "update_skill" | "create_project" | "list_project_repositories" | "list_projects" diff --git a/packages/paperclip-runner/src/eval/workflow-evals.test.ts b/packages/paperclip-runner/src/eval/workflow-evals.test.ts index 916a413f1a6..284eb92476b 100644 --- a/packages/paperclip-runner/src/eval/workflow-evals.test.ts +++ b/packages/paperclip-runner/src/eval/workflow-evals.test.ts @@ -466,13 +466,13 @@ describe("workflow reports and stress traceability", () => { candidateFailures: 36, }); expect(report.coverage).toMatchObject({ - canonicalOperations: 53, + canonicalOperations: 54, capabilityCases: 106, workflows: 12, stressFindings: 44, stressExclusions: 1, }); - expect(report.coverage.operations).toHaveLength(53); + expect(report.coverage.operations).toHaveLength(54); expect(report.coverage.composedWorkflows).toHaveLength(12); expect( report.coverage.operations.find( @@ -480,7 +480,7 @@ describe("workflow reports and stress traceability", () => { )?.workflowIds.length, ).toBeGreaterThan(0); expect(renderRunnerWorkflowMarkdown(report)).toContain( - "53 operations · 106 capability cases · 12 workflows", + "54 operations · 106 capability cases · 12 workflows", ); expect(renderRunnerWorkflowJUnit(report)).toContain( 'tests="36" failures="36" skipped="0"', diff --git a/packages/paperclip-runner/src/mock-core/capability-control-plane-types.ts b/packages/paperclip-runner/src/mock-core/capability-control-plane-types.ts index 7b646b8f1fc..d61fdcaf596 100644 --- a/packages/paperclip-runner/src/mock-core/capability-control-plane-types.ts +++ b/packages/paperclip-runner/src/mock-core/capability-control-plane-types.ts @@ -359,6 +359,7 @@ interface CapabilityBaseCommand { export type CapabilitySemanticCommand = | (CapabilityBaseCommand & { kind: "create_skill"; name: string; slug?: string; description: string; markdown: string }) + | (CapabilityBaseCommand & { kind: "update_skill"; skillId: string; markdown: string; expectedVersionId: string }) | (CapabilityBaseCommand & { kind: "report_progress"; body: string }) | (CapabilityBaseCommand & { kind: "write_document"; @@ -479,6 +480,7 @@ export type CapabilityCommandOutcome = */ export const CAPABILITY_COMMAND_REQUIRED_CLAIMS = { create_skill: [], + update_skill: [], report_progress: [], write_document: [], request_human_input: [], diff --git a/packages/paperclip-runner/src/mock-core/capability-mock-control-plane-adapter.ts b/packages/paperclip-runner/src/mock-core/capability-mock-control-plane-adapter.ts index bb8de32b494..e6dcdc75f0f 100644 --- a/packages/paperclip-runner/src/mock-core/capability-mock-control-plane-adapter.ts +++ b/packages/paperclip-runner/src/mock-core/capability-mock-control-plane-adapter.ts @@ -734,6 +734,21 @@ export class CapabilityMockControlPlaneAdapter implements CapabilityMockControlP entityRefs.push(`skill:${skill.id}`); break; } + case "update_skill": { + const skill = this.#state.skills?.find(candidate => candidate.id === command.skillId && candidate.companyId === run.companyId); + if (!skill) throw new CapabilityMockControlPlaneError("fixture_state_invalid", "Skill not found"); + if (skill.versionId !== command.expectedVersionId) throw new CapabilityMockControlPlaneError("fixture_state_invalid", "Skill version changed"); + const document = parseFrontmatterMarkdown(command.markdown); + if (!document.hasFrontmatter || !validateSkillFrontmatter(document.frontmatter) || !document.body.trim()) { + throw new CapabilityMockControlPlaneError("invalid_skill_document", "Provide a complete SKILL.md with valid frontmatter and a nonempty body"); + } + skill.name = String(document.frontmatter.name); + skill.description = String(document.frontmatter.description); + skill.markdown = command.markdown; + skill.versionId = this.#id("skill-version"); + entityRefs.push(`skill:${skill.id}`); + break; + } case "write_document": { requireText(command.key, "document key"); requireText(command.title, "document title"); diff --git a/packages/paperclip-runner/src/protocol-actions/index.ts b/packages/paperclip-runner/src/protocol-actions/index.ts index 3cad6cc1bc5..e4ebbb7bb82 100644 --- a/packages/paperclip-runner/src/protocol-actions/index.ts +++ b/packages/paperclip-runner/src/protocol-actions/index.ts @@ -5,6 +5,7 @@ import { getAgentInstructionHistoryAction } from "./get-agent-instruction-histor import { restoreAgentInstructionsAction } from "./restore-agent-instructions.js"; import { reassignTaskAction } from "./reassign-task.js"; import { createSkillAction } from "./create-skill.js"; +import { updateSkillAction } from "./update-skill.js"; import { createProjectAction } from "./create-project.js"; import { listProjectRepositoriesAction } from "./list-project-repositories.js"; import { searchApiAction } from "./search-api.js"; @@ -61,6 +62,7 @@ export const PAPERCLIP_PROTOCOL_ACTIONS = deepFreezeProtocolAction([ restoreAgentInstructionsAction, createSkillAction, + updateSkillAction, createProjectAction, listProjectRepositoriesAction, searchApiAction, diff --git a/packages/paperclip-runner/src/protocol-actions/update-skill.ts b/packages/paperclip-runner/src/protocol-actions/update-skill.ts new file mode 100644 index 00000000000..52f189d1b3e --- /dev/null +++ b/packages/paperclip-runner/src/protocol-actions/update-skill.ts @@ -0,0 +1,54 @@ +/** Canonical definition and documentation for `update_skill`. */ +export const updateSkillAction = { + id: "update_skill", + canonical: { + operationId: "update_skill", surfaces: ["scenario", "live"], placement: "optional_agent_tool", + optionalGroup: "company_skills", requiredClaims: [], taskModes: ["standard", "skill_test"], + sideEffectClass: "company_write", idempotency: "required", disabledByDefault: false, + realBindingStatus: "live_codex", realServiceBinding: "PaperclipRunnerToolAuthority", + prpEvidence: "Authenticated skill file API with version guard, durable receipt, and attributed activity.", + prpBindingStatus: "bound", legacyAliases: [], + }, + documentation: { + title: "Update skill", + description: "Replace the primary SKILL.md of an existing company skill. Read the current skill and version first. Supply the complete file, expectedVersionId, and an idempotencyKey; reuse the key on retries. A version conflict requires a fresh read. This uses the existing file API and company edit policy.", + note: null, + }, + examples: { + call: { operationId: "update_skill", input: { skillId: "00000000-0000-4000-8000-000000000001", expectedVersionId: "00000000-0000-4000-8000-000000000002", idempotencyKey: "edit-1", markdown: "---\nname: review\ndescription: Review changes.\n---\n\n# Review\nInspect the changes.\n" } }, + scenarioCall: { operationId: "update_skill", idempotencyKey: "edit-1", input: { skillId: "skill-example", expectedVersionId: "version-example", markdown: "---\nname: review\ndescription: Review changes.\n---\n\n# Review\nInspect the changes.\n" } }, + success: { ok: true, operationId: "update_skill", result: { skillId: "skill-example", path: "SKILL.md", versionId: "version-next", studioPath: "/skills/studio/skill-example" } }, + }, + live: { + order: 47, + descriptor: { + schema: "paperclip.semantic-tool.v1", operationId: "update_skill", version: 1, + title: "Update skill", description: "Replace an existing company skill's complete SKILL.md using the current version as a guard. Reuse idempotencyKey on lost-response retries; read again after a version conflict.", + exposure: "optional", requiredClaims: [], allowedModes: ["standard", "skill_test"], + inputSchema: { type: "object", properties: { + skillId: { type: "string", minLength: 1 }, + markdown: { type: "string", minLength: 1, maxLength: 200000 }, + expectedVersionId: { type: "string", minLength: 1 }, + idempotencyKey: { type: "string", minLength: 1, maxLength: 240 }, + }, required: ["skillId", "markdown", "expectedVersionId", "idempotencyKey"], additionalProperties: false }, + outputSchema: { type: "object", additionalProperties: true }, + }, + }, + scenario: { + order: 47, + successExample: { schema: "paperclip.capability.tool-result.v1", ok: true, operationId: "update_skill", operationResultId: "example-result", value: { commandId: "example", disposition: "applied", stateRevision: 1, entityRefs: ["skill:skill-example"], scheduledWakeIds: [] }, commandResult: null, authorization: {} }, + descriptor: { + operationId: "update_skill", version: 1, title: "Update skill", + description: "Replace an existing company skill's complete SKILL.md with an expected version guard and a retry key.", + inputSchema: { type: "object", properties: { + skillId: { type: "string", minLength: 1 }, + markdown: { type: "string", minLength: 1, maxLength: 200000 }, + expectedVersionId: { type: "string", minLength: 1 }, + }, required: ["skillId", "markdown", "expectedVersionId"], additionalProperties: false }, + outputSchema: { type: "object", additionalProperties: true }, + disposition: "optional_agent_tool", optionalGroup: "company_skills", requiredClaims: [], + taskModes: ["standard", "skill_test"], sideEffectClass: "company_write", idempotency: "required", redaction: [], + mockCommandMapping: { kind: "semantic_command", commandKind: "update_skill" }, + }, + }, +} as const; diff --git a/packages/paperclip-runner/src/semantic-tools/discovery.ts b/packages/paperclip-runner/src/semantic-tools/discovery.ts index 4b3631c8d63..fd5b075dbc6 100644 --- a/packages/paperclip-runner/src/semantic-tools/discovery.ts +++ b/packages/paperclip-runner/src/semantic-tools/discovery.ts @@ -33,7 +33,7 @@ const NAMESPACE: Readonly> = Objec update_agent_instructions: "agent_instructions", get_agent_instruction_history: "agent_instructions", restore_agent_instructions: "agent_instructions", - create_skill: "skills", create_task: "delegation", create_project: "projects", list_project_repositories: "projects", list_projects: "projects", + create_skill: "skills", update_skill: "skills", create_task: "delegation", create_project: "projects", list_project_repositories: "projects", list_projects: "projects", set_dependencies: "delegation", reassign_task: "delegation", list_approvals: "governance", get_approval: "governance", get_approval_context: "governance", request_approval: "governance", decide_approval: "governance", comment_on_approval: "governance", diff --git a/packages/paperclip-runner/src/semantic-tools/dispatcher.ts b/packages/paperclip-runner/src/semantic-tools/dispatcher.ts index 4436d04e24c..ad58bb77b0a 100644 --- a/packages/paperclip-runner/src/semantic-tools/dispatcher.ts +++ b/packages/paperclip-runner/src/semantic-tools/dispatcher.ts @@ -331,6 +331,10 @@ export class CapabilitySemanticDispatcher { slug: typeof input.slug === "string" ? input.slug : undefined, description: requiredString(input.description), markdown: requiredString(input.markdown) }; break; + case "update_skill": + command = { kind: "update_skill", taskId, skillId: requiredString(input.skillId), + expectedVersionId: requiredString(input.expectedVersionId), markdown: requiredString(input.markdown) }; + break; case "write_document": command = { kind: "write_document", @@ -420,6 +424,13 @@ export class CapabilitySemanticDispatcher { versionId: skill.versionId, studioPath: `/skills/studio/${skill.id}`, }); } + if (operationId === "update_skill" && outcome.ok) { + const id = outcome.result.entityRefs.find(ref => ref.startsWith("skill:"))?.slice(6); + const skill = this.port.snapshot().skills?.find(candidate => candidate.id === id); + if (skill) return readSuccess(outcome.result.stateRevision, { + skillId: skill.id, path: "SKILL.md", versionId: skill.versionId, studioPath: `/skills/studio/${skill.id}`, + }); + } return commandOutcome(outcome); } diff --git a/packages/paperclip-runner/src/semantic-tools/paperclip-discovery.ts b/packages/paperclip-runner/src/semantic-tools/paperclip-discovery.ts index 5e6a68a9195..1388697928e 100644 --- a/packages/paperclip-runner/src/semantic-tools/paperclip-discovery.ts +++ b/packages/paperclip-runner/src/semantic-tools/paperclip-discovery.ts @@ -38,7 +38,7 @@ const NAMESPACE: Readonly> = get_workspace_runtime: "workspace", control_workspace_service: "workspace", set_dependencies: "delegation", reassign_task: "delegation", - create_skill: "skills", create_task: "delegation", create_project: "projects", list_project_repositories: "projects", list_projects: "projects", + create_skill: "skills", update_skill: "skills", create_task: "delegation", create_project: "projects", list_project_repositories: "projects", list_projects: "projects", request_approval: "governance", decide_approval: "governance", comment_on_approval: "governance", diff --git a/packages/paperclip-runner/src/semantic-tools/semantic-tools.test.ts b/packages/paperclip-runner/src/semantic-tools/semantic-tools.test.ts index ed4097a8760..f187a3d6e02 100644 --- a/packages/paperclip-runner/src/semantic-tools/semantic-tools.test.ts +++ b/packages/paperclip-runner/src/semantic-tools/semantic-tools.test.ts @@ -43,6 +43,25 @@ async function running( } describe("Capability semantic catalog and authorization", () => { + it("updates an existing skill once, rejects a stale version, and denies an ungranted capability", async () => { + const adapter = await running(); + const dispatcher = new CapabilitySemanticDispatcher(adapter); + const markdown = "---\nname: release-review\ndescription: Review release notes.\n---\n# Review\nCheck each note."; + const created = await dispatcher.dispatch({ runId: OPEN.identity.runId, callId: "create", operationId: "create_skill", + input: { name: "release-review", description: "Review release notes.", markdown, idempotencyKey: "create" } }); + expect(created.ok).toBe(true); + if (!created.ok) return; + const args = { skillId: (created.result as { id: string }).id, expectedVersionId: (created.result as { versionId: string }).versionId, + markdown: markdown + " Inspect tests.", idempotencyKey: "update-once" }; + const call = { runId: OPEN.identity.runId, callId: "update", operationId: "update_skill" as const, input: args }; + const first = await dispatcher.dispatch(call); + expect(first).toMatchObject({ ok: true, result: { skillId: args.skillId, path: "SKILL.md", versionId: expect.any(String) } }); + expect(await dispatcher.dispatch({ ...call, callId: "retry" })).toMatchObject({ ok: true, result: first.ok ? first.result : {} }); + expect(await dispatcher.dispatch({ ...call, callId: "stale", input: { ...args, idempotencyKey: "next" } })).toMatchObject({ ok: false }); + const denied = new CapabilitySemanticDispatcher(adapter, { scenario: { id: "no-update", claims: [], denyOperations: ["update_skill"] } }); + expect(await denied.dispatch({ ...call, callId: "denied", input: { ...args, idempotencyKey: "denied" } })).toMatchObject({ ok: false, denial: { code: "scenario_denied" } }); + }); + it("creates a durable skill once and returns its reference on retry", async () => { const adapter = await running(); const dispatcher = new CapabilitySemanticDispatcher(adapter); @@ -94,7 +113,7 @@ describe("Capability semantic catalog and authorization", () => { it("publishes a stable narrow catalog without credentials or control-plane-owned tools", () => { const names = CAPABILITY_SEMANTIC_TOOL_CATALOG.map((tool) => tool.operationId); expect(new Set(names).size).toBe(names.length); - expect(names).toHaveLength(41); + expect(names).toHaveLength(42); expect(names).toContain("get_task_context"); expect(names).toContain("finish_task"); expect(names).not.toContain("checkout_task"); diff --git a/packages/paperclip-runner/src/semantic-tools/types.ts b/packages/paperclip-runner/src/semantic-tools/types.ts index 2e95fe1cce8..343410f1501 100644 --- a/packages/paperclip-runner/src/semantic-tools/types.ts +++ b/packages/paperclip-runner/src/semantic-tools/types.ts @@ -47,6 +47,7 @@ export type CapabilitySemanticOperationId = | "reassign_task" | "set_dependencies" | "create_skill" + | "update_skill" | "create_project" | "list_project_repositories" | "list_projects" diff --git a/packages/paperclip-runner/src/tools/capability-semantic-tool-runtime.ts b/packages/paperclip-runner/src/tools/capability-semantic-tool-runtime.ts index 5cb627c1c5d..39b92e13b33 100644 --- a/packages/paperclip-runner/src/tools/capability-semantic-tool-runtime.ts +++ b/packages/paperclip-runner/src/tools/capability-semantic-tool-runtime.ts @@ -1543,6 +1543,9 @@ function commandForOperation( return { kind: "create_skill", name: requireString(input.name), slug: typeof input.slug === "string" ? input.slug : undefined, description: requireString(input.description), markdown: requireString(input.markdown) }; + case "update_skill": + return { kind: "update_skill", skillId: requireString(input.skillId), + expectedVersionId: requireString(input.expectedVersionId), markdown: requireString(input.markdown) }; case "write_document": return { kind: "write_document", diff --git a/packages/shared/src/validators/company-skill.ts b/packages/shared/src/validators/company-skill.ts index a1742825701..b75d5bd3c63 100644 --- a/packages/shared/src/validators/company-skill.ts +++ b/packages/shared/src/validators/company-skill.ts @@ -395,6 +395,8 @@ export const companySkillFileUpdateSchema = z.object({ content: z.string(), encoding: z.enum(["utf8", "base64"]).optional(), executable: z.boolean().optional(), + expectedVersionId: z.string().guid().nullable().optional(), + idempotencyKey: z.string().min(1).max(240).optional(), }); export const companySkillFileDeleteSchema = z.object({ diff --git a/server/src/routes/company-skills.ts b/server/src/routes/company-skills.ts index 7654829f49f..daa61597b8a 100644 --- a/server/src/routes/company-skills.ts +++ b/server/src/routes/company-skills.ts @@ -7,7 +7,7 @@ import { once } from "node:events"; import type { SkillSourceDiscoveryEvent } from "@paperclipai/shared"; import { createHash } from "node:crypto"; import { and, eq, sql } from "drizzle-orm"; -import { activityLog } from "@paperclipai/db"; +import { activityLog, companySkills } from "@paperclipai/db"; import { persistActivity, publishActivity } from "../services/activity-log.js"; import { projectToolContext } from "../services/project-tool-context.js"; import { Router, type Request } from "express"; @@ -1219,13 +1219,66 @@ export function companySkillRoutes(db: Db) { const companyId = req.params.companyId as string; const skillId = req.params.skillId as string; await assertCanMutateCompanySkills(req, companyId, "skills.edit", () => skillPolicyResource({ companyId, skillId })); + const { idempotencyKey, expectedVersionId, ...input } = req.body; + if (idempotencyKey) { + const actor = getActorInfo(req); + const runContext = req.actor.type === "agent" && req.actor.source === "agent_jwt" && req.actor.runId + ? await projectToolContext(db, req.actor, true, "Skill") : null; + const receiptKey = `skill-file:${companyId}:${actor.actorId}:${runContext?.issue.id ?? "board"}:${skillId}:${idempotencyKey}`; + const fingerprint = createHash("sha256").update(JSON.stringify({ ...input, expectedVersionId })).digest("hex"); + const rollbackState: { current: { restore: () => Promise; versionId: string | null } | null } = { current: null }; + const outcome = await db.transaction(async tx => { + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${receiptKey}, 0))`); + if (runContext) await projectToolContext(tx as unknown as Db, req.actor, true, "Skill"); + const [prior] = await tx.select().from(activityLog).where(and( + eq(activityLog.companyId, companyId), eq(activityLog.action, "company.skill_file_updated"), + sql`${activityLog.details}->>'idempotencyKey' = ${receiptKey}`, + )); + if (prior) { + if (prior.details?.fingerprint !== fingerprint) throw conflict("Skill file idempotency key was used with different inputs"); + return { receipt: prior.details?.receipt, publication: null }; + } + let publication: ActivityPublication | null = null; + let receipt: { skillId: string; path: string; versionId: string | null; studioPath: string } | null = null; + await companySkillService(tx as unknown as Db).updateFile(companyId, skillId, input.path, input.content, skillActor(req), { + encoding: input.encoding, executable: input.executable, expectedVersionId, + onRollback: (restore, versionId) => { rollbackState.current = { restore, versionId }; }, + afterUpdate: async (versionId) => { + receipt = { skillId, path: input.path, versionId, studioPath: `/skills/studio/${encodeURIComponent(skillId)}` }; + const activity = await persistActivity(tx as unknown as Db, { + companyId, actorType: actor.actorType, actorId: actor.actorId, agentId: actor.agentId, + runId: actor.runId, agentApiKeyId: actor.agentApiKeyId, issueId: runContext?.issue.id, + action: "company.skill_file_updated", entityType: "company_skill", entityId: skillId, + details: { path: input.path, markdown: input.path === "SKILL.md", versionId, + sourceIssueId: runContext?.issue.id ?? null, idempotencyKey: receiptKey, fingerprint, receipt }, + }); + publication = activity.publication; + }, + }); + return { receipt, publication }; + }).catch(async (error) => { + if (rollbackState.current) { + const { restore, versionId } = rollbackState.current; + await db.transaction(async tx => { + await tx.execute(sql`select ${companySkills.id} from ${companySkills} where ${companySkills.id} = ${skillId} and ${companySkills.companyId} = ${companyId} for update`); + const [current] = await tx.select({ versionId: companySkills.currentVersionId }).from(companySkills) + .where(and(eq(companySkills.id, skillId), eq(companySkills.companyId, companyId))); + if (current && current.versionId === versionId) await restore(); + }); + } + throw error; + }); + if (outcome.publication) publishActivity(outcome.publication); + res.json(outcome.receipt); + return; + } const result = await svc.updateFile( companyId, skillId, String(req.body.path ?? ""), String(req.body.content ?? ""), skillActor(req), - { encoding: req.body.encoding, executable: req.body.executable }, + { encoding: req.body.encoding, executable: req.body.executable, expectedVersionId }, ); const actor = getActorInfo(req); diff --git a/server/src/routes/openapi.ts b/server/src/routes/openapi.ts index f37ba2eba83..dfda2d4d572 100644 --- a/server/src/routes/openapi.ts +++ b/server/src/routes/openapi.ts @@ -7740,12 +7740,12 @@ registry.registerPath({ method: "patch", path: "/api/companies/{companyId}/skills/{skillId}/files", tags: ["skills"], - summary: "Update a skill file", + summary: "Update a skill file (optional expectedVersionId and idempotencyKey guard agent retries)", request: { params: z.object({ companyId: z.string(), skillId: z.string() }), body: jsonBody(companySkillFileUpdateSchema), }, - responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized }, + responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized, 409: r.conflict }, }); registry.registerPath({ diff --git a/server/src/services/company-skills.ts b/server/src/services/company-skills.ts index 1392c69fecb..98d5400060a 100644 --- a/server/src/services/company-skills.ts +++ b/server/src/services/company-skills.ts @@ -4643,6 +4643,11 @@ export function companySkillService(db: Db) { for (const slug of [...new Set([initial.slug, ...additionalSlugs])].filter(Boolean).sort()) { await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`${companyId}:${slug}`}, 0))`); } + await tx.execute(sql` + select ${companySkills.id} from ${companySkills} + where ${companySkills.id} = ${skillId} and ${companySkills.companyId} = ${companyId} + for update + `); const skill = await getById(companyId, skillId, tx); if (!skill) throw notFound("Skill not found"); if (skill.slug !== initial.slug) throw conflict("Skill was renamed. Retry the operation."); @@ -4669,10 +4674,16 @@ export function companySkillService(db: Db) { relativePath: string, content: string, actor: SkillActor | null = null, - format: { encoding?: "utf8" | "base64"; executable?: boolean } = {}, + format: { encoding?: "utf8" | "base64"; executable?: boolean; expectedVersionId?: string | null; + afterUpdate?: (versionId: string | null) => Promise; + onRollback?: (restore: () => Promise, versionId: string | null) => void } = {}, ): Promise { return withSkillFileMutation(companyId, skillId, async (skill, tx) => { + if (format.expectedVersionId !== undefined && skill.currentVersionId !== format.expectedVersionId) { + throw conflict("Skill version changed. Read the current version before retrying."); + } + const source = deriveSkillSourceInfo(skill); if (!source.editable || skill.sourceType !== "local_path") { throw unprocessable(source.editableReason ?? "This skill cannot be edited."); @@ -4689,41 +4700,55 @@ export function companySkillService(db: Db) { } const previousContent = await fs.readFile(absolutePath).catch(() => null); const previousMode = (await fs.stat(absolutePath).catch(() => null))?.mode ?? 0o644; + const restore = async () => { + if (previousContent === null) await fs.rm(absolutePath, { force: true }); + else { + await fs.writeFile(absolutePath, previousContent); + await fs.chmod(absolutePath, previousMode); + } + }; + format.onRollback?.(restore, skill.currentVersionId); const mode = (format.executable ?? Boolean(previousMode & 0o111)) ? 0o755 : 0o644; - await fs.mkdir(path.dirname(absolutePath), { recursive: true }); - await fs.writeFile(absolutePath, bytes); - await fs.chmod(absolutePath, mode); + try { + await fs.mkdir(path.dirname(absolutePath), { recursive: true }); + await fs.writeFile(absolutePath, bytes); + await fs.chmod(absolutePath, mode); - if (normalizedPath === "SKILL.md") { - const parsed = parseFrontmatterMarkdown(content); - await tx - .update(companySkills) - .set({ - name: asString(parsed.frontmatter.name) ?? skill.name, - description: asString(parsed.frontmatter.description) ?? skill.description, - markdown: content, - ...readSkillStoreMetadata(parsed.frontmatter, skill.metadata), - updatedAt: new Date(), - }) - .where(eq(companySkills.id, skill.id)); - } else { - await tx - .update(companySkills) - .set({ updatedAt: new Date() }) - .where(eq(companySkills.id, skill.id)); + if (normalizedPath === "SKILL.md") { + const parsed = parseFrontmatterMarkdown(content); + await tx + .update(companySkills) + .set({ + name: asString(parsed.frontmatter.name) ?? skill.name, + description: asString(parsed.frontmatter.description) ?? skill.description, + markdown: content, + ...readSkillStoreMetadata(parsed.frontmatter, skill.metadata), + updatedAt: new Date(), + }) + .where(eq(companySkills.id, skill.id)); + } else { + await tx + .update(companySkills) + .set({ updatedAt: new Date() }) + .where(eq(companySkills.id, skill.id)); + } + + const inventory = await refreshEditedSkillInventory(skill, tx); + if (!previousContent?.equals(bytes) || Boolean(previousMode & 0o111) !== Boolean(mode & 0o111)) { + await createVersion(companyId, skillId, {}, actor, { + database: tx, skipInventoryRefresh: true, skill, fileInventory: inventory, + }); + } + + const updated = await getById(companyId, skillId, tx); + const detail = updated ? await readLoadedSkillFile(updated, normalizedPath) : null; + if (!detail) throw notFound("Skill file not found"); + await format.afterUpdate?.(updated?.currentVersionId ?? null); + return detail; + } catch (error) { + await restore(); + throw error; } - - const inventory = await refreshEditedSkillInventory(skill, tx); - if (!previousContent?.equals(bytes) || Boolean(previousMode & 0o111) !== Boolean(mode & 0o111)) { - await createVersion(companyId, skillId, {}, actor, { - database: tx, skipInventoryRefresh: true, skill, fileInventory: inventory, - }); - } - - const updated = await getById(companyId, skillId, tx); - const detail = updated ? await readLoadedSkillFile(updated, normalizedPath) : null; - if (!detail) throw notFound("Skill file not found"); - return detail; }); } diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts index 026ec993182..00d042b45d2 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts @@ -102,7 +102,7 @@ describe("PaperclipRunnerToolAuthority", () => { issueId, runId, }); - expect(authority.definitions()).toHaveLength(35); + expect(authority.definitions()).toHaveLength(36); const questions = authority.definitions().find(tool => tool.name === "request_human_input")!; expect(questions.description).toContain("ask only the next unanswered question"); expect(questions.description).toContain("Never fabricate answers"); @@ -131,6 +131,7 @@ describe("PaperclipRunnerToolAuthority", () => { "list_document_revisions", "write_document", "create_skill", + "update_skill", "list_agents", "get_agent", "list_approvals", diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts index 5af91c09202..c3dab94a6ab 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts @@ -13,7 +13,7 @@ import { resolveCoreTrustPreset } from "../trust-preset-resolver.js"; import { normalizeIssueExecutionPolicy } from "../issue-execution-policy.js"; import { buildLowTrustSourceTrust } from "../source-trust.js"; import { handoffPlanContext } from "./handoff-plan-context.js"; -import { callCreateSkillTool } from "../skill-tools.js"; +import { callCreateSkillTool, callUpdateSkillTool } from "../skill-tools.js"; import { callProjectTool } from "../project-tools.js"; import { isConnectorTool, executeConnectorTool, type ConnectorAssignment } from "../connector-runtime.js"; import { resolveNativeRuntimeMcpSnapshot } from "./runtime-context.js"; @@ -97,7 +97,7 @@ const IMPLEMENTED_OPERATIONS = new Set([ "search_api", "call_api", "hire_agent", "get_task_context", "get_task_history", "search_tasks", "report_progress", "set_task_title", "request_human_input", - "create_skill", "create_task", "reassign_task", "set_dependencies", "create_project", "list_project_repositories", "list_projects", "register_deliverable", + "create_skill", "update_skill", "create_task", "reassign_task", "set_dependencies", "create_project", "list_project_repositories", "list_projects", "register_deliverable", "list_documents", "read_document", "list_document_revisions", "write_document", "list_agents", "get_agent", "list_approvals", "get_approval", "get_approval_context", ]); @@ -414,6 +414,12 @@ export class PaperclipRunnerToolAuthority { if (!apiUrl || !token) throw new Error("Skill tool authentication is unavailable"); return callCreateSkillTool({ arguments: input, apiUrl, token, companyId: this.binding.companyId }); } + case "update_skill": { + const apiUrl = this.binding.apiUrl ?? process.env.PAPERCLIP_API_URL; + const token = createLocalAgentJwt(this.binding.agentId, this.binding.companyId, context.actor.adapterType, this.binding.runId, context.run.responsibleUserId); + if (!apiUrl || !token) throw new Error("Skill tool authentication is unavailable"); + return callUpdateSkillTool({ arguments: input, apiUrl, token, companyId: this.binding.companyId }); + } case "create_project": case "list_project_repositories": case "list_projects": { diff --git a/server/src/services/native-runtime/runner-api-catalog.ts b/server/src/services/native-runtime/runner-api-catalog.ts index 0a6ed482f5f..4dddb822a8d 100644 --- a/server/src/services/native-runtime/runner-api-catalog.ts +++ b/server/src/services/native-runtime/runner-api-catalog.ts @@ -48,6 +48,7 @@ function dedicatedTools(method: string, path: string): string[] { if (/\/agents\/\{[^}]+\}\/instructions-bundle\/revision\/\{[^}]+\}$/.test(path) && method === "GET") return ["read_agent_instructions"]; if (/\/agents\/\{[^}]+\}\/instructions-bundle\/restore$/.test(path) && method === "POST") return ["restore_agent_instructions"]; if (/\/companies\/\{[^}]+\}\/skills$/.test(path) && method === "POST") return ["create_skill"]; + if (/\/companies\/\{[^}]+\}\/skills\/\{[^}]+\}\/files$/.test(path) && method === "PATCH") return ["update_skill"]; if (/\/companies\/\{[^}]+\}\/agent-hires$/.test(path) && method === "POST") return ["hire_agent"]; if (/\/projects$/.test(path)) return method === "GET" ? ["list_projects"] : method === "POST" ? ["create_project"] : []; if (/\/project-repositories$/.test(path) && method === "GET") return ["list_project_repositories"]; diff --git a/server/src/services/native-runtime/update-skill.integration.test.ts b/server/src/services/native-runtime/update-skill.integration.test.ts new file mode 100644 index 00000000000..78dc5966ec9 --- /dev/null +++ b/server/src/services/native-runtime/update-skill.integration.test.ts @@ -0,0 +1,116 @@ +import { randomUUID } from "node:crypto"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { eq } from "drizzle-orm"; +import { activityLog, companyMemberships, heartbeatRuns, issues } from "@paperclipai/db"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { startRunnerApiTestServer } from "../../__tests__/helpers/runner-api-server.js"; +import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js"; +import { companySkillPolicyService } from "../company-skill-policy.js"; +import { companySkillService } from "../company-skills.js"; + +describe("runner update_skill through the real skill API", () => { + let server: Awaited>; + let home: string; + const previousSecret = process.env.PAPERCLIP_AGENT_JWT_SECRET; + const previousHome = process.env.PAPERCLIP_HOME; + beforeAll(async () => { + process.env.PAPERCLIP_AGENT_JWT_SECRET = randomUUID(); + home = await mkdtemp(join(tmpdir(), "paperclip-update-skill-")); + process.env.PAPERCLIP_HOME = home; + server = await startRunnerApiTestServer(); + }, 60_000); + afterAll(async () => { + await server?.close(); + if (previousSecret === undefined) delete process.env.PAPERCLIP_AGENT_JWT_SECRET; + else process.env.PAPERCLIP_AGENT_JWT_SECRET = previousSecret; + if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; + else process.env.PAPERCLIP_HOME = previousHome; + if (home) await rm(home, { recursive: true, force: true }); + }); + + async function fixture(options: Parameters[0] = {}) { + const result = await server.fixture(options); + await server.db.insert(companyMemberships).values({ companyId: result.companyId, principalType: "agent", principalId: result.agentId, status: "active", membershipRole: "member" }); + const skill = await companySkillService(server.db).createLocalSkill(result.companyId, { + name: "release-review", description: "Review releases.", + markdown: "---\nname: release-review\ndescription: Review releases.\n---\n\n# Review\nInspect the release.\n", + }); + const args = { skillId: skill.id, expectedVersionId: skill.currentVersionId, idempotencyKey: randomUUID(), + markdown: "---\nname: release-review\ndescription: Review releases.\n---\n\n# Review\nInspect the release and its tests.\n" }; + return { ...result, skill, args }; + } + + it("persists exactly one version and attributed activity across retries and a replacement run", async () => { + const testFixture = await fixture(); + const { authority, args, companyId, skill, issueId } = testFixture; + expect(authority.definitions().some(tool => tool.name === "update_skill")).toBe(true); + const first = await authority.execute({ tool: "update_skill", callId: "first", arguments: args }); + const second = await authority.execute({ tool: "update_skill", callId: "retry", arguments: args }); + expect(second).toEqual(first); + expect(first).toMatchObject({ skillId: skill.id, path: "SKILL.md", versionId: expect.any(String) }); + const replacementRunId = randomUUID(); + await server.db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, testFixture.runId)); + await server.db.insert(heartbeatRuns).values({ id: replacementRunId, companyId, agentId: testFixture.agentId, + status: "running", runtimeMode: "native", nativeIssueId: issueId, invocationSource: "assignment", triggerDetail: "system", contextSnapshot: { issueId } }); + await server.db.update(issues).set({ executionRunId: replacementRunId }).where(eq(issues.id, issueId)); + const replacement = new PaperclipRunnerToolAuthority(server.db, { ...testFixture, runId: replacementRunId }); + expect(await replacement.execute({ tool: "update_skill", callId: "replacement", arguments: args })).toEqual(first); + expect(await companySkillService(server.db).listVersions(companyId, skill.id)).toHaveLength(2); + expect((await server.db.select().from(activityLog).where(eq(activityLog.action, "company.skill_file_updated"))) + .filter(event => event.entityId === skill.id)) + .toEqual([expect.objectContaining({ entityId: skill.id, details: expect.objectContaining({ sourceIssueId: issueId, versionId: (first as { versionId: string }).versionId }) })]); + await expect(replacement.execute({ tool: "update_skill", callId: "changed-key", arguments: { ...args, markdown: args.markdown + "More." } })).rejects.toThrow(/different inputs/); + await expect(replacement.execute({ tool: "update_skill", callId: "stale", arguments: { ...args, idempotencyKey: randomUUID() } })).rejects.toThrow(/version changed/); + }); + + it("denies policy changes and cross-company skill IDs", async () => { + const first = await fixture(); + const second = await fixture(); + await expect(second.authority.execute({ tool: "update_skill", callId: "foreign", arguments: { ...first.args, idempotencyKey: randomUUID() } })).rejects.toThrow(); + await companySkillPolicyService(server.db).replace({ companyId: first.companyId, expectedRevision: 0, + policy: { schemaVersion: 1, defaultEffect: "deny", rules: [] }, activity: { actorType: "user", actorId: "test-board" } }); + await expect(first.authority.execute({ tool: "update_skill", callId: "denied", arguments: first.args })).rejects.toThrow(/company policy/); + expect(await companySkillService(server.db).listVersions(first.companyId, first.skill.id)).toHaveLength(1); + }); + + it("serializes simultaneous retries and checks policy again after a successful edit", async () => { + const testFixture = await fixture(); + const calls = await Promise.all(["parallel-a", "parallel-b"].map(callId => + testFixture.authority.execute({ tool: "update_skill", callId, arguments: testFixture.args }))); + expect(calls[0]).toEqual(calls[1]); + expect(await companySkillService(server.db).listVersions(testFixture.companyId, testFixture.skill.id)).toHaveLength(2); + await companySkillPolicyService(server.db).replace({ companyId: testFixture.companyId, expectedRevision: 0, + policy: { schemaVersion: 1, defaultEffect: "deny", rules: [] }, activity: { actorType: "user", actorId: "test-board" } }); + await expect(testFixture.authority.execute({ tool: "update_skill", callId: "revoked-retry", arguments: testFixture.args })).rejects.toThrow(/company policy/); + expect(await companySkillService(server.db).listVersions(testFixture.companyId, testFixture.skill.id)).toHaveLength(2); + }); + + it("restores the file when the required audit callback fails", async () => { + const testFixture = await fixture(); + const service = companySkillService(server.db); + const original = await service.readFile(testFixture.companyId, testFixture.skill.id, "SKILL.md"); + await expect(service.updateFile(testFixture.companyId, testFixture.skill.id, "SKILL.md", testFixture.args.markdown, + null, { expectedVersionId: testFixture.skill.currentVersionId, afterUpdate: async () => { throw new Error("audit unavailable"); } })) + .rejects.toThrow("audit unavailable"); + expect((await service.readFile(testFixture.companyId, testFixture.skill.id, "SKILL.md"))?.content).toBe(original?.content); + expect(await service.listVersions(testFixture.companyId, testFixture.skill.id)).toHaveLength(1); + }); + + it("rejects a competing edit with a different key and the same expected version", async () => { + const testFixture = await fixture(); + const attempts = await Promise.allSettled([testFixture.args, { ...testFixture.args, + idempotencyKey: randomUUID(), markdown: testFixture.args.markdown + "Another edit." }].map((argumentsValue, index) => + testFixture.authority.execute({ tool: "update_skill", callId: `race-${index}`, arguments: argumentsValue }))); + expect(attempts.filter(attempt => attempt.status === "fulfilled")).toHaveLength(1); + expect(attempts.filter(attempt => attempt.status === "rejected")).toHaveLength(1); + expect(await companySkillService(server.db).listVersions(testFixture.companyId, testFixture.skill.id)).toHaveLength(2); + }); + + it("hides the tool and rejects invocation in planning mode", async () => { + const testFixture = await fixture({ mode: "planning" }); + expect(new PaperclipRunnerToolAuthority(server.db, { ...testFixture, workMode: "planning" }).definitions().some(tool => tool.name === "update_skill")).toBe(false); + await expect(testFixture.authority.execute({ tool: "update_skill", callId: "planning", arguments: testFixture.args })).rejects.toThrow(); + }); +}); diff --git a/server/src/services/skill-tools.test.ts b/server/src/services/skill-tools.test.ts index a1794dbcaef..113c684e8c5 100644 --- a/server/src/services/skill-tools.test.ts +++ b/server/src/services/skill-tools.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it, vi } from "vitest"; -import { callCreateSkillTool, createSkillToolInput } from "./skill-tools.js"; +import { callCreateSkillTool, createSkillToolInput, callUpdateSkillTool, updateSkillToolInput } from "./skill-tools.js"; const input = { name: "release-review", description: "Review release notes.", idempotencyKey: "release-review-1", @@ -36,3 +36,29 @@ describe("create skill tool", () => { expect(result).toEqual({ id: "skill-1", name: input.name, slug: input.name, description: input.description, versionId: "version-1", studioPath: "/skills/studio/skill-1" }); }); }); + +describe("update skill tool", () => { + const args = { skillId: "00000000-0000-4000-8000-000000000001", expectedVersionId: "00000000-0000-4000-8000-000000000002", + markdown: input.markdown, idempotencyKey: "update-1" }; + + it("rejects incomplete content and untrusted scope without a request", async () => { + const fetcher = vi.fn(); + expect(updateSkillToolInput.parse(args)).toEqual(args); + for (const invalid of [{ markdown: "# Missing frontmatter" }, { markdown: input.markdown.replace("# Review\nCheck each release note against the change.", " ") }, + { companyId: "other" }, { expectedVersionId: "not-a-uuid" }]) { + await expect(callUpdateSkillTool({ arguments: { ...args, ...invalid }, apiUrl: "http://localhost", token: "token", companyId: "company" }, fetcher)).rejects.toThrow(); + } + expect(fetcher).not.toHaveBeenCalled(); + }); + + it("calls the company file endpoint and returns a compact receipt", async () => { + const receipt = { skillId: args.skillId, versionId: "next-version", path: "SKILL.md", studioPath: `/skills/studio/${args.skillId}` }; + const fetcher = vi.fn().mockResolvedValue(new Response(JSON.stringify({ ...receipt, content: args.markdown }), { status: 200 })); + expect(await callUpdateSkillTool({ arguments: args, apiUrl: "http://localhost/api", token: "token", companyId: "company" }, fetcher)).toEqual(receipt); + expect(fetcher).toHaveBeenCalledWith(`http://localhost/api/companies/company/skills/${args.skillId}/files`, expect.objectContaining({ method: "PATCH" })); + expect(JSON.parse(fetcher.mock.calls[0]![1].body)).toEqual({ path: "SKILL.md", content: args.markdown, + expectedVersionId: args.expectedVersionId, idempotencyKey: args.idempotencyKey }); + fetcher.mockResolvedValueOnce(new Response(JSON.stringify({ error: "Skill version changed" }), { status: 409 })); + await expect(callUpdateSkillTool({ arguments: args, apiUrl: "http://localhost", token: "token", companyId: "company" }, fetcher)).rejects.toThrow("Skill version changed"); + }); +}); diff --git a/server/src/services/skill-tools.ts b/server/src/services/skill-tools.ts index 371d507a5ec..c0df4a3a318 100644 --- a/server/src/services/skill-tools.ts +++ b/server/src/services/skill-tools.ts @@ -18,6 +18,35 @@ export const createSkillToolInput = z.object({ } }); +/** Replace the complete primary file of an existing skill, never a partial patch. */ +export const updateSkillToolInput = z.object({ + skillId: z.string().guid(), + markdown: z.string().min(1).max(200000), + expectedVersionId: z.string().guid(), + idempotencyKey: z.string().min(1).max(240), +}).strict().superRefine((input, context) => { + const document = parseFrontmatterMarkdown(input.markdown); + const metadata = skillFrontmatterSchema.safeParse(document.frontmatter); + if (!document.hasFrontmatter || !metadata.success || !document.body.trim()) { + context.addIssue({ code: "custom", path: ["markdown"], message: "Provide a complete SKILL.md with valid name and description frontmatter and a nonempty body." }); + } +}); + +export async function callUpdateSkillTool(input: { + arguments: Record; apiUrl: string; token: string; companyId: string; +}, fetcher: typeof fetch = fetch) { + const { skillId, markdown, expectedVersionId, idempotencyKey } = updateSkillToolInput.parse(input.arguments); + const response = await fetcher(`${input.apiUrl.replace(/\/+$/, "").replace(/\/api$/, "")}/api/companies/${encodeURIComponent(input.companyId)}/skills/${encodeURIComponent(skillId)}/files`, { + method: "PATCH", + headers: { Authorization: `Bearer ${input.token}`, "Content-Type": "application/json" }, + body: JSON.stringify({ path: "SKILL.md", content: markdown, expectedVersionId, idempotencyKey }), + signal: AbortSignal.timeout(60_000), + }); + const result = await response.json(); + if (!response.ok) throw new Error(typeof result.error === "string" ? result.error : `Skill update failed (${response.status})`); + return { skillId: result.skillId, versionId: result.versionId, path: result.path, studioPath: result.studioPath }; +} + /** Use the same API and company skill policy as Skill Studio. */ export async function callCreateSkillTool(input: { arguments: Record; apiUrl: string; token: string; companyId: string; From 1c3abf5075306129fd08263d47483466430e7f76 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 3 Oct 2026 12:39:18 -0500 Subject: [PATCH 11/37] fix(ui): use available space for composer labels (#15050) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip helps people manage AI agents and their work. > - The task composer shows the next assignee and model before a message starts work. > - Fixed width limits shortened both labels when the composer had unused space. > - People could not read the selected agent and model even when the full text could fit. > - This pull request makes the capsule use the available composer width. > - It keeps the ellipsis when Plan mode or a narrow layout causes real space pressure. > - The benefit is clearer run settings without damage to the compact composer layout. ## Linked Issues or Issue Description **What happened?** The task composer truncated the assignee name at 6rem and the complete assignee and model capsule at 16rem. It did this even when the composer had more available space. **Expected behavior** The composer must show the complete assignee and model labels when they fit. It must use an ellipsis only when another control or a narrow viewport limits the available width. **Steps to reproduce** 1. Open a task composer with a long assignee name and a long model name. 2. Use a wide desktop layout. 3. Observe that the old capsule shortened both labels while unused space remained. **Paperclip version or commit** Current `master` before this change. **Deployment mode** Local dev (`pnpm dev`). **Installation method** Built from source. **Agent adapter(s) involved** Not adapter-specific. This is a core UI bug. **Database mode** Not database-related. **Access context** Board. **Additional context** The Storybook cases cover a wide composer and a narrow composer with Plan mode. ## What Changed - Removed the fixed maximum width from the assignee and model capsule. - Removed the fixed maximum width from the assignee label. - Kept overflow ellipsis behavior when the parent row has insufficient space. - Added stable label selectors and focused component coverage. - Added Storybook cases for complete labels and Plan-mode truncation. ## Verification - `pnpm --filter @paperclipai/plugin-sdk build` - `pnpm --filter @paperclipai/ui typecheck` - `vitest run ui/src/components/task-chat/ComposerRunSettingsPicker.test.tsx` - `node scripts/check-token-gates.mjs` - Storybook production build under Node.js 24.20.0 - Captured and inspected the two new Storybook cases. - The complete repository typecheck and build reach the Rust Runner step. This local environment does not have `cargo`. Hosted CI supplies the Rust toolchain. - The repository test suite reaches workspace-runtime tests. This local runner does not allow their required temporary home directories. Hosted CI supplies a writable test home. ## Risks - Low risk. The change only removes fixed width limits from one flex item. - A very narrow composer can still shorten both labels. This is the intended fallback. - The Storybook constrained case verifies that Plan mode and Send remain usable. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI GPT-5 through the Paperclip Codex runner. The run used reasoning, tool use, code execution, browser automation, and image inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- .../ComposerRunSettingsPicker.test.tsx | 15 ++++++++++++ .../task-chat/ComposerRunSettingsPicker.tsx | 6 ++--- .../composer-model-picker/fixtures.ts | 3 +++ .../stories/composer-model-picker.stories.tsx | 24 +++++++++++++++++++ 4 files changed, 45 insertions(+), 3 deletions(-) diff --git a/ui/src/components/task-chat/ComposerRunSettingsPicker.test.tsx b/ui/src/components/task-chat/ComposerRunSettingsPicker.test.tsx index e09a165b4cf..3185f1d905b 100644 --- a/ui/src/components/task-chat/ComposerRunSettingsPicker.test.tsx +++ b/ui/src/components/task-chat/ComposerRunSettingsPicker.test.tsx @@ -55,6 +55,21 @@ afterEach(() => { }); describe("composer assignee picker", () => { + it("lets the assignee and model use the available composer width", () => { + render(vi.fn(), vi.fn()); + const trigger = container!.querySelector('[data-testid="task-chat-composer-assignee"]'); + const assignee = trigger!.querySelector('[data-testid="task-chat-composer-assignee-label"]'); + const model = trigger!.querySelector('[data-testid="task-chat-composer-model-label"]'); + + expect(trigger?.className).toContain("max-w-full"); + expect(trigger?.className).not.toContain("max-w-64"); + expect(assignee?.className).toContain("min-w-0"); + expect(assignee?.className).not.toContain("max-w-24"); + expect(model?.className).toContain("min-w-0"); + expect(assignee?.className).toContain("truncate"); + expect(model?.className).toContain("truncate"); + }); + it("finds assignees by their displayed role and harness", async () => { render(vi.fn(), vi.fn()); await click("Select assignee, model and effort"); diff --git a/ui/src/components/task-chat/ComposerRunSettingsPicker.tsx b/ui/src/components/task-chat/ComposerRunSettingsPicker.tsx index f3b6025126a..63f42235d25 100644 --- a/ui/src/components/task-chat/ComposerRunSettingsPicker.tsx +++ b/ui/src/components/task-chat/ComposerRunSettingsPicker.tsx @@ -167,10 +167,10 @@ export function ComposerRunSettingsPicker({ const reset = () => onSettingsChange(DEFAULT_COMPOSER_RUN_SETTINGS); const closeButton = mobile ? : null; const trigger = ; diff --git a/ui/storybook/prototypes/composer-model-picker/fixtures.ts b/ui/storybook/prototypes/composer-model-picker/fixtures.ts index c194f2b7732..2ea83b97711 100644 --- a/ui/storybook/prototypes/composer-model-picker/fixtures.ts +++ b/ui/storybook/prototypes/composer-model-picker/fixtures.ts @@ -86,6 +86,9 @@ export const composerAgents: ComposerAgent[] = [ { id: "kimi-code/kimi-for-coding", label: "K2.8 Preview" }, { id: "kimi-code/k3", label: "K3" }, ], manualPattern: "kimi-code/model" }, + { id: "long-labels", name: "Alexandra Engineering Coordinator", role: "Engineering", harness: "Codex", adapterType: "codex_local", defaultModel: "gpt-5.6-sol", models: [ + { id: "gpt-5.6-sol", label: "GPT-5.6 Sol Extended Context Preview", detail: "Agent default" }, + ], manualPattern: "Model ID, e.g. gpt-5.6-sol" }, ]; /** Share the capsule-avatar palettes used by the agent persona stories. */ diff --git a/ui/storybook/stories/composer-model-picker.stories.tsx b/ui/storybook/stories/composer-model-picker.stories.tsx index dbe7cf8c050..391eddb75c2 100644 --- a/ui/storybook/stories/composer-model-picker.stories.tsx +++ b/ui/storybook/stories/composer-model-picker.stories.tsx @@ -414,3 +414,27 @@ export const ProductionIntermediateWidthPlan: Story = { await expect(screen.getByTestId("composer-model-popover")).toBeVisible(); }, }; + +export const ProductionLongLabelsWide: Story = { + name: "25 · Long labels with available space", + render: () => , + play: async ({ canvasElement }) => { + const screen = within(canvasElement.ownerDocument.body); + const assignee = screen.getByTestId("task-chat-composer-assignee-label"); + const model = screen.getByTestId("task-chat-composer-model-label"); + await expect(assignee.scrollWidth).toBeLessThanOrEqual(assignee.clientWidth); + await expect(model.scrollWidth).toBeLessThanOrEqual(model.clientWidth); + }, +}; + +export const ProductionLongLabelsConstrained: Story = { + name: "25b · Long labels constrained by Plan mode", + render: () => , + play: async ({ canvasElement }) => { + const screen = within(canvasElement.ownerDocument.body); + const assignee = screen.getByTestId("task-chat-composer-assignee-label"); + const model = screen.getByTestId("task-chat-composer-model-label"); + await expect(assignee.scrollWidth > assignee.clientWidth || model.scrollWidth > model.clientWidth).toBe(true); + await expect(screen.getByRole("button", { name: "Remove Plan mode" })).toBeVisible(); + }, +}; From 2a8a99e4a5f69aa803b3f10b982f583e75a87042 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 3 Oct 2026 12:41:22 -0500 Subject: [PATCH 12/37] fix(ui): reveal completed thinking caret beside label (#15048) ## Thinking Path > - Paperclip helps people manage AI agents and their work. > - The task thread shows completed agent work and its reasoning. > - The completed work row places the disclosure caret at the far right. > - The caret stays visible even when the reader does not inspect the row. > - The row needs a quiet control that stays close to the work label. > - This pull request places the caret after that label and shows it on hover or keyboard focus. > - The change keeps the timestamp on the right and does not move content on hover. ## Linked Issues or Issue Description **What happened?** The completed agent work row shows a disclosure caret at the far right of the task thread. The caret stays visible when the pointer is elsewhere. **Expected behavior** The caret stays near the work label. It appears on hover or keyboard focus. The timestamp stays on the right. **Steps to reproduce** 1. Open a task with a completed agent run. 2. Look at the completed work row without hovering over it. 3. Hover over the row and then use the keyboard to focus its button. Related: #11772 addresses a different caret and composer layout. ## What Changed - Move the completed work caret next to the work label. - Keep its space reserved. Show it on hover or keyboard focus. - Add a test for caret position, visibility classes, and the open state. ## Verification - `node node_modules/vitest/vitest.mjs run ui/src/components/IssueChatThread.test.tsx` passes all 100 tests. - `node scripts/check-token-gates.mjs` passes all token gates. - `git diff origin/master...HEAD --check` passes. - CI passes on the latest head, including typecheck, build, tests, and verification. The local isolated worktree could not run typecheck because dependency installation failed while applying the existing `codex-acp` patch. ## Risks - Low risk. The same button still opens the work detail. Only the caret position and visibility change. - On a touch device, the caret is not visible without hover. The work row stays a button that users can tap. > This is a focused UI fix. It does not add a planned core feature from `ROADMAP.md`. ## Model Used - OpenAI Codex coding agent. The runner does not expose the exact model ID or context window. The agent used reasoning, shell tools, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with the details available in this runner - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked public issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket ID - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have checked documentation impact; no documentation change is needed for this visual fix - [x] I have considered and documented the risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- ui/src/components/IssueChatThread.test.tsx | 47 ++++++++++++++++++++++ ui/src/components/IssueChatThread.tsx | 25 ++++++------ 2 files changed, 59 insertions(+), 13 deletions(-) diff --git a/ui/src/components/IssueChatThread.test.tsx b/ui/src/components/IssueChatThread.test.tsx index 384fa952a3d..72d4dd8968a 100644 --- a/ui/src/components/IssueChatThread.test.tsx +++ b/ui/src/components/IssueChatThread.test.tsx @@ -4813,6 +4813,53 @@ describe("IssueChatThread", () => { }); }); + it("keeps the completed chain-of-thought caret beside its label and reveals it on hover or focus", () => { + const root = createRoot(container); + const run = issueChatLongThreadLinkedRuns[2]; + + act(() => { + root.render( + + {}} + showComposer={false} + enableLiveTranscriptPolling={false} + transcriptsByRunId={issueChatLongThreadTranscriptsByRunId} + hasOutputForRun={() => true} + /> + , + ); + }); + + const header = Array.from(container.querySelectorAll("button")).find((button) => + button.textContent?.includes("worked for 4 minutes"), + ); + expect(header).toBeDefined(); + const label = Array.from(header!.children).find((child) => + child.textContent?.includes("worked for 4 minutes"), + ); + const caret = label?.nextElementSibling; + expect(caret?.tagName.toLowerCase()).toBe("svg"); + expect(header?.classList.contains("group")).toBe(true); + expect(caret?.classList.contains("opacity-0")).toBe(true); + expect(caret?.classList.contains("group-hover:opacity-100")).toBe(true); + expect(caret?.classList.contains("group-focus-visible:opacity-100")).toBe(true); + expect(caret?.nextElementSibling?.classList.contains("ml-auto")).toBe(true); + + act(() => { + header!.click(); + }); + expect(caret?.classList.contains("rotate-180")).toBe(true); + + act(() => { + root.unmount(); + }); + }); + it("folds chain-of-thought when the same message transitions from running to complete", () => { expect( resolveAssistantMessageFoldedState({ diff --git a/ui/src/components/IssueChatThread.tsx b/ui/src/components/IssueChatThread.tsx index f04a25585a9..4a9aef03b1f 100644 --- a/ui/src/components/IssueChatThread.tsx +++ b/ui/src/components/IssueChatThread.tsx @@ -2659,19 +2659,18 @@ function IssueChatAssistantMessage({ {chainOfThoughtLabel?.toLowerCase()} - - {message.createdAt ? ( - - {commentDateLabel(message.createdAt)} - - ) : null} - - +
), }; + +const LOCAL_BRANCH: IssueWorkProduct = { + ...product(KINDS[2], STATES[0]), + id: "local-branch-no-remote", + title: "Company skill update entrypoint", + provider: "git", + updatedAt: new Date("2026-10-03T11:52:54.448Z"), + url: null, + metadata: null, + status: "active", + summary: "Uncommitted implementation changes are in the execution working tree on branch company-skill-update-entrypoint. Includes a guarded skill API, runner tool, docs, and focused regressions. No commit or push was requested.", +}; + +/** A work product without a remote URL still explains what it is and opens its own details. */ +export const BranchWithoutRemoteLink: Story = { + args: { workProduct: LOCAL_BRANCH, href: null }, + render: (args) => ( +
+ +
+ ), + play: async ({ canvasElement }) => { + expect(canvasElement.textContent).toContain("Branch · no remote link"); + expect(canvasElement.textContent).not.toContain("Uncommitted implementation changes"); + expect(canvasElement.querySelector('button[aria-label^="Show details:"]')).not.toBeNull(); + }, +}; + +export const BranchDetailsExpanded: Story = { + ...BranchWithoutRemoteLink, + play: async ({ canvasElement, userEvent }) => { + const button = canvasElement.querySelector('button[aria-label^="Show details:"]'); + await userEvent.click(button!); + expect(canvasElement.textContent).toContain("git · active · Updated"); + expect(canvasElement.textContent).toContain("Saved description"); + expect(canvasElement.querySelector("details")?.open).toBe(false); + }, +}; From 994d6edcdd4e15d5f9cc5cf8c135ac599104b86a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 3 Oct 2026 21:19:55 -0500 Subject: [PATCH 14/37] Fix Markdown in provisional task titles (#15047) ## Thinking Path > - Paperclip uses issues to organize work for AI agents. > - Paperclip creates a provisional issue title from the issue description. > - A description can start with a Markdown image or other Markdown syntax. > - The old title logic truncated the raw Markdown source. > - This pull request parses Markdown before it truncates the title. > - The benefit is a short and readable provisional title. ## Linked Issues or Issue Description **What happened?** When an issue description started with a Markdown image, Paperclip used the raw image syntax and URL in the provisional title. Other common Markdown markers could also appear in the title. **Expected behavior** Paperclip must remove Markdown syntax before it creates the 120-character provisional title. It must keep useful text, such as link labels and inline code content. **Steps to reproduce** 1. Create an issue without an explicit title. 2. Start the issue description with a Markdown image. 3. Add Markdown text after the image. 4. Observe that the provisional title contains raw Markdown syntax or an image URL. **Paperclip version or commit** The problem reproduces on `master` before this change. **Deployment mode** Local development with the embedded PGlite database. ## What Changed - Parse the description and remove image nodes before title generation. - Convert the remaining Markdown to plain text before the 120-character limit. - Keep link labels, inline code, and identifier punctuation. - Use image alt text, or `Image`, for an image-only description. - Fall back to a simple 120-character title if Markdown parsing or conversion fails. - Add regression tests for image and inline-code titles, parser failures, and text-conversion failures. ## Verification - `./node_modules/.bin/vitest run server/src/__tests__/task-title-routes.test.ts` - `./node_modules/.bin/tsc -p server/tsconfig.json --noEmit` - Full GitHub CI matrix passed on commit `663f3a0517c9e450f174b6c37970756457067ba5`. - Greptile reviewed the latest commit with a 5/5 confidence score and no unresolved findings. ## Risks - Low risk. This change only affects generated provisional titles. It does not change explicit titles or full issue descriptions. - The implementation uses the Markdown parser and plain-text converter that are already present in the server. > This is a focused bug fix. It does not add planned core work from `ROADMAP.md`. ## Model Used - OpenAI Codex with GPT-5. The run used reasoning, tool use, code execution, and repository access. The runtime did not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../src/__tests__/task-title-routes.test.ts | 58 ++++++++++++++++++- server/src/services/issues.ts | 39 ++++++++++++- 2 files changed, 95 insertions(+), 2 deletions(-) diff --git a/server/src/__tests__/task-title-routes.test.ts b/server/src/__tests__/task-title-routes.test.ts index 77ab74d5388..5146729da07 100644 --- a/server/src/__tests__/task-title-routes.test.ts +++ b/server/src/__tests__/task-title-routes.test.ts @@ -1,5 +1,15 @@ import { randomUUID } from "node:crypto"; -import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import * as chat from "chat"; + +vi.mock("chat", async importOriginal => { + const original = await importOriginal(); + return { + ...original, + parseMarkdown: vi.fn(original.parseMarkdown), + markdownToPlainText: vi.fn(original.markdownToPlainText), + }; +}); import { and, eq } from "drizzle-orm"; import { activityLog, externalObjectMentions, heartbeatRuns, issues } from "@paperclipai/db"; import { createChildIssueSchema, createIssueSchema, setIssueTitleSchema } from "@paperclipai/shared"; @@ -49,6 +59,52 @@ describe("task titles", () => { expect(setIssueTitleSchema.safeParse({ title: "x".repeat(241) }).success).toBe(false); }); + it("removes leading Markdown before truncating a provisional title", async () => { + const f = await server.fixture(); + const description = [ + "![Screenshot](https://example.com/a-very-long-image-name.png)", + "# Fix the [sign-in redirect](https://example.com/issues/42)", + "Keep `returnTo` working with **saved sessions**.", + ].join("\n\n"); + const result = await issueService(server.db).create(f.companyId, { description }); + expect(result).toMatchObject({ + title: "Fix the sign-in redirect Keep returnTo working with saved sessions.", + description, + titleNeedsGeneration: true, + }); + await expect(issueService(server.db).create(f.companyId, { + description: "![Error dialog](https://example.com/error.png)", + })).resolves.toMatchObject({ title: "Error dialog", titleNeedsGeneration: true }); + await expect(issueService(server.db).create(f.companyId, { + description: "![](https://example.com/image.png)", + })).resolves.toMatchObject({ title: "Image", titleNeedsGeneration: true }); + await expect(issueService(server.db).create(f.companyId, { + description: "![Map](https://example.com/Map_(1).png)", + })).resolves.toMatchObject({ title: "Map", titleNeedsGeneration: true }); + await expect(issueService(server.db).create(f.companyId, { + description: "Fix `set_task_title`", + })).resolves.toMatchObject({ title: "Fix set_task_title", titleNeedsGeneration: true }); + await expect(issueService(server.db).create(f.companyId, { + description: "![Screenshot][img]\n\n[img]: https://example.com/image.png\n\nFix login", + })).resolves.toMatchObject({ title: "Fix login", titleNeedsGeneration: true }); + }); + + it.each(["parseMarkdown", "markdownToPlainText"] as const)("falls back to a simple title if %s throws", async parser => { + const f = await server.fixture(); + const description = ` Fix the login flow after a failed parser\n${"long ".repeat(30)}`; + vi.mocked(chat[parser]).mockImplementationOnce(() => { throw new Error("parser failed"); }); + try { + const result = await issueService(server.db).create(f.companyId, { description }); + expect(result).toMatchObject({ + title: description.trim().replace(/\s+/g, " ").slice(0, 120), + description, + titleNeedsGeneration: true, + }); + } finally { + vi.mocked(chat[parser]).mockClear(); + } + }); + it("creates prompt-only children and still accepts explicit child titles", async () => { const f = await server.fixture(); const description = "Investigate the child sign-in redirect"; diff --git a/server/src/services/issues.ts b/server/src/services/issues.ts index a214dbd595b..fbf98682852 100644 --- a/server/src/services/issues.ts +++ b/server/src/services/issues.ts @@ -9,6 +9,7 @@ import { executionProjectionsForRuns } from "./execution-projection.js"; import type { ExecutionProjection } from "@paperclipai/shared"; import { Buffer } from "node:buffer"; import { createHash, randomUUID } from "node:crypto"; +import { markdownToPlainText, parseMarkdown } from "chat"; import { and, asc, @@ -6621,6 +6622,40 @@ export function issueService(db: Db) { const instanceSettings = instanceSettingsService(db); const treeControlSvc = issueTreeControlService(db); + function provisionalTitleFromDescription(description: string) { + const simpleTitle = description.trim().replace(/\s+/g, " ").slice(0, 120); + try { + type MarkdownNode = { + type: string; + alt?: string | null; + children?: MarkdownNode[]; + position?: { start: { offset?: number }; end: { offset?: number } }; + }; + const imageRanges: Array<{ start: number; end: number }> = []; + const imageAlts: string[] = []; + const visit = (node: MarkdownNode) => { + if (node.type === "image" || node.type === "imageReference") { + const start = node.position?.start.offset; + const end = node.position?.end.offset; + if (start !== undefined && end !== undefined) imageRanges.push({ start, end }); + if (node.alt?.trim()) imageAlts.push(node.alt.trim()); + } + node.children?.forEach(visit); + }; + visit(parseMarkdown(description) as MarkdownNode); + const withoutImages = imageRanges + .sort((a, b) => b.start - a.start) + .reduce((text, range) => `${text.slice(0, range.start)} ${text.slice(range.end)}`, description); + const plainText = markdownToPlainText(withoutImages).trim().replace(/\s+/g, " "); + const fallback = imageRanges.length > 0 + ? imageAlts.join(" ") || "Image" + : simpleTitle; + return (plainText || fallback).slice(0, 120); + } catch { + return simpleTitle; + } + } + function normalizeCreateIssueTitle(title: string) { return title.trim().replace(/\s+/g, " ").toLowerCase(); } @@ -9759,7 +9794,9 @@ export function issueService(db: Db) { ...issueData } = data; const explicitTitle = issueData.title?.trim(); - const provisionalTitle = issueData.description?.trim().replace(/\s+/g, " ").slice(0, 120); + const provisionalTitle = issueData.description + ? provisionalTitleFromDescription(issueData.description) + : undefined; const resolvedTitle = explicitTitle || provisionalTitle; if (!resolvedTitle) throw unprocessable("Provide a title or task description"); const titleNeedsGeneration = !explicitTitle; From 1c07b5903b1b11139b1e1ce052a3cd4885865d90 Mon Sep 17 00:00:00 2001 From: scotttong Date: Sun, 4 Oct 2026 02:13:46 -0700 Subject: [PATCH 15/37] feat: Chat leads the left nav, agent work beside chats, and a Combined Inbox + Task List flag (#15100) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The left nav is the main way people move between tasks, the inbox, and Agent Chat > - The nav has separate Inbox and Tasks rows that show overlapping work, and Chat is one row among many > - The side panel beside a chat shows the conversation's own artifacts, not the work the agent did > - People want Chat to be easy to find, and they want one place for their task views > - This pull request moves Chat to the top of Work, shows the agent's tasks and artifacts beside each chat, and adds an experimental flag that folds Inbox into Tasks > - The benefit is a shorter nav and a chat view that shows what the agent is working on. Both changes stay off until an operator enables them ## Linked Issues or Issue Description Refs #14706 (the secondary Agent Chat navigation this change builds on) Refs #14848 (reopen the last visited agent chat) **Subsystem affected** UI navigation (left nav, mobile tab bar), Agent Chat side panel, task list and inbox, and the company artifacts API. **Problem or motivation** Inbox and Tasks are two nav rows for overlapping work. Chat sits in the top group with no clear home. The chat rail lists only agents you already talked to, so you cannot see your other teammates there. The side panel beside a chat shows only the conversation's own artifacts. It does not show the tasks and files the agent made. **Proposed solution** With Agent Chat on, Chat leads the Work section and the rail lists every eligible agent. The chat side panel opens on the agent's tasks as cards, and the agent's artifacts are available from +. A new experimental flag, Combined Inbox + Task List, makes Inbox a set of views inside Tasks. **Alternatives considered** Rebuilding the inbox inside the task list. Instead, `/issues` hosts the existing Inbox component for inbox views and the existing task list for status views, so all inbox behaviour stays the same. **Roadmap alignment** Agent Chat (ROADMAP.md, "Agent Chat (including CEO Chat)"). All changes are behind experimental flags that are off by default. ## What Changed - **Agent Chat nav (streamlined shell):** Chat is the first row of Work, not a top-group row. Workspaces leaves the nav while Agent Chat is on. The mobile tab bar is Home · Chat · + · Tasks · Agents. The legacy shell keeps master's top-group Chat row. - **Chat rail:** `AgentConversationsSidebar` lists every eligible agent. The open chat is first, then conversations by recent activity, then the rest of the roster alphabetically. Terminated agents and agents you left are omitted unless you have history with them. The picker still marks only real conversations as "Open chat". - **Chat side panel:** a new default Tasks tab shows one card per task the agent created, was assigned, commented on, or acted on, newest first. It has the task list's filter popover and a sort control. **+ → Artifacts** shows the agent's artifacts as cards. Cards open in a new tab. Agent Chat off keeps the old Artifacts tab. - **Artifacts API:** `GET /api/companies/:companyId/artifacts` accepts `agentId`. The filter applies to documents, work products, and attachments by the agent each result is attributed to. The shared validator and the UI client carry the new parameter, and the OpenAPI entry picks it up from the shared schema. - **Combined Inbox + Task List flag (`enableCombinedInboxTasks`, off by default):** new card in Settings > Experimental. The Inbox row goes away and its badge moves to Tasks. A Views menu on `/issues` covers Mine, Unread, Blocked, Recent, Everything, All, Active, Backlog, and Done. Bare `/issues` opens the last-used view (default Mine). Links that carry `assignee`, `workspace`, `participantAgentId`, or `q` open All so the filter is kept. `/inbox/*` and `/issues/{all,active,backlog,done,recent}` redirect to the matching view. `/inbox/requests` stays its own page. - **Task detail breadcrumb:** the view key now decides the source, so quick-archive still works after a reload from an inbox view. - **Docs:** `doc/PRODUCT.md` and `doc/SPEC.md` describe the chat rail, the chat side panel, and the new flag. ## Verification - `cd ui && npx vitest run --no-file-parallelism src/components/chat src/components/task-side-panel/TaskSidePanel.test.tsx src/components/AgentConversationsSidebar.test.tsx src/components/Sidebar.test.tsx src/components/SidebarCompanyMenu.test.tsx src/components/Layout.test.tsx src/pages/AgentChats.test.tsx src/pages/InstanceExperimentalSettings.test.tsx src/lib/task-views.test.ts src/lib/issueDetailBreadcrumb.test.ts src/pages/Inbox.test.tsx src/pages/Issues.test.tsx src/App.test.tsx src/App.activity-routing.test.tsx src/components/MobileBottomNav.test.tsx src/components/CommandPalette.test.tsx`: 20 files, 356 tests pass. - `cd server && npx vitest run src/__tests__/company-artifacts-service.test.ts`: 13/13 pass, including the new agent-filter test across all three artifact sources. - The new rail test fails against the unmodified rail. - `pnpm check:token-gates`: all gates clean. - Manual: enable Agent Chat in Settings > Experimental. Open Chat. The rail lists all agents. Open a chat. The side panel shows the agent's tasks. Use **+ → Artifacts** to see the agent's artifacts. Then enable Combined Inbox + Task List. The Inbox row goes away, and Tasks shows a Views menu. - Snapshot baselines are intentionally not updated. See `doc/design/DECISION-SHEET.md`, "Per-change snapshot verification demoted to dormant (Jul 13 2026)". ## Risks - With both flags off, the app behaves like master. The only exception is the API: it accepts a new optional query parameter. - With Agent Chat on, the rail can list many agents in a large company. It uses the agent list the app already loads, and search filters it. - The Tasks panel reads at most 200 recently updated tasks per agent and says so when it reaches the limit. The Artifacts panel reads at most 500 of the agent's artifacts. - Combined Inbox + Task List changes what bare `/issues` opens for people who enable it. Deep links with a task filter still open All. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, through Claude Code with tool use (shell, file edit, test runs). Extended thinking was enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: scotttong Co-authored-by: Claude Opus 5.5 --- doc/PRODUCT.md | 26 +- doc/SPEC.md | 2 +- packages/shared/src/feature-catalog.ts | 11 +- packages/shared/src/types/instance.ts | 2 + packages/shared/src/validators/artifact.ts | 2 + packages/shared/src/validators/instance.ts | 1 + .../company-artifacts-service.test.ts | 26 ++ .../instance-settings-service.test.ts | 1 + server/src/services/company-artifacts.ts | 6 + server/src/services/instance-settings.ts | 2 + tests/e2e/agent-chat-sessions.spec.ts | 12 +- ui/src/App.tsx | 78 ++++- ui/src/api/artifacts.ts | 3 + .../components/AgentConversationSidebar.tsx | 6 +- .../AgentConversationsSidebar.test.tsx | 29 +- .../components/AgentConversationsSidebar.tsx | 14 +- ui/src/components/CommandPalette.tsx | 6 +- ui/src/components/IssuesList.tsx | 62 +++- ui/src/components/MobileBottomNav.tsx | 36 +- ui/src/components/Sidebar.test.tsx | 121 ++++++- ui/src/components/Sidebar.tsx | 55 +++- ui/src/components/TaskViewsMenu.tsx | 80 +++++ .../components/chat/AgentWorkPanels.test.tsx | 128 ++++++++ ui/src/components/chat/AgentWorkPanels.tsx | 308 ++++++++++++++++++ .../task-side-panel/TaskSidePanel.test.tsx | 90 +++++ .../task-side-panel/TaskSidePanel.tsx | 47 ++- .../useCombinedInboxTasksEnabled.test.ts | 14 + ui/src/hooks/useCombinedInboxTasksEnabled.ts | 47 +++ ui/src/lib/issueDetailBreadcrumb.ts | 15 +- ui/src/lib/queryKeys.ts | 4 + ui/src/lib/task-side-panel-state.ts | 7 + ui/src/lib/task-views.test.ts | 97 ++++++ ui/src/lib/task-views.ts | 152 +++++++++ ui/src/pages/Inbox.tsx | 54 ++- .../InstanceExperimentalSettings.test.tsx | 15 + ui/src/pages/InstanceExperimentalSettings.tsx | 14 +- ui/src/pages/Issues.tsx | 102 +++++- 37 files changed, 1590 insertions(+), 85 deletions(-) create mode 100644 ui/src/components/TaskViewsMenu.tsx create mode 100644 ui/src/components/chat/AgentWorkPanels.test.tsx create mode 100644 ui/src/components/chat/AgentWorkPanels.tsx create mode 100644 ui/src/hooks/useCombinedInboxTasksEnabled.test.ts create mode 100644 ui/src/hooks/useCombinedInboxTasksEnabled.ts create mode 100644 ui/src/lib/task-views.test.ts create mode 100644 ui/src/lib/task-views.ts diff --git a/doc/PRODUCT.md b/doc/PRODUCT.md index 15ab6854106..502b1d42dd9 100644 --- a/doc/PRODUCT.md +++ b/doc/PRODUCT.md @@ -209,10 +209,26 @@ telemetry. See [Announcements](ANNOUNCEMENTS.md). ### Agent chat discovery -With Agent Chat enabled, the Chats sidebar always includes the company's -earliest-created agent, plus personal starred agents and up to four other recent -conversations. First use has the same compact rows as returning use. The compose -icon shares a column with stars and appears on hover or keyboard focus (always on -touch). It opens a company-wide name/role search, independent of sidebar membership. +With Agent Chat enabled, Chat is the first row of the Work section and opens a +secondary sidebar beside the primary nav. It lists every agent you can chat +with: the open conversation first, then your other conversations by recent +activity, then the rest of the roster alphabetically. Terminated agents and +agents you have left are omitted unless you have history with them. Search +filters by name, title, or role; **+** starts or reopens a conversation. Selecting an agent opens their persistent conversation; it does not reset history or create a task until the existing first-write flow requires one. + +Beside a conversation, the side panel opens on the agent's tasks: one card per +task the agent created, was assigned, commented on, or acted on, newest first, +with the task list's filters and a sort control. The agent's artifacts are a +second card stack available from the panel's **+** menu. Both open in a new tab +so the conversation stays open. + +### Combined Inbox + Task List + +An opt-in experimental setting (`enableCombinedInboxTasks`, off by default) +folds Inbox into Tasks. The Inbox nav row goes away and its unread badge moves +to Tasks. A Views menu on the task list covers the inbox views (Mine, Unread, +Blocked, Recent, Everything) and the task-status views (All, Active, Backlog, +Done). Bare `/issues` opens the last-used view, defaulting to Mine; links that +carry a task filter open All. Old `/inbox` links redirect to the matching view. diff --git a/doc/SPEC.md b/doc/SPEC.md index ab162e3df95..d2003e10805 100644 --- a/doc/SPEC.md +++ b/doc/SPEC.md @@ -299,7 +299,7 @@ should name the rejected action and the specific restriction. There is no separate messaging or chat system. Tasks are the communication channel. This keeps all context attached to the work it relates to and creates a natural audit trail. -Experimental Agent Chat presents one persistent task per person and agent as a simplified conversation. Chat has a searchable secondary sidebar with agent avatars; adding an agent starts or reopens their single conversation. It retains the task composer, transcript, tools, attachments, documents, and existing Subtasks panel, with ordinary company visibility. New execution tasks are ordinary project tasks, not children of the conversation. Idle conversations wait for a message without entering execution-task work queues. Agents clarify goals here and create assigned tasks for substantial execution. `/new` resets provider context at an ordered session boundary within the same task while preserving visible history. `enableAgentChat` is disabled by default; the V1 lifecycle and rollout contract is specified in `SPEC-implementation.md`. +Experimental Agent Chat presents one persistent task per person and agent as a simplified conversation. Chat has a searchable secondary sidebar with agent avatars that lists every eligible agent, conversations first; adding an agent starts or reopens their single conversation. It retains the task composer, transcript, tools, attachments, documents, and existing Subtasks panel, with ordinary company visibility. New execution tasks are ordinary project tasks, not children of the conversation. Idle conversations wait for a message without entering execution-task work queues. Agents clarify goals here and create assigned tasks for substantial execution. `/new` resets provider context at an ordered session boundary within the same task while preserving visible history. `enableAgentChat` is disabled by default; the V1 lifecycle and rollout contract is specified in `SPEC-implementation.md`. ### Question recipients diff --git a/packages/shared/src/feature-catalog.ts b/packages/shared/src/feature-catalog.ts index f2c404a3bd8..457172b2fa2 100644 --- a/packages/shared/src/feature-catalog.ts +++ b/packages/shared/src/feature-catalog.ts @@ -147,7 +147,16 @@ export const INSTANCE_FEATURE_CATALOG: Record { expect(afterPrimaryCursor.artifacts.map((artifact) => artifact.title)).toEqual(["notes.txt"]); }); + it("filters every artifact source by the agent it is attributed to", async () => { + const { companyId } = await seedArtifacts(); + const writerId = "21212121-2121-4121-8121-212121212121"; + await db.insert(agents).values({ id: writerId, companyId, name: "Writer", role: "engineer" }); + await db.update(documents).set({ createdByAgentId: writerId }) + .where(eq(documents.id, "cccccccc-cccc-4ccc-8ccc-cccccccccccc")); + const storage = createStorageService({ "notes.txt": Buffer.from("notes") }); + + const writer = await companyArtifactsService(db, storage).list(companyId, { agentId: writerId, limit: 20 }); + expect(writer.artifacts.map((artifact) => artifact.title)).toEqual(["Review Notes"]); + + const original = await companyArtifactsService(db, storage).list(companyId, { + agentId: "33333333-3333-4333-8333-333333333333", + limit: 20, + }); + expect(original.artifacts.map((artifact) => artifact.title)).toEqual(["direct-video.mp4", "Primary Cut", "notes.txt"]); + expect(new Set(original.artifacts.map((artifact) => artifact.createdByAgent?.id))) + .toEqual(new Set(["33333333-3333-4333-8333-333333333333"])); + + const foreign = await companyArtifactsService(db, storage).list(companyId, { + agentId: "44444444-4444-4444-8444-444444444444", + limit: 20, + }); + expect(foreign.artifacts).toEqual([]); + }); + it("deduplicates work product attachments beyond the work product fetch window", async () => { const { companyId, projectId, issueId } = await seedArtifacts(); const dedupedAttachmentId = "abababab-abab-4bab-8bab-abababababab"; diff --git a/server/src/__tests__/instance-settings-service.test.ts b/server/src/__tests__/instance-settings-service.test.ts index 31b6d1751ba..32ce6a2c01d 100644 --- a/server/src/__tests__/instance-settings-service.test.ts +++ b/server/src/__tests__/instance-settings-service.test.ts @@ -43,6 +43,7 @@ describe("instance settings service", () => { enableApps: true, enableMcpAggregators: true, enableAgentChat: false, + enableCombinedInboxTasks: false, enableChatConnectors: false, enableMemoryConnectors: false, enableConferenceRoomChat: false, diff --git a/server/src/services/company-artifacts.ts b/server/src/services/company-artifacts.ts index 8b75a1015f5..621e90ee037 100644 --- a/server/src/services/company-artifacts.ts +++ b/server/src/services/company-artifacts.ts @@ -148,6 +148,7 @@ function buildArtifactsGroupHref( params.set("groupIssueId", groupIssueId); if (query.kind !== "all") params.set("kind", query.kind); if (query.projectId) params.set("projectId", query.projectId); + if (query.agentId) params.set("agentId", query.agentId); if (query.q) params.set("q", query.q); return `/${encodeURIComponent(companyPrefix)}/artifacts?${params.toString()}`; } @@ -374,6 +375,7 @@ export function companyArtifactsService(db: Db, storage?: StorageService) { if (documentCursor) documentConditions.push(documentCursor); if (groupBy === "task" && query.groupIssueId) documentConditions.push(eq(issues.id, query.groupIssueId)); if (query.projectId) documentConditions.push(eq(issues.projectId, query.projectId)); + if (query.agentId) documentConditions.push(sql`coalesce(${createdAgent.id}, ${updatedAgent.id}) = ${query.agentId}`); if (q) { documentConditions.push(sql`( coalesce(${documents.title}, '') ILIKE ${q} ESCAPE '\\' @@ -502,6 +504,9 @@ export function companyArtifactsService(db: Db, storage?: StorageService) { workProductBaseConditions.push(projectCondition); workProductConditions.push(projectCondition); } + // Agent filter applies to the listed rows only; the attachment dedupe + // below stays company-wide so a work product's file never re-lists. + if (query.agentId) workProductConditions.push(eq(workProductAgent.id, query.agentId)); if (q) { const searchCondition = sql`( ${issueWorkProducts.title} ILIKE ${q} ESCAPE '\\' @@ -627,6 +632,7 @@ export function companyArtifactsService(db: Db, storage?: StorageService) { if (groupBy === "task" && query.groupIssueId) attachmentConditions.push(eq(issues.id, query.groupIssueId)); if (attachmentKind) attachmentConditions.push(attachmentKind); if (query.projectId) attachmentConditions.push(eq(issues.projectId, query.projectId)); + if (query.agentId) attachmentConditions.push(eq(attachmentAgent.id, query.agentId)); if (q) { attachmentConditions.push(sql`( coalesce(${assets.originalFilename}, '') ILIKE ${q} ESCAPE '\\' diff --git a/server/src/services/instance-settings.ts b/server/src/services/instance-settings.ts index 2e7d555f0c2..1191d7afab3 100644 --- a/server/src/services/instance-settings.ts +++ b/server/src/services/instance-settings.ts @@ -237,6 +237,7 @@ export function normalizeExperimentalSettings(raw: unknown): InstanceExperimenta enablePipelines: parsed.data.enablePipelines ?? false, enableCases: parsed.data.enableCases ?? false, enableAgentChat: parsed.data.enableAgentChat ?? false, + enableCombinedInboxTasks: parsed.data.enableCombinedInboxTasks ?? false, enableConferenceRoomChat: parsed.data.enableConferenceRoomChat ?? false, enableClassicTaskInterface: parsed.data.enableClassicTaskInterface ?? false, enableIssuePlanDecompositions: parsed.data.enableIssuePlanDecompositions ?? false, @@ -280,6 +281,7 @@ export function normalizeExperimentalSettings(raw: unknown): InstanceExperimenta enablePipelines: false, enableCases: false, enableAgentChat: false, + enableCombinedInboxTasks: false, enableConferenceRoomChat: false, enableClassicTaskInterface: false, enableIssuePlanDecompositions: false, diff --git a/tests/e2e/agent-chat-sessions.spec.ts b/tests/e2e/agent-chat-sessions.spec.ts index 185d78548ac..44a6084b1bc 100644 --- a/tests/e2e/agent-chat-sessions.spec.ts +++ b/tests/e2e/agent-chat-sessions.spec.ts @@ -264,7 +264,9 @@ test("secondary chat navigation preserves layout, unique conversations, history, await page.getByRole("link", { name: "Chat", exact: true }).click(); await expect(page.getByRole("heading", { name: "Who would you like to talk to?" })).toBeVisible(); const sidebar = page.getByRole("complementary", { name: "Chat", exact: true }); - await expect(sidebar.getByText("No chats yet", { exact: true })).toBeVisible(); + // The rail lists every eligible agent before any conversation exists. + const nav = sidebar.getByRole("navigation", { name: "Agent conversations" }); + await expect(nav.getByRole("link")).toHaveText([/^Alpha/, /^Beta/, /^Delta/, /^Epsilon/, /^Gamma/, /^Zeta/]); const landingBounds = await sidebar.boundingBox(); const compose = sidebar.getByRole("button", { name: "Add chat", exact: true }); const picker = page.getByRole("dialog", { name: "Chat with an agent", exact: true }); @@ -298,10 +300,10 @@ test("secondary chat navigation preserves layout, unique conversations, history, await picker.getByRole("combobox").press("Enter"); await expect(picker).not.toBeVisible(); await expect(page.getByRole("link", { name: "Configure Alpha", exact: true })).toBeVisible(); - const nav = sidebar.getByRole("navigation", { name: "Agent conversations" }); const alphaLink = nav.getByRole("link", { name: /^Alpha / }); const betaLink = nav.getByRole("link", { name: /^Beta / }); - await expect(nav.getByRole("link")).toHaveCount(2); + // Open chat first, then other conversations, then the rest alphabetically. + await expect(nav.getByRole("link")).toHaveText([/^Alpha/, /^Beta/, /^Delta/, /^Epsilon/, /^Gamma/, /^Zeta/]); const editor = page.getByTestId("task-chat-composer-input").locator('[contenteditable="true"]'); await editor.fill("Unsent draft for Alpha"); await betaLink.click(); @@ -316,10 +318,10 @@ test("secondary chat navigation preserves layout, unique conversations, history, await expect(nav.getByRole("link")).toHaveCount(1); await expect(betaLink).toBeVisible(); await search.press("Escape"); - await expect(nav.getByRole("link")).toHaveCount(2); + await expect(nav.getByRole("link")).toHaveCount(6); await page.reload(); await expect(editor).toContainText("Unsent draft for Alpha"); - await expect(nav.getByRole("link")).toHaveCount(2); + await expect(nav.getByRole("link")).toHaveCount(6); await page.getByRole("link", { name: "Chat", exact: true }).click(); await expect(page.getByRole("link", { name: "Configure Alpha", exact: true })).toBeVisible(); diff --git a/ui/src/App.tsx b/ui/src/App.tsx index 706991339c3..1b1b1eeed32 100644 --- a/ui/src/App.tsx +++ b/ui/src/App.tsx @@ -54,6 +54,7 @@ import { ApprovalDetail } from "./pages/ApprovalDetail"; import { CompanyActivity } from "./pages/audit/CompanyActivity"; import { AuditHub } from "./pages/audit/AuditHub"; import { Inbox } from "./pages/Inbox"; +import { useCombinedInboxTasksEnabled } from "./hooks/useCombinedInboxTasksEnabled"; import { WhatNeedsMe } from "./pages/WhatNeedsMe"; import { DecisionQueuePage } from "./pages/DecisionQueuePage"; import { BoardChat } from "./pages/BoardChat"; @@ -104,6 +105,7 @@ import { NotFoundPage } from "./pages/NotFound"; import { useCompany } from "./context/CompanyContext"; import { useDialogActions, useDialogState } from "./context/DialogContext"; import { loadLastInboxTab } from "./lib/inbox"; +import { TASK_VIEW_PARAM, taskViewForInboxTab, type TaskViewKey } from "./lib/task-views"; import { isOnboardingWizardActive, onboardingStepForCompany, @@ -145,7 +147,9 @@ function ProductionSurface({ children }: { children: ReactNode }) { return }>{children}; } -function boardRoutes(streamlinedUiEnabled: boolean) { +function boardRoutes(streamlinedUiEnabled: boolean, combinedInboxTasksEnabled: boolean) { + // Combined Inbox + Task List (PAP-670) only exists in the streamlined shell. + const mergedTasks = streamlinedUiEnabled && combinedInboxTasksEnabled; return ( <> } /> @@ -303,11 +307,24 @@ function boardRoutes(streamlinedUiEnabled: boolean) { } /> } /> } /> - } /> - } /> - } /> - } /> - } /> + {mergedTasks ? ( + <> + {/* Combined Inbox + Task List: the status presets are real views, not aliases of /issues. */} + } /> + } /> + } /> + } /> + } /> + + ) : ( + <> + } /> + } /> + } /> + } /> + } /> + + )} } /> } /> } /> @@ -423,14 +440,32 @@ function boardRoutes(streamlinedUiEnabled: boolean) { ) : null} } /> } /> - } /> - } /> - } /> - } /> - } /> - } /> + {/* Combined Inbox + Task List: Inbox is a view inside Tasks. Every /inbox/* URL still + resolves — it redirects into the matching view — and with the flag off + (or in the legacy shell) the standalone pages stay. /inbox/requests + stays its own page either way; Settings → Members links straight to it. */} + {mergedTasks ? ( + <> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + + ) : ( + <> + } /> + } /> + } /> + } /> + } /> + } /> + } /> + + )} } /> - } /> } /> } /> } /> @@ -454,7 +489,19 @@ function AppsConnectEntryRoute({ } function InboxRootRedirect() { - return ; + const { enabled: streamlinedUiEnabled } = useStreamlinedUiEnabled(); + const { enabled: combinedInboxTasksEnabled } = useCombinedInboxTasksEnabled(); + return streamlinedUiEnabled && combinedInboxTasksEnabled + ? + : ; +} + +/** Sends a retired Inbox/Tasks URL to its view on the merged Tasks surface. */ +function TaskViewRedirect({ view }: { view: TaskViewKey }) { + const location = useLocation(); + const params = new URLSearchParams(location.search); + params.set(TASK_VIEW_PARAM, view); + return ; } function LegacySkillStudioRedirect() { @@ -745,6 +792,7 @@ function NoCompaniesStartPage() { export function App() { const { enabled: streamlinedUiEnabled, loaded: streamlinedUiLoaded } = useStreamlinedUiEnabled(); + const { enabled: combinedInboxTasksEnabled } = useCombinedInboxTasksEnabled(); return ( <> @@ -835,7 +883,7 @@ export function App() { } /> } /> : }> - {boardRoutes(streamlinedUiEnabled)} + {boardRoutes(streamlinedUiEnabled, combinedInboxTasksEnabled)} } /> diff --git a/ui/src/api/artifacts.ts b/ui/src/api/artifacts.ts index e0653245d38..7d7fa0e157f 100644 --- a/ui/src/api/artifacts.ts +++ b/ui/src/api/artifacts.ts @@ -34,6 +34,8 @@ export type ArtifactKindFilter = Exclude | "a export interface ListArtifactsParams { kind?: ArtifactKindFilter; projectId?: string; + /** Only artifacts attributed to this agent. */ + agentId?: string; q?: string; /** Grouping mode. `none` (default) returns the flat artifact grid. */ groupBy?: CompanyArtifactGroupBy; @@ -47,6 +49,7 @@ function buildArtifactsQuery(params?: ListArtifactsParams): string { const search = new URLSearchParams(); if (params?.kind && params.kind !== "all") search.set("kind", params.kind); if (params?.projectId) search.set("projectId", params.projectId); + if (params?.agentId) search.set("agentId", params.agentId); if (params?.q) search.set("q", params.q); if (params?.groupBy && params.groupBy !== "none") search.set("groupBy", params.groupBy); if (params?.groupIssueId) search.set("groupIssueId", params.groupIssueId); diff --git a/ui/src/components/AgentConversationSidebar.tsx b/ui/src/components/AgentConversationSidebar.tsx index 1d32fee7dad..a500d2e863d 100644 --- a/ui/src/components/AgentConversationSidebar.tsx +++ b/ui/src/components/AgentConversationSidebar.tsx @@ -10,6 +10,8 @@ import { agentRouteRef, cn } from "@/lib/utils"; export interface AgentConversationSidebarProps { agents: Agent[]; + /** Agents with a saved conversation; defaults to every listed agent. */ + existingChatAgentIds?: readonly string[]; availableAgents?: Agent[]; activeId?: string; previews?: Record; @@ -26,7 +28,7 @@ export interface AgentConversationSidebarProps { } /** Searchable navigation for one conversation per agent. */ -export function AgentConversationSidebar({ agents, availableAgents = agents, activeId, previews = {}, loading = false, error, onRetry, initialSearch = "", onSelect, onBrowse, onAddChat, historyLoading, historyError, onRetryHistory }: AgentConversationSidebarProps) { +export function AgentConversationSidebar({ agents, existingChatAgentIds, availableAgents = agents, activeId, previews = {}, loading = false, error, onRetry, initialSearch = "", onSelect, onBrowse, onAddChat, historyLoading, historyError, onRetryHistory }: AgentConversationSidebarProps) { const navigate = useNavigate(); const [pickerOpen, setPickerOpen] = useState(false); const [search, setSearch] = useState(initialSearch); @@ -94,7 +96,7 @@ export function AgentConversationSidebar({ agents, availableAgents = agents, act agent.id)} loading={loading} error={error} onRetry={onRetry} + existingChatAgentIds={existingChatAgentIds ?? agents.map(agent => agent.id)} loading={loading} error={error} onRetry={onRetry} renderAgentIcon={agent => } onSelect={async (agent, signal) => { if (onAddChat) await onAddChat(agent, signal); diff --git a/ui/src/components/AgentConversationsSidebar.test.tsx b/ui/src/components/AgentConversationsSidebar.test.tsx index 9f442f39036..b45eeceda84 100644 --- a/ui/src/components/AgentConversationsSidebar.test.tsx +++ b/ui/src/components/AgentConversationsSidebar.test.tsx @@ -18,6 +18,10 @@ vi.mock("@/api/agents", () => ({ agentsApi: { list: vi.fn(), get: state.getAgent vi.mock("@/api/agentChats", () => ({ agentChatsApi: { list: vi.fn(), ensure: state.ensure } })); vi.mock("@/api/auth", () => ({ authApi: { getSession: () => ({ user: { id: "user-a" } }) } })); vi.mock("./AgentAvatar", () => ({ AgentAvatar: () => null })); +const memberships = vi.hoisted(() => ({ agentMemberships: {} as Record })); +vi.mock("@/api/resourceMemberships", () => ({ resourceMembershipsApi: { + listMine: async () => ({ projectMemberships: {}, agentMemberships: memberships.agentMemberships, starredProjectIds: [], starredAgentIds: [], starredDocumentIds: [] }), +} })); let root: Root; let container: HTMLDivElement; let client: QueryClient; @@ -36,6 +40,7 @@ beforeEach(() => { state.companyId = "company-a"; state.navigate.mockReset(); state.closeSidebar.mockReset(); state.ensure.mockReset(); state.getAgent.mockReset(); state.ensure.mockImplementation(async (_company, id) => chat(id)); + memberships.agentMemberships = {}; client = new QueryClient({ defaultOptions: { queries: { retry: false, staleTime: Infinity } } }); client.setQueryData(queryKeys.auth.session, { user: { id: "user-a" } }); client.setQueryData(queryKeys.agents.list("company-a"), roster); @@ -45,8 +50,10 @@ beforeEach(() => { afterEach(async () => { await act(async () => root.unmount()); client.clear(); container.remove(); vi.unstubAllGlobals(); }); it("adds through the server and reopens one cached conversation without a duplicate row", async () => { await render(); - expect(container.querySelectorAll('nav[aria-label="Agent conversations"] a')).toHaveLength(1); - await addChat(); await choose("bob"); + expect(container.querySelectorAll('nav[aria-label="Agent conversations"] a')).toHaveLength(2); + await addChat(); + expect([...document.querySelectorAll("[role=option]")].find(option => option.textContent?.startsWith("bob"))?.textContent).not.toContain("Open chat"); + await choose("bob"); expect(state.ensure).toHaveBeenCalledWith("company-a", "bob"); expect(state.navigate).toHaveBeenLastCalledWith("/chats/bob"); expect(state.closeSidebar).toHaveBeenCalledWith(false); @@ -135,3 +142,21 @@ it("keeps healthy chats usable while a historical lookup fails and retries", asy expect(container.querySelector('[role="alert"]')).toBeNull(); expect(container.querySelector('a[href="/chats/alice"]')).not.toBeNull(); }); + +it("lists every eligible agent: open chat, then conversations by recency, then the rest alphabetically", async () => { + const more = [ + { ...roster[0], id: "zed", name: "Zed", urlKey: "zed" }, + { ...roster[0], id: "carol", name: "carol", urlKey: "carol" }, + { ...roster[0], id: "gone", name: "Gone", urlKey: "gone", status: "terminated" }, + { ...roster[0], id: "left", name: "Left", urlKey: "left" }, + ] as Agent[]; + memberships.agentMemberships = { left: "left" }; + client.setQueryData(queryKeys.agents.list("company-a"), [...more, ...roster]); + client.setQueryData(queryKeys.agentChats.list("company-a", "user-a"), [chat("bob")]); + await render(); + await vi.waitFor(async () => { + await act(async () => {}); + expect([...container.querySelectorAll('nav[aria-label="Agent conversations"] a')].map(link => link.getAttribute("href"))) + .toEqual(["/chats/alice", "/chats/bob", "/chats/carol", "/chats/zed"]); // the open chat (alice) is pinned first + }); +}); diff --git a/ui/src/components/AgentConversationsSidebar.tsx b/ui/src/components/AgentConversationsSidebar.tsx index 049de33ff4f..bb173be4a17 100644 --- a/ui/src/components/AgentConversationsSidebar.tsx +++ b/ui/src/components/AgentConversationsSidebar.tsx @@ -6,12 +6,14 @@ import { agentRouteRef } from "@/lib/utils"; import { useQueries } from "@tanstack/react-query"; import { agentsApi } from "@/api/agents"; import { queryKeys } from "@/lib/queryKeys"; +import { resourceMembershipState, useResourceMemberships } from "@/hooks/useResourceMemberships"; /** Account- and company-scoped data for the secondary chat navigation. */ export function AgentConversationsSidebar() { const { companyId, userId, agents, chats, session } = useAgentChatNavigation(); const openChat = useOpenAgentChat(companyId, userId); const navigate = useNavigate(); + const memberships = useResourceMemberships(companyId); const { isMobile, setSidebarOpen } = useSidebar(); const { pathname } = useLocation(); const activeRef = pathname.split("/chats/")[1]?.split("/")[0]; @@ -30,9 +32,19 @@ export function AgentConversationsSidebar() { return agent ? [agent] : []; }); if (active && !conversations.some(agent => agent.id === active.id)) conversations.unshift(active); + // Everyone you can chat with is listed, not only agents you've already + // talked to: conversations first by recent activity, then the rest of the + // roster alphabetically. Same eligibility rule as the Agents nav section — + // terminated agents and agents you've left are omitted. + const listedIds = new Set(conversations.map(agent => agent.id)); + const teammates = roster + .filter(agent => !listedIds.has(agent.id) && agent.status !== "terminated" + && (!memberships.isSuccess || resourceMembershipState(memberships.data, "agent", agent.id) !== "left")) + .sort((left, right) => left.name.localeCompare(right.name, undefined, { sensitivity: "base" })); const previews = Object.fromEntries((chats.data ?? []).filter(chat => chat.conversationState === "active") .map(chat => [chat.conversationAgentId!, "Working…"])); - return agent.id)} activeId={active?.id} previews={previews} loading={agents.isPending || chats.isPending || session.isPending} error={agents.error ?? chats.error ?? session.error} diff --git a/ui/src/components/CommandPalette.tsx b/ui/src/components/CommandPalette.tsx index 25f3d2ace6d..85dc925a18a 100644 --- a/ui/src/components/CommandPalette.tsx +++ b/ui/src/components/CommandPalette.tsx @@ -1,4 +1,5 @@ import { AgentIdentity } from "@/components/AgentIdentity"; +import { useCombinedInboxTasksEnabled } from "@/hooks/useCombinedInboxTasksEnabled"; import { useState, useEffect, useMemo } from "react"; import { useLocation, useNavigate } from "@/lib/router"; import { useQuery } from "@tanstack/react-query"; @@ -103,6 +104,7 @@ export function CommandPalette() { queryFn: () => instanceSettingsApi.getExperimental(), retry: false, }); + const { enabled: combinedInboxTasksEnabled } = useCombinedInboxTasksEnabled(); const fileViewerEnabled = experimentalSettings?.enableExperimentalFileViewer === true; useEffect(() => { @@ -358,9 +360,9 @@ export function CommandPalette() { Dashboard - go("/inbox")}> + go(combinedInboxTasksEnabled ? "/issues?view=mine" : "/inbox")}> - Inbox + {combinedInboxTasksEnabled ? "My work" : "Inbox"} go("/issues")}> diff --git a/ui/src/components/IssuesList.tsx b/ui/src/components/IssuesList.tsx index f5718914b45..da467d40fda 100644 --- a/ui/src/components/IssuesList.tsx +++ b/ui/src/components/IssuesList.tsx @@ -250,14 +250,16 @@ function getInitialWorkspaceViewState( initialAssignees?: string[], initialWorkspaces?: string[], defaultSortField?: IssueSortField, + initialStatuses?: string[], ): IssueViewState { const initial = getInitialViewState(stored, initialAssignees, defaultSortField); - if (!initialWorkspaces) return initial; - return { - ...initial, - workspaces: initialWorkspaces, - statuses: [], - }; + const scoped = initialWorkspaces + ? { ...initial, workspaces: initialWorkspaces, statuses: [] } + : initial; + // A status preset (Active / Backlog / Done, and All as the empty set) is the + // view's definition, so it wins over whatever the last session persisted. + // `undefined` means "no preset" and leaves the stored statuses alone. + return initialStatuses ? { ...scoped, statuses: initialStatuses } : scoped; } function getIssueColumnsStorageKey(key: string): string { @@ -471,6 +473,12 @@ interface IssuesListProps { issueLinkState?: unknown; initialAssignees?: string[]; initialWorkspaces?: string[]; + /** + * Status preset applied on entry and whenever it changes, overriding the + * persisted status filter. `[]` clears it; `undefined` leaves it alone. + * Used by the Tasks view presets (PAP-670). + */ + initialStatuses?: string[]; initialSearch?: string; searchFilters?: Omit; searchWithinLoadedIssues?: boolean; @@ -495,6 +503,11 @@ interface IssuesListProps { rowPresentation?: IssueRowPresentation; /** Opt in per surface while the shared collection toolbar rolls out. */ toolbarPresentation?: "legacy" | "collection"; + /** + * Rendered before the create button in the toolbar's context slot — the hook + * the merged Tasks surface uses to put its Views control there (PAP-670). + */ + toolbarContext?: ReactNode; onUpdateIssue: (id: string, data: Record) => void; } @@ -710,6 +723,7 @@ function StreamlinedIssuesList({ issueLinkState, initialAssignees, initialWorkspaces, + initialStatuses, initialSearch, searchFilters, searchWithinLoadedIssues = false, @@ -726,6 +740,7 @@ function StreamlinedIssuesList({ onLoadMoreIssues, onSearchChange, rowPresentation = "legacy", + toolbarContext, toolbarPresentation = "legacy", onUpdateIssue, }: IssuesListProps) { @@ -788,6 +803,7 @@ function StreamlinedIssuesList({ }; const initialAssigneesKey = initialAssignees?.join("|") ?? ""; const initialWorkspacesKey = initialWorkspaces?.join("|") ?? ""; + const initialStatusesKey = initialStatuses ? `set:${initialStatuses.join("|")}` : ""; const initialPreferencesRef = useRef | null>(null); if (initialPreferencesRef.current === null) { initialPreferencesRef.current = loadIssueCollectionPreferences(preferenceLocation); @@ -795,7 +811,13 @@ function StreamlinedIssuesList({ const initialPreferences = initialPreferencesRef.current; const [viewState, setViewState] = useState(() => - getInitialWorkspaceViewState(initialPreferences, initialAssignees, initialWorkspaces, defaultSortField), + getInitialWorkspaceViewState( + initialPreferences, + initialAssignees, + initialWorkspaces, + defaultSortField, + initialStatuses, + ), ); const [assigneePickerIssueId, setAssigneePickerIssueId] = useState(null); const [assigneeSearch, setAssigneeSearch] = useState(""); @@ -812,13 +834,21 @@ function StreamlinedIssuesList({ }, [initialSearch]); // Reload view state whenever the persisted context changes. - const prevViewStateContextKey = useRef(`${scopedKey}::${initialAssigneesKey}::${initialWorkspacesKey}`); + const prevViewStateContextKey = useRef( + `${scopedKey}::${initialAssigneesKey}::${initialWorkspacesKey}::${initialStatusesKey}`, + ); useEffect(() => { - const nextContextKey = `${scopedKey}::${initialAssigneesKey}::${initialWorkspacesKey}`; + const nextContextKey = `${scopedKey}::${initialAssigneesKey}::${initialWorkspacesKey}::${initialStatusesKey}`; if (prevViewStateContextKey.current !== nextContextKey) { prevViewStateContextKey.current = nextContextKey; const preferences = loadIssueCollectionPreferences(preferenceLocation); - setViewState(getInitialWorkspaceViewState(preferences, initialAssignees, initialWorkspaces, defaultSortField)); + setViewState(getInitialWorkspaceViewState( + preferences, + initialAssignees, + initialWorkspaces, + defaultSortField, + initialStatuses, + )); setVisibleIssueColumns(preferences.columns); } }, [ @@ -827,6 +857,8 @@ function StreamlinedIssuesList({ initialAssigneesKey, initialWorkspaces, initialWorkspacesKey, + initialStatuses, + initialStatusesKey, defaultSortField, preferenceLocation.companyId, preferenceLocation.collectionKey, @@ -1726,7 +1758,15 @@ function StreamlinedIssuesList({ + {toolbarContext} + + + ) : ( + + + {TASK_VIEW_GROUPS.map((group, groupIndex) => ( +
+ {groupIndex > 0 ? : null} + + {group.label} + {groupIndex === 0 && badgeCount != null && badgeCount > 0 ? ( + + {badgeCount > 99 ? "99+" : badgeCount} + + ) : null} + + {group.views.map((view) => { + const selected = view.key === value; + return ( + onChange(view.key)} + className="items-start gap-2" + aria-current={selected ? "true" : undefined} + > + + ); + })} +
+ ))} + + + ); +} diff --git a/ui/src/components/chat/AgentWorkPanels.test.tsx b/ui/src/components/chat/AgentWorkPanels.test.tsx new file mode 100644 index 00000000000..d845f952984 --- /dev/null +++ b/ui/src/components/chat/AgentWorkPanels.test.tsx @@ -0,0 +1,128 @@ +// @vitest-environment jsdom + +import { act, type ReactNode } from "react"; +import { createRoot, type Root } from "react-dom/client"; +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; +import type { CompanyArtifact, Issue } from "@paperclipai/shared"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { TooltipProvider } from "@/components/ui/tooltip"; +import { AgentArtifactsPanel, AgentTasksPanel, sortAgentTasks } from "./AgentWorkPanels"; + +(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true; + +const api = vi.hoisted(() => ({ + issuesList: vi.fn(), + artifactsList: vi.fn(), +})); + +vi.mock("@/api/issues", () => ({ issuesApi: { list: api.issuesList } })); +vi.mock("@/api/artifacts", () => ({ artifactsApi: { list: api.artifactsList } })); +vi.mock("@/api/projects", () => ({ projectsApi: { list: async () => [] } })); +vi.mock("@/components/IssueFiltersPopover", () => ({ IssueFiltersPopover: () => })); +vi.mock("@/lib/router", () => ({ + Link: ({ to, children, className, target, rel }: { to: string; children: ReactNode; className?: string; target?: string; rel?: string }) => + {children}, +})); + +function task(overrides: Partial): Issue { + return { + id: "task", + identifier: "PAP-1", + title: "Task", + status: "todo", + priority: "medium", + createdAt: "2026-09-01T00:00:00.000Z", + updatedAt: "2026-09-01T00:00:00.000Z", + ...overrides, + } as Issue; +} + +function artifact(overrides: Partial): CompanyArtifact { + return { + id: "artifact", + source: "document", + mediaKind: "document", + title: "report.md", + issue: { id: "issue-1", identifier: "PAP-9", title: "Issue" }, + createdByAgent: { id: "agent-1", name: "CEO" }, + updatedAt: "2026-09-30T00:00:00.000Z", + href: "/PAP/issues/PAP-9#document-report", + ...overrides, + } as CompanyArtifact; +} + +describe("agent work panels", () => { + let root: Root; + let container: HTMLDivElement; + let queryClient: QueryClient; + + beforeEach(() => { + api.issuesList.mockReset(); + api.artifactsList.mockReset(); + container = document.createElement("div"); + document.body.appendChild(container); + root = createRoot(container); + queryClient = new QueryClient({ defaultOptions: { queries: { retry: false } } }); + }); + + afterEach(() => { + act(() => root.unmount()); + queryClient.clear(); + container.remove(); + }); + + async function render(node: ReactNode) { + await act(async () => { + root.render( + + {node} + , + ); + }); + await act(async () => { await new Promise((resolve) => setTimeout(resolve, 0)); }); + } + + it("lists the agent's tasks newest first as linked cards, without the chat itself", async () => { + api.issuesList.mockResolvedValue([ + task({ id: "old", identifier: "PAP-1", title: "Older task", updatedAt: new Date("2026-09-01T00:00:00.000Z") }), + task({ id: "chat", identifier: "PAP-2", title: "The conversation" }), + task({ id: "new", identifier: "PAP-3", title: "Newer task", status: "in_progress", updatedAt: new Date("2026-09-20T00:00:00.000Z") }), + ]); + await render(); + + expect(api.issuesList).toHaveBeenCalledWith("company-1", expect.objectContaining({ participantAgentId: "agent-1" })); + const cards = Array.from(container.querySelectorAll("a")); + expect(cards.map((card) => card.getAttribute("href"))).toEqual(["/issues/PAP-3", "/issues/PAP-1"]); + expect(cards.every((card) => card.getAttribute("target") === "_blank")).toBe(true); + expect(cards[0]?.textContent).toContain("Newer task"); + expect(cards[0]?.textContent).toContain("PAP-3"); + expect(container.querySelector("time")).not.toBeNull(); + expect(container.textContent).not.toContain("The conversation"); + }); + + it("sorts by status and title on request", () => { + const tasks = [ + task({ id: "a", title: "Bravo", status: "done" }), + task({ id: "b", title: "Alpha", status: "in_progress" }), + ]; + expect(sortAgentTasks(tasks, "status", "asc").map((item) => item.id)).toEqual(["b", "a"]); + expect(sortAgentTasks(tasks, "title", "asc").map((item) => item.id)).toEqual(["b", "a"]); + }); + + it("asks the server for the agent's artifacts and shows filename, date and task id", async () => { + api.artifactsList.mockResolvedValue({ + artifacts: [artifact({ id: "mine", title: "plan.md" })], + nextCursor: null, + }); + await render(); + + expect(api.artifactsList).toHaveBeenCalledWith("company-1", expect.objectContaining({ agentId: "agent-1" })); + const cards = Array.from(container.querySelectorAll("a")); + expect(cards).toHaveLength(1); + expect(cards[0]?.getAttribute("href")).toBe("/PAP/issues/PAP-9#document-report"); + expect(cards[0]?.getAttribute("target")).toBe("_blank"); + expect(cards[0]?.textContent).toContain("plan.md"); + expect(cards[0]?.textContent).toContain("PAP-9"); + expect(cards[0]?.textContent).toContain("Updated"); + }); +}); diff --git a/ui/src/components/chat/AgentWorkPanels.tsx b/ui/src/components/chat/AgentWorkPanels.tsx new file mode 100644 index 00000000000..2e7c6361088 --- /dev/null +++ b/ui/src/components/chat/AgentWorkPanels.tsx @@ -0,0 +1,308 @@ +import { useMemo, useState } from "react"; +import { useQuery } from "@tanstack/react-query"; +import { ArrowUpDown, File, FileText, Image, Search, Video } from "lucide-react"; +import type { CompanyArtifact, Issue } from "@paperclipai/shared"; +import { artifactsApi } from "@/api/artifacts"; +import { issuesApi } from "@/api/issues"; +import { projectsApi } from "@/api/projects"; +import { IssueFiltersPopover } from "@/components/IssueFiltersPopover"; +import { StatusIcon } from "@/components/StatusIcon"; +import { Button } from "@/components/ui/button"; +import { Input } from "@/components/ui/input"; +import { Popover, PopoverContent, PopoverTrigger } from "@/components/ui/popover"; +import { + applyIssueFilters, + countActiveIssueFilters, + defaultIssueFilterState, + issueStatusOrder, + type IssueFilterState, +} from "@/lib/issue-filters"; +import { queryKeys } from "@/lib/queryKeys"; +import { Link } from "@/lib/router"; +import { cn, formatDate, relativeTime } from "@/lib/utils"; + +/** + * The agent-scoped side panels shown beside an agent chat: every task the + * agent has worked on, and every artifact it produced. Both are card stacks + * so the chat's right column reads as "what this agent has been doing" + * rather than the properties of the conversation issue itself. + */ + +const AGENT_TASK_LIMIT = 200; +const ARTIFACT_PAGE_SIZE = 100; +const ARTIFACT_MAX_PAGES = 5; + +export type AgentTaskSortField = "updated" | "created" | "status" | "title"; + +const SORT_OPTIONS: ReadonlyArray<[AgentTaskSortField, string]> = [ + ["updated", "Last updated"], + ["created", "Created"], + ["status", "Status"], + ["title", "Title"], +]; + +export function sortAgentTasks( + tasks: Issue[], + field: AgentTaskSortField, + dir: "asc" | "desc", +): Issue[] { + const sign = dir === "asc" ? 1 : -1; + return [...tasks].sort((a, b) => { + switch (field) { + case "status": + return sign * (issueStatusOrder.indexOf(a.status) - issueStatusOrder.indexOf(b.status)); + case "title": + return sign * a.title.localeCompare(b.title); + case "created": + return sign * (new Date(a.createdAt).getTime() - new Date(b.createdAt).getTime()); + default: + return sign * (new Date(a.updatedAt).getTime() - new Date(b.updatedAt).getTime()); + } + }); +} + +function PanelMessage({ children, tone = "muted" }: { children: React.ReactNode; tone?: "muted" | "error" }) { + return ( +

+ {children} +

+ ); +} + +const cardClassName = + "flex flex-col gap-1 rounded-lg border border-border bg-card p-3 transition-colors hover:border-foreground/20 hover:bg-accent/40 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring"; + +export function AgentTaskCard({ task }: { task: Issue }) { + return ( + card; Card renders a div and would break anchor semantics + to={`/issues/${task.identifier ?? task.id}`} + disableIssueQuicklook + target="_blank" + rel="noreferrer" + data-testid="agent-task-card" + className={cardClassName} + > +
+ + + {task.title} + +
+
+ {task.identifier ?? task.id.slice(0, 8)} + · + +
+ + ); +} + +export function AgentTasksPanel({ + companyId, + agentId, + excludeIssueId, +}: { + companyId: string; + agentId: string; + /** The conversation issue itself, which is not one of the agent's tasks. */ + excludeIssueId?: string; +}) { + const [query, setQuery] = useState(""); + const [filters, setFilters] = useState(defaultIssueFilterState); + const [sortField, setSortField] = useState("updated"); + const [sortDir, setSortDir] = useState<"asc" | "desc">("desc"); + + const tasksQuery = useQuery({ + queryKey: queryKeys.issues.listParticipatedByAgent(companyId, agentId), + queryFn: () => issuesApi.list(companyId, { + participantAgentId: agentId, + sortField: "updated", + sortDir: "desc", + limit: AGENT_TASK_LIMIT, + }), + }); + const projectsQuery = useQuery({ + queryKey: queryKeys.projects.list(companyId), + queryFn: () => projectsApi.list(companyId), + }); + + const visible = useMemo(() => { + const needle = query.trim().toLowerCase(); + const scoped = (tasksQuery.data ?? []).filter((task) => { + if (task.id === excludeIssueId) return false; + if (!needle) return true; + return task.title.toLowerCase().includes(needle) + || (task.identifier ?? "").toLowerCase().includes(needle); + }); + return sortAgentTasks(applyIssueFilters(scoped, filters), sortField, sortDir); + }, [tasksQuery.data, excludeIssueId, query, filters, sortField, sortDir]); + + const activeFilterCount = countActiveIssueFilters(filters); + const total = (tasksQuery.data ?? []).filter((task) => task.id !== excludeIssueId).length; + // Search and filters run on the most recently updated tasks only. + const capped = (tasksQuery.data?.length ?? 0) >= AGENT_TASK_LIMIT; + + return ( +
+
+
+ + setQuery(event.target.value)} + placeholder="Search tasks" + aria-label="Search tasks" + className="h-8 pl-7 text-sm" + /> +
+ setFilters((current) => ({ ...current, ...patch }))} + activeFilterCount={activeFilterCount} + projects={projectsQuery.data?.map((project) => ({ id: project.id, name: project.name }))} + enableExternalObjectFilters={false} + buttonVariant="outline" + iconOnly + presentation="streamlined" + /> + + + + + +
+ {SORT_OPTIONS.map(([field, label]) => ( + + ))} +
+ + +
+ + {tasksQuery.isPending ? ( + Loading tasks… + ) : tasksQuery.isError ? ( + Could not load this agent's tasks. + ) : total === 0 ? ( + This agent hasn't worked on any tasks yet. + ) : visible.length === 0 ? ( + + {capped + ? `No tasks match these filters among the ${AGENT_TASK_LIMIT} most recently updated.` + : "No tasks match these filters."} + + ) : ( +
+ {visible.map((task) => )} + {capped ? ( +

+ Showing the {AGENT_TASK_LIMIT} most recently updated tasks. +

+ ) : null} +
+ )} +
+ ); +} + +function ArtifactKindIcon({ artifact }: { artifact: CompanyArtifact }) { + const className = "mt-0.5 h-4 w-4 shrink-0 text-muted-foreground"; + switch (artifact.mediaKind) { + case "image": return ; + case "video": return