fix: complete native chat responses and improve API discovery (#18364)

## Thinking Path

> - Paperclip must perform work and report its outcome accurately.
> - A chat response and its delegated or operator-owned work are
separate stages.
> - Native chat contracts instead required literal completion of the
user's whole request.
> - Agents kept searching or waiting even when a truthful handoff should
finish the response.
> - OpenCode also has separate acknowledgement backlog and
empty-response failure paths.
> - This PR aligns chat completion with its current stage and makes
OpenCode completion truthful.

## Linked Issues or Issue Description

Refs #18098 for retained OpenCode evidence and #18377 for separate eval
setup, oracle and reporting repairs.

**What happened?** Real pinned OpenCode 1.18.34 reproduced an MCP
timeout behind 800 text fragments. Pilot
[41222196525](https://github.com/paperclipai/paperclip/actions/runs/41222196525)
separately exposed a fabricated successful reply after a reasoning-only
response. Later restart attempts timed out without an answer. Their
native criterion literally required restarting the instance, although
the chat instructions and authority required an operator handoff.
General authority guidance alone did not resolve it.

**Expected behavior:** A standing chat completes its current authorized
response. It performs and verifies permitted actions, verifies delegated
handoffs, or explains the specific unavailable action and operator path.
The worker's task remains separate. Ordinary execution tasks must still
complete their requested work. Empty provider responses must not become
invented success.

**Version and reproduction:** Actual pinned OpenCode and native Rust
Runner, with retained full-stack AWS attempts. Product baseline master
`51ea624344`. This PR adds no historical catalog to master.

## What Changed

- Build standing-chat completion contracts from the server-owned
conversation identity. Scope each current request to the response stage;
preserve request text, criterion IDs, order, human-response scope and
source revisions. Require verified authorized actions and truthful
handoffs. Ordinary task contracts stay strict. Existing admitted inputs
retain their saved contract during recovery; new contract content uses
the existing revision/hash mechanism.
- Guide prospective defaults to their owning resource and preserve
existing work unless backfill is requested. Guide unavailable
board/operator operations to the supported handoff. These are general
chat instructions with no eval identities or expected answers.
- Batch adjacent assistant text for at most 50 ms and 4,000 UTF-8 bytes.
Preserve text, metadata and order; flush before non-text frames and
shutdown.
- Give an empty OpenCode response one correction within its existing
turn, model, reasoning, context and deadline. Accept actual completed
answer parts with incomplete headers. Preserve actual active tool work,
bound idle evidence grace and fail truthfully if correction stays empty.
- Reject empty OpenCode terminals in Rust without actual prose or an
accepted semantic result. Never invent successful replies or evidence.
Refresh generated project-discovery contracts.
- Preserve accurate public API-discovery descriptions for existing
project defaults and managed instance restart operations, including
their field scope and authority limits.

## Verification

**Qualified: [pilot
41256592937](https://github.com/paperclipai/paperclip/actions/runs/41256592937)
passes 3/3, with all 35 independent checks passing.**

| Case | Checks | Full case duration | Evidence and cleanup |
| --- | --- | --- | --- |
| Instance restart | 9/9 | 140.1 s | complete |
| Project defaults | 11/11 | 204.2 s | complete |
| Project work | 15/15 | 204.0 s | complete |

All three are original-attempt results from source
`c51e079f7f2364305b131ffd549e263ff78c5335`, containing the exact product
files from `13c2b61f31d873a475ea0c3c026b19926ff42301`. Actual saved
conversation contracts were inspected, as were ordinary native worker
contracts where present. Each protected archive includes the actual API
state and final PNG; cleanup completes. Original queries,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, finite case clocks,
concurrency one and zero eval retries are preserved. Earlier passes do
not transfer into this qualification.

Both current heads pass all checks: product **62** (56 success, one
neutral, five skipped), catalog **56** (54 success, one neutral, one
skipped). Both have exact-head Greptile **5/5**, zero unresolved review
threads and clean mergeability. The three trusted private-evidence
protocol files remain byte-identical to the dispatch workflow. Raw
evidence stays protected.

- **2,044 API discovery tests pass.** Red controls showed that “restart
instance” selected unrelated drain/settings operations and “save project
defaults” selected repository replacement. Accurate public summaries now
rank the existing operations correctly; route registration preserves
descriptions. No endpoint, permission or field behavior changes.
- **112 real-database, authenticated HTTP, native Runner and OpenAPI
consumer tests pass locally**, using disposable task-owned Docker
PostgreSQL. It was removed afterward; shared IPC was untouched. These
cover persisted contract selection, admitted native input and the model
envelope together. The isolated CI suites also pass.
- **77 focused contract/input/context tests pass.** Red controls catch
the wrong conversation predicate; canonical identity requires both agent
and user IDs, including the actual heartbeat projection. Ordinary tasks,
agent-only issues and planner tasks remain strict. Coalesced requests
preserve criterion identities, source hashes and human-response scope.
- **144 OpenCode driver tests pass**; native core **441 pass, one
ignored**; packaging **56 pass, six skipped**. Server typecheck and
build pass.
- Real pinned OpenCode/native Runner: a delayed 25-second
acknowledgement completes within the original 30-second MCP deadline.
The text-fragment control changes 817 notifications / 54.1 seconds /
timeout to 45 / 19.0 seconds / accepted finish. Separate fast,
six-minute and real authenticated API probes complete 32 tool calls plus
finish. These do not reproduce every remaining historical MCP timeout,
and no common cause is claimed for all such failures.

Retained failures remain attributed to their original source. Earlier
runs exposed fabricated empty-response success, prospective-default
backfill, a literal whole-task chat criterion, fixture readiness and
oracle/reporting defects. The immediately preceding
[41250502608](https://github.com/paperclipai/paperclip/actions/runs/41250502608),
product `c0bf249807` / source `7b1de24ce1`, is **1/3**: work 15/15;
restart 4/9 with the repaired contract actually admitted but no answer;
defaults timed out before settlement, then exceeded the unchanged 256
MiB original-evidence allowance. Defaults domain behavior on that
attempt remains inconclusive. #18377 retains the full attempt ledger,
including missing archives and pre-paid cancellations. The fresh 3/3
establishes these selected outcomes; it does not establish full-catalog
reliability.

## Risks

The shared contract change applies to standing native chats across
providers. It changes model instructions and completion scope, while
permission, approval and review gates remain enforced by their existing
owners. Incorrect model claims remain possible; the product evals
independently require actual mutations, native work and delivery. The
correction adds at most one provider request within the existing
deadline; text batching can delay adjacent text by 50 ms. Three
qualified attempts do not establish full-catalog reliability.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, tool use, code
execution and live eval tooling. Exact hosted model revision and
context-window size are not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

- [x] Final same-source three-cell pilot passes 3/3 with required
evidence and cleanup

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts

@@ -540,6 +540,12 @@ class OpenCodeHarnessSession implements HarnessSession {
   readonly #companyObservability: ReturnType<typeof createOpenCodeCompanyObservability>;
   #closed = false;
   #completionSettlement: Promise<void> | null = null;
+  #turnStartSourceSeq = 0;
+  #turnReasoningMode: OpenCodeReasoningMode = "default";
+  #emptyResponseRecovery: { turnId: string; priorMessages: Set<string>; responseStarted: boolean; responseFinished: boolean } | null = null;
+  #emptyResponseAckTimer: ReturnType<typeof setTimeout> | undefined;
+  #pendingToolCalls = 0;
+  readonly #runningNativeToolParts = new Set<string>();
   #abort = new AbortController();
 
   constructor(input: {
@@ -686,14 +692,16 @@ class OpenCodeHarnessSession implements HarnessSession {
     const turnId = `turn-${randomBytes(12).toString("hex")}`;
     this.#streamingParts.clear();
     this.#activeTurnId = turnId;
+    this.#turnStartSourceSeq = this.#sourceSequence;
+    this.#turnReasoningMode = reasoningMode;
+    this.#emptyResponseRecovery = null;
+    this.#runningNativeToolParts.clear();
     this.#emit("turn.submitted", {
       envelopeSchema: this.#taskEnvelope.schema,
       text: input.message.text,
     }, { turnId });
     this.#emit("turn.accepted", { turnId }, { turnId });
     this.#emit("turn.started", { status: "inProgress" }, { turnId });
-    const [providerID, ...modelParts] = this.#model.split("/");
-    const modelID = modelParts.join("/");
     // Keep the task envelope only on the initial task-mode wake. OpenCode
     // rebuilds system instructions from the latest user message, so system
     // instructions must still accompany every prompt (including recovery).
@@ -703,12 +711,21 @@ class OpenCodeHarnessSession implements HarnessSession {
           message: input.message.text,
         })
       : input.message.text;
+    await this.#submitPrompt(prompt);
+    this.#sendFullContext = false;
+    return { turnId };
+  }
+
+  async #submitPrompt(prompt: string): Promise<void> {
+    const [providerID, ...modelParts] = this.#model.split("/");
+    const modelID = modelParts.join("/");
     await api(
       this.#fetch,
       this.#runtime,
       `/session/${encodeURIComponent(this.#providerSessionId)}/prompt_async`,
       {
         method: "POST",
+        signal: this.#abort.signal,
         body: JSON.stringify({
           // OpenCode 1.18's PromptPayload carries the provider and model at the
           // top level.  Older examples used a nested `model` object; 1.18
@@ -721,7 +738,7 @@ class OpenCodeHarnessSession implements HarnessSession {
           // Always select the empty default variant on an ordinary turn so a
           // previous disabled turn cannot change this turn's provider defaults.
           ...(providerID === "openrouter"
-            ? { variant: reasoningMode === "disabled" ? "paperclip-no-reasoning" : "paperclip-default" }
+            ? { variant: this.#turnReasoningMode === "disabled" ? "paperclip-no-reasoning" : "paperclip-default" }
             : {}),
           // Keep question enabled in the isolated config. OpenCode 1.18.32
           // turns a prompt's deprecated `tools` map into replacement session
@@ -731,8 +748,55 @@ class OpenCodeHarnessSession implements HarnessSession {
         }),
       },
     );
-    this.#sendFullContext = false;
-    return { turnId };
+  }
+
+  #clearEmptyResponseAckTimer(): void {
+    clearTimeout(this.#emptyResponseAckTimer);
+    this.#emptyResponseAckTimer = undefined;
+  }
+
+  #armEmptyResponseAckTimer(turnId: string, message: string): void {
+    if (this.#emptyResponseAckTimer) return;
+    this.#emptyResponseAckTimer = setTimeout(() => {
+      this.#emptyResponseAckTimer = undefined;
+      this.#failEmptyResponse(turnId, message);
+    }, 20_000);
+    this.#emptyResponseAckTimer.unref();
+  }
+
+  #failEmptyResponse(turnId: string, message: string): void {
+    if (this.#closed || this.#activeTurnId !== turnId || this.#result !== null || this.#pendingToolCalls > 0 || this.#runningNativeToolParts.size > 0) return;
+    this.#emit("turn.failed", { status: "failed", error: { code: "opencode_empty_response", message, retryable: false } }, { turnId });
+    this.#terminalTurns.set(turnId, canonicalJson({ status: "failed" }));
+    this.#activeTurnId = null;
+  }
+
+  #recoverEmptyResponse(turnId: string): boolean {
+    if (this.#result !== null || this.#pendingRuntimeRequests.size > 0) return false;
+    if (this.#pendingToolCalls > 0 || this.#runningNativeToolParts.size > 0) return true;
+    const recovery = this.#emptyResponseRecovery;
+    const boundary = Math.max(this.#turnStartSourceSeq, this.#lastNonTerminalToolSourceSeq);
+    if (this.#completedTextParts.some(part => part.observedSourceSeq > boundary)) return false;
+    // Both idle event forms can remain queued from the empty response. Wait
+    // for correction evidence. A completed answer part is sufficient even
+    // when its message header omits completion metadata. An idle response
+    // with neither answer nor finish evidence gets a bounded grace period;
+    // active provider/tool work still follows the original outer deadline.
+    if (recovery && !recovery.responseFinished) {
+      if (recovery.responseStarted)
+        this.#armEmptyResponseAckTimer(turnId, "OpenCode became idle without completing its requested final response.");
+      return true;
+    }
+    if (recovery) {
+      this.#failEmptyResponse(turnId, "OpenCode ended without a final answer or accepted completion report after one correction.");
+      return true;
+    }
+    this.#emptyResponseRecovery = { turnId, priorMessages: new Set(this.#messageRoles.keys()), responseStarted: false, responseFinished: false };
+    this.#emit("harness.diagnostic", { code: "opencode_empty_response_correction", message: "Requesting one final response after an empty OpenCode completion." }, { turnId });
+    this.#armEmptyResponseAckTimer(turnId, "OpenCode did not start its requested final response.");
+    void this.#submitPrompt("Your response ended without a user-visible final answer or an accepted completion report. Continue this same request from the work already done. Give the user a concise final answer and submit paperclip_finish (or paperclip_block when blocked) with the current completion contract. Do not repeat completed actions.")
+      .catch(() => this.#failEmptyResponse(turnId, "OpenCode could not request a final response after an empty completion."));
+    return true;
   }
 
   async interrupt(input: { turnId?: string; reason?: string }): Promise<void> {
@@ -992,6 +1056,7 @@ class OpenCodeHarnessSession implements HarnessSession {
     await this.#completionSettlement;
     if (this.#closed) return;
     this.#closed = true;
+    this.#clearEmptyResponseAckTimer();
     this.#abort.abort();
     // Settle whatever is still pending, including a request whose turn
     // already went terminal without the driver observing it (see
@@ -1040,6 +1105,13 @@ class OpenCodeHarnessSession implements HarnessSession {
     callId: string;
     arguments: unknown;
   }): Promise<unknown> {
+    this.#pendingToolCalls += 1;
+    this.#clearEmptyResponseAckTimer();
+    try { return await this.#dispatchToolCall(call); }
+    finally { this.#pendingToolCalls -= 1; }
+  }
+
+  async #dispatchToolCall(call: { tool: string; callId: string; arguments: unknown }): Promise<unknown> {
     // The authenticated MCP bridge has resolved this against its closed
     // declared catalog. A second prefix strip changes legitimate operation IDs.
     const tool = call.tool;
@@ -1583,7 +1655,8 @@ class OpenCodeHarnessSession implements HarnessSession {
     // adjacent tokens. Content deduplication would silently lose the second.
     // Key id-less events by digest: a retained canonical copy of each recent
     // event would hold up to 10,000 large tool payloads in memory.
-    const eventId = text(event.id) || (type === "message.part.delta"
+    const idle = type === "session.idle" || type === "session.status" && text(record(properties.status).type) === "idle";
+    const eventId = idle ? null : text(event.id) || (type === "message.part.delta"
       ? null
       : `sha256:${createHash("sha256").update(canonicalJson(value)).digest("hex")}`);
     if (eventId !== null && this.#seenProviderEvents.has(eventId)) return;
@@ -1598,6 +1671,8 @@ class OpenCodeHarnessSession implements HarnessSession {
     );
     if (sessionId && sessionId !== this.#providerSessionId) return;
     const turnId = this.#activeTurnId;
+    if (this.#emptyResponseRecovery?.responseStarted && type === "session.status" && text(record(properties.status).type) === "busy")
+      this.#clearEmptyResponseAckTimer();
     if (type === "question.asked" && turnId) {
       this.#acceptQuestion(properties);
       return;
@@ -1701,6 +1776,11 @@ class OpenCodeHarnessSession implements HarnessSession {
         this.#streamingParts.set(text(part.id), part);
       }
       const role = this.#messageRoles.get(messageId);
+      const recovery = this.#emptyResponseRecovery;
+      if (role === "assistant" && recovery?.responseStarted && !recovery.priorMessages.has(messageId) && this.#emptyResponseAckTimer) {
+        this.#clearEmptyResponseAckTimer();
+        this.#armEmptyResponseAckTimer(owningTurnId, "OpenCode became idle without completing its requested final response.");
+      }
       if (role === "assistant") this.#emitAssistantPart(part, owningTurnId);
       else if (role === undefined) {
         const pending = this.#pendingMessageParts.get(messageId) ?? [];
@@ -1722,6 +1802,14 @@ class OpenCodeHarnessSession implements HarnessSession {
         text(info.messageID, text(info.messageId)),
       );
       const role = text(info.role);
+      const recovery = this.#emptyResponseRecovery;
+      if (recovery?.turnId === turnId && role === "assistant" && messageId && !recovery.priorMessages.has(messageId)) {
+        const toolCalls = ["tool-calls", "tool_calls"].includes(text(info.finish));
+        const finished = !!(Number.isFinite(record(info.time).completed) || text(info.finish)) && !toolCalls;
+        if (!recovery.responseStarted || finished) this.#clearEmptyResponseAckTimer();
+        recovery.responseStarted = true;
+        if (finished) recovery.responseFinished = true;
+      }
       // Record the message's owning turn at the moment OpenCode first
       // reports it. A later turn that reuses the same native message id
       // legitimately reclaims ownership; a stale message never sees this
@@ -1775,6 +1863,7 @@ class OpenCodeHarnessSession implements HarnessSession {
           text(record(properties.status).type) === "idle")) &&
       turnId
     ) {
+      if (this.#recoverEmptyResponse(turnId)) return;
       const fingerprint = canonicalJson({
         status: "completed",
         semanticResult: this.#resultFingerprint,
@@ -1865,6 +1954,12 @@ class OpenCodeHarnessSession implements HarnessSession {
     const partId = text(part.id, `${turnId}:part`);
     const partType = text(part.type, "unknown");
     const messageId = text(part.messageID, text(part.messageId)) || null;
+    if (turnId === this.#activeTurnId && ["tool", "tool-call", "tool_call"].includes(partType)) {
+      if (text(record(part.state).status) === "running") {
+        this.#runningNativeToolParts.add(partId);
+        this.#clearEmptyResponseAckTimer();
+      } else this.#runningNativeToolParts.delete(partId);
+    }
     const canonicalToolName = canonicalOpenCodeMcpToolName(
       canonicalOpenCodeDisplayToolName(
         text(part.tool, text(part.name)),
@@ -2147,7 +2242,11 @@ class OpenCodeHarnessSession implements HarnessSession {
       });
       return;
     }
+    if (eventType === "run.result.proposed" || eventType === "runtime_request.created") {
+      this.#clearEmptyResponseAckTimer();
+    }
     if (["turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"].includes(eventType)) {
+      this.#clearEmptyResponseAckTimer();
       const performance = this.#runtime.performanceDiagnostics?.take(this.#performanceBoundaryMs);
       this.#performanceBoundaryMs = this.#now().getTime();
       if (performance) this.#emit("harness.diagnostic", performance, refs);