diff --git a/doc/connections/GITHUB.md b/doc/connections/GITHUB.md index 90fa895341b..0f93999ab86 100644 --- a/doc/connections/GITHUB.md +++ b/doc/connections/GITHUB.md @@ -329,6 +329,36 @@ another recipient is named. The routing rule does not change automatic issue or event policies. Archived bots and identities from other companies or providers are excluded from the routing lookup. +## Retrying a saved request + +A normal task Retry can continue a newer saved GitHub message after an earlier +attempt failed. Execution revalidates the exact admitted successor, its current +linked principal and repository access, and the durable continuation receipt; +it does not repeat the admission that already consumed the recovery hold. +A failed native predecessor still requires its committed failed result and +exact executor termination proof. Successful results and uncertain delivery do +not grant a replay. A certified failed-chat Retry retains the exact accepted +message identity when its predecessor stopped before identity initialization, +so a published review can expose its authorized passive response wait. +If an inherited task-resume attempt already recorded a result, ordinary Retry +explains that exact replay is unavailable and directs the user to send a new +message to continue from saved work. That new turn still passes normal current +requester, access and executor-containment checks. + +If that successor failed at Retry authorization before runtime or provider +startup, the next normal Retry follows its canonical source receipt. The exact +system diagnostic, settlement events, empty execution evidence and absence of +allocated resources establish that narrow recovery path. A generic setup +failure or a claimed `providerWorkStarted: false` does not establish it. + +The canonical task Retry also recognizes an isolated-directory binding failure +at the workspace preparation boundary before allocation. Its structured reason +and task binding, trusted failed settlement, dispatch-only profile and absence +of execution resources must all agree. Other workspace validation failures do +not inherit this permission. Current request authorization still runs at both +admission and execution; a failed unadmitted Retry intent does not replace the +saved request or prevent a later deliberate Retry. + ## Run projection The resolved token is leased at run start as an audited class-3 secret and is diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 8269a7f4589..17411564865 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -521,13 +521,25 @@ of the invalid call. Unauthorized tools still receive a generic rejection, and corrected calls retain the same permission and approval checks. A native provider may omit `continuation` (or send `null`) when it reports -`yielded` after saving a monitor or requesting repository preparation. The controller reads current authorized task +`yielded` after saving a monitor, requesting repository preparation, or publishing +an authorized current-head GitHub review. The controller reads current authorized task context and fills the canonical continuation only if that same task, run, assignee, and work mode remain eligible. An eligible monitor takes precedence. A pending repository requested by that task in its current workspace offers `same_agent`; the existing status arbiter and wake queue own the next admission. Prepared repositories and requests belonging to another task cannot offer it. -An explicit `done` result does not acquire a continuation. Completion admission +A completed GitHub review with its processed publication may offer `response_wake` +for that exact head and accepted requester under current conversation and +connection authority. A normal tree resume may inherit the published review +through its accepted parent identity. An ordinary GitHub follow-up may inherit it +only after this run has a processed useful reply bound to its exact current +inbound comment; acknowledgements, pending or ambiguous writes and partial +assessment receipts do not suffice. Completion admission, finalization and +status commit recheck the server-issued run/review/head binding; a pending +publication, newer unreviewed head or revoked access cannot fall through to an +immediate continuation. An explicit malformed continuation remains invalid. +An explicit `done` result does not acquire a continuation or change its completion +policy; review scores are not interpreted as task-completion instructions. Completion admission checks that persisted monitor again. This default does not create or renew a schedule. A consumed, cleared, exhausted, or ineligible monitor cannot authorize a passive wait. An explicit `response_wake` still identifies a real response @@ -757,7 +769,7 @@ Explicit retry admission checks the local parent adapter controller and verified A new user message can continue a terminal native run whose process fields were cleared before local stop receipts existed. Admission must verify the exact run, runner, workspace, and provider session in the retained suspended state, with no active provider turn, pending tool call, or undelivered output. Missing or mismatched state keeps the hold. A later recorded process launch also keeps the hold until its stop is verified. Normal assignment, decision, controller, environment cleanup, and active-run gates still apply. The message starts one fresh conversation turn; it does not replay the failed run, reset its recovery budget, or certify unknown action outcomes. -The task thread exposes the guarded Retry action for failed or timed-out conversation runs and native preparation cancelled before provider startup when the never-started proof is verified. The server projects this eligibility on the recovery notice and rechecks it on Retry. Cleanup quarantine retains its inspection path, and non-conversation reconciliation gates remain enforced. Pending decisions, active execution, pause, budget, dependency, and ownership gates remain in force. A refused Retry reports its reason inline and remains available for another attempt. +The task thread exposes the guarded Retry action for failed or timed-out conversation runs and native preparation cancelled before provider startup when the never-started proof is verified. The server projects this eligibility on the recovery notice and rechecks it on Retry. Cleanup quarantine retains its inspection path, and non-conversation reconciliation gates remain enforced. Pending decisions, active execution, pause, budget, dependency, and ownership gates remain in force. A refused Retry reports its reason inline and remains available for another attempt. When a newer linked-human GitHub message was saved but its admission was skipped by a budget or agent-availability gate, ordinary task Retry selects that unstarted message before considering the older failed run. The existing chat task-Retry owner snapshots the complete pending batch, including eligible skipped input, into a new admission generation bound to the canonical processed deliveries, comments, principal, session generation, admitted instruction snapshots, and task Retry intent. It retains the original skipped receipt and failed result unchanged. A terminal failed publication of that old result also remains history for a distinct selected request; pending or uncertain publication still holds work, and the old output is never replayed. Board Retry refusals expose only stable allowlisted authority categories, never source payloads, identifiers, or credential fingerprints. Admission rechecks source integrity and current access, then uses the normal new-user-turn continuation proof: a committed failed result can remain history only after controller, successor, process, workspace, and exact remote-termination guards pass. Unknown ownership stays deferred; Retry cannot certify it. The existing chat retry queue owns delivery and bounded infrastructure retries; a committed receipt settles a lost acknowledgment without another execution. An accepted saved-input receipt discharges only its matching unadmitted-input gate, allowing later genuine conversation or tree-resume wakes without replaying the skipped request. A continuation cancelled by the queued-run gate while waiting for review did not start a provider. Its matching dispatch-gate receipt and unclaimed execution fields keep this deliberate wait out of unknown-action reconciliation. Pending review and decision gates still apply. If an older server already created a recovery hold for this exact state, Retry or a new user message can request one fresh turn after the server verifies there is no provider, coordinator, or unfinished cleanup. Saved user input is reconsidered through the same admission path; the cancelled turn is not replayed. @@ -838,6 +850,7 @@ A healthy `in_review` issue has at least one valid action path: - a human owner via `assigneeUserId` - an active run or queued wake that is expected to process the review state - an active one-shot monitor for an external service or async review loop that the assignee owns +- an authorized GitHub conversation waiting for its next provider event after a published review; a tree resume or same-agent continuation retains this path only through its accepted requester lineage and the latest saved review publication - an open explicit recovery action for an ambiguous review handoff Agent-assigned `in_review` with no typed participant is only healthy when one of the other paths exists. Assignment to the same agent that produced the handoff is not, by itself, a review path. @@ -1674,6 +1687,23 @@ Delivery still requires the provider's verified stop receipt. Periodic queue retries do not gain extra cleanup attempts, and the queue displays the server's waiting reason while cleanup remains unresolved. +A connected task can also fail while preparing an accepted board-authored +message, before the provider starts. Its Retry belongs to the canonical task +intent rather than the external inbound-message ledger. The server certifies +its unchanged, undeleted comment, original user wake and accepted identity in +the same company, task, agent and user scope. A settled bootstrap diagnostic +must agree with the absence of a native result, coordinator, environment lease, +controller, process or provider output. A previous session display hint does +not establish that the failed attempt started a provider. + +That intent pins the saved comment and accepted identity. Queue admission +rechecks the source; dispatch rechecks the coalesced intent and its exact claimed +successor receipt before inheriting the accepted identity. Current permissions, +assignment, pauses, budgets and containment remain normal queue requirements. +This path neither fabricates external inbound provenance nor replays a recorded +terminal result. If this source cannot be certified, Retry directs the user to +send a new task message to continue from saved work. + The legacy task recovery notice shows “Automatic recovery of this task stopped.” in a bordered container with Retry for a failed or timed-out run. A failed Retry shows its error in the same container. New user messages and saved undelivered @@ -1943,6 +1973,16 @@ re-read dependency readiness, since new dependencies need not change the displayed task status. Old blocked rows without a receipt remain held; no historical status backfill is performed. +For legacy GitHub input without a saved App ID or its own response receipt, +Task Retry may prove the endpoint's earlier non-null App claim from a processed +response ledger created before that input. The proof stays within the same +company, endpoint, conversation and agent, with a processed prior delivery and +its inbound source link. Supported setup and reconnect writers preserve an +existing non-null App claim. This historical identity proof grants no publication +replay or access: the saved input and current principal, repository, conversation, +runtime fence and execution containment still pass normal admission. Explicit +conflicting App bindings remain denials. + ### Queued input after a native Stop A run-only Stop ends the current response. It does not discard queued user diff --git a/doc/task-recovery.md b/doc/task-recovery.md index 714d921a97b..cede3a25dc2 100644 --- a/doc/task-recovery.md +++ b/doc/task-recovery.md @@ -28,6 +28,12 @@ comment receipt, then checks current sender, repository, conversation and task access. The retry records the current runtime generation and credential-reference fingerprint. Another reconnect invalidates that admission. Historical deliveries, stale webhook callbacks and the exact-run retry contract retain their fences. +Legacy input that predates saved App IDs can also use an earlier processed +response ledger from the same company, endpoint, conversation and agent. Its +server-owned App binding must predate that input and retain a processed delivery +and inbound source link. The endpoint's non-null App claim cannot be replaced; +this proves historical identity without replaying the earlier response. Missing +proof or an explicit conflicting App binding still rejects Retry. Task Retry keeps the complete saved-input batch, including batches larger than the exact-run endpoint's limit. Source rows, receipts and invalidations are read diff --git a/maps/agent-access-map/chat-channels.md b/maps/agent-access-map/chat-channels.md index d14f5c4237a..aaa0b34679b 100644 --- a/maps/agent-access-map/chat-channels.md +++ b/maps/agent-access-map/chat-channels.md @@ -283,6 +283,20 @@ real human dependencies retain the ordinary attention-request path. A timed wait requires a confirmed `set_task_monitor` schedule. Database coverage lives in `external-chat-wait.integration.test.ts`; this is not live model evidence. +Inherited GitHub review waits (2026-10-26): an explicit native `response_wake` +after tree resume or same-agent continuation may retain the provider-event path +through accepted requester identity lineage and the latest processed review +publication for that task/conversation. This does not grant monitor management +or publication authority. Current task/agent/company binding, repository, +principal and endpoint checks remain effective. PostgreSQL regressions cover +quiet repeated finalization, historical publication reuse, newer unreviewed +heads and revoked access; live GitHub delivery remains separate acceptance. +Omitted/null yielded reports may use the current run's server-projected published +review wait. The same proof is rechecked at feedback, finalization and commit; +older/different-run review, pending publication and revoked connection cannot +supply it. This grants no new publication/monitor authority and leaves Done policy +unchanged; explicit malformed input is not coerced. + ## `outbound-email` Capability ID: `chat-channels/outbound-email`. @@ -300,3 +314,11 @@ Evidence: `source-only` — reviewed at `9ef434fe79c57d976b8b03b14c5242db270b832 Verification: `not-run` — Use the actual permitted writer and run to create one child/publication, confirm recipient delivery and an exact-message reply, and repeat idempotent requests without duplicate sends. Assert run/owner/mode/budget/company/credential/attachment denials and separate read access. Inspect the provider before resolving an unknown write outcome; a 202 or queued publication alone is not proof of delivery. Gaps: Real AgentMail transport, recipient delivery, and each native tool binding still need live proof. Controlled provider receipts and inbound-reply recipes do not establish a newly initiated external email journey. + +Ordinary task Retry can select newer linked-human GitHub input whose original +admission was skipped by a budget or agent-availability gate. The chat owner +revalidates its exact processed delivery, company/task/agent/principal, unchanged +comment, session generation, and admitted instruction snapshot. A separate retry +receipt preserves the original skipped receipt and older failed native result. +Current access and normal native stop/cleanup guards still govern execution; +board wake metadata alone cannot manufacture chat authority or replay old work. diff --git a/maps/agent-access-map/recovery.md b/maps/agent-access-map/recovery.md index c9df2c9b13e..481c17abf8e 100644 --- a/maps/agent-access-map/recovery.md +++ b/maps/agent-access-map/recovery.md @@ -17,8 +17,16 @@ Ordinary board Retry can reauthorize saved GitHub input for the same App after reconnect. It pins current runtime authority and preserves sender, repository, conversation and task checks. Pending input takes precedence over the older failed request and transfers once to the retry owner. This grants no new agent -Retry API or stale webhook authority. Local PostgreSQL and HTTP-route regressions +Retry API or stale webhook authority. Legacy input can use a server-owned, +processed earlier response ledger to prove the same endpoint's App claim before +that input. Company, conversation, agent and prior delivery/source linkage remain +required. Explicit conflicting App IDs cannot use that fallback. Local PostgreSQL and HTTP-route regressions cover these transitions; hosted execution and delivery remain separate acceptance. +Dispatch validates the exact claimed wake and resolved user-continuation receipt +for the current successor. A later board Retry can follow that successor only +when exact durable diagnostics and absent execution/resources prove it never +started. This preserves the original sender/source and grants no agent Retry +permission or accepted-result replay. Source revision: `3bbc2104b65caf9f7763938a01260a5673e12f7a`. @@ -248,6 +256,24 @@ replace the owner's generation or revision. Resolved files-only recovery retains its no-replay restriction. The local PostgreSQL regression checks the producer, worker, and physical source receipt; it is not deployed provider acceptance. +Connected board-message preparation failures use the canonical board Retry +owner only after the exact unchanged comment, original user wake, accepted +same-user identity and pre-provider settlement are certified. Admission and +execution recheck the pinned source and claimed successor receipts. Repeated +pre-provider failures retain the source only through each canonical intent, +failed dispatch and accepted parent identity; cycles and chains beyond twenty +links fail closed. This grants +no external inbound authority or permission to replay a terminal result; an +uncertifiable source directs the user to send a new task message. The real +HTTP/queue regression proves this boundary locally, not hosted provider recovery. + +For external chat, the same canonical Retry owner may certify the structured +isolated-directory binding failure that occurs before environment allocation. +It requires exact task/source authority and trusted settlement with no process, +lease, native result or unknown execution evidence. Other validation reasons and +the older direct chat-retry entry point remain denied. Local HTTP/dispatch proof +does not establish hosted provider completion. + An authorized deliberate Retry also registers missing historical native preservation from an exact prepared descriptor and an unowned terminal finalization. It adopts only the existing bounded maintenance capability. A live diff --git a/maps/feature-map/chat-channels.md b/maps/feature-map/chat-channels.md index 6c98a4e5376..a5a490d0a33 100644 --- a/maps/feature-map/chat-channels.md +++ b/maps/feature-map/chat-channels.md @@ -72,6 +72,17 @@ database suite checks these states and revoked endpoints using stored provider receipts; live model adherence remains a separate check. It also checks saved monitors, disabled repositories, multiple reviewed heads, read-only finish validation, withdrawal rollback, and separate approval gates. +Repeat the review wait after a tree resume and a same-agent continuation with +no new provider message. Accepted requester lineage and the latest saved review +publication keep the task in review without a self-wake or duplicate publication. +A newer unreviewed head, changed requester/repository, or revoked access cannot +inherit the prior head's passive wait. Omitted/null yielded continuation is filled +from authorized server task context after a current-head review, an accepted tree +resume inheriting it, or a same-head follow-up with this run's processed useful +reply bound to its exact current source. Pending or ambiguous replies and +acknowledgements do not authorize this fallback. A malformed first report can recover by omitting the continuation; +explicit malformed input remains rejected. Revocation between context read, +finish and commit preserves the task without self-waking. Done policy is unchanged. Manual: For each supported provider being changed, send a unique message, verify the intended agent/company receives it, and inspect the actual external reply in the original thread. Repeat with an unlinked/unauthorized sender, duplicate inbound delivery, and one supported attachment. Record provider-specific results separately. @@ -217,3 +228,14 @@ Sources: Source: [TaskChatBubble.tsx](../../ui/src/components/task-chat/TaskChat ### Documentation scope: Chat and email channel setup Shared recipes name provider checks; each provider transport/thread/identity/file-size/type combination needs separate live evidence and deeper instructions. + +Saved-input Retry regression: fail a disposable GitHub turn, change the +configured instruction skill, and send a new authorized request while an agent +budget gate blocks admission. Clear that gate and use the task thread's ordinary +**Retry** control. Verify the new saved message and its admitted skill snapshot +reach one run, rather than replaying the previous failed request. Preserve both +original receipts. With a committed failed native result, continuation requires +verified controller and environment cleanup; an unknown executor remains held. +After successful admission, reload and verify that the prior skipped receipt does +not strand a later real provider reply or tree resume. Provider fixtures support +this behavior; live fixed-source runtime and destination proof remain separate. diff --git a/maps/feature-map/recovery.md b/maps/feature-map/recovery.md index fcf47da1e27..50699dee967 100644 --- a/maps/feature-map/recovery.md +++ b/maps/feature-map/recovery.md @@ -82,6 +82,15 @@ stopped-run recovery checks. ### `operator-retry` +The [connected-task regression](../../server/src/__tests__/chat-channels.integration.test.ts) +uses HTTP Retry after an accepted board message fails before provider startup. +It checks the canonical intent, same-user identity and actual queue dispatch, +then repeats Retry after another contained preparation failure with the same +source and a new certified successor; +edited source, contradictory execution evidence and mismatched receipts refuse +or wait. This local boundary probe deliberately stops before a provider call, +so it does not establish hosted recovery of the task. + Use the task's Retry action. It can create an isolated attempt after old execution is contained even when old-file preservation remains unresolved. Approval, credentials, budget and deliberate pause use their existing specific controls. @@ -103,6 +112,15 @@ there must be one successor and no duplicate publication. A second reconnect after admission, changed App, revoked sender or changed repository must prevent dispatch. An unadmitted rejection keeps the follow-up available for another deliberate Retry. Historical deliveries must remain unchanged. +Repeat with legacy input that has no saved App ID and was refused by a budget +before acknowledgement. An earlier processed response from the same endpoint, +conversation and agent can prove its App claim. Missing, late, unlinked or +conflicting proof must reject Retry. Verify actual execution separately. +At dispatch, confirm the admitted successor retains the same source and resolved +continuation receipt. If it fails at Retry authorization before provider startup, +use the next ordinary Retry against that newest failed run. It must retain the +saved follow-up and create one successor; output, an allocated environment or an +active controller must prevent treating that failure as never started. ### `post-stop-message` diff --git a/packages/paperclip-runner/acpx-profiles.json b/packages/paperclip-runner/acpx-profiles.json index 90da6bee2c4..fbe3f2753ad 100644 --- a/packages/paperclip-runner/acpx-profiles.json +++ b/packages/paperclip-runner/acpx-profiles.json @@ -32,12 +32,12 @@ "requiresProviderPolicy": true }, "copilot": { - "agentProfileVersion": 62, + "agentProfileVersion": 63, "agentServerPackage": "@github/copilot", "agentServerVersion": "1.0.88", "agentRuntimePackage": null, "agentRuntimeVersion": null, - "commandDigest": "sha256:aeeb80bc186569c78911aeaa445c1fdf5e81420cfeadb563d5620497c58dd0f4", + "commandDigest": "sha256:1412842a36a7211d84b6b98a3d1f69e3388bc4814e5825da890a774ae0e3460e", "requiresProviderPolicy": true }, "claude": { diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs index 587a7bd5752..ed82e55fea9 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs @@ -47,7 +47,7 @@ pub(crate) fn acpx_release_profile(agent: &str) -> Option { agent_runtime_package: None, agent_runtime_version: None, command_digest: - "sha256:aeeb80bc186569c78911aeaa445c1fdf5e81420cfeadb563d5620497c58dd0f4", + "sha256:1412842a36a7211d84b6b98a3d1f69e3388bc4814e5825da890a774ae0e3460e", requires_provider_policy: true, }, "claude" => AcpxReleaseProfile { diff --git a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs index bc0b7f399bd..770540a02c5 100644 --- a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs +++ b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs @@ -7,9 +7,9 @@ import { fileURLToPath } from "node:url"; import { resolve, dirname } from "node:path"; const root = resolve(dirname(fileURLToPath(import.meta.url)), ".."); -const fixture = resolve(root, "test/fixtures/copilot-profile-v62-identity.json"); +const fixture = resolve(root, "test/fixtures/copilot-profile-v63-identity.json"); const historical = JSON.parse(readFileSync(resolve(root, "test/fixtures/copilot-profile-v12-identity.json"), "utf8")); -const declaration = { ...historical.declaration, agentProfileVersion: 62 }; +const declaration = { ...historical.declaration, agentProfileVersion: 63 }; delete declaration.permissionIdentitySourceSha256; const sources = { displayEventSourceSha256: "src/drivers/acpx/copilot-events.ts", diff --git a/packages/paperclip-runner/src/contracts/completion-result.ts b/packages/paperclip-runner/src/contracts/completion-result.ts index 47862d1fdcb..8542380bbb6 100644 --- a/packages/paperclip-runner/src/contracts/completion-result.ts +++ b/packages/paperclip-runner/src/contracts/completion-result.ts @@ -3,7 +3,7 @@ export const PRP_COMPLETION_TOOL_NAME = "paperclip_finish" as const; export const PRP_BLOCK_TOOL_NAME = "paperclip_block" as const; /** Native Runner guidance; legacy adapters use their own skill/API completion paths. */ export const PRP_COMPLETION_TOOL_DESCRIPTION = - "Report completed work (done), work requiring review (needs_review), or an explicit wait (yielded with response_wake for a response, monitor after set_task_monitor confirms a schedule on this task, or omit continuation to use the saved repository-preparation continuation). Use the current completion contract and supporting evidence. If rejected, correct the report and retry. After acceptance, read the returned outcome; do not claim completion while gated. Explain any required approval with its supplied link and action, then write the final response and end the turn without further tool calls."; + "Report completed work (done), work requiring review (needs_review), or an explicit wait (yielded with response_wake for a response, monitor after set_task_monitor confirms a schedule on this task, or omit continuation to use this run's authorized published-review wait or saved repository-preparation continuation). Use the current completion contract and supporting evidence. If rejected, correct the report and retry. After acceptance, read the returned outcome; do not claim completion while gated. Explain any required approval with its supplied link and action, then write the final response and end the turn without further tool calls."; export const PRP_BLOCK_TOOL_DESCRIPTION = "Report work that cannot continue because of a concrete blocker. Routine repository preparation is not a blocker: use paperclip_finish with yielded and omit continuation to resume through the saved workspace continuation. Identify the blocker, its owner, and the action needed to unblock it; use the current completion contract and supporting evidence. If rejected, correct the report and retry. After acceptance, read the returned outcome, explain the blocker and any required action in the final response, and end the turn without further tool calls."; export const PRP_SEMANTIC_TOOL_NAMES = [ @@ -372,7 +372,7 @@ export const PRP_COMPLETION_RESULT_PROVIDER_INPUT_SCHEMA = { ...waitContinuationSchema, type: ["object", "null"], description: - "For yielded, omit or use null to reuse this task's eligible saved monitor or repository-preparation continuation; the controller reads current task state. Otherwise provide a real response_wake or monitor with summary and idempotencyKey. Use null or omit for done, completed, or needs_review.", + "For yielded, omit or use null to reuse this run's authorized published-review wait, eligible saved monitor, or repository-preparation continuation; the controller reads current task state. Otherwise provide a real response_wake or monitor with summary and idempotencyKey. Use null or omit for done, completed, or needs_review.", }, }, // Keep the provider-facing root a concrete object. Codex code-mode renders a diff --git a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts index 2a96fdc9e51..2aa9af9391b 100644 --- a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts @@ -87,9 +87,28 @@ vi.mock("./installation-integrity.js", () => ({ verifyNativeAcpxInstallation: vi describe("Copilot build-owned installation", () => { it("admits the current profile with environment launch preparation", () => { - expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(62); + expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(63); expect(QUALIFIED_ACPX_PROFILES.copilot.qualificationStatus).toBeUndefined(); - expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:aeeb80bc186569c78911aeaa445c1fdf5e81420cfeadb563d5620497c58dd0f4"); + expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:1412842a36a7211d84b6b98a3d1f69e3388bc4814e5825da890a774ae0e3460e"); + }); + + it("preserves both published v62 identities while binding their merged source in v63", async () => { + const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v63-identity.json", import.meta.url), "utf8")); + for (const [name, checksum, changed] of [ + ["copilot-profile-v62-identity.json", "faa4d1b35267b54e134a8e0c84586fe1196d8fad5525cb53aa195aae53ca6e83", ["agentProfileVersion", "profileCompatibilitySourceSha256", "semanticCompletionContractSourceSha256", "semanticContinuationSourceSha256"]], + ["copilot-profile-v62-candidate-historical-identity.json", "da3f44a3cfda31cbff558d618e1e9cb1cfb80379b613ff1266e8b866ea666df8", ["agentProfileVersion", "profileCompatibilitySourceSha256", "runtimeRequestBridgeSourceSha256"]], + ] as const) { + const bytes = readFileSync(new URL(`../../../test/fixtures/${name}`, import.meta.url)); + expect(createHash("sha256").update(bytes).digest("hex")).toBe(checksum); + const prior = JSON.parse(bytes.toString("utf8")); + expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual(changed); + expect(Object.keys(current.declaration).sort()).toEqual(Object.keys(prior.declaration).sort()); + const sorted = Object.fromEntries(Object.entries(prior.declaration).sort(([a], [b]) => a.localeCompare(b))); + expect(`sha256:${createHash("sha256").update(JSON.stringify(sorted)).digest("hex")}`).toBe(prior.commandDigest); + expect(isSupportedAcpxProfileVersion("copilot", 62)).toBe(true); + await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot, + agentProfileVersion: 62, commandDigest: prior.commandDigest })).rejects.toThrow("exact pinned"); + } }); it("preserves v61 while binding uncertain process checks and controller retirement in v62", async () => { diff --git a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts index 5aaf18f97c3..4cf0892c5c5 100644 --- a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts +++ b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts @@ -48,12 +48,12 @@ export const QUALIFIED_ACPX_PROFILE_DATA = { "protocolVersion": 1, "acpxVersion": "0.13.1", "agent": "copilot", - "agentProfileVersion": 62, + "agentProfileVersion": 63, "agentServerPackage": "@github/copilot", "agentServerVersion": "1.0.88", "agentRuntimePackage": null, "agentRuntimeVersion": null, - "commandDigest": "sha256:aeeb80bc186569c78911aeaa445c1fdf5e81420cfeadb563d5620497c58dd0f4", + "commandDigest": "sha256:1412842a36a7211d84b6b98a3d1f69e3388bc4814e5825da890a774ae0e3460e", "permissionPolicy": "interactive" }, "claude": { diff --git a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts index e8091c079e2..8f5fd0a22d9 100644 --- a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts +++ b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts @@ -2,13 +2,13 @@ import { QUALIFIED_ACPX_PROFILE_DATA } from "./generated-profiles.js"; /** Decodable historical profile revisions, not permission to launch them. * Exact current profile/package/digest admission is checked separately. */ -type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61; +type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62; type CurrentAcpxProfileVersion = typeof QUALIFIED_ACPX_PROFILE_DATA[keyof typeof QUALIFIED_ACPX_PROFILE_DATA]["agentProfileVersion"]; export type AcpxProfileVersion = HistoricalAcpxProfileVersion | CurrentAcpxProfileVersion; const historicalVersions: Readonly> = { pi: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28], claude: [1, 2, 3, 4, 5], codex: [1, 2, 3, 4, 5], - grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61], + grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62], cursor: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], }; diff --git a/packages/paperclip-runner/src/drivers/completion-continuation.test.ts b/packages/paperclip-runner/src/drivers/completion-continuation.test.ts index 62051110a60..5a9b60c2118 100644 --- a/packages/paperclip-runner/src/drivers/completion-continuation.test.ts +++ b/packages/paperclip-runner/src/drivers/completion-continuation.test.ts @@ -25,6 +25,13 @@ describe("controller-owned completion continuation", () => { .toEqual({ ...yielded, continuation: preparation }); }); + it.each([undefined, null])("fills a server-owned published response wait (%s)", async absent => { + const responseWait = { kind: "response_wake", summary: "Wait for the author.", idempotencyKey: "published-review:run:receipt" }; + expect(await resolveCompletionContinuation({ ...yielded, continuation: absent }, "run", async () => ({ + ...context, completionContinuation: responseWait, + }))).toEqual({ ...yielded, continuation: responseWait }); + }); + it("rechecks a sidecar-resolved workspace continuation against current server authority", async () => { const preparation = { kind: "same_agent", summary: "Continue with the prepared repository.", idempotencyKey: "repository:task:receipt:run" }; const report = { ...yielded, continuation: preparation }; diff --git a/packages/paperclip-runner/src/drivers/completion-continuation.ts b/packages/paperclip-runner/src/drivers/completion-continuation.ts index 05fd4ad8b75..2f817bb4e28 100644 --- a/packages/paperclip-runner/src/drivers/completion-continuation.ts +++ b/packages/paperclip-runner/src/drivers/completion-continuation.ts @@ -15,10 +15,10 @@ export async function resolveCompletionContinuation( const task = object(context.activeTask); const continuation = object(context.completionContinuation); if (object(context.run).id !== runId || task.executionRunId !== runId || - !["monitor", "same_agent"].includes(String(continuation.kind)) || + !["monitor", "same_agent", "response_wake"].includes(String(continuation.kind)) || typeof continuation.summary !== "string" || !continuation.summary.trim() || typeof continuation.idempotencyKey !== "string" || !continuation.idempotencyKey.trim()) { - throw new Error("Cannot yield without a continuation: no eligible saved monitor or workspace continuation exists on this task. Save a monitor for a deferred check, provide an explicit response_wake for a real response wait, or continue the unfinished work. Do not repeat completed actions."); + throw new Error("Cannot yield without a continuation: no eligible saved monitor or workspace continuation or authorized response wait exists on this task. Save a monitor for a deferred check, provide an explicit response_wake for a real response wait, or continue the unfinished work. Do not repeat completed actions."); } if (report.continuation != null && (continuation.kind !== "same_agent" || supplied.summary !== continuation.summary || supplied.idempotencyKey !== continuation.idempotencyKey)) { diff --git a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts index a44136bd364..c8e2ed1c292 100644 --- a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts +++ b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts @@ -2184,12 +2184,13 @@ describe("OpenCodeServerDriver", () => { await session.close({ reason: "test" }); }); - it.each([true, false])("resolves an omitted continuation through the real OpenCode MCP boundary only with a saved monitor: %s", async eligible => { + it.each(["monitor", "response_wake"] as const)("resolves an omitted %s through the real OpenCode MCP boundary only from current server state", async kind => { + for (const eligible of [true, false]) { await chmod(fixture, 0o755); const root = await mkdtemp(join(tmpdir(), "paperclip-opencode-monitor-")); const workspace = await mkdtemp(join(tmpdir(), "paperclip-opencode-monitor-workspace-")); roots.push(root, workspace); - const continuation = { kind: "monitor", summary: "Wait for the saved task monitor.", idempotencyKey: "task-monitor:saved" }; + const continuation = { kind, summary: "Wait for the saved task path.", idempotencyKey: "task-wait:saved" }; const reads: string[] = []; const accepted: unknown[] = []; const driver = new OpenCodeServerDriver({ @@ -2216,7 +2217,8 @@ describe("OpenCodeServerDriver", () => { expect(events.filter(event => event.eventType === "run.result.proposed")).toEqual([]); expect((await session.snapshot()).semanticResult).toBeNull(); } - } finally { await session.close({ reason: "monitor default test complete" }); } + } finally { await session.close({ reason: "saved wait default test complete" }); } + } }); it("keeps assigned discovery operation IDs at the authenticated MCP and controller boundary", async () => { diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v62-candidate-historical-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v62-candidate-historical-identity.json new file mode 100644 index 00000000000..7266c5e13ca --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v62-candidate-historical-identity.json @@ -0,0 +1,85 @@ +{ + "commandDigest": "sha256:1251b53cb4716273c74e5839eb112ef8ca639e0967eb90a9f1a5f5ddafdb642c", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 62, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "17361d1782c0dcebb2b7972635f45c890f2a4081f8bc0ef2d2276550e0b91629", + "semanticDirectDriverSourceSha256": "ff572638f2cb5d9f8249ff8e2f412311c1e3af209ef0bf7e5ea9f10a21d91103", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "f49398b682a5623812f5d3c33d6bb9a749008e47cc0ed969072041615f66df59", + "semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9", + "semanticSchemaBundleSourceSha256": "016391a087cc217aafed4ba1f4efafc6e4881a7525952dce824b7f4d24530603", + "semanticSidecarProtocolSourceSha256": "03c0c95b900f9dc18f29f6be27b8c79bc0bcab29780190d46a74e95328081236", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "103b83066da72434602e9e461e6278dfb40bb993d2f3b49b2e93df81bbb08b72", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "03f4cd7f76f34f6d82e009b66aa51f18f6e23dc235b660778060e35b6ea07412", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "e7b12d4418c7a71f5f2474b046697d3009eae3613b284bec381df5e9965a1c37", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "9173ebccf09cb56a781d2bf5c7eb27fdafb13d3450d99342c6aff86b42d6b43a", + "providerLaunchSourceSha256": "fa8db1c42c378898287c141d8cea9069166981e369c9b9c51422cb6177de5a2c", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "e6c98128d98a4177c42d68a04c1f749554fd5ec844da1bbbdb3f682670189445", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "3a30d52bffad68c7f02349193be84761050eb99039fc0556c2b449891a36e918", + "profileCompatibilitySourceSha256": "be60656e66d7f61a290f8d1c9a3c9ec511fb13b6541825a087b470e971ccc020", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "c9cc0fd516b3df92a6655f56ee9bef37615a889c371e2e137d8ac5c1176de811", + "runtimeHostSourceSha256": "94a73749267f5a80b5df19413a65b4b9263c00a9bf7e98fda00f00b8f9fd32ec", + "runtimeAdapterSourceSha256": "e4c764c2c3c7dd655a965b39472bd1ad8d577d1e7258b25594b171c9429752b8", + "runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +} diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v63-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v63-identity.json new file mode 100644 index 00000000000..10ea2c41afc --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v63-identity.json @@ -0,0 +1,85 @@ +{ + "commandDigest": "sha256:1412842a36a7211d84b6b98a3d1f69e3388bc4814e5825da890a774ae0e3460e", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 63, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "17361d1782c0dcebb2b7972635f45c890f2a4081f8bc0ef2d2276550e0b91629", + "semanticDirectDriverSourceSha256": "ff572638f2cb5d9f8249ff8e2f412311c1e3af209ef0bf7e5ea9f10a21d91103", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "f49398b682a5623812f5d3c33d6bb9a749008e47cc0ed969072041615f66df59", + "semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9", + "semanticSchemaBundleSourceSha256": "016391a087cc217aafed4ba1f4efafc6e4881a7525952dce824b7f4d24530603", + "semanticSidecarProtocolSourceSha256": "03c0c95b900f9dc18f29f6be27b8c79bc0bcab29780190d46a74e95328081236", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "103b83066da72434602e9e461e6278dfb40bb993d2f3b49b2e93df81bbb08b72", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "03f4cd7f76f34f6d82e009b66aa51f18f6e23dc235b660778060e35b6ea07412", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "e7b12d4418c7a71f5f2474b046697d3009eae3613b284bec381df5e9965a1c37", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "9173ebccf09cb56a781d2bf5c7eb27fdafb13d3450d99342c6aff86b42d6b43a", + "providerLaunchSourceSha256": "fa8db1c42c378898287c141d8cea9069166981e369c9b9c51422cb6177de5a2c", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "9be35e006ffffcca48205f9e53548dd453ec6a8cbcf0aa4dd735194e9619dfc7", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "3a30d52bffad68c7f02349193be84761050eb99039fc0556c2b449891a36e918", + "profileCompatibilitySourceSha256": "5b00c3f78ad12321d48e66a90338af2d53d59dd17bb4b72b10ccdb4240b5a150", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "c9cc0fd516b3df92a6655f56ee9bef37615a889c371e2e137d8ac5c1176de811", + "runtimeHostSourceSha256": "94a73749267f5a80b5df19413a65b4b9263c00a9bf7e98fda00f00b8f9fd32ec", + "runtimeAdapterSourceSha256": "e4c764c2c3c7dd655a965b39472bd1ad8d577d1e7258b25594b171c9429752b8", + "runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +} diff --git a/server/src/__tests__/chat-channels.integration.test.ts b/server/src/__tests__/chat-channels.integration.test.ts index 347ec611be1..f7b82550879 100644 --- a/server/src/__tests__/chat-channels.integration.test.ts +++ b/server/src/__tests__/chat-channels.integration.test.ts @@ -1,6 +1,14 @@ +import { adapterExecutionControls, createAdapterExecutionControl } from "../services/adapter-execution-control.js"; +import { chatRunSourceMessageId } from "../services/chat-channels/run-origin.js"; +import * as durableChatWakeup from "../services/durable-chat-wakeup.js"; +import { conflict } from "../errors.js"; +import * as githubTaskPreparation from "../services/chat-channels/github/task-preparation.js"; +import { configureAgentLifecycle } from "../services/agent-lifecycle.js"; +import { budgetPolicies } from "@paperclipai/db"; +import { budgetService } from "../services/budgets.js"; import { idleWorkSnapshot } from "../services/task-admission.js"; import { readFreshWorkspaceReadiness } from "../services/fresh-workspace-continuation.js"; -import { requestTaskRetry } from "../services/task-retry.js"; +import { requestTaskRetry, reconcileTaskChatRetryReceipts } from "../services/task-retry.js"; import { createSystemRecovery } from "../services/system-recovery-state.js"; import { remoteTerminationReceipt } from "../services/remote-execution-termination.js"; import { stageReceiptReactionRemovals } from "../services/chat-receipt-reactions.js"; @@ -39,6 +47,11 @@ import { createAssignedMcpTools } from "../services/native-runtime/assigned-mcp- import { createToolGatewayService } from "../services/tool-gateway.js"; import { resolveGitHubOperationCredentials } from "../services/github-operation-credentials.js"; import { initializeRunIdentity } from "../services/run-identity.js"; +import { prepareHeartbeatExecutionConfiguration } from "../services/heartbeat/execution-configuration.js"; +import { appendHeartbeatRunEvent } from "../services/heartbeat-run-events.js"; +import { companySkillService } from "../services/company-skills.js"; +import { companyObservability } from "@paperclipai/db"; +import * as runIdentityModule from "../services/run-identity.js"; import { chatGitHubRegistrations, chatGitHubReviews, chatGitHubConfigurations, toolCatalogEntries } from "@paperclipai/db"; import { AsyncLocalStorage } from "node:async_hooks"; import * as cloudRuntimeIdentity from "../services/cloud-runtime-identity.js"; @@ -60722,6 +60735,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { externalActorId?: string, linkedOriginalActor = false, githubUnavailableFile = false, + currentGitHubPr = false, ) { // Keep the policy under which the source was admitted. Inserting an // upgraded GitHub policy after receipt correctly invalidates retry proof. @@ -60733,6 +60747,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { linkedOriginalActor, githubUnavailableFile, false, + currentGitHubPr, ); const [issue] = await readIssuesById(context.conversation.issueId); const [action] = await db @@ -61129,6 +61144,901 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { }, ); + it.each(["contained", "prior_denied_intent", "observability_warning", "observability_reason", "observability_revision", "observability_provenance", "observability_extra", "other_reason", "wrong_issue", "provider_started", "result_extra", "profile", "usage", "output", + "process", "controller", "active_control", "lease", "accepted_result", "event_source", "extra_event", "source_edited", "principal_revoked"] as const)( + "retries an authenticated GitHub request after isolated-directory validation failed before allocation: %s", async mode => { + const f = await failedChatRetryFixture("github", "42", true, false, true); + const companyId = f.fixture.companyId, agentId = f.fixture.assignedAgentId, issueId = f.issue.id; + const diagnostic = "The existing execution workspace is not this task's isolated directory."; + try { + await initializeRunIdentity(db, { companyId, runId: f.runId, issueId, responsibleUserId: "owner-user", + messageIds: [String(f.action.payload.commentId)], cause: "chat_message" }); + await db.update(heartbeatRuns).set({ errorCode: "workspace_validation_failed", error: diagnostic, + runtimeMode: "legacy", sessionIdBefore: "prior-provider-display-hint", + usageJson: { ledgerScope: { issueId, projectId: null, billingCode: null } }, + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } }, + resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false }, stopReason: "adapter_failed", + workspaceValidation: { reason: "isolated_task_directory_binding_mismatch", issueId } } }) + .where(eq(heartbeatRuns.id, f.runId)); + if (mode.startsWith("observability_")) { + await db.insert(companyObservability).values({ companyId, revision: randomUUID(), enabled: true, + destination: { provider: "otlp", endpointBaseUrl: "https://collector.example/otel" }, + capture: { userPrompts: false, assistantResponses: false, toolDetails: false, toolContent: false } }); + const [agent] = await db.update(agents).set({ adapterConfig: { provider: "opencode", model: "openrouter/synthetic", homeMode: "preconfigured" } }) + .where(eq(agents.id, agentId)).returning(); + const [run] = await readHeartbeatRunsById(f.runId); + const prepared = await prepareHeartbeatExecutionConfiguration(db, { run, agent, + task: { issueId, context: { ...run.contextSnapshot }, responsibleUserId: "owner-user", + acceptedPlanContinuationWake: false, acceptedPlanWakeRoutingDecision: null, taskProjectContext: null, + routineEnvContext: { routineId: null, env: null, responsibleUserId: null } }, + config: { sourceWorkspaceConfig: agent.adapterConfig, aiBinding: undefined }, + policy: { trustPreset: { kind: "standard", preset: "standard", boundary: null, sourcePresets: {} }, + useIsolatedTaskDirectory: false, explicitTaskDirectory: false, requestedExecutionWorkspaceMode: "shared_workspace" }, + environment: { selectedEnvironmentId: null, selectedEnvironmentForConfig: null, + resolvedInstanceSettings: await instanceSettingsService(db).get(), persistedNativeExecutionInput: null }, + services: { secretsSvc: secretService(db), companySkills: companySkillService(db), + appendRunEvent: async (source, event) => { await appendHeartbeatRunEvent(db, { companyId: source.companyId, + agentId: source.agentId, runId: source.id, ...event }); }, finalizeAiConnectionBusyDeferral: vi.fn() }, + onFailureSecretsResolved: () => {}, onManagedAiRuntimeAcquired: () => {}, onIdentityRedactorPrepared: () => {}, + }); + expect(prepared.prepared).toBe(true); + expect(await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.runId))) + .toMatchObject([{ eventType: "agent_observability.unavailable", payload: { reason: "unsupported_harness" } }]); + } + await appendHeartbeatRunEvent(db, { companyId, agentId, runId: f.runId, + eventType: "error", stream: "system", level: "error", message: diagnostic }); + await appendHeartbeatRunEvent(db, { companyId, agentId, runId: f.runId, eventType: "lifecycle", stream: "system", level: "info", + message: "task.run.measured", payload: { durableSettlement: true, durationMs: 2505, outcome: "failed", phase: "task.run.measured" } }); + await db.update(agents).set({ status: "running", runtimeConfig: { heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 1 } } }) + .where(eq(agents.id, agentId)); + const [capacity] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running" }).returning(); + const heartbeat = heartbeatService(db); + f.wakeup.mockImplementation(heartbeat.wakeup); + const app = express(); + app.use(express.json()); + app.use((req, _res, next) => { req.actor = boardActor(companyId); next(); }); + app.use("/api", issueRoutes(db, createStorageService().storage, { chatRunRetries: f.service })); + app.use(errorHandler); + const [failed] = await readHeartbeatRunsById(f.runId); + if (mode === "observability_reason" || mode === "observability_revision" || mode === "observability_extra") { + const [warning] = await db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, f.runId), eq(heartbeatRunEvents.seq, 1))); + await db.update(heartbeatRunEvents).set({ payload: { ...warning.payload, + ...(mode === "observability_reason" ? { reason: "unknown_preparation" } : mode === "observability_revision" ? { revision: null } : { providerWork: true }) } }) + .where(eq(heartbeatRunEvents.id, warning.id)); + } + if (mode === "observability_provenance") await db.update(heartbeatRunEvents).set({ sourceEventId: "untrusted-provider-warning" }) + .where(and(eq(heartbeatRunEvents.runId, f.runId), eq(heartbeatRunEvents.seq, 1))); + const validation = failed.resultJson!.workspaceValidation as object; + if (mode === "other_reason" || mode === "wrong_issue") await db.update(heartbeatRuns).set({ resultJson: { ...failed.resultJson, + workspaceValidation: { ...validation, ...(mode === "other_reason" ? { reason: "plugin_workspace_strategy_unsupported" } : { issueId: randomUUID() }) } } }) + .where(eq(heartbeatRuns.id, f.runId)); + if (mode === "provider_started" || mode === "result_extra") await db.update(heartbeatRuns).set({ resultJson: { ...failed.resultJson, + ...(mode === "provider_started" ? { executionRecovery: { kind: "bootstrap", providerWorkStarted: true } } : { unknownExecution: true }) } }) + .where(eq(heartbeatRuns.id, f.runId)); + if (mode === "profile") await db.update(heartbeatRuns).set({ runnerProfileJson: { ...failed.runnerProfileJson, nativeExecutionInput: {} } }).where(eq(heartbeatRuns.id, f.runId)); + if (mode === "usage") await db.update(heartbeatRuns).set({ usageJson: { ...failed.usageJson, tokens: 1 } }).where(eq(heartbeatRuns.id, f.runId)); + if (mode === "output") await db.update(heartbeatRuns).set({ stdoutExcerpt: "provider output" }).where(eq(heartbeatRuns.id, f.runId)); + if (mode === "process") await db.update(heartbeatRuns).set({ processPid: 999999 }).where(eq(heartbeatRuns.id, f.runId)); + if (mode === "controller") await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60000) }).where(eq(heartbeatRuns.id, f.runId)); + if (mode === "active_control") adapterExecutionControls.set(f.runId, createAdapterExecutionControl()); + if (mode === "lease") await db.insert(environmentLeases).values({ companyId, heartbeatRunId: f.runId, issueId, + provider: "daytona", providerLeaseId: "contradictory-preparation-allocation", status: "active", leasePolicy: "ephemeral" }); + if (mode === "accepted_result") { + const contractId = randomUUID(), resultId = randomUUID(); + await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1, + schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "standard", + completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", contractJson: {}, + canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "test" }); + await db.update(heartbeatRuns).set({ nativeIssueId: issueId, completionContractId: contractId }).where(eq(heartbeatRuns.id, f.runId)); + await db.insert(nativeRunResults).values({ id: resultId, companyId, issueId, runId: f.runId, completionContractId: contractId, + schemaStatus: "accepted", serverFingerprint: resultId, canonicalSha256: resultId, + resultJson: { terminal: { runTerminalState: "succeeded" } } }); + } + if (mode === "event_source") await db.update(heartbeatRunEvents).set({ sourceEventId: "provider-event" }).where(and(eq(heartbeatRunEvents.runId, f.runId), eq(heartbeatRunEvents.seq, 1))); + if (mode === "extra_event") await db.insert(heartbeatRunEvents).values({ companyId, agentId, runId: f.runId, + seq: 3, eventType: "lifecycle", stream: "system", level: "info", message: "unexpected provider preparation" }); + if (mode === "principal_revoked") await db.update(chatIdentityLinks).set({ status: "revoked", revokedAt: new Date() }).where(eq(chatIdentityLinks.endpointId, f.endpoint.id)); + if (mode === "contained") await expect(db.transaction(tx => f.service.prepareFailedChatRunRetry(tx, { + companyId, issueId, agentId, failedRunId: f.runId, initiatedByUserId: "owner-user", + }))).rejects.toMatchObject({ details: { code: "chat_failed_run_retry_not_authorized" } }); + const send = () => request(app).post(`/api/issues/${issueId}/retry`).send({ idempotencyKey: randomUUID(), observedRunId: f.runId }); + if (mode === "prior_denied_intent") { + const prior = await send().expect(202); + // Reproduce a terminal unadmitted authorization refusal, retaining the + // real HTTP intent/action receipts; the next normal click sees them. + await db.update(heartbeatRuns).set({ logBytes: 1 }).where(eq(heartbeatRuns.id, f.runId)); + await f.service.processFailedChatRunRetry(prior.body.chatActionId); + await db.update(heartbeatRuns).set({ logBytes: null }).where(eq(heartbeatRuns.id, f.runId)); + await reconcileTaskChatRetryReceipts(db); + expect((await readAgentWakeupRequestsById(prior.body.requestId))[0]).toMatchObject({ status: "failed", runId: null }); + expect((await db.select().from(chatActions).where(eq(chatActions.id, prior.body.chatActionId)))[0]).toMatchObject({ status: "failed" }); + } + const response = await send(); + if (["principal_revoked", "accepted_result"].includes(mode)) { + expect(response.status).toBe(409); + } else { + expect(response.status).toBe(202); + if (mode === "source_edited") await db.update(issueComments).set({ body: "Changed after Retry acceptance" }).where(eq(issueComments.id, String(f.action.payload.commentId))); + await f.service.processFailedChatRunRetry(response.body.chatActionId); + } + if (!["contained", "prior_denied_intent", "observability_warning"].includes(mode)) { + expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(0); + return; + } + const [wake] = await readAgentWakeupRequestsById(response.body.chatActionId); + expect(wake).toMatchObject({ status: "queued", runId: expect.any(String) }); + const [successor] = await readHeartbeatRunsById(wake.runId!); + expect(successor).toMatchObject({ retryOfRunId: f.runId, contextSnapshot: { + source: "chat:github", wakeCommentId: f.action.payload.commentId, taskRetryRequestId: response.body.requestId } }); + const realInitialize = runIdentityModule.initializeRunIdentity; + const initialized: string[] = []; + const probe = vi.spyOn(runIdentityModule, "initializeRunIdentity").mockImplementation(async (database, input) => { + const identity = await realInitialize(database, input); + if (input.companyId !== companyId) return identity; + expect(identity.messageId).toBe(f.action.payload.commentId); + initialized.push(input.runId); + throw new Error("workspace_retry_identity_probe_complete"); + }); + try { + await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.id, capacity.id)); + await db.update(agents).set({ status: "idle" }).where(eq(agents.id, agentId)); + await heartbeat.resumeQueuedRuns(); + await heartbeat.drainActiveRunExecutions(); + expect(initialized).toEqual([successor.id]); + expect((await readHeartbeatRunsById(successor.id))[0]).toMatchObject({ error: "workspace_retry_identity_probe_complete" }); + expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(1); + } finally { probe.mockRestore(); await heartbeat.drainActiveRunExecutions(); } + } finally { + adapterExecutionControls.get(f.runId)?.finish(); adapterExecutionControls.delete(f.runId); + await db.update(agentWakeupRequests).set({ status: "cancelled", finishedAt: new Date() }).where(and( + eq(agentWakeupRequests.companyId, companyId), inArray(agentWakeupRequests.status, ["queued", "deferred_issue_execution"]))); + await f.service.shutdown(); + } + }); + + it("retries the saved board-message turn in a connected task after environment qualification failed before provider startup", async () => { + const f = await failedChatRetryFixture("github", "42", true, false, true); + try { + const companyId = f.fixture.companyId, agentId = f.fixture.assignedAgentId, issueId = f.issue.id; + await initializeRunIdentity(db, { companyId, runId: f.runId, issueId, responsibleUserId: "owner-user", + messageIds: [String(f.action.payload.commentId)], cause: "chat_message" }); + const [comment] = await db.insert(issueComments).values({ companyId, issueId, authorType: "user", + authorUserId: "owner-user", body: "Continue from saved work and review the current head." }).returning(); + await db.update(agents).set({ status: "running", runtimeConfig: { heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 1 } } }) + .where(eq(agents.id, agentId)); + const [capacity] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running" }).returning(); + const heartbeat = heartbeatService(db); + const current = await heartbeat.wakeup(agentId, { source: "automation", triggerDetail: "system", + reason: "issue_reopened_via_comment", requestedByActorType: "user", requestedByActorId: "owner-user", + payload: { issueId, commentId: comment.id }, contextSnapshot: { issueId, taskKey: issueId, + source: "issue.comment.reopen", wakeCommentId: comment.id, wakeCommentIds: [comment.id] } }); + expect(current).toMatchObject({ status: "queued" }); + await initializeRunIdentity(db, { companyId, runId: current!.id, issueId, responsibleUserId: "owner-user", + parentRunId: f.runId, messageIds: [comment.id], cause: "issue_reopened_via_comment" }); + const diagnostic = 'Failed to acquire lease for environment "Managed computer" (sandbox): sandbox_snapshot_qualification_required: The snapshot qualification is missing or stale. Test this environment again before running Paperclip Runner.'; + await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(), runtimeMode: "legacy", + errorCode: "setup_failed", error: diagnostic, sessionIdBefore: "prior-provider-session-hint", + runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } }, + usageJson: { ledgerScope: { issueId, projectId: null } }, + resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false }, stopReason: "adapter_failed" } }) + .where(eq(heartbeatRuns.id, current!.id)); + await db.update(agentWakeupRequests).set({ status: "failed", finishedAt: new Date() }).where(eq(agentWakeupRequests.id, current!.wakeupRequestId!)); + await db.insert(heartbeatRunEvents).values([ + { companyId, agentId, runId: current!.id, seq: 1, eventType: "agent_observability.unavailable", stream: "system", level: "warn", + message: "Company observability was not applied: this harness is not supported.", payload: { reason: "unsupported_harness", revision: randomUUID() } }, + { companyId, agentId, runId: current!.id, seq: 2, eventType: "error", stream: "system", level: "error", message: diagnostic }, + { companyId, agentId, runId: current!.id, seq: 3, eventType: "lifecycle", stream: "system", level: "info", + message: "task.run.measured", payload: { durableSettlement: true, durationMs: 18000, outcome: "failed", phase: "task.run.measured" } }, + ]); + await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.id, capacity.id)); + await db.update(issues).set({ status: "blocked", executionRunId: null }).where(eq(issues.id, issueId)); + await db.update(agents).set({ status: "idle" }).where(eq(agents.id, agentId)); + const [failed] = await readHeartbeatRunsById(current!.id); + await expect(chatRunSourceMessageId(db, failed, { provider: "github" })).resolves.toBe(f.action.payload.commentId); + const app = express(); + app.use(express.json()); + app.use((req, _res, next) => { req.actor = boardActor(companyId); next(); }); + app.use("/api", issueRoutes(db, createStorageService().storage, { chatRunRetries: f.service })); + app.use(errorHandler); + const denySource = async (apply: () => Promise, restore: () => Promise, stale = false) => { + try { + await apply(); + const denied = await request(app).post(`/api/issues/${issueId}/retry`).send({ idempotencyKey: randomUUID(), observedRunId: current!.id }).expect(stale ? 200 : 409); + if (stale) expect(denied.body).toMatchObject({ accepted: false, stale: true }); + expect(await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and( + eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.reason, "task_retry")))).toHaveLength(0); + } finally { await restore(); } + }; + await denySource(() => db.update(issueComments).set({ body: "Edited after acceptance", updatedAt: new Date() }).where(eq(issueComments.id, comment.id)), + () => db.update(issueComments).set({ body: comment.body, updatedAt: sql`${issueComments.createdAt}` }).where(eq(issueComments.id, comment.id))); + await denySource(() => db.update(issueComments).set({ authorUserId: "another-user" }).where(eq(issueComments.id, comment.id)), + () => db.update(issueComments).set({ authorUserId: comment.authorUserId, updatedAt: sql`${issueComments.createdAt}` }).where(eq(issueComments.id, comment.id))); + await denySource(() => db.update(issueComments).set({ deletedAt: new Date() }).where(eq(issueComments.id, comment.id)), + () => db.update(issueComments).set({ deletedAt: null, updatedAt: sql`${issueComments.createdAt}` }).where(eq(issueComments.id, comment.id))); + const [sourceWake] = await readAgentWakeupRequestsById(current!.wakeupRequestId!); + await denySource(() => db.update(agentWakeupRequests).set({ requestedByActorId: "another-user" }).where(eq(agentWakeupRequests.id, sourceWake.id)), + () => db.update(agentWakeupRequests).set({ requestedByActorId: sourceWake.requestedByActorId }).where(eq(agentWakeupRequests.id, sourceWake.id))); + for (const patch of [ + { responsibleUserId: "another-user" }, { agentId: f.fixture.replacementAgentId }, + { contextSnapshot: { ...failed.contextSnapshot, issueId: randomUUID() } }, + { activeIdentityContextId: null }, { sessionIdAfter: "provider-started" }, { processPid: 999999 }, + { controllerLeaseExpiresAt: new Date(Date.now() + 60000) }, { stdoutExcerpt: "provider output" }, + { resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: true } } }, + { runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" }, unexpectedExecution: true } }, + ]) { + const restore = Object.fromEntries(Object.keys(patch).map(key => [key, failed[key as keyof typeof failed]])); + await denySource(() => db.update(heartbeatRuns).set(patch).where(eq(heartbeatRuns.id, failed.id)), + () => db.update(heartbeatRuns).set(restore).where(eq(heartbeatRuns.id, failed.id)), "agentId" in patch); + } + const contractId = randomUUID(), resultId = randomUUID(); + await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1, + schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "standard", + completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", contractJson: {}, + canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "test" }); + await denySource(async () => { + await db.update(heartbeatRuns).set({ nativeIssueId: issueId, completionContractId: contractId }).where(eq(heartbeatRuns.id, failed.id)); + await db.insert(nativeRunResults).values({ id: resultId, companyId, issueId, runId: failed.id, + completionContractId: contractId, schemaStatus: "accepted", serverFingerprint: resultId, canonicalSha256: resultId, + resultJson: { terminal: { runTerminalState: "succeeded" } } }); + }, async () => { + await db.delete(nativeRunResults).where(eq(nativeRunResults.id, resultId)); + await db.update(heartbeatRuns).set({ nativeIssueId: null, completionContractId: null }).where(eq(heartbeatRuns.id, failed.id)); + await db.delete(completionContracts).where(eq(completionContracts.id, contractId)); + }); + const [liveLease] = await db.insert(environmentLeases).values({ companyId, issueId, heartbeatRunId: failed.id, + provider: "daytona", providerLeaseId: "contradictory-board-startup", status: "active", leasePolicy: "ephemeral" }).returning(); + await denySource(async () => {}, () => db.delete(environmentLeases).where(eq(environmentLeases.id, liveLease.id))); + adapterExecutionControls.set(failed.id, createAdapterExecutionControl()); + await denySource(async () => {}, async () => { adapterExecutionControls.get(failed.id)?.finish(); adapterExecutionControls.delete(failed.id); }); + const idempotencyKey = randomUUID(); + const response = await request(app).post(`/api/issues/${issueId}/retry`).send({ idempotencyKey, observedRunId: current!.id }).expect(202); + expect(response.body).toMatchObject({ accepted: true, observedRunId: current!.id }); + expect(response.body.chatActionId).toBeUndefined(); + const [retry] = await readAgentWakeupRequestsById(response.body.requestId); + expect(retry).toMatchObject({ reason: "task_retry", requestedByActorId: "owner-user", + payload: { issueId, taskRetry: { sourceRunId: current!.id } } }); + expect(await db.select().from(chatActions).where(and(eq(chatActions.companyId, companyId), eq(chatActions.kind, "failed_run_retry")))).toHaveLength(0); + await db.update(agents).set({ status: "running" }).where(eq(agents.id, agentId)); + const [dispatchCapacity] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running" }).returning(); + await db.update(issueComments).set({ body: "Changed after Retry acceptance", updatedAt: new Date() }).where(eq(issueComments.id, comment.id)); + await heartbeat.resumeExecutionWaitComments(); + expect((await readAgentWakeupRequestsById(retry.id))[0]).toMatchObject({ status: "deferred_issue_execution", runId: null }); + await db.update(issueComments).set({ body: comment.body, updatedAt: sql`${issueComments.createdAt}` }).where(eq(issueComments.id, comment.id)); + await heartbeat.resumeExecutionWaitComments(); + const [successor] = (await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.contextSnapshot?.taskRetryRequestId === retry.id); + expect(successor).toMatchObject({ status: "queued", responsibleUserId: "owner-user", + contextSnapshot: { taskRetryRequestId: retry.id, previousRunId: current!.id } }); + expect((await readAgentWakeupRequestsById(retry.id))[0]).toMatchObject({ runId: successor.id }); + const realInitialize = runIdentityModule.initializeRunIdentity; + const initialized: string[] = []; + const probe = vi.spyOn(runIdentityModule, "initializeRunIdentity").mockImplementation(async (database, input) => { + if (input.runId !== successor.id) return realInitialize(database, input); + const [canonical] = await readAgentWakeupRequestsById(retry.id); + const [dispatch] = await readAgentWakeupRequestsById(successor.wakeupRequestId!); + const denyIdentity = async (apply: () => Promise, restore: () => Promise) => { + try { await apply(); await expect(realInitialize(database, input)).rejects.toThrow("This saved message can no longer be retried safely"); } + finally { await restore(); } + expect((await readHeartbeatRunsById(input.runId))[0].activeIdentityContextId).toBeNull(); + }; + await denyIdentity(() => db.update(agentWakeupRequests).set({ payload: { ...canonical.payload, + taskRetry: { ...(canonical.payload!.taskRetry as object), boardMessageOrigin: undefined } } }).where(eq(agentWakeupRequests.id, canonical.id)), + () => db.update(agentWakeupRequests).set({ payload: canonical.payload }).where(eq(agentWakeupRequests.id, canonical.id))); + await denyIdentity(() => db.update(agentWakeupRequests).set({ runId: failed.id }).where(eq(agentWakeupRequests.id, canonical.id)), + () => db.update(agentWakeupRequests).set({ runId: canonical.runId }).where(eq(agentWakeupRequests.id, canonical.id))); + await denyIdentity(() => db.update(agentWakeupRequests).set({ requestedByActorId: "another-user" }).where(eq(agentWakeupRequests.id, canonical.id)), + () => db.update(agentWakeupRequests).set({ requestedByActorId: canonical.requestedByActorId }).where(eq(agentWakeupRequests.id, canonical.id))); + await denyIdentity(() => db.update(agentWakeupRequests).set({ runId: failed.id }).where(eq(agentWakeupRequests.id, dispatch.id)), + () => db.update(agentWakeupRequests).set({ runId: dispatch.runId }).where(eq(agentWakeupRequests.id, dispatch.id))); + await denyIdentity(() => db.update(issueComments).set({ body: "Changed before provider dispatch", updatedAt: new Date() }).where(eq(issueComments.id, comment.id)), + () => db.update(issueComments).set({ body: comment.body, updatedAt: sql`${issueComments.createdAt}` }).where(eq(issueComments.id, comment.id))); + await denyIdentity(() => db.update(heartbeatRuns).set({ contextSnapshot: { ...successor.contextSnapshot, taskId: randomUUID() } }).where(eq(heartbeatRuns.id, successor.id)), + () => db.update(heartbeatRuns).set({ contextSnapshot: successor.contextSnapshot }).where(eq(heartbeatRuns.id, successor.id))); + await denyIdentity(() => db.update(issues).set({ executionRunId: failed.id }).where(eq(issues.id, issueId)), + () => db.update(issues).set({ executionRunId: successor.id }).where(eq(issues.id, issueId))); + const identity = await realInitialize(database, input); + expect(identity).toMatchObject({ responsibleUserId: "owner-user", parentContextId: retry.payload!.taskRetry && + (retry.payload!.taskRetry as { boardMessageOrigin: { identityContextId: string } }).boardMessageOrigin.identityContextId }); + const [executing] = await readHeartbeatRunsById(input.runId); + await expect(chatRunSourceMessageId(db, executing, { provider: "github" })).resolves.toBe(f.action.payload.commentId); + initialized.push(input.runId); + // This fixture tests the real dispatch and identity boundary, without + // allocating a provider or pretending its task work has completed. + throw new Error("board_retry_identity_probe_complete"); + }); + try { + await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.id, dispatchCapacity.id)); + await db.update(agents).set({ status: "idle" }).where(eq(agents.id, agentId)); + await heartbeat.resumeQueuedRuns(); + await heartbeat.drainActiveRunExecutions(); + expect(initialized).toEqual([successor.id]); + expect((await readHeartbeatRunsById(successor.id))[0]).toMatchObject({ status: "failed", error: "board_retry_identity_probe_complete" }); + expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.contextSnapshot?.taskRetryRequestId === retry.id)).toHaveLength(1); + expect((await readHeartbeatRunsById(current!.id))[0]).toMatchObject({ status: "failed", activeIdentityContextId: failed.activeIdentityContextId }); + } finally { probe.mockRestore(); await heartbeat.drainActiveRunExecutions(); } + const [firstRetry] = await readHeartbeatRunsById(successor.id); + const nextRequest = () => request(app).post(`/api/issues/${issueId}/retry`).send({ + idempotencyKey: randomUUID(), observedRunId: firstRetry.id }); + await db.update(heartbeatRuns).set({ sessionIdAfter: "contradictory-provider-session" }).where(eq(heartbeatRuns.id, firstRetry.id)); + await nextRequest().expect(409); + await db.update(heartbeatRuns).set({ sessionIdAfter: null }).where(eq(heartbeatRuns.id, firstRetry.id)); + await db.update(issueComments).set({ body: "Changed original accepted message", updatedAt: new Date() }).where(eq(issueComments.id, comment.id)); + await nextRequest().expect(409); + await db.update(issueComments).set({ body: comment.body, updatedAt: sql`${issueComments.createdAt}` }).where(eq(issueComments.id, comment.id)); + const [firstIntent] = await readAgentWakeupRequestsById(retry.id); + await db.update(agentWakeupRequests).set({ payload: { ...firstIntent.payload, + taskRetry: { ...(firstIntent.payload!.taskRetry as object), sourceRunId: firstRetry.id } } }).where(eq(agentWakeupRequests.id, firstIntent.id)); + await db.update(heartbeatRuns).set({ contextSnapshot: { ...firstRetry.contextSnapshot, previousRunId: firstRetry.id } }).where(eq(heartbeatRuns.id, firstRetry.id)); + await nextRequest().expect(409); + await db.update(agentWakeupRequests).set({ payload: firstIntent.payload }).where(eq(agentWakeupRequests.id, firstIntent.id)); + await db.update(heartbeatRuns).set({ contextSnapshot: firstRetry.contextSnapshot }).where(eq(heartbeatRuns.id, firstRetry.id)); + const second = await nextRequest().expect(202); + expect(second.body).toMatchObject({ accepted: true, observedRunId: firstRetry.id }); + expect(second.body.chatActionId).toBeUndefined(); + const [secondIntent] = await readAgentWakeupRequestsById(second.body.requestId); + expect(secondIntent.payload).toMatchObject({ taskRetry: { sourceRunId: firstRetry.id, + boardMessageOrigin: { commentId: comment.id, identityContextId: firstRetry.activeIdentityContextId } } }); + const secondIdentities: string[] = []; + const secondProbe = vi.spyOn(runIdentityModule, "initializeRunIdentity").mockImplementation(async (database, input) => { + const identity = await realInitialize(database, input); + if (input.companyId === companyId) { + expect(identity).toMatchObject({ responsibleUserId: "owner-user", parentContextId: firstRetry.activeIdentityContextId }); + const [executing] = await readHeartbeatRunsById(input.runId); + await expect(chatRunSourceMessageId(db, executing, { provider: "github" })).resolves.toBe(f.action.payload.commentId); + secondIdentities.push(input.runId); + throw new Error("board_retry_identity_probe_complete"); + } + return identity; + }); + try { + await heartbeat.resumeExecutionWaitComments(); + await heartbeat.resumeQueuedRuns(); + await heartbeat.drainActiveRunExecutions(); + const [secondRun] = (await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.contextSnapshot?.taskRetryRequestId === secondIntent.id); + expect(secondRun).toMatchObject({ status: "failed", error: "board_retry_identity_probe_complete", responsibleUserId: "owner-user" }); + expect(secondIdentities).toEqual([secondRun.id]); + expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.contextSnapshot?.taskRetryRequestId === secondIntent.id)).toHaveLength(1); + } finally { secondProbe.mockRestore(); await heartbeat.drainActiveRunExecutions(); } + + } finally { + await db.update(agentWakeupRequests).set({ status: "failed", finishedAt: new Date() }).where(and( + eq(agentWakeupRequests.companyId, f.fixture.companyId), eq(agentWakeupRequests.reason, "task_retry"), + eq(agentWakeupRequests.status, "deferred_issue_execution"))); + await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(and( + eq(heartbeatRuns.companyId, f.fixture.companyId), eq(heartbeatRuns.status, "queued"))); + await f.service.shutdown(); + } + }); + + it("distinguishes an inherited GitHub requester from permission to replay a committed provider failure", async () => { + const f = await failedChatRetryFixture("github", "42", true, false, true); + try { + const companyId = f.fixture.companyId, agentId = f.fixture.assignedAgentId, issueId = f.issue.id; + await initializeRunIdentity(db, { companyId, runId: f.runId, issueId, responsibleUserId: "owner-user", + messageIds: [String(f.action.payload.commentId)], cause: "chat_message" }); + await db.update(agents).set({ status: "running", runtimeConfig: { heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 1 } } }) + .where(eq(agents.id, agentId)); + const [capacity] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running" }).returning(); + const heartbeat = heartbeatService(db); + const resumed = await heartbeat.wakeup(agentId, { source: "on_demand", triggerDetail: "manual", + reason: "issue_tree_resumed", requestedByActorType: "user", requestedByActorId: "owner-user", + payload: { issueId }, contextSnapshot: { issueId, taskKey: f.issue.identifier, source: "issue.tree_resume" } }); + expect(resumed).toMatchObject({ status: "queued" }); + await initializeRunIdentity(db, { companyId, runId: resumed!.id, issueId, responsibleUserId: "owner-user", + parentRunId: f.runId, cause: "issue_tree_resumed" }); + const contractId = randomUUID(), resultId = randomUUID(); + await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1, + schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "standard", + completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", contractJson: {}, + canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "test" }); + await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(), runtimeMode: "native", + nativeIssueId: issueId, nativePhase: "committed", errorCode: "adapter_failed", error: "Native session failed", + completionContractId: contractId }).where(eq(heartbeatRuns.id, resumed!.id)); + await db.update(agentWakeupRequests).set({ status: "failed", finishedAt: new Date() }).where(eq(agentWakeupRequests.id, resumed!.wakeupRequestId!)); + await db.insert(nativeRunResults).values({ id: resultId, companyId, issueId, runId: resumed!.id, + completionContractId: contractId, schemaStatus: "accepted", serverFingerprint: resultId, canonicalSha256: resultId, + resultJson: { result: { reportedWorkDisposition: "needs_review", completionClaim: { objectiveSatisfied: false } }, + terminal: { runTerminalState: "failed", turnTerminalState: "failed" } } }); + await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId: resumed!.id, phase: "committed", resultId }); + const lease = { id: randomUUID(), companyId, heartbeatRunId: resumed!.id, provider: "daytona", providerLeaseId: "contained-resume-provider" }; + await db.insert(environmentLeases).values({ ...lease, issueId, status: "expired", leasePolicy: "ephemeral", + cleanupStatus: "success", releasedAt: new Date(), metadata: { remoteExecutionTermination: + remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "destroyed" }) } }); + await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.id, capacity.id)); + await db.update(issues).set({ status: "blocked", executionRunId: null }).where(eq(issues.id, issueId)); + await db.update(agents).set({ status: "idle" }).where(eq(agents.id, agentId)); + const [failedResume] = await readHeartbeatRunsById(resumed!.id); + expect(failedResume.contextSnapshot).toMatchObject({ source: "issue.tree_resume" }); + expect(failedResume.contextSnapshot).not.toHaveProperty("wakeCommentIds"); + await expect(chatRunSourceMessageId(db, failedResume, { provider: "github" })) + .resolves.toBe(f.action.payload.commentId); + const input = { db, companyId, issueId, actorId: "owner-user", actorSource: "cloud_tenant", + observedRunId: resumed!.id, idempotencyKey: randomUUID(), prepareChatRetry: f.service.prepareFailedChatRunRetry }; + const app = express(); + app.use(express.json()); + app.use((req, _res, next) => { req.actor = boardActor(companyId); next(); }); + app.use("/api", issueRoutes(db, createStorageService().storage, { chatRunRetries: f.service })); + app.use(errorHandler); + const denied = await request(app).post(`/api/issues/${issueId}/retry`).send({ + idempotencyKey: input.idempotencyKey, observedRunId: input.observedRunId, + }).expect(409); + expect(denied.body).toMatchObject({ error: "This attempt already recorded a result. Send a new message on the task to continue from the saved work.", + details: { code: "chat_failed_run_retry_not_authorized", reason: "accepted_result_without_new_input" } }); + // Isolate the later accepted-result guard using only owned fixture data. + // Resolving inherited provenance cannot by itself authorize replay. + await db.update(heartbeatRuns).set({ contextSnapshot: { ...failedResume.contextSnapshot, + source: "chat:github", wakeCommentId: f.action.payload.commentId, wakeCommentIds: [f.action.payload.commentId] } }) + .where(eq(heartbeatRuns.id, failedResume.id)); + try { + await expect(requestTaskRetry({ ...input, idempotencyKey: randomUUID() })).rejects.toMatchObject({ details: { + code: "chat_failed_run_retry_not_authorized", reason: "accepted_result_without_new_input" } }); + } finally { await db.update(heartbeatRuns).set({ contextSnapshot: failedResume.contextSnapshot }).where(eq(heartbeatRuns.id, failedResume.id)); } + expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.retryOfRunId === resumed!.id)).toHaveLength(0); + } finally { await f.service.shutdown(); } + }); + + it.each([ + ...(["real_budget_ingress", "legacy_budget_ingress", "legacy_prior_response", "legacy_execution", "legacy_execution_log", "legacy_execution_lease", "legacy_execution_profile", "legacy_execution_event", "legacy_execution_controller", "legacy_execution_intent", "legacy_no_prior", "legacy_prior_late", "legacy_prior_unprocessed", "legacy_prior_delivery_unprocessed", "legacy_prior_numeric_app", "legacy_response_conflict", "legacy_prior_wrong_app", "legacy_prior_wrong_agent", "legacy_prior_unlinked", "legacy_prior_retargeted", "legacy_explicit_null", "legacy_explicit_empty", "legacy_explicit_malformed", "legacy_explicit_mismatch", "legacy_current_app_changed", "source_app_changed", "source_destination_changed", "accepted", "multiple_saved", "real_queue", "committed_succeeded", "committed_consumed", "committed_failure", "committed_failed_publication", "committed_unknown_publication", "committed_pending_publication", "committed_mixed_publication", "committed_consumed_publication", "committed_succeeded_publication", "committed_delayed_cleanup", "committed_unknown_cleanup", "committed_live_controller", "unreconciled", "source_edited", "source_claimed", "malformed_previous", "endpoint_paused", "principal_revoked", "still_budget_blocked"] as const) + .map(mode => ({ provider: "github" as const, mode })), + ])("retries the newer saved $provider request instead of the failed request after a budget hold: $mode", async ({ provider, mode }) => { + const legacyIngress = mode.startsWith("legacy_"); + const realBudgetIngress = mode === "real_budget_ingress" || legacyIngress; + const executionRevalidation = mode.startsWith("legacy_execution"); + if (legacyIngress) { + // The legacy intake producer retained requester authority but no App ID. + // A budget refusal never reaches the acknowledgement which could bind it. + const historical = vi.spyOn(githubTaskPreparation, "retainGitHubDeliveryAuthority").mockImplementation( + async (tx, endpoint, deliveryId, linkedUserId, taskUserId) => { + const guest = !linkedUserId; + await tx.update(chatDeliveries).set({ normalizedEvent: sql`${chatDeliveries.normalizedEvent} || ${JSON.stringify({ + githubAuthority: { guest, responsibleUserId: taskUserId, sponsorUserId: guest ? endpoint.sponsorUserId : null }, + })}::jsonb` }).where(eq(chatDeliveries.id, deliveryId)); + }); + onTestFinished(() => historical.mockRestore()); + } + const context = await failedChatRetryFixture(provider, realBudgetIngress ? "42" : undefined, true, false, realBudgetIngress); + try { + const companyId = context.fixture.companyId; + const [skill] = await db.insert(companySkills).values({ companyId, key: `company/${companyId}/saved-review`, + slug: "saved-review", name: "Saved review", markdown: "# Saved review" }).returning(); + if (provider === "github") { + const configured = await githubChatManagementService(db).configuration(context.endpoint.id, "owner-user"); + const configuration = { ...configured.configuration, defaults: { ...configured.configuration.defaults, + instructions: `Use [/saved-review](skill://${skill.id}).` } }; + if (realBudgetIngress) await githubChatManagementService(db).saveConfiguration(context.endpoint.id, + { expectedRevision: configured.revision, configuration }, "owner-user"); + else await db.insert(chatGitHubConfigurations).values({ companyId, endpointId: context.endpoint.id, + configuration, updatedByUserId: "owner-user", revision: 2 }); + } + if (legacyIngress) { + await githubResponseCommentService(db, context.providerFetch).tryAcknowledge(context.action.deliveryId!); + const [prior] = await db.select().from(chatActions).where(and( + eq(chatActions.deliveryId, context.action.deliveryId!), eq(chatActions.kind, "github_response_comment"))); + expect(prior).toMatchObject({ status: "processed", payload: { + botExternalId: (await context.service.get(context.endpoint.id)).botExternalId, agentId: context.fixture.assignedAgentId, + } }); + } + if (mode === "legacy_no_prior") await db.delete(chatActions).where(and( + eq(chatActions.deliveryId, context.action.deliveryId!), eq(chatActions.kind, "github_response_comment"))); + if (realBudgetIngress) { + await db.update(heartbeatRuns).set({ costAccountingPending: true }).where(eq(heartbeatRuns.id, context.runId)); + await db.insert(budgetPolicies).values({ companyId, scopeType: "agent", scopeId: context.fixture.assignedAgentId, + metric: "billed_cents", windowKind: "lifetime", amount: 100, notifyEnabled: false }); + context.wakeup.mockImplementation(heartbeatService(db).wakeup); + } else context.wakeup.mockImplementation(async (agentId, opts) => { + const request = opts.durableChatRequest!; + await request.authorize(db); + await db.insert(agentWakeupRequests).values({ id: request.id, companyId, + agentId, source: opts.source!, triggerDetail: opts.triggerDetail, reason: "budget.blocked", + requestedByActorType: request.requestedByActorType, requestedByActorId: request.requestedByActorId, + requestedAt: request.requestedAt, idempotencyKey: request.idempotencyKey, + payload: { ...opts.payload, executionWait: { scopeType: "agent", scopeId: agentId } }, + status: "skipped", finishedAt: new Date() }); + return null; + }); + if (realBudgetIngress) { + const endpoint = await context.service.get(context.endpoint.id); + attachFakeGitHubIssueCommentWebhook({ botUsername: endpoint.botUsername!, callbacks: context.runtime.configurations.get(endpoint.id)!.callbacks, + endpointId: endpoint.id, runtime: context.runtime.endpoints.get(endpoint.id)! }); + const ref = (await db.select().from(toolConnections).where(eq(toolConnections.id, endpoint.connectionId)))[0]! + .credentialSecretRefs.find(ref => ref.configPath === "credentials.webhookSecret")!; + const secret = await secretService(db).resolveSecretValue(companyId, ref.secretId, ref.versionSelector ?? "latest"); + const response = await context.service.handleWebhook(endpoint.publicId, "github", signedGitHubWebhookRequest({ + delivery: randomUUID(), event: "issue_comment", webhookSecret: secret, + payload: { action: "created", installation: { id: 2468 }, + repository: { id: 97531, full_name: "paperclipai/paperclip", owner: { id: 1357, login: "paperclipai" } }, + issue: { number: 791, pull_request: { url: "https://api.github.com/repos/paperclipai/paperclip/pulls/791" } }, + sender: { id: 42, login: "octocat", type: "User" }, + comment: { id: 990092, body: `@${endpoint.botUsername} review using the selected instructions`, + user: { id: 42, login: "octocat", type: "User" }, created_at: new Date().toISOString() } }, + })); + expect(response.status).toBeLessThan(300); + } else await deliverMessage({ callbacks: context.runtime.configurations.get(context.endpoint.id)!.callbacks, + endpointId: context.endpoint.id, provider, thread: context.thread.thread, + message: makeMessage({ id: provider === "telegram" ? `${context.thread.thread.channelId}:102` : "saved-after-budget-hold", text: "@maya review using the selected instructions", + userId: provider === "telegram" ? context.thread.thread.channelId : "U-SAFE-PROGRESS", mentioned: true }), + trigger: context.thread.thread.isDM ? "direct_message" : "mention" }); + if (mode === "multiple_saved") await deliverMessage({ + callbacks: context.runtime.configurations.get(context.endpoint.id)!.callbacks, + endpointId: context.endpoint.id, provider, thread: context.thread.thread, + message: makeMessage({ id: "second-saved-after-budget-hold", text: "@maya include the follow-up", + userId: "U-SAFE-PROGRESS", mentioned: true }), trigger: "mention" }); + const savedInputs = await db.select().from(chatActions).where(and( + eq(chatActions.companyId, companyId), eq(chatActions.conversationId, context.conversation.id), + eq(chatActions.kind, "inbound_wakeup"), sql`${chatActions.id} <> ${context.action.id}`)) + .orderBy(asc(chatActions.createdAt), asc(chatActions.id)); + const saved = savedInputs.at(-1)!; + const expectedComments = savedInputs.map(action => action.payload.commentId); + expect(saved, JSON.stringify((await db.select({ state: chatDeliveries.state, error: chatDeliveries.redactedError, event: chatDeliveries.eventKind }).from(chatDeliveries).where(eq(chatDeliveries.endpointId, context.endpoint.id))))).toMatchObject({ status: "failed", result: { receiptStatus: "skipped" } }); + expect(saved.payload.commentId).not.toBe(context.action.payload.commentId); + if (realBudgetIngress) { + expect((await readAgentWakeupRequestsById(saved.id))[0]).toMatchObject({ status: "skipped", reason: "budget.blocked", + runId: null, requestedByActorType: "user", requestedByActorId: "owner-user" }); + await db.update(budgetPolicies).set({ isActive: false }).where(eq(budgetPolicies.companyId, companyId)); + await budgetService(db).getInvocationBlock(companyId, context.fixture.assignedAgentId); + const lifecycle = configureAgentLifecycle(db, { requiredPluginIds: async () => [], + runPlugin: async () => "complete", runHost: async () => "complete" }); + try { await lifecycle.process(context.fixture.assignedAgentId); } finally { await lifecycle.stop(); } + expect((await db.select().from(agents).where(eq(agents.id, context.fixture.assignedAgentId)))[0].lifecycleHolds).not.toContain("budget"); + expect((await db.select().from(agents).where(eq(agents.id, context.fixture.assignedAgentId)))[0]).toMatchObject({status: "idle"}); + expect((await readHeartbeatRunsById(context.runId))[0]).toMatchObject({status: "failed"}); + } + const unadmitted = () => db.select({ id: issues.id }).from(issues).where(and(eq(issues.id, context.issue.id), + unadmittedChatWakeupCondition(issues.id, issues.companyId))); + expect(await unadmitted()).toHaveLength(1); + // The budget was cleared. Retry must reconsider this independent saved + // request through chat admission, not replay the older failed turn. + const held = mode === "unreconciled" || mode === "committed_delayed_cleanup" || mode === "committed_unknown_cleanup" || mode === "committed_live_controller"; + if (mode === "unreconciled" || mode.startsWith("committed_") || executionRevalidation) { + await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: context.issue.id, + nativePhase: "terminal_failure", runnerProfileJson: {} }).where(eq(heartbeatRuns.id, context.runId)); + await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: context.issue.id, + kind: "active_run_watchdog", cause: "native_continuation_requires_reconciliation", status: "active", + ownerType: "board", fingerprint: randomUUID(), nextAction: "Reconcile the previous executor.", + evidence: { runId: context.runId, automaticRecovery: { replay: "blocked" } } }); + if (mode.startsWith("committed_") || executionRevalidation) { + const contractId = randomUUID(), resultId = randomUUID(); + await db.insert(completionContracts).values({ id: contractId, companyId, issueId: context.issue.id, + revision: 1, schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "standard", + completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", contractJson: {}, + canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "test" }); + await db.update(heartbeatRuns).set({ nativePhase: "committed", completionContractId: contractId }) + .where(eq(heartbeatRuns.id, context.runId)); + await db.insert(nativeRunResults).values({ id: resultId, companyId, issueId: context.issue.id, + runId: context.runId, completionContractId: contractId, serverFingerprint: resultId, + schemaStatus: "accepted", canonicalSha256: resultId, + resultJson: { terminal: { runTerminalState: mode.startsWith("committed_succeeded") ? "succeeded" : "failed" } } }); + await db.insert(nativeRunFinalizations).values({ companyId, issueId: context.issue.id, + runId: context.runId, resultId, phase: "committed", + leaseOwner: mode === "committed_live_controller" ? "active-controller" : null }); + const lease = { id: randomUUID(), companyId, heartbeatRunId: context.runId, + provider: "daytona", providerLeaseId: "owned-saved-input-sandbox" }; + await db.insert(environmentLeases).values({ ...lease, issueId: context.issue.id, + status: "expired", cleanupStatus: "success", releasedAt: new Date(), leasePolicy: "ephemeral", + metadata: mode === "committed_unknown_cleanup" || mode === "committed_delayed_cleanup" ? {} : { + remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "destroyed" }) } }); + } + } + if (executionRevalidation || ["committed_failed_publication", "committed_unknown_publication", "committed_pending_publication", "committed_mixed_publication", "committed_consumed_publication", "committed_succeeded_publication"].includes(mode)) { + // A failed native run materializes its summary before delivery. Keep + // this historical output distinct from the newer user's request. + const [resultComment] = await db.insert(issueComments).values({ companyId, issueId: context.issue.id, + authorAgentId: context.fixture.assignedAgentId, createdByRunId: context.runId, + body: "The OpenCode run failed before it completed." }).returning(); + await db.insert(chatPublications).values({ companyId, endpointId: context.endpoint.id, + conversationId: context.conversation.id, issueId: context.issue.id, commentId: resultComment.id, + idempotencyKey: `run:${context.runId}:failed-result`, payload: { text: resultComment.body }, + state: executionRevalidation || ["committed_failed_publication", "committed_mixed_publication", "committed_consumed_publication", "committed_succeeded_publication"].includes(mode) ? "failed" + : mode === "committed_unknown_publication" ? "delivery_unknown" : "pending", + redactedError: "GitHub failure has no request binding" }); + if (mode === "committed_mixed_publication") await db.insert(chatPublications).values({ companyId, + endpointId: context.endpoint.id, conversationId: context.conversation.id, issueId: context.issue.id, + commentId: resultComment.id, idempotencyKey: `run:${context.runId}:uncertain-result`, + payload: { text: resultComment.body }, state: "delivery_unknown" }); + } + if (mode.startsWith("committed_consumed")) await db.update(heartbeatRuns).set({ contextSnapshot: { + ...((await readHeartbeatRunsById(context.runId))[0]).contextSnapshot, + wakeCommentId: saved.payload.commentId, wakeCommentIds: [saved.payload.commentId], + } }).where(eq(heartbeatRuns.id, context.runId)); + if (mode === "malformed_previous") await db.update(heartbeatRuns).set({ contextSnapshot: { + ...((await readHeartbeatRunsById(context.runId))[0]).contextSnapshot, wakeCommentIds: ["unbound"], + } }).where(eq(heartbeatRuns.id, context.runId)); + if (mode === "source_app_changed" || mode === "source_destination_changed") { + const [delivery] = await db.select().from(chatDeliveries).where(eq(chatDeliveries.id, saved.deliveryId!)); + await db.update(chatDeliveries).set({ normalizedEvent: { ...delivery.normalizedEvent, + ...(mode === "source_app_changed" ? { githubAuthority: { ...(delivery.normalizedEvent.githubAuthority as object), botExternalId: "different-app" } } + : { resource: { ...(delivery.normalizedEvent.resource as object), providerResourceId: "another/repository" } }), + } }).where(eq(chatDeliveries.id, delivery.id)); + } + const previousPublications = await readChatPublicationsByEndpointId(context.endpoint.id); + const request = { db, companyId, issueId: context.issue.id, actorId: "owner-user", actorSource: "cloud_tenant", + observedRunId: context.runId, idempotencyKey: randomUUID(), prepareChatRetry: context.service.prepareFailedChatRunRetry }; + if (legacyIngress) { + const [delivery] = await db.select().from(chatDeliveries).where(eq(chatDeliveries.id, saved.deliveryId!)); + expect(delivery.normalizedEvent.githubAuthority).not.toHaveProperty("botExternalId"); + expect(await db.select({ id: chatActions.id }).from(chatActions).where(and( + eq(chatActions.deliveryId, saved.deliveryId!), eq(chatActions.kind, "github_response_comment")))).toHaveLength(0); + const priorWhere = and(eq(chatActions.deliveryId, context.action.deliveryId!), eq(chatActions.kind, "github_response_comment")); + if (mode === "legacy_prior_late") await db.update(chatActions).set({ createdAt: new Date(delivery.createdAt.getTime() + 1) }).where(priorWhere); + if (mode === "legacy_prior_unprocessed") await db.update(chatActions).set({ status: "received" }).where(priorWhere); + if (mode === "legacy_prior_delivery_unprocessed") await db.update(chatDeliveries).set({ state: "failed" }) + .where(eq(chatDeliveries.id, context.action.deliveryId!)); + if (mode === "legacy_prior_numeric_app") await db.update(chatActions).set({ + payload: sql`jsonb_set(${chatActions.payload}, '{botExternalId}', ${JSON.stringify(Number((await context.service.get(context.endpoint.id)).botExternalId))}::jsonb)`, + }).where(priorWhere); + if (mode === "legacy_response_conflict") await db.insert(chatActions).values({ companyId, + endpointId: context.endpoint.id, conversationId: context.conversation.id, deliveryId: delivery.id, + principalId: delivery.principalId, kind: "github_response_comment", providerActionId: `github_response:${delivery.id}`, + status: "processed", payload: { botExternalId: "conflicting-app", agentId: context.fixture.assignedAgentId } }); + if (mode === "legacy_prior_wrong_app" || mode === "legacy_prior_wrong_agent") await db.update(chatActions).set({ + payload: sql`jsonb_set(${chatActions.payload}, ${mode === "legacy_prior_wrong_app" ? "{botExternalId}" : "{agentId}"}, '"another-identity"'::jsonb)`, + }).where(priorWhere); + if (mode === "legacy_prior_unlinked") await db.delete(chatMessageLinks).where(eq(chatMessageLinks.deliveryId, context.action.deliveryId!)); + if (mode === "legacy_prior_retargeted") await db.update(chatActions).set({ providerActionId: "github_response:unbound" }).where(priorWhere); + if (mode.startsWith("legacy_explicit_")) await db.update(chatDeliveries).set({ normalizedEvent: { + ...delivery.normalizedEvent, githubAuthority: { ...(delivery.normalizedEvent.githubAuthority as object), + botExternalId: mode === "legacy_explicit_null" ? null : mode === "legacy_explicit_empty" ? "" + : mode === "legacy_explicit_malformed" ? { appId: "untrusted" } : "another-app" }, + } }).where(eq(chatDeliveries.id, delivery.id)); + if (mode === "legacy_current_app_changed") await db.update(chatEndpoints).set({ botExternalId: randomUUID() }) + .where(eq(chatEndpoints.id, context.endpoint.id)); + if (mode !== "legacy_budget_ingress" && mode !== "legacy_prior_response" && !executionRevalidation) { + await expect(requestTaskRetry(request)).rejects.toMatchObject({ details: { + code: "chat_failed_run_retry_not_authorized", reason: "source_authorization", authorityReason: "app_binding", + } }); + expect((await readAgentWakeupRequestsById(saved.id))[0]).toMatchObject({ status: "skipped", runId: null }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual(previousPublications); + return; + } + } + if (mode === "source_app_changed" || mode === "source_destination_changed") { + await expect(requestTaskRetry(request)).rejects.toMatchObject({ details: { + code: "chat_failed_run_retry_not_authorized", reason: "source_authorization", + authorityReason: mode === "source_app_changed" ? "app_binding" : "destination_binding", + } }); + expect((await readAgentWakeupRequestsById(saved.id))[0]).toMatchObject({ status: "skipped", runId: null }); + expect((await readHeartbeatRunsById(context.runId))[0]).toMatchObject({ status: "failed" }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual(previousPublications); + return; + } + if (mode.startsWith("committed_succeeded") || mode.startsWith("committed_consumed") || mode === "malformed_previous" || mode === "committed_unknown_publication" || mode === "committed_pending_publication" || mode === "committed_mixed_publication") { + await expect(requestTaskRetry(request)).rejects.toMatchObject({ details: { + code: "chat_failed_run_retry_not_authorized", + ...(mode.endsWith("publication") ? { reason: mode === "committed_consumed_publication" || mode === "committed_succeeded_publication" + ? "accepted_result_without_new_input" : "prior_result_publication" } : {}), + } }); + expect(await unadmitted()).toHaveLength(1); + return; + } + const receipt = await requestTaskRetry(request); + expect(receipt).toMatchObject({ accepted: true }); + const [retry] = await db.select().from(chatActions).where(eq(chatActions.id, receipt.chatActionId!)); + expect(retry.payload).toMatchObject({ commentIds: expectedComments }); + expect(await db.select({ id: chatActions.id }).from(chatActions).where(and(eq(chatActions.id, retry.id), dueChatAction("failed_run_retry")))).toHaveLength(1); + expect((await readAgentWakeupRequestsById(saved.id))[0]).toMatchObject({ status: "skipped", runId: null }); + expect((await readHeartbeatRunsById(context.runId))[0]).toMatchObject({ status: "failed" }); + if (mode === "source_claimed") await db.update(agentWakeupRequests).set({ status: "queued", runId: context.runId }) + .where(eq(agentWakeupRequests.id, saved.id)); + if (mode === "source_edited") await db.update(issueComments).set({ body: "Edited after Retry" }) + .where(eq(issueComments.id, String(saved.payload.commentId))); + if (mode === "endpoint_paused") await db.update(chatEndpoints).set({ status: "paused" }) + .where(eq(chatEndpoints.id, context.endpoint.id)); + if (mode === "principal_revoked") await db.update(companyMemberships).set({ status: "suspended" }) + .where(and(eq(companyMemberships.companyId, companyId), eq(companyMemberships.principalId, "owner-user"))); + if (realBudgetIngress || mode === "real_queue" || mode === "unreconciled" || mode.startsWith("committed_")) { + // Capacity prevents a real provider start while exercising production + // admission, persisted source identity, and queue ownership. + await db.update(agents).set({ adapterType: "paperclip_runner", status: "running", + runtimeConfig: { heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 1 } } }) + .where(eq(agents.id, context.fixture.assignedAgentId)); + await db.insert(heartbeatRuns).values({ companyId, agentId: context.fixture.assignedAgentId, + status: "running", startedAt: new Date() }); + context.wakeup.mockImplementation(heartbeatService(db).wakeup); + } else if (mode !== "still_budget_blocked") context.wakeup.mockImplementation(async () => ({ accepted: true })); + if (mode === "source_edited" || mode === "source_claimed" || mode === "endpoint_paused" || mode === "principal_revoked") { + expect(await context.service.processFailedChatRunRetry(receipt.chatActionId!)).toMatchObject({ status: "failed" }); + expect(await readAgentWakeupRequestsById(receipt.chatActionId!)).toHaveLength(0); + } else { + await context.service.processFailedChatRunRetry(receipt.chatActionId!); + const [admitted] = await readAgentWakeupRequestsById(receipt.chatActionId!); + expect(admitted, JSON.stringify({ action: (await db.select().from(chatActions).where(eq(chatActions.id, receipt.chatActionId!)))[0], calls: context.wakeup.mock.calls.length })).toMatchObject({ requestedByActorId: "owner-user", + payload: { wakeCommentId: saved.payload.commentId }, + status: mode === "still_budget_blocked" ? "skipped" : held ? "deferred_issue_execution" : "queued" }); + expect(context.wakeup.mock.calls.at(-1)![1].contextSnapshot).toMatchObject({ + source: `chat:${provider}`, wakeCommentId: saved.payload.commentId, + }); + + if (executionRevalidation) { + const heartbeat = heartbeatService(db); + const originalAuthorize = durableChatWakeup.authorizeFailedChatRunRetryWake; + const verified: string[] = []; + const identitySources: Array<{ runId: string; messageId: string | null }> = []; + const realInitialize = runIdentityModule.initializeRunIdentity; + const identityProof = vi.spyOn(runIdentityModule, "initializeRunIdentity").mockImplementation(async (database, input) => { + const identity = await realInitialize(database, input); + if (input.companyId === companyId && mode === "legacy_execution" && verified.length > 1) { + identitySources.push({ runId: input.runId, messageId: identity.messageId }); + // Exercise the actual initializer after the real Retry execution + // authority, without allocating a native provider for this probe. + throw new Error("certified_retry_identity_probe_complete"); + } + return identity; + }); + const executionProof = vi.spyOn(durableChatWakeup, "authorizeFailedChatRunRetryWake").mockImplementation(async (...args) => { + const input = args[2]; + const authorized = await originalAuthorize(...args); + if (input.companyId === companyId && input.phase === "execution") { + expect(authorized).toBe(true); + if (!verified.length && mode === "legacy_execution") { + // Exercise the actual dispatch authorization transaction after + // real admission; restore each contradiction before the next. + const tx = args[1]; + for (const changed of [{ runId: context.runId }, { companyId: randomUUID() }, + { agentId: randomUUID() }, { issueId: randomUUID() }]) { + await expect(originalAuthorize(args[0], tx, { ...input, ...changed })).rejects.toThrow(); + } + const [current] = await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId!)); + const [task] = await tx.select().from(issues).where(eq(issues.id, context.issue.id)); + const [wake] = await tx.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, input.wakeupRequestId!)); + const [coordinator] = await tx.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, context.runId)); + const [nativeResult] = await tx.select().from(nativeRunResults).where(eq(nativeRunResults.runId, context.runId)); + const [lease] = await tx.select().from(environmentLeases).where(eq(environmentLeases.heartbeatRunId, context.runId)); + const [sourceComment] = await tx.select().from(issueComments).where(eq(issueComments.id, saved.payload.commentId as string)); + const [endpoint] = await tx.select().from(chatEndpoints).where(eq(chatEndpoints.id, context.endpoint.id)); + const resolved = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, companyId), + eq(issueRecoveryActions.sourceIssueId, context.issue.id), eq(issueRecoveryActions.status, "resolved"))); + const deny = async (mutate: () => Promise, restore: () => Promise) => { + await mutate(); + try { await expect(originalAuthorize(...args)).rejects.toThrow(); } + finally { await restore(); } + }; + await deny(() => tx.update(heartbeatRuns).set({ contextSnapshot: { ...current.contextSnapshot, issueId: randomUUID() } }).where(eq(heartbeatRuns.id, current.id)), + () => tx.update(heartbeatRuns).set({ contextSnapshot: current.contextSnapshot }).where(eq(heartbeatRuns.id, current.id))); + await deny(() => tx.update(heartbeatRuns).set({ contextSnapshot: { ...current.contextSnapshot, explicitUserContinuation: { + ...current.contextSnapshot!.explicitUserContinuation as object, commentId: randomUUID() } } }).where(eq(heartbeatRuns.id, current.id)), + () => tx.update(heartbeatRuns).set({ contextSnapshot: current.contextSnapshot }).where(eq(heartbeatRuns.id, current.id))); + await deny(() => tx.update(agentWakeupRequests).set({ runId: context.runId }).where(eq(agentWakeupRequests.id, wake.id)), + () => tx.update(agentWakeupRequests).set({ runId: wake.runId }).where(eq(agentWakeupRequests.id, wake.id))); + await deny(() => tx.update(issues).set({ executionRunId: context.runId }).where(eq(issues.id, task.id)), + () => tx.update(issues).set({ executionRunId: task.executionRunId }).where(eq(issues.id, task.id))); + await deny(() => tx.update(issues).set({ assigneeAgentId: null }).where(eq(issues.id, task.id)), + () => tx.update(issues).set({ assigneeAgentId: task.assigneeAgentId }).where(eq(issues.id, task.id))); + await deny(() => tx.update(issueComments).set({ body: "Changed authorized message" }).where(eq(issueComments.id, sourceComment.id)), + () => tx.update(issueComments).set({ body: sourceComment.body, updatedAt: sourceComment.updatedAt }).where(eq(issueComments.id, sourceComment.id))); + await deny(() => tx.update(chatEndpoints).set({ status: "paused" }).where(eq(chatEndpoints.id, endpoint.id)), + () => tx.update(chatEndpoints).set({ status: endpoint.status }).where(eq(chatEndpoints.id, endpoint.id))); + await deny(() => tx.update(agents).set({ status: "paused" }).where(eq(agents.id, input.agentId)), + () => tx.update(agents).set({ status: "running" }).where(eq(agents.id, input.agentId))); + await deny(() => tx.update(nativeRunFinalizations).set({ leaseOwner: "still-running" }).where(eq(nativeRunFinalizations.runId, coordinator.runId)), + () => tx.update(nativeRunFinalizations).set({ leaseOwner: coordinator.leaseOwner }).where(eq(nativeRunFinalizations.runId, coordinator.runId))); + await deny(async () => { adapterExecutionControls.set(context.runId, createAdapterExecutionControl()); }, + async () => { adapterExecutionControls.get(context.runId)?.finish(); adapterExecutionControls.delete(context.runId); }); + await deny(() => tx.update(agentWakeupRequests).set({ status: "running" }).where(eq(agentWakeupRequests.id, wake.id)), + () => tx.update(agentWakeupRequests).set({ status: wake.status }).where(eq(agentWakeupRequests.id, wake.id))); + await deny(() => tx.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60000) }).where(eq(heartbeatRuns.id, context.runId)), + () => tx.update(heartbeatRuns).set({ controllerLeaseExpiresAt: null }).where(eq(heartbeatRuns.id, context.runId))); + await deny(() => tx.update(environmentLeases).set({ metadata: {} }).where(eq(environmentLeases.id, lease.id)), + () => tx.update(environmentLeases).set({ metadata: lease.metadata }).where(eq(environmentLeases.id, lease.id))); + await deny(() => tx.update(nativeRunResults).set({ resultJson: { terminal: { runTerminalState: "succeeded" } } }).where(eq(nativeRunResults.id, nativeResult.id)), + () => tx.update(nativeRunResults).set({ resultJson: nativeResult.resultJson }).where(eq(nativeRunResults.id, nativeResult.id))); + for (const action of resolved) await deny(() => tx.update(issueRecoveryActions).set({ evidence: {} }).where(eq(issueRecoveryActions.id, action.id)), + () => tx.update(issueRecoveryActions).set({ evidence: action.evidence }).where(eq(issueRecoveryActions.id, action.id))); + } + verified.push(input.runId!); + if (mode === "legacy_execution" && verified.length > 1) return authorized; + // Stop after the real execution authority boundary. This produces + // the exact never-started bootstrap shape without a provider call. + throw conflict("The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state.", + { code: "chat_failed_run_retry_not_authorized" }); + } + return authorized; + }); + try { + await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(and( + eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.status, "running"), isNull(heartbeatRuns.wakeupRequestId))); + await db.update(agents).set({ status: "idle" }).where(eq(agents.id, context.fixture.assignedAgentId)); + await heartbeat.resumeQueuedRuns(); + await heartbeat.drainActiveRunExecutions(); + const [firstAttempt] = await readHeartbeatRunsById(admitted.runId!); + expect(firstAttempt).toMatchObject({ status: "failed", errorCode: "chat_failed_run_retry_not_authorized", + resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } }); + expect(verified).toEqual([firstAttempt.id]); + if (mode === "legacy_execution_log") await db.update(heartbeatRuns).set({ logBytes: 1 }).where(eq(heartbeatRuns.id, firstAttempt.id)); + if (mode === "legacy_execution_profile") await db.update(heartbeatRuns).set({ runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" }, unexpectedExecution: true } }).where(eq(heartbeatRuns.id, firstAttempt.id)); + if (mode === "legacy_execution_controller") await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60000) }).where(eq(heartbeatRuns.id, firstAttempt.id)); + if (mode === "legacy_execution_event") await db.update(heartbeatRunEvents).set({ message: "different diagnostic" }).where(and(eq(heartbeatRunEvents.runId, firstAttempt.id), eq(heartbeatRunEvents.seq, 1))); + if (mode === "legacy_execution_intent") await db.update(agentWakeupRequests).set({ runId: context.runId }).where(eq(agentWakeupRequests.id, receipt.requestId)); + if (mode === "legacy_execution_lease") await db.insert(environmentLeases).values({ companyId, heartbeatRunId: firstAttempt.id, + issueId: context.issue.id, provider: "daytona", providerLeaseId: "unresolved-bootstrap-lease", status: "active", leasePolicy: "ephemeral" }); + const next = await requestTaskRetry({ ...request, observedRunId: firstAttempt.id, idempotencyKey: randomUUID() }); + expect(next).toMatchObject({ accepted: true }); + await context.service.processFailedChatRunRetry(next.chatActionId!); + await heartbeat.resumeQueuedRuns(); + await heartbeat.drainActiveRunExecutions(); + const [nextWake] = await readAgentWakeupRequestsById(next.chatActionId!); + if (mode !== "legacy_execution") { + expect(nextWake).toBeUndefined(); + expect(verified).toEqual([firstAttempt.id]); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual(previousPublications); + return; + } + expect(nextWake.runId).toBeTruthy(); + expect(nextWake.runId).not.toBe(firstAttempt.id); + const [nextRun] = await readHeartbeatRunsById(nextWake.runId!); + expect(nextRun).toMatchObject({ retryOfRunId: firstAttempt.id, wakeupRequestId: next.chatActionId, + contextSnapshot: { source: "chat:github", wakeCommentId: saved.payload.commentId, + wakeCommentIds: [saved.payload.commentId] } }); + expect(verified).toEqual([firstAttempt.id, nextRun.id]); + expect(identitySources).toEqual([{ runId: nextRun.id, messageId: saved.payload.commentId }]); + await expect(chatRunSourceMessageId(db, nextRun, { provider: "github" })).resolves.toBe(saved.payload.commentId); + expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.contextSnapshot?.wakeCommentId === saved.payload.commentId)) + .toHaveLength(2); + expect((await readHeartbeatRunsById(context.runId))[0]).toMatchObject({ status: "failed" }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual(previousPublications); + return; + } finally { executionProof.mockRestore(); identityProof.mockRestore(); await heartbeat.drainActiveRunExecutions(); } + } + await context.service.processFailedChatRunRetry(receipt.chatActionId!); + expect(await unadmitted()).toHaveLength(mode === "still_budget_blocked" ? 1 : 0); + if (held) { + expect(admitted.runId).toBeNull(); + await reconcileTaskChatRetryReceipts(db); + await context.service.processFailedChatRunRetry(receipt.chatActionId!); + expect((await readAgentWakeupRequestsById(receipt.chatActionId!))[0]).toMatchObject({ + status: "deferred_issue_execution", runId: null }); + expect((await db.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, context.issue.id))))) + .toEqual(expect.arrayContaining([expect.objectContaining({ status: "active" })])); + if (mode === "committed_delayed_cleanup") { + const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.heartbeatRunId, context.runId)); + await db.update(environmentLeases).set({ metadata: { ...lease.metadata, + remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "destroyed" }) }, + }).where(eq(environmentLeases.id, lease.id)); + await reconcileTaskChatRetryReceipts(db); + await context.service.processFailedChatRunRetry(receipt.chatActionId!); + const [resumed] = await readAgentWakeupRequestsById(receipt.chatActionId!); + expect(resumed).toMatchObject({ id: admitted.id, status: "queued" }); + expect(resumed.runId).toBeTruthy(); + expect((await readHeartbeatRunsById(resumed.runId!))[0]).toMatchObject({ status: "queued", + contextSnapshot: { wakeCommentId: saved.payload.commentId } }); + await context.service.processFailedChatRunRetry(receipt.chatActionId!); + await reconcileTaskChatRetryReceipts(db); + expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => + run.contextSnapshot?.taskRetryRequestId === receipt.requestId)).toHaveLength(1); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, context.runId)))[0]) + .toMatchObject({ phase: "committed" }); + } + } + if (mode === "real_queue" || mode === "committed_failure" || mode === "committed_failed_publication") { + expect(admitted.runId).toBeTruthy(); + expect((await readHeartbeatRunsById(admitted.runId!))[0]).toMatchObject({ status: "queued", + contextSnapshot: { source: `chat:${provider}`, wakeCommentId: saved.payload.commentId } }); + if (provider === "github") expect(await resolveRunScopedMentionedSkillKeys({ db, companyId, + issueId: context.issue.id, commentIds: [String(saved.payload.commentId)] })).toEqual([skill.key]); + await reconcileTaskChatRetryReceipts(db); + expect((await readAgentWakeupRequestsById(receipt.requestId!))[0]).toMatchObject({ + status: "coalesced", runId: admitted.runId }); + const calls = context.wakeup.mock.calls.length; + await db.update(chatActions).set({ status: "issued", result: null, updatedAt: new Date() }) + .where(eq(chatActions.id, receipt.chatActionId!)); + await context.service.processFailedChatRunRetry(receipt.chatActionId!); + expect(context.wakeup.mock.calls).toHaveLength(calls); + expect((await db.select().from(chatActions).where(eq(chatActions.id, receipt.chatActionId!)))[0]) + .toMatchObject({ status: "processed" }); + if (mode === "committed_failure") { + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, context.runId)))[0]) + .toMatchObject({ phase: "committed" }); + expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, context.runId)))[0]) + .toMatchObject({ schemaStatus: "accepted", resultJson: { terminal: { runTerminalState: "failed" } } }); + } + } + } + expect((await readAgentWakeupRequestsById(saved.id))[0]).toMatchObject(mode === "source_claimed" + ? { status: "queued", runId: context.runId } : { status: "skipped", runId: null }); + expect((await readHeartbeatRunsById(context.runId))[0]).toMatchObject({ status: "failed" }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual(previousPublications); + } finally { await context.service.shutdown(); } + }); + it.each([ "settled", "distinct_account", @@ -62777,8 +63687,26 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { fields?: Record, ) => { if (!injected && fields?.status === agents.status) { - injected = true; - throw new Error("temporary source read unavailable"); + const selection = select(fields as Parameters[0]); + const from = selection.from.bind(selection); + vi.spyOn(selection, "from").mockImplementation((table) => { + const query = from(table); + const where = query.where.bind(query); + vi.spyOn(query, "where").mockImplementation((condition) => { + const scoped = where(condition); + const { params } = scoped.toSQL(); + // Maintenance may inspect another task's queued notice first. + // Fail only this final's own source-authorization read. + if (!injected && params.includes(context.fixture.companyId) && + params.includes(context.fixture.assignedAgentId)) { + injected = true; + throw new Error("temporary source read unavailable"); + } + return scoped; + }); + return query; + }); + return selection; } return fields ? select(fields as Parameters[0]) @@ -63379,6 +64307,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { linkedOriginalActor = false, githubUnavailableFile = false, upgradedGitHub = true, + currentGitHubPr = false, ) { const fixture = await seedCompany(); const configured = @@ -63419,8 +64348,8 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { }) : provider === "github" ? makeThread({ - channelId: "github:paperclipai/paperclip", - id: `github:paperclipai/paperclip:issue:${700 + Number(suffix)}`, + channelId: currentGitHubPr ? "paperclipai/paperclip" : "github:paperclipai/paperclip", + id: currentGitHubPr ? `github:paperclipai/paperclip:${700 + Number(suffix)}` : `github:paperclipai/paperclip:issue:${700 + Number(suffix)}`, name: "paperclipai/paperclip", }) : provider === "discord" @@ -63472,6 +64401,12 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { externalId: originalActorId, }); } + if (provider === "github" && currentGitHubPr) { + const config = await githubChatManagementService(db).configuration(endpoint.id, "owner-user"); + await db.insert(chatGitHubConfigurations).values({ companyId: fixture.companyId, endpointId: endpoint.id, + configuration: { ...config.configuration, toolsEnabled: true, defaults: { ...config.configuration.defaults, + invocation: "mentions_only", events: [] } }, updatedByUserId: "owner-user" }); + } const sourceMessage = makeMessage({ id: messageId, mentioned: !thread.thread.isDM, @@ -63482,6 +64417,9 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { externalActorId ?? (provider === "telegram" ? telegramChatId : "U-SAFE-PROGRESS"), }); + if (currentGitHubPr) sourceMessage.raw = { action: "created", installation: { id: 2468 }, + repository: { id: 97531, full_name: "paperclipai/paperclip" }, issue: { number: 791, pull_request: {} }, + sender: { id: 42, login: "octocat", type: "User" }, comment: { id: Number(messageId), body: sourceMessage.text, user: { id: 42 } } }; if (githubUnavailableFile) { const url = "https://github.com/user-attachments/files/31967808/private-current.txt"; diff --git a/server/src/services/chat-channels/failed-run-retry.ts b/server/src/services/chat-channels/failed-run-retry.ts index c18a15bdcb8..0792b140497 100644 --- a/server/src/services/chat-channels/failed-run-retry.ts +++ b/server/src/services/chat-channels/failed-run-retry.ts @@ -1,5 +1,8 @@ +import { safeChatWakeAuthorityReason, type ChatWakeAuthorityReason } from "./inbound/wakeup-authorization-error.js"; +import { adapterExecutionControls } from "../adapter-execution-control.js"; +import { admitExplicitNativeContinuation } from "../explicit-native-continuation.js"; import { continuingTeamsFilePublication, type ContinuingTeamsFileTransfer } from "./microsoft-teams/file-transfer-source.js"; -import { readTaskChatRetryIntent, taskChatRetryCanAwaitRecovery, taskChatRetryExecutionStopped } from "./task-retry-authority.js"; +import { readTaskChatRetryIntent, taskChatRetryCanAwaitRecovery, taskChatRetryExecutionStopped, taskChatRetryAdmittedContinuationStopped } from "./task-retry-authority.js"; import { notifyChatActionWork, notifyChatPublicationWork } from "../chat-work-notifications.js"; import { FAILED_RUN_RETRY_STALE_MS, dueChatAction } from "../chat-action-work.js"; import { readChatAuthorizationRun } from "./authorization-run-read.js"; @@ -125,10 +128,13 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti } = options; - const failedChatRetryDenied = () => + const failedChatRetryDenied = (reason?: "accepted_result_without_new_input" | + "prior_result_publication" | "prior_interaction" | "source_authorization", authorityReason?: ChatWakeAuthorityReason) => conflict( - "The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state.", - { code: "chat_failed_run_retry_not_authorized" }, + reason === "accepted_result_without_new_input" + ? "This attempt already recorded a result. Send a new message on the task to continue from the saved work." + : "The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state.", + { code: "chat_failed_run_retry_not_authorized", ...(reason ? { reason } : {}), ...(reason === "source_authorization" && authorityReason ? { authorityReason } : {}) }, ); class FailedChatRetryPublicationReadError extends Error { constructor() { @@ -197,24 +203,48 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti } } - /** The reviewed-chat ownership check runs before runtime/profile resolution - * or provider startup. Recognize only that exact persisted failure together - * with its sole system diagnostic and absence of execution evidence. A - * generic setup error (or a prior-session display hint) is not this proof. */ - async function isPreProviderReviewedChatFailure( + /** Certify known preparation boundaries from their server-owned failure + * contract and absence of execution evidence. Other workspace validations + * may run after acquisition and must not inherit this exception. A saved + * session display hint never establishes containment or resume authority. */ + async function isPreProviderChatFailure( tx: DbOrTransaction, run: typeof heartbeatRuns.$inferSelect, + allowWorkspacePreparation = false, ): Promise { - const diagnostic = "reviewed_chat_execution_binding_not_authorized"; + const retryDiagnostic = "The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state."; + const failedRetry = run.errorCode === "chat_failed_run_retry_not_authorized" && run.error === retryDiagnostic; + const validation = run.resultJson?.workspaceValidation as Record | undefined; + const recovery = run.resultJson?.executionRecovery as Record | undefined; + const workspacePreparation = allowWorkspacePreparation && run.errorCode === "workspace_validation_failed" && + validation?.reason === "isolated_task_directory_binding_mismatch" && + typeof validation.issueId === "string" && validation.issueId === run.contextSnapshot?.issueId && + (run.issueId === null || run.issueId === validation.issueId) && + (run.contextSnapshot?.taskId === undefined || run.contextSnapshot.taskId === validation.issueId) && + Object.keys(validation).every(key => ["reason", "issueId"].includes(key)) && + recovery?.kind === "bootstrap" && recovery.providerWorkStarted === false && + Object.keys(recovery).every(key => ["kind", "providerWorkStarted"].includes(key)) && + run.resultJson?.stopReason === "adapter_failed" && + Object.keys(run.resultJson).every(key => ["executionRecovery", "workspaceValidation", "stopReason", + "effectiveTimeoutSec", "timeoutConfigured", "timeoutFired", "timeoutSource"].includes(key)); + const diagnostic = failedRetry ? retryDiagnostic : workspacePreparation ? run.error : "reviewed_chat_execution_binding_not_authorized"; + const dispatchOnlyProfile = failedRetry || workspacePreparation; if ( + adapterExecutionControls.has(run.id) || + (run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt > new Date()) || run.runtimeMode !== "legacy" || - run.errorCode !== "setup_failed" || - run.error !== diagnostic || + (!failedRetry && !workspacePreparation && run.errorCode !== "setup_failed") || + run.status !== "failed" || !run.finishedAt || + (failedRetry && run.sessionIdBefore !== null) || + (dispatchOnlyProfile && JSON.stringify(run.runnerProfileJson) !== JSON.stringify({ adapterDispatch: { adapterType: "paperclip_runner" } })) || + !diagnostic || run.error !== diagnostic || + (workspacePreparation && run.usageJson !== null && + Object.keys(run.usageJson).some(key => key !== "ledgerScope")) || [ run.runtimeModeResolverVersion, run.runtimeModeReason, run.runtimeModeResolvedAt, - run.runnerProfileJson, + ...(dispatchOnlyProfile ? [] : [run.runnerProfileJson]), run.runnerInstanceId, run.nativeSessionId, run.nativeIssueId, @@ -237,7 +267,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti run.lastOutputAt, run.lastOutputStream, run.lastOutputBytes, - run.usageJson, + ...(workspacePreparation ? [] : [run.usageJson]), run.exitCode, run.signal, ].some((value) => value !== null) || @@ -254,14 +284,52 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti eq(heartbeatRunEvents.runId, run.id), ), ) - .limit(2) + .orderBy(asc(heartbeatRunEvents.seq)) + .limit(workspacePreparation ? 4 : 3) .for("share", { noWait: true }); - const event = events[0]; + let preparationOffset = 0; + const warning = events[0]; + if (workspacePreparation && warning?.eventType === "agent_observability.unavailable") { + // This known warning is written by configuration preparation, before + // workspace allocation. It is not evidence that a provider executed. + if (warning.seq !== 1 || warning.agentId !== run.agentId || warning.stream !== "system" || warning.level !== "warn" || + warning.message !== "Company observability was not applied: this harness is not supported." || + warning.payload?.reason !== "unsupported_harness" || (typeof warning.payload.revision !== "string" || !isUuidLike(warning.payload.revision)) || + Object.keys(warning.payload).some(key => !["reason", "revision"].includes(key)) || + warning.sourceInstanceId !== null || warning.sourceEventId !== null || warning.sourceSeq !== null || + warning.sourcePayloadSha256 !== null || warning.protocolSchemaVersion !== null) return false; + preparationOffset = 1; + } + const event = events[preparationOffset]; + const measured = events[preparationOffset + 1]; + if (failedRetry) { + const explicit = run.contextSnapshot?.explicitUserContinuation as Record | undefined; + const hint = run.contextSnapshot?.chatFailedRunRetry as Record | undefined; + const intentId = run.contextSnapshot?.taskRetryRequestId; + if (typeof intentId !== "string" || !run.retryOfRunId || !run.wakeupRequestId || + hint?.version !== 1 || hint.actionId !== run.wakeupRequestId || hint.failedRunId !== run.retryOfRunId || + explicit?.previousRunId !== run.retryOfRunId || explicit.taskRetryRequestId !== intentId || + explicit.commentId !== run.contextSnapshot?.wakeCommentId) return false; + const prior = await readTaskChatRetryIntent(tx as unknown as Db, { companyId: run.companyId, + issueId: String(run.contextSnapshot?.issueId), agentId: run.agentId, + sourceRunId: run.retryOfRunId, intentId, chatActionId: run.wakeupRequestId }); + if (!prior || prior.wake.status !== "coalesced" || prior.wake.runId !== run.id) return false; + } + if (failedRetry || workspacePreparation) { + if (!measured || measured.seq !== 2 + preparationOffset || measured.agentId !== run.agentId || + measured.eventType !== "lifecycle" || measured.stream !== "system" || + measured.message !== "task.run.measured" || measured.sourceInstanceId !== null || + measured.sourceEventId !== null || measured.sourceSeq !== null || measured.sourcePayloadSha256 !== null || + measured.protocolSchemaVersion !== null || typeof measured.payload?.durationMs !== "number" || + !Number.isFinite(measured.payload.durationMs) || measured.payload.durationMs < 0 || + measured.payload?.durableSettlement !== true || measured.payload?.outcome !== "failed" || + measured.payload?.phase !== "task.run.measured") return false; + } if ( - events.length !== 1 || + events.length !== (failedRetry || workspacePreparation ? 2 + preparationOffset : 1) || !event || event.agentId !== run.agentId || - event.seq !== 1 || + event.seq !== 1 + preparationOffset || event.eventType !== "error" || event.stream !== "system" || event.level !== "error" || @@ -553,10 +621,11 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti async function assertFailedNativeRetryState( tx: DbOrTransaction, - source: Pick, + source: Pick, companyId: string, taskRetryIntentId?: string, allowPendingRecovery = false, + executionRunId?: string, ) { const [run] = await tx .select() @@ -573,7 +642,25 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti if (taskRetryIntentId) { if (!await readTaskChatRetryIntent(tx as unknown as Db, { companyId, issueId: source.issueId, agentId: source.agentId, sourceRunId: run.id, intentId: taskRetryIntentId })) throw failedChatRetryDenied(); - if (await taskChatRetryExecutionStopped(tx as unknown as Db, run)) return; + const previousComments = new Set(source.taskRetryInput?.sourceActionIds.length + ? retryCommentIds(run.contextSnapshot ?? {}, source.taskRetryInput) : []); + const newInput = source.taskRetryInput?.sourceActionIds.length && source.requestedByActorType === "user" && + source.commentIds.length && source.commentIds.every(id => !previousComments.has(id)); + if (newInput) { + // Persisting independent input may wait behind an unknown executor. + // Promotion still requires the ordinary new-user-turn stop proof. + if (allowPendingRecovery) return; + if (executionRunId && await taskChatRetryAdmittedContinuationStopped(tx as unknown as Db, run, { + issueId: source.issueId, intentId: taskRetryIntentId, successorRunId: executionRunId, + commentId: source.commentIds.at(-1)!, + })) return; + if (await admitExplicitNativeContinuation({ db: tx as Db, companyId, issueId: source.issueId, + agentId: source.agentId, actorType: "user", actorId: source.requestedByActorId, + reason: "issue_commented", commentId: source.commentIds.at(-1)!, successorRunId: taskRetryIntentId, + dryRun: true })) return; + } + if (await taskChatRetryExecutionStopped(tx as unknown as Db, run) || + await isPreProviderChatFailure(tx, run, true)) return; if (allowPendingRecovery && await taskChatRetryCanAwaitRecovery( tx as unknown as Db, run, source.issueId, taskRetryIntentId, )) return; @@ -719,7 +806,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti "process_exit", "timeout", ].includes(failedRun.errorCode ?? "") || - (evidence.kind === "retry" && await isPreProviderReviewedChatFailure(tx, evidence.run)) + (evidence.kind === "retry" && await isPreProviderChatFailure(tx, evidence.run)) )) || (!input.publication && ["backlog", "done", "cancelled"].includes(issue.status)) @@ -732,6 +819,22 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti if (failedRun.runtimeMode === "native" && evidence.kind === "retry" && !input.taskRetryIntentId) await failedNativeRetryCoordinator(tx, evidence.run, issue.id); const context = failedRun.contextSnapshot ?? {}; + const selection = input.taskRetryInput && taskRetryInputSchema.safeParse(input.taskRetryInput); + if (selection && !selection.success) throw failedChatRetryDenied(); + const selectedInput = selection?.success ? selection.data : undefined; + if (input.taskRetryIntentId && !input.committedResponse && failedRun.status === "failed" && + failedRun.runtimeMode === "native" && failedRun.nativeIssueId === issue.id && + context.issueId === issue.id && context.source === "issue.tree_resume" && + !selectedInput?.sourceActionIds.length) { + // An inherited requester is not permission to replay a recorded result. + // Explain the existing refusal before raw chat-source validation hides + // the supported new-message path. This branch grants no continuation. + const [recorded] = await tx.select({ id: nativeRunResults.id }).from(nativeRunResults).where(and( + eq(nativeRunResults.companyId, input.companyId), eq(nativeRunResults.issueId, issue.id), + eq(nativeRunResults.runId, failedRun.id), eq(nativeRunResults.schemaStatus, "accepted"), + )).limit(1).for("share", { noWait: true }); + if (recorded) throw failedChatRetryDenied("accepted_result_without_new_input"); + } const provider = CHAT_PROVIDERS.find( (value) => context.source === `chat:${value}`, ); @@ -742,13 +845,11 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti (context.taskKey !== issue.identifier && context.taskKey !== issue.id) ) throw failedChatRetryDenied(); - const selection = input.taskRetryInput && taskRetryInputSchema.safeParse(input.taskRetryInput); - if (selection && !selection.success) throw failedChatRetryDenied(); - const selectedInput = selection?.success ? selection.data : undefined; if (selectedInput && (!input.taskRetryIntentId || provider !== "github")) throw failedChatRetryDenied(); const reauthorization = input.taskRetryReauthorization ?? (selectedInput ? { intentId: input.taskRetryIntentId!, sourceRunId: failedRun.id, runtimeFence: selectedInput.runtimeFence, } : undefined); + if (reauthorization && selectedInput?.skippedActionIds) reauthorization.skippedActionIds = selectedInput.skippedActionIds; const pendingSources = selectedInput?.sourceActionIds.length ? await tx.select().from(chatActions).where(and( eq(chatActions.companyId, input.companyId), eq(chatActions.kind, "inbound_wakeup"), sql`${chatActions.payload}->>'issueId' = ${issue.id}`, @@ -859,6 +960,11 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti ), ) .limit(1); + const previousComments = new Set(pendingSources?.length ? retryCommentIds(context, selectedInput) : []); + const independentFailedInput = input.taskRetryIntentId && pendingSources?.length && + pendingSources.every(action => action.payload.requestedByActorType === "user") && + commentIds.every(id => !previousComments.has(id)) && failedRun.status === "failed" && + (accepted?.resultJson.terminal as Record | undefined)?.runTerminalState === "failed"; const [resultPublication] = await tx .select({ id: chatPublications.id }) .from(chatPublications) @@ -869,6 +975,10 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti input.committedResponse ? eq(chatPublications.state, "delivery_unknown") : undefined, + // A distinct admitted user request does not replay the failed + // result's output. Preserve that terminal publication as history; + // pending or uncertain delivery still owns its external effects. + independentFailedInput ? ne(chatPublications.state, "failed") : undefined, continuingUnknownPublicationId ? ne(chatPublications.id, continuingUnknownPublicationId) : undefined, @@ -884,13 +994,10 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti ), ) .limit(1); - if ( - (!input.committedResponse && accepted) || - resultPublication || - question || - answerDelivery - ) - throw failedChatRetryDenied(); + if (!input.committedResponse && accepted && !independentFailedInput) + throw failedChatRetryDenied(input.taskRetryIntentId ? "accepted_result_without_new_input" : undefined); + if (resultPublication) throw failedChatRetryDenied(input.taskRetryIntentId ? "prior_result_publication" : undefined); + if (question || answerDelivery) throw failedChatRetryDenied(input.taskRetryIntentId ? "prior_interaction" : undefined); if (context.chatFailedRunRetry && !pendingSources) { const hint = context.chatFailedRunRetry as Record; const [priorIntent] = await tx @@ -1069,7 +1176,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti const invalidated = new Set(lifecycleRows.map(row => lifecycleKey(row.target, row.context))); const batch = reauthorization ? await authorizeGitHubTaskRetryBatch(tx, ordered, { nonblocking: true, terminal: input.publication === true, taskRetry: reauthorization, - }).catch(error => { if (isExternalActionAuthorizationChange(error)) throw failedChatRetryDenied(); throw error; }) : null; + }).catch(error => { if (isExternalActionAuthorizationChange(error)) throw failedChatRetryDenied(input.taskRetryIntentId ? "source_authorization" : undefined, safeChatWakeAuthorityReason(error)); throw error; }) : null; const canonicalSources: Array> = []; let destination: Awaited> | null = null; @@ -1080,16 +1187,21 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti ...(reauthorization ? { taskRetry: reauthorization } : {}), }).catch((error) => { if (isExternalActionAuthorizationChange(error)) - throw failedChatRetryDenied(); + throw failedChatRetryDenied(input.taskRetryIntentId ? "source_authorization" : undefined, safeChatWakeAuthorityReason(error)); throw error; }); if (reauthorization && !reauthorization.runtimeFence) reauthorization.runtimeFence = destination.retryRuntimeFence; const delivery = deliveriesById.get(action.deliveryId!); const current = actionsById.get(action.id); const comment = commentsById.get(String(action.payload.commentId)); + const receipt = receiptsById.get(action.id); + const skippedInput = pendingSources && selectedInput?.skippedActionIds?.includes(action.id) && + receipt?.status === "skipped" && !receipt.runId && + ["budget.blocked", "agent.not_invokable"].includes(receipt.reason ?? "") && + action.payload.requestedByActorType === "user"; if ( !current || - current.status !== "processed" || + (current.status !== "processed" && !(skippedInput && current.status === "failed")) || inboundWakeScope(current) !== inboundWakeScope(action) || !delivery || delivery.state !== "processed" || @@ -1106,7 +1218,6 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti ) throw failedChatRetryDenied(); if (invalidated.has(lifecycleKey(delivery.providerEventId, lifecycleRuntimeFence(delivery)))) throw failedChatRetryDenied(); - const receipt = receiptsById.get(action.id); if (!receipt) throw failedChatRetryDenied(); assertFailedRetryReceipt( createDurableChatWakeupRequest({ @@ -1125,7 +1236,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti receipt, ); if (pendingSources) { - const direct = receipt.status === "deferred_issue_execution" && !receipt.runId + const direct = (receipt.status === "deferred_issue_execution" || skippedInput) && !receipt.runId && receipt.payload?.coalescedIntoWakeupRequestId === undefined; const owned = input.retryActionId && receipt.status === "coalesced" && receipt.payload?.coalescedIntoWakeupRequestId === input.retryActionId; @@ -1164,7 +1275,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti owner.agentId !== input.agentId || owner.requestedByActorType !== receipt.requestedByActorType || owner.requestedByActorId !== receipt.requestedByActorId)) || - ["skipped", "cancelled"].includes(receipt.status) + (!skippedInput && ["skipped", "cancelled"].includes(receipt.status)) ) throw failedChatRetryDenied(); canonicalSources.push({ @@ -1457,7 +1568,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti }); await assertFailedNativeRetryState(tx, source, input.companyId, typeof source.intent.payload.taskRetryIntentId === "string" ? source.intent.payload.taskRetryIntentId : undefined, - input.phase === "admission"); + input.phase === "admission", input.phase === "execution" ? input.runId : undefined); const context = input.contextSnapshot; const hint = context.chatFailedRunRetry as | Record diff --git a/server/src/services/chat-channels/github/review-wait.ts b/server/src/services/chat-channels/github/review-wait.ts index e34b3a576c7..9b688564be8 100644 --- a/server/src/services/chat-channels/github/review-wait.ts +++ b/server/src/services/chat-channels/github/review-wait.ts @@ -1,6 +1,8 @@ import { createHash } from "node:crypto"; -import { and, eq, or } from "drizzle-orm"; -import { chatActions, chatGitHubReviews, type Db } from "@paperclipai/db"; +import { HttpError } from "../../../errors.js"; +import { chatRunSourceMessageId } from "../run-origin.js"; +import { and, desc, eq, or, sql } from "drizzle-orm"; +import { chatActions, chatConversations, chatEndpointResources, chatEndpoints, chatGitHubReviews, chatMessageLinks, heartbeatRuns, toolConnections, type Db } from "@paperclipai/db"; export async function hasPublishedGitHubReview(db: Db, binding: { companyId: string; @@ -27,3 +29,69 @@ export async function hasPublishedGitHubReview(db: Db, binding: { )).limit(1); return !!publication; } + +type WaitRun = Pick; + +/** Passive review waits inherit the same accepted requester lineage as GitHub + * tools. This is not a grant to publish, run a monitor, or reuse attachments. */ +export async function readInheritedGitHubReviewWait(db: Db, run: WaitRun) { + if (!run.activeIdentityContextId || !run.nativeIssueId) return null; + let commentId: string | null; + try { + commentId = await chatRunSourceMessageId(db, run, { provider: "github" }); + } catch (error) { + if (error instanceof HttpError && error.status === 403) return { kind: "revoked" as const }; + throw error; + } + if (!commentId) return null; + const [review] = await db.select({ + id: chatGitHubReviews.id, endpointId: chatGitHubReviews.endpointId, + state: chatGitHubReviews.state, repository: chatGitHubReviews.repository, + repositoryId: chatGitHubReviews.repositoryId, headSha: chatGitHubReviews.headSha, + assessment: chatGitHubReviews.assessment, reviewRunId: chatGitHubReviews.runId, + connectionEnabled: toolConnections.enabled, connectionStatus: toolConnections.status, + reviewAgentId: heartbeatRuns.agentId, reviewCompanyId: heartbeatRuns.companyId, + resourceRepository: chatEndpointResources.providerResourceId, + resourceRepositoryId: sql`${chatEndpointResources.metadata}->>'providerRepositoryId'`, + pullNumber: chatGitHubReviews.pullNumber, thread: chatConversations.externalThreadId, + }).from(chatMessageLinks) + .innerJoin(chatConversations, and(eq(chatConversations.id, chatMessageLinks.conversationId), + eq(chatConversations.companyId, run.companyId), eq(chatConversations.issueId, run.nativeIssueId))) + .innerJoin(chatEndpoints, and(eq(chatEndpoints.id, chatConversations.endpointId), + eq(chatEndpoints.companyId, run.companyId), eq(chatEndpoints.provider, "github"), + eq(chatEndpoints.assignedAgentId, run.agentId))) + .leftJoin(toolConnections, and(eq(toolConnections.id, chatEndpoints.connectionId), + eq(toolConnections.companyId, run.companyId))) + .innerJoin(chatEndpointResources, and(eq(chatEndpointResources.id, chatConversations.resourceId), + eq(chatEndpointResources.companyId, run.companyId), eq(chatEndpointResources.endpointId, chatEndpoints.id))) + .innerJoin(chatGitHubReviews, and(eq(chatGitHubReviews.endpointId, chatEndpoints.id), + eq(chatGitHubReviews.companyId, run.companyId), eq(chatGitHubReviews.issueId, run.nativeIssueId))) + .leftJoin(heartbeatRuns, eq(heartbeatRuns.id, chatGitHubReviews.runId)) + .where(and(eq(chatMessageLinks.companyId, run.companyId), eq(chatMessageLinks.commentId, commentId), + eq(chatMessageLinks.direction, "inbound"))) + .orderBy(desc(chatGitHubReviews.createdAt), desc(chatGitHubReviews.id)).limit(1); + // A newer unreviewed head cannot inherit the prior head's publication. + if (!review) return null; + if (!review.connectionEnabled || review.connectionStatus !== "active" || review.state !== "completed" || + review.reviewCompanyId !== run.companyId || review.reviewAgentId !== run.agentId || + review.assessment?.reviewedCommit !== review.headSha || + review.resourceRepository.toLowerCase() !== review.repository.toLowerCase() || + review.resourceRepositoryId !== review.repositoryId || + review.thread.toLowerCase() !== `github:${review.repository}:${review.pullNumber}`.toLowerCase()) return { kind: "revoked" as const }; + const [publication] = await db.select({ id: chatActions.id }).from(chatActions).where(and( + eq(chatActions.companyId, run.companyId), eq(chatActions.endpointId, review.endpointId), + eq(chatActions.kind, "github_review_publication"), eq(chatActions.status, "processed"), + eq(chatActions.providerActionId, `github_publication:${run.nativeIssueId}:assessment:${createHash("sha256").update(JSON.stringify(review.id)).digest("hex")}`), + )).limit(1); + if (!publication) return { kind: "revoked" as const }; + // The caller must still verify live task ownership, destination and principal + // policy. This normalized source is used only by the passive-wait boundary. + return { kind: "published" as const, endpointId: review.endpointId, reviewId: review.id, reviewRunId: review.reviewRunId, + headSha: review.headSha, context: { + source: "chat:github", paperclipHarnessCheckedOut: true, + wakeCommentId: commentId, wakeCommentIds: [commentId], + paperclipWake: { checkedOutByHarness: true, externalChatProvider: "github", + issue: { id: run.nativeIssueId }, commentIds: [commentId] }, + } }; +} diff --git a/server/src/services/chat-channels/github/task-retry-authorization.ts b/server/src/services/chat-channels/github/task-retry-authorization.ts index 52e51f38ebf..d27edc958cf 100644 --- a/server/src/services/chat-channels/github/task-retry-authorization.ts +++ b/server/src/services/chat-channels/github/task-retry-authorization.ts @@ -1,6 +1,6 @@ -import { and, eq, inArray } from "drizzle-orm"; -import { chatActions, chatConversations, chatDeliveries, chatEndpointResources, chatMessageLinks, issueComments, toolConnections } from "@paperclipai/db"; -import { forbidden } from "../../../errors.js"; +import { and, eq, inArray, isNull, lt, sql } from "drizzle-orm"; +import { agentWakeupRequests, chatActions, chatConversations, chatDeliveries, chatEndpointResources, chatMessageLinks, issueComments, toolConnections } from "@paperclipai/db"; +import { chatWakeAuthorityDenied } from "../inbound/wakeup-authorization-error.js"; import type { DbOrTransaction, EndpointRow, LifecycleRuntimeFence } from "../runtime-types.js"; import { credentialFingerprint } from "../credential-fingerprint.js"; import { readTaskChatRetryIntent } from "../task-retry-authority.js"; @@ -9,8 +9,25 @@ export type TaskRetryWakeAuthority = { intentId: string; sourceRunId: string; runtimeFence?: LifecycleRuntimeFence; + skippedActionIds?: string[]; }; +async function skippedTaskRetryInputs(tx: DbOrTransaction, actions: Array, + authority: TaskRetryWakeAuthority) { + const first = actions[0]; + const ids = actions.filter(action => action.status === "failed" && action.payload.requestedByActorType === "user" && + authority.skippedActionIds?.includes(action.id)).map(action => action.id); + if (!first || !ids.length) return new Set(); + const rows = await tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and( + eq(agentWakeupRequests.companyId, first.companyId), inArray(agentWakeupRequests.id, ids), + eq(agentWakeupRequests.agentId, String(first.payload.agentId)), eq(agentWakeupRequests.status, "skipped"), + isNull(agentWakeupRequests.runId), inArray(agentWakeupRequests.reason, ["budget.blocked", "agent.not_invokable"]), + eq(agentWakeupRequests.requestedByActorType, "user"), + eq(agentWakeupRequests.requestedByActorId, String(first.payload.requestedByActorId)), + )); + return new Set(rows.map(row => row.id)); +} + export function githubTaskRetryAppMatches(endpoint: EndpointRow, delivery: typeof chatDeliveries.$inferSelect, response: { payload: Record } | undefined) { const original = delivery.normalizedEvent.githubAuthority as { botExternalId?: unknown } | undefined; @@ -18,6 +35,37 @@ export function githubTaskRetryAppMatches(endpoint: EndpointRow, delivery: typeo : response?.payload.botExternalId === endpoint.botExternalId && response.payload.agentId === endpoint.assignedAgentId; } +/** Legacy intake did not persist an App ID when admission stopped before + * acknowledgement. An earlier request ledger proves this endpoint had already + * claimed the App: response payloads are server-owned and a non-null endpoint + * claim cannot be replaced by setup, reconnect, or lifecycle repair. This is + * identity evidence only; the earlier response is never replayed. */ +async function githubTaskRetryAppAuthorized(tx: DbOrTransaction, endpoint: EndpointRow, + delivery: typeof chatDeliveries.$inferSelect, response: { payload: Record } | undefined) { + if (githubTaskRetryAppMatches(endpoint, delivery, response)) return true; + const original = delivery.normalizedEvent.githubAuthority as { botExternalId?: unknown } | undefined; + // Never substitute historical evidence for an explicit conflicting binding. + if (original?.botExternalId !== undefined || response || !endpoint.botExternalId || !delivery.conversationId) return false; + const [prior] = await tx.select({ id: chatActions.id }).from(chatActions) + .innerJoin(chatDeliveries, and(eq(chatDeliveries.id, chatActions.deliveryId), + eq(chatDeliveries.companyId, chatActions.companyId), eq(chatDeliveries.endpointId, chatActions.endpointId), + eq(chatDeliveries.conversationId, chatActions.conversationId), eq(chatDeliveries.principalId, chatActions.principalId))) + .innerJoin(chatMessageLinks, and(eq(chatMessageLinks.deliveryId, chatDeliveries.id), + eq(chatMessageLinks.companyId, chatDeliveries.companyId), eq(chatMessageLinks.endpointId, chatDeliveries.endpointId), + eq(chatMessageLinks.conversationId, chatDeliveries.conversationId), eq(chatMessageLinks.direction, "inbound"))) + .where(and(eq(chatActions.companyId, endpoint.companyId), eq(chatActions.endpointId, endpoint.id), + eq(chatActions.conversationId, delivery.conversationId), eq(chatActions.kind, "github_response_comment"), + eq(chatActions.status, "processed"), eq(chatDeliveries.state, "processed"), + eq(chatActions.providerActionId, sql`'github_response:' || ${chatDeliveries.id}::text`), + sql`jsonb_typeof(${chatActions.payload}->'botExternalId') = 'string'`, + sql`${chatActions.payload}->>'botExternalId' = ${endpoint.botExternalId}`, + sql`jsonb_typeof(${chatActions.payload}->'agentId') = 'string'`, + sql`${chatActions.payload}->>'agentId' = ${endpoint.assignedAgentId}`, + lt(chatActions.createdAt, delivery.createdAt), lt(chatDeliveries.createdAt, delivery.createdAt))) + .limit(1); + return Boolean(prior); +} + export function githubTaskRetryDestinationMatches(delivery: typeof chatDeliveries.$inferSelect, conversation: typeof chatConversations.$inferSelect, resource: typeof chatEndpointResources.$inferSelect | null) { const originalResource = delivery.normalizedEvent.resource as { providerResourceId?: unknown } | undefined; @@ -33,27 +81,28 @@ export function githubTaskRetryDestinationMatches(delivery: typeof chatDeliverie export async function githubTaskRetryRuntimeFence(tx: DbOrTransaction, endpoint: EndpointRow, action: typeof chatActions.$inferSelect, delivery: typeof chatDeliveries.$inferSelect, authority: TaskRetryWakeAuthority): Promise { - const deny = () => forbidden("This accepted chat message is no longer authorized to start work", - { code: "chat_action_authorization_changed" }); + const deny = chatWakeAuthorityDenied; const retry = await readTaskChatRetryIntent(tx as Parameters[0], { companyId: action.companyId, issueId: String(action.payload.issueId), agentId: String(action.payload.agentId), sourceRunId: authority.sourceRunId, intentId: authority.intentId, }); - if (!retry || endpoint.provider !== "github" || !endpoint.botExternalId || action.kind !== "inbound_wakeup" - || action.status !== "processed" || delivery.state !== "processed") throw deny(); + const skipped = await skippedTaskRetryInputs(tx, [action], authority); + if (!retry) throw deny("retry_intent"); + if (endpoint.provider !== "github" || !endpoint.botExternalId || action.kind !== "inbound_wakeup" + || (action.status !== "processed" && !skipped.has(action.id)) || delivery.state !== "processed") throw deny("source_receipt"); const [response] = await tx.select({ payload: chatActions.payload }).from(chatActions).where(and( eq(chatActions.companyId, action.companyId), eq(chatActions.endpointId, endpoint.id), eq(chatActions.conversationId, action.conversationId!), eq(chatActions.deliveryId, delivery.id), eq(chatActions.kind, "github_response_comment"), )); - if (!githubTaskRetryAppMatches(endpoint, delivery, response)) throw deny(); + if (!(await githubTaskRetryAppAuthorized(tx, endpoint, delivery, response))) throw deny("app_binding"); const [connection] = await tx.select({ refs: toolConnections.credentialSecretRefs, enabled: toolConnections.enabled, status: toolConnections.status }).from(toolConnections).where(and( eq(toolConnections.companyId, endpoint.companyId), eq(toolConnections.id, endpoint.connectionId), )); - if (!connection?.enabled || connection.status !== "active") throw deny(); + if (!connection?.enabled || connection.status !== "active") throw deny("connection_access"); const generation = Number((endpoint.setup as { runtimeGeneration?: unknown }).runtimeGeneration ?? 0); - if (!Number.isSafeInteger(generation) || generation < 0) throw deny(); + if (!Number.isSafeInteger(generation) || generation < 0) throw deny("runtime_generation"); return authority.runtimeFence ?? { generation, credentialFingerprint: credentialFingerprint(connection.refs ?? []) }; } @@ -71,16 +120,17 @@ export function createGitHubTaskRetryBatchAuthority Promise) { async function authorizeGitHubTaskRetryBatch(tx: DbOrTransaction, actions: Array, notice: { nonblocking: true; terminal: boolean; taskRetry: TaskRetryWakeAuthority }) { - const deny = () => forbidden("This accepted chat message is no longer authorized to start work", - { code: "chat_action_authorization_changed" }); + const deny = chatWakeAuthorityDenied; const first = actions[0]; - if (!first) throw deny(); + if (!first) throw deny("batch_scope"); + const skipped = await skippedTaskRetryInputs(tx, actions, notice.taskRetry); const commonScope = (action: typeof chatActions.$inferSelect) => JSON.stringify([ - action.companyId, action.endpointId, action.conversationId, action.principalId, action.kind, action.status, + action.companyId, action.endpointId, action.conversationId, action.principalId, action.kind, + skipped.has(action.id) ? "processed" : action.status, ...["version", "issueId", "agentId", "sessionGeneration", "requestedByActorType", "requestedByActorId"] .map(key => action.payload[key]), ]); - if (actions.some(action => commonScope(action) !== commonScope(first))) throw deny(); + if (actions.some(action => commonScope(action) !== commonScope(first))) throw deny("batch_scope"); const deliveries = await tx.select().from(chatDeliveries).where(and(eq(chatDeliveries.companyId, first.companyId), inArray(chatDeliveries.id, actions.map(action => action.deliveryId!)))); const byDelivery = new Map(deliveries.map(delivery => [delivery.id, delivery])); @@ -102,21 +152,21 @@ export function createGitHubTaskRetryBatchAuthority(); for (const action of actions) { const delivery = byDelivery.get(action.deliveryId!); - if (!delivery || !linked.has(`${delivery.id}:${action.payload.commentId}`)) throw deny(); + if (!delivery || !linked.has(`${delivery.id}:${action.payload.commentId}`)) throw deny("batch_source_link"); // Reuse only the same live grant within this transaction. Automatic review // policies with different contexts still receive their own full check. const grant = JSON.stringify(delivery.normalizedEvent.githubAutomatic ?? delivery.normalizedEvent.githubIssue ?? null); let current = groups.get(grant); if (!current) { current = await authorize(tx, action, notice); - if (current.endpoint.provider !== "github") throw deny(); + if (current.endpoint.provider !== "github") throw deny("app_binding"); notice.taskRetry.runtimeFence ??= current.retryRuntimeFence; groups.set(grant, current); } if (delivery.state !== "processed" || delivery.endpointId !== current.endpoint.id || delivery.conversationId !== current.conversation.id || delivery.principalId !== action.principalId || - !githubTaskRetryAppMatches(current.endpoint, delivery, legacyByDelivery.get(delivery.id)) || - !githubTaskRetryDestinationMatches(delivery, current.conversation, current.resource)) throw deny(); + !(await githubTaskRetryAppAuthorized(tx, current.endpoint, delivery, legacyByDelivery.get(delivery.id))) || + !githubTaskRetryDestinationMatches(delivery, current.conversation, current.resource)) throw deny("destination_binding"); authorized.set(action.id, { ...current, delivery }); } return authorized; diff --git a/server/src/services/chat-channels/inbound/wakeup-authorization-error.test.ts b/server/src/services/chat-channels/inbound/wakeup-authorization-error.test.ts new file mode 100644 index 00000000000..2a4d4f2c0ae --- /dev/null +++ b/server/src/services/chat-channels/inbound/wakeup-authorization-error.test.ts @@ -0,0 +1,16 @@ +import { describe, expect, it } from "vitest"; +import { forbidden } from "../../../errors.js"; +import { chatWakeAuthorityDenied, safeChatWakeAuthorityReason } from "./wakeup-authorization-error.js"; + +describe("safe chat wake authority reason", () => { + it("retains only the stable category from a known authority refusal", () => { + expect(safeChatWakeAuthorityReason(chatWakeAuthorityDenied("source_receipt"))).toBe("source_receipt"); + }); + it.each([ + forbidden("private provider response", { code: "chat_action_authorization_changed", authorityReason: "private-source-payload" }), + forbidden("private provider response", { code: "another_error", authorityReason: "source_receipt" }), + { details: { code: "chat_action_authorization_changed", authorityReason: "source_receipt" } }, + ])("does not copy arbitrary error fields or untrusted error shapes", error => { + expect(safeChatWakeAuthorityReason(error)).toBeUndefined(); + }); +}); diff --git a/server/src/services/chat-channels/inbound/wakeup-authorization-error.ts b/server/src/services/chat-channels/inbound/wakeup-authorization-error.ts new file mode 100644 index 00000000000..8edf2730346 --- /dev/null +++ b/server/src/services/chat-channels/inbound/wakeup-authorization-error.ts @@ -0,0 +1,19 @@ +import { forbidden, HttpError } from "../../../errors.js"; + +// Stable categories only. Never surface stored provider payloads, identifiers, +// credential fingerprints, or arbitrary downstream error text to the board. +const reasons = ["source_shape", "task_assignment", "delivery_binding", "runtime_fence", + "conversation_binding", "destination_binding", "automatic_policy", "task_privacy", + "principal_access", "source_comment", "retry_intent", "source_receipt", "app_binding", + "connection_access", "runtime_generation", "batch_scope", "batch_source_link"] as const; +export type ChatWakeAuthorityReason = typeof reasons[number]; +export function chatWakeAuthorityDenied(authorityReason: ChatWakeAuthorityReason) { + return forbidden("This accepted chat message is no longer authorized to start work", + { code: "chat_action_authorization_changed", authorityReason }); +} +export function safeChatWakeAuthorityReason(error: unknown): ChatWakeAuthorityReason | undefined { + if (!(error instanceof HttpError) || !error.details || typeof error.details !== "object") return undefined; + const details = error.details as Record; + return details.code === "chat_action_authorization_changed" && reasons.includes(details.authorityReason as ChatWakeAuthorityReason) + ? details.authorityReason as ChatWakeAuthorityReason : undefined; +} diff --git a/server/src/services/chat-channels/inbound/wakeup-authorization.ts b/server/src/services/chat-channels/inbound/wakeup-authorization.ts index f4da4fa7cce..0d45fcb7606 100644 --- a/server/src/services/chat-channels/inbound/wakeup-authorization.ts +++ b/server/src/services/chat-channels/inbound/wakeup-authorization.ts @@ -1,3 +1,4 @@ +import { chatWakeAuthorityDenied } from "./wakeup-authorization-error.js"; import { nonDirectDestinationAllowed } from "./surfaces.js"; import { lifecycleRuntimeFence } from "./delivery-ledger.js"; import type { DbTransaction, DbOrTransaction } from "../runtime-types.js"; @@ -45,11 +46,7 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions notice?: { nonblocking: true; terminal: boolean; taskRetry?: TaskRetryWakeAuthority }, ) { const payload = action.payload; - const deny = () => - forbidden( - "This accepted chat message is no longer authorized to start work", - { code: "chat_action_authorization_changed" }, - ); + const deny = chatWakeAuthorityDenied; if ( payload.version !== 1 || typeof payload.issueId !== "string" || @@ -61,7 +58,7 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions !["user", "system"].includes(String(payload.requestedByActorType)) || typeof payload.requestedByActorId !== "string" ) - throw deny(); + throw deny("source_shape"); const issue = await tx .select() .from(issues) @@ -80,7 +77,7 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions (!notice?.terminal && ["backlog", "done", "cancelled"].includes(issue.status)) ) - throw deny(); + throw deny("task_assignment"); // The scheduler already owns the issue lock. Do not wait in the opposite // order behind an ingress transaction which owns the endpoint first. const endpoint = await tx @@ -117,13 +114,11 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions delivery.endpointId !== endpoint.id || delivery.conversationId !== action.conversationId || delivery.principalId !== action.principalId || - !currentFence || - !(await runtimeCallbackEndpoint(tx as unknown as DbTransaction, endpoint.id, currentFence, [ - "verifying", - "active", - ])) + !currentFence ) - throw deny(); + throw deny("delivery_binding"); + if (!(await runtimeCallbackEndpoint(tx as unknown as DbTransaction, endpoint.id, currentFence, ["verifying", "active"]))) + throw deny("runtime_fence"); const conversation = await tx .select() .from(chatConversations) @@ -143,7 +138,7 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions conversation.sessionGeneration !== payload.sessionGeneration || !["active", "waiting"].includes(conversation.state) ) - throw deny(); + throw deny("conversation_binding"); const resource = conversation.resourceId ? await tx .select() @@ -163,15 +158,15 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions endpoint, action.principalId, ); - if (notice?.taskRetry && !githubTaskRetryDestinationMatches(delivery, conversation, resource)) throw deny(); + if (notice?.taskRetry && !githubTaskRetryDestinationMatches(delivery, conversation, resource)) throw deny("destination_binding"); if (endpoint.provider === "github") { const automatic = await resolveGitHubWakeAuthority(tx, endpoint, delivery); - if (automatic.kind === "denied") throw deny(); + if (automatic.kind === "denied") throw deny("automatic_policy"); if (automatic.kind === "allowed") authorization.userId = automatic.userId; } if (!(await canActorReadIssuePrivacy(tx, authorization.userId ? { type: "board", userId: authorization.userId } - : { type: "none" }, issue))) throw deny(); + : { type: "none" }, issue))) throw deny("task_privacy"); const expectedUserId = payload.requestedByActorType === "user" ? payload.requestedByActorId @@ -185,7 +180,7 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions (payload.requestedByActorType === "system" && payload.requestedByActorId !== action.principalId) ) - throw deny(); + throw deny("principal_access"); const source = await tx .select({ authorUserId: issueComments.authorUserId, @@ -217,7 +212,7 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions (source.deletedAt && !notice?.terminal) || source.authorUserId !== expectedUserId ) - throw deny(); + throw deny("source_comment"); return { issue, delivery, endpoint, conversation, resource, ...(notice?.taskRetry && currentFence ? { retryRuntimeFence: currentFence } : {}) }; } diff --git a/server/src/services/chat-channels/task-retry-authority.ts b/server/src/services/chat-channels/task-retry-authority.ts index 340f2a67b54..8bb532d9c0e 100644 --- a/server/src/services/chat-channels/task-retry-authority.ts +++ b/server/src/services/chat-channels/task-retry-authority.ts @@ -1,5 +1,6 @@ import { and, eq, sql } from "drizzle-orm"; -import { agentWakeupRequests, environmentLeases, heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import { agentWakeupRequests, environmentLeases, heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, nativeRunResults, issues, type Db } from "@paperclipai/db"; +import { buildExecutionContinuation } from "../execution-continuation.js"; import { readTaskRetryIntent } from "../task-retry.js"; import { hasNativeLocalProcessStop, hasHistoricalSuspendedNativeSession, recordedLocalExecutionStopped } from "../native-local-process-stop.js"; import { hasRemoteTerminationReceipt } from "../remote-execution-termination.js"; @@ -30,13 +31,29 @@ export async function readTaskChatRetryIntent(db: Db, input: { /** An unusable checkpoint does not prevent an isolated attempt. The old * executor must still be contained; a terminal database row is insufficient. */ export async function taskChatRetryExecutionStopped(db: Db, run: typeof heartbeatRuns.$inferSelect) { + return taskChatRetryExecutionContained(db, run, false); +} + +async function taskChatRetryExecutionContained(db: Db, run: typeof heartbeatRuns.$inferSelect, admittedFailure: boolean) { if (adapterExecutionControls.has(run.id) || (run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt > new Date())) return false; const [coordinator] = await db.select().from(nativeRunFinalizations).where(and( eq(nativeRunFinalizations.companyId, run.companyId), eq(nativeRunFinalizations.runId, run.id), )).for("share", { noWait: true }); - if (coordinator && (coordinator.leaseOwner || coordinator.resultId || coordinator.failureDetail?.successorRunId || + if (coordinator && (coordinator.leaseOwner || coordinator.failureDetail?.successorRunId || !["terminal_failure", "committed"].includes(coordinator.phase))) return false; + if (coordinator?.resultId) { + if (!admittedFailure || run.runtimeMode !== "native" || run.status !== "failed" || + run.nativePhase !== "committed" || !run.completionContractId || coordinator.phase !== "committed" || + coordinator.issueId !== run.nativeIssueId) return false; + const [failedResult] = await db.select({ id: nativeRunResults.id }).from(nativeRunResults).where(and( + eq(nativeRunResults.companyId, run.companyId), eq(nativeRunResults.runId, run.id), + eq(nativeRunResults.issueId, run.nativeIssueId!), eq(nativeRunResults.id, coordinator.resultId), + eq(nativeRunResults.schemaStatus, "accepted"), eq(nativeRunResults.completionContractId, run.completionContractId), + sql`${nativeRunResults.resultJson}#>>'{terminal,runTerminalState}' = 'failed'`, + )).limit(1); + if (!failedResult) return false; + } const leases = await db.select().from(environmentLeases).where(and( eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id), )).for("share", { noWait: true }); @@ -46,6 +63,42 @@ export async function taskChatRetryExecutionStopped(db: Db, run: typeof heartbea await hasHistoricalSuspendedNativeSession(db, run); } +/** Dispatch revalidates the canonical admission receipt, not an admission which + * has already resolved its recovery holds. This exception certifies only the + * exact current successor; it never makes the failed result replayable. */ +export async function taskChatRetryAdmittedContinuationStopped(db: Db, run: typeof heartbeatRuns.$inferSelect, + input: { issueId: string; intentId: string; successorRunId: string; commentId: string }) { + if (run.nativeIssueId !== input.issueId) return false; + const retry = await readTaskChatRetryIntent(db, { companyId: run.companyId, issueId: input.issueId, + agentId: run.agentId, sourceRunId: run.id, intentId: input.intentId }); + if (!retry?.intent.chatActionId) return false; + const [task] = await db.select({ assignee: issues.assigneeAgentId, executionRunId: issues.executionRunId }) + .from(issues).where(and(eq(issues.companyId, run.companyId), eq(issues.id, input.issueId))); + const [successor] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, run.companyId), + eq(heartbeatRuns.id, input.successorRunId), eq(heartbeatRuns.agentId, run.agentId))).for("share", { noWait: true }); + const [receipt] = await db.select({ runId: agentWakeupRequests.runId, status: agentWakeupRequests.status }) + .from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, run.companyId), + eq(agentWakeupRequests.id, retry.intent.chatActionId), eq(agentWakeupRequests.agentId, run.agentId))); + const explicit = successor?.contextSnapshot?.explicitUserContinuation as Record | undefined; + if (!successor || task?.assignee !== run.agentId || task.executionRunId !== successor.id || + successor.contextSnapshot?.issueId !== input.issueId || + (successor.nativeIssueId !== null && successor.nativeIssueId !== input.issueId) || + successor.status !== "running" || successor.wakeupRequestId !== retry.intent.chatActionId || + successor.retryOfRunId !== run.id || receipt?.runId !== successor.id || receipt.status !== "claimed" || + successor.contextSnapshot?.taskRetryRequestId !== input.intentId || + explicit?.previousRunId !== run.id || explicit.commentId !== input.commentId || + explicit.taskRetryRequestId !== input.intentId || await getExecutionBlocker(db, run.companyId, input.issueId)) return false; + try { + await buildExecutionContinuation({ db, companyId: run.companyId, issueId: input.issueId, + agentId: run.agentId, runId: successor.id, context: successor.contextSnapshot!, summary: null, exposeLowTrustRaw: false }); + } catch (error) { + if (error instanceof Error && ["continuation_user_authorization_missing", "continuation_source_context_missing", + "continuation_task_ownership_changed"].includes(error.message)) return false; + throw error; + } + return taskChatRetryExecutionContained(db, run, true); +} + /** Admission may persist a wait for the exact recovery this user requested. * This grants no execution authority: promotion and execution still require * containment, and the queue retains the source's current execution blocker. */ diff --git a/server/src/services/chat-channels/task-retry-input.ts b/server/src/services/chat-channels/task-retry-input.ts index b5fb8a30233..200f77bda1c 100644 --- a/server/src/services/chat-channels/task-retry-input.ts +++ b/server/src/services/chat-channels/task-retry-input.ts @@ -1,5 +1,5 @@ import { z } from "zod"; -import { and, asc, eq, inArray, isNull, sql } from "drizzle-orm"; +import { and, asc, eq, gt, inArray, isNull, or, sql } from "drizzle-orm"; import { agentWakeupRequests, chatActions, chatConversations, chatEndpoints, heartbeatRuns, issues, type Db } from "@paperclipai/db"; import type { DbOrTransaction } from "./runtime-types.js"; @@ -8,15 +8,19 @@ export const taskRetryInputSchema = z.object({ // Truncating it strands older requests or makes the newer-input guard reject // every Retry. Dispatch already carries the full queue with bounded inline text. sourceActionIds: z.array(z.string().uuid()).refine(ids => new Set(ids).size === ids.length), + skippedActionIds: z.array(z.string().uuid()).optional(), runtimeFence: z.object({ generation: z.number().int().nonnegative(), credentialFingerprint: z.string().regex(/^[a-f0-9]{64}$/) }).optional(), -}); +}).refine(value => !value.skippedActionIds || ( + new Set(value.skippedActionIds).size === value.skippedActionIds.length && + value.skippedActionIds.every(id => value.sourceActionIds.includes(id)) +)); export type TaskRetryInput = z.infer; export async function selectTaskRetryInput(tx: DbOrTransaction, input: { companyId: string; issueId: string; agentId: string; failedRunId: string; }) { - const [conversation] = await tx.select({ id: chatConversations.id }).from(chatConversations) + const [conversation] = await tx.select({ id: chatConversations.id, runCreatedAt: heartbeatRuns.createdAt }).from(chatConversations) .innerJoin(chatEndpoints, eq(chatEndpoints.id, chatConversations.endpointId)) .innerJoin(chatActions, and(eq(chatActions.companyId, chatConversations.companyId), eq(chatActions.conversationId, chatConversations.id))) @@ -28,15 +32,20 @@ export async function selectTaskRetryInput(tx: DbOrTransaction, input: { eq(chatEndpoints.externalExecutionPolicy, "restricted"), )).limit(1); if (!conversation) return undefined; - const pending = await tx.select({ id: chatActions.id }).from(chatActions) + const pending = await tx.select({ id: chatActions.id, status: agentWakeupRequests.status }).from(chatActions) .innerJoin(agentWakeupRequests, and(eq(agentWakeupRequests.companyId, chatActions.companyId), eq(agentWakeupRequests.id, chatActions.id))).where(and( eq(chatActions.companyId, input.companyId), eq(chatActions.conversationId, conversation.id), - eq(chatActions.kind, "inbound_wakeup"), eq(chatActions.status, "processed"), - eq(agentWakeupRequests.agentId, input.agentId), eq(agentWakeupRequests.status, "deferred_issue_execution"), + eq(chatActions.kind, "inbound_wakeup"), eq(agentWakeupRequests.agentId, input.agentId), + or(and(eq(chatActions.status, "processed"), eq(agentWakeupRequests.status, "deferred_issue_execution")), + and(eq(chatActions.status, "failed"), eq(agentWakeupRequests.status, "skipped"), + inArray(agentWakeupRequests.reason, ["budget.blocked", "agent.not_invokable"]), + sql`${chatActions.payload}->>'requestedByActorType' = 'user'`, + gt(chatActions.createdAt, conversation.runCreatedAt))), isNull(agentWakeupRequests.runId), )).orderBy(asc(chatActions.createdAt), asc(chatActions.id)).for("update", { of: agentWakeupRequests }); - return taskRetryInputSchema.parse({ sourceActionIds: pending.map(row => row.id) }); + return taskRetryInputSchema.parse({ sourceActionIds: pending.map(row => row.id), + skippedActionIds: pending.filter(row => row.status === "skipped").map(row => row.id) }); } export async function claimTaskRetryInput(tx: DbOrTransaction, input: { diff --git a/server/src/services/chat-run-publications.ts b/server/src/services/chat-run-publications.ts index 25ac411e74b..c3d3bc0d478 100644 --- a/server/src/services/chat-run-publications.ts +++ b/server/src/services/chat-run-publications.ts @@ -131,14 +131,18 @@ export async function resolveChatRunPresentationAuthorizationReason( /** A missing receipt is not proof that a provider write never arrived. */ export async function githubRunReplyState( db: Db, - input: { companyId: string; issueId: string; runId: string; endpointId: string }, + input: { companyId: string; issueId: string; runId: string; endpointId: string; sourceCommentId?: string; requireSettled?: boolean }, ): Promise<"none" | "unsettled" | "confirmed"> { const actions = await db .select({ status: chatActions.status, operation: sql`${chatActions.payload}->>'operation'`, reviewId: sql`${chatActions.payload}->>'reviewId'`, - result: chatActions.result, + result: sql | null>`case when ${chatActions.result} is null then null else jsonb_build_object( + 'reviewId', ${chatActions.result}->'reviewId', 'id', ${chatActions.result}->'id', 'url', ${chatActions.result}->'url', 'summaryUrl', ${chatActions.result}->'summaryUrl', + 'replyWriteStarted', ${chatActions.result}->'replyWriteStarted', 'retryable', ${chatActions.result}->'retryable', + 'code', ${chatActions.result}->'code', 'attempts', ${chatActions.result}->'attempts', + 'deliveryUnknown', ${chatActions.result}->'deliveryUnknown') end`, }) .from(chatActions) .innerJoin(chatConversations, and( @@ -161,12 +165,19 @@ export async function githubRunReplyState( sql`${chatActions.payload}->'session'->>'issueId' = ${input.issueId}`, sql`${chatActions.payload}->'session'->>'runId' = ${input.runId}`, sql`${chatActions.payload}->>'operation' in ('comment', 'formal_review', 'assessment')`, + input.sourceCommentId ? sql`exists (select 1 from chat_message_links source_link + where source_link.company_id = ${chatActions.companyId} and source_link.endpoint_id = ${chatActions.endpointId} + and source_link.conversation_id = ${chatActions.conversationId} and source_link.delivery_id = ${chatActions.deliveryId} + and source_link.direction = 'inbound' and source_link.comment_id = ${input.sourceCommentId}::uuid)` : undefined, )); - let unsettled = false; + let unsettled = false, confirmed = false; for (const action of actions) { + if (input.requireSettled && (action.status !== "processed" || action.result?.deliveryUnknown === true)) { + unsettled = true; + } // An assessment can publish findings before a later step fails or loses // authority. Those durable receipts still prove a reply was delivered. - if (action.operation === "assessment") { + if (action.operation === "assessment" && !input.requireSettled) { const [review] = await db .select({ receipts: chatGitHubReviews.publicationReceipts }) .from(chatGitHubReviews) @@ -181,11 +192,16 @@ export async function githubRunReplyState( } if (action.status === "processed") { if (action.operation === "comment" || action.operation === "formal_review") { - if (action.result?.id && action.result?.url) return "confirmed"; - unsettled = true; + if (action.result?.id && action.result?.url) { + if (!input.requireSettled) return "confirmed"; + confirmed = true; + } else unsettled = true; } if (action.operation === "assessment") { - if (action.result?.summaryUrl) return "confirmed"; + if (action.result?.summaryUrl) { + if (!input.requireSettled) return "confirmed"; + confirmed = true; + } else if (input.requireSettled) unsettled = true; } } else if (action.result?.replyWriteStarted === true || ["received", "processing"].includes(action.status) || (action.status === "failed" && (action.result?.retryable === true || action.result?.code === "publication_failed")) || @@ -194,7 +210,7 @@ export async function githubRunReplyState( unsettled = true; } } - return unsettled ? "unsettled" : "none"; + return unsettled ? "unsettled" : confirmed ? "confirmed" : "none"; } type ChatRunMilestoneCandidate = { diff --git a/server/src/services/durable-chat-wakeup.ts b/server/src/services/durable-chat-wakeup.ts index 9a689254e03..65a150f8a26 100644 --- a/server/src/services/durable-chat-wakeup.ts +++ b/server/src/services/durable-chat-wakeup.ts @@ -181,6 +181,19 @@ export function unadmittedChatWakeupCondition( and chat_admission.agent_id::text = chat_unadmitted.payload->>'agentId' and chat_admission.status <> 'skipped' ) + and not exists ( + select 1 from chat_actions chat_retry + join agent_wakeup_requests chat_retry_admission on chat_retry_admission.id = chat_retry.id + where chat_retry.company_id = chat_unadmitted.company_id + and chat_retry.endpoint_id = chat_unadmitted.endpoint_id + and chat_retry.conversation_id = chat_unadmitted.conversation_id + and chat_retry.kind = 'failed_run_retry' + and chat_retry.payload->'taskRetryInput'->'skippedActionIds' ? chat_unadmitted.id::text + and chat_retry.payload->>'issueId' = chat_unadmitted.payload->>'issueId' + and chat_retry_admission.company_id = chat_retry.company_id + and chat_retry_admission.agent_id::text = chat_retry.payload->>'agentId' + and chat_retry_admission.status <> 'skipped' + ) and (chat_unadmitted.status <> 'failed' or not exists ( select 1 from chat_actions chat_later join agent_wakeup_requests chat_later_admission on chat_later_admission.id = chat_later.id diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 8cda4456d6c..72a79b53199 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -2322,7 +2322,7 @@ export function heartbeatService( interactionId: readNonEmptyString(context.interactionId), issueId, messageIds: - run.retryOfRunId || context.retryOfRunId + (run.retryOfRunId || context.retryOfRunId) && !isFailedChatRunRetry ? [] : queuedCommentIdsFromRunContext(context).length ? queuedCommentIdsFromRunContext(context) diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts index 7d337f06cfb..debcd1c18e0 100644 --- a/server/src/services/heartbeat/queue.ts +++ b/server/src/services/heartbeat/queue.ts @@ -1,4 +1,4 @@ -import { assertTaskRetryConversationAllowed, readTaskRetryIntent, retireSupersededTaskRetry, taskRetryMutationAllowed } from "../task-retry.js"; +import { assertTaskRetryConversationAllowed, readTaskRetryIntent, readTaskRetryBoardMessageOrigin, retireSupersededTaskRetry, taskRetryMutationAllowed } from "../task-retry.js"; import { insertIssueComment } from "../issue-comment-writes.js"; import { withExternalAdmissionGuard, assertNoExternalOverlap } from "../../modules/external-agents/index.js"; import { isAgentAwaitingSetup } from "../../modules/agent-lifecycle/index.js"; @@ -2195,6 +2195,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende if (executionReconciliationWake && !issueId) return null; const durableRequest = opts.durableChatRequest; let taskRetryActorId: string | null = null; + delete enrichedContextSnapshot.boardMessageRetry; if (opts.taskRetryRequestId) { const [receipt] = await db.select().from(agentWakeupRequests).where(and( eq(agentWakeupRequests.id, opts.taskRetryRequestId), eq(agentWakeupRequests.agentId, agentId), @@ -2213,6 +2214,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende enrichedContextSnapshot.forceFreshSession = true; enrichedContextSnapshot.previousRunId = retry.sourceRunId; enrichedContextSnapshot.taskRetryRequestId = receipt.id; + if (retry.boardMessageOrigin) enrichedContextSnapshot.boardMessageRetry = true; } @@ -2859,6 +2861,12 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende return { kind: "deferred" as const }; if (!await taskRetryMutationAllowed(tx as unknown as Db, { companyId: agent.companyId, issueId, actorId: retry.actorId, actorSource: retry.actorSource })) return { kind: "deferred" as const }; + if (retry.boardMessageOrigin) { + const origin = await readTaskRetryBoardMessageOrigin(tx as unknown as Db, { companyId: agent.companyId, + issueId, agentId, sourceRunId: retry.sourceRunId, actorId: retry.actorId }); + if (!origin || origin.commentId !== retry.boardMessageOrigin.commentId || + origin.identityContextId !== retry.boardMessageOrigin.identityContextId) return { kind: "deferred" as const }; + } } if (opts.queuedCommentRequestId) { const ids = await undeliveredLegacyUserCommentIds(tx as unknown as Db, diff --git a/server/src/services/native-runtime/chat-attachment-reuse.ts b/server/src/services/native-runtime/chat-attachment-reuse.ts index 3d30cf42f74..cc7990e9b7a 100644 --- a/server/src/services/native-runtime/chat-attachment-reuse.ts +++ b/server/src/services/native-runtime/chat-attachment-reuse.ts @@ -19,6 +19,7 @@ import { issueComments, issues, issueWorkProducts, + runIdentityContexts, } from "@paperclipai/db"; import { @@ -32,8 +33,33 @@ import { loadConfig } from "../../config.js"; import { HttpError } from "../../errors.js"; import { voiceSessionStore, type VoiceTransaction } from "../voice/voice-session-store.js"; import { issueService } from "../issues.js"; +import { readInheritedGitHubReviewWait } from "../chat-channels/github/review-wait.js"; import { boundExternalChatProvider } from "./external-chat-provider.js"; import { resolveExternalChatQuestionResponse } from "./external-chat-question-response.js"; +import { githubRunReplyState } from "../chat-run-publications.js"; + +// This boundary reads identity and wake authority, never launch renderings, +// saved provider profiles, or run output. Keep the nested wake proof narrow too. +const responseWaitContextKeys = ["source", "issueId", "wakeReason", "wakeCommentId", "wakeCommentIds", + "interactionId", "interactionKind", "interactionStatus", "sourceRunId", "sourceCommentId", + "externalChatContinuation", "paperclipExternalChatQuestionResponse", "paperclipHarnessCheckedOut", + "paperclipExternalChatExecutionBound"] as const; +const responseWaitRun = { + id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + nativeIssueId: heartbeatRuns.nativeIssueId, issueId: heartbeatRuns.issueId, status: heartbeatRuns.status, + responsibleUserId: heartbeatRuns.responsibleUserId, activeIdentityContextId: heartbeatRuns.activeIdentityContextId, + contextSnapshot: sql | null>`case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then (select coalesce(jsonb_object_agg(field.key, field.value), '{}'::jsonb) + from jsonb_each(${heartbeatRuns.contextSnapshot}) as field(key, value) + where field.key in (${sql.join(responseWaitContextKeys.map(key => sql`${key}`), sql`, `)})) + || jsonb_build_object('paperclipWake', jsonb_build_object( + 'checkedOutByHarness', ${heartbeatRuns.contextSnapshot}#>'{paperclipWake,checkedOutByHarness}', + 'externalChatExecutionBound', ${heartbeatRuns.contextSnapshot}#>'{paperclipWake,externalChatExecutionBound}', + 'externalChatProvider', ${heartbeatRuns.contextSnapshot}#>'{paperclipWake,externalChatProvider}', + 'issue', jsonb_build_object('id', ${heartbeatRuns.contextSnapshot}#>'{paperclipWake,issue,id}'), + 'commentIds', ${heartbeatRuns.contextSnapshot}#>'{paperclipWake,commentIds}')) + else null end`, +}; export const LIST_CHAT_ATTACHMENTS_TOOL_NAME = "list_chat_attachments"; export const REUSE_CHAT_ATTACHMENT_TOOL_NAME = "reuse_chat_attachment"; @@ -652,7 +678,7 @@ export async function resolveExternalChatResponseWaitAuthorizationInTransaction( : issueQuery.for("update").limit(1)); const runQuery = tx - .select() + .select(responseWaitRun) .from(heartbeatRuns) .where( and( @@ -684,12 +710,12 @@ export async function resolveExternalChatResponseWaitAuthorizationInTransaction( !answerContext ) return "revoked"; - const candidate = run - ? externalChatWaitCandidate( - answerContext?.authorizationContext ?? run.contextSnapshot, - binding, - ) - : null; + const directContext = answerContext?.authorizationContext ?? run?.contextSnapshot; + const inheritedReviewWait = run && !externalChatWaitCandidate(directContext, binding) + ? await readInheritedGitHubReviewWait(tx, run) : null; + if (inheritedReviewWait?.kind === "revoked") return "revoked"; + const authorizationContext = inheritedReviewWait?.context ?? directContext; + const candidate = run ? externalChatWaitCandidate(authorizationContext, binding) : null; if (!candidate) return "not_applicable"; const actorQuery = tx @@ -779,7 +805,7 @@ export async function resolveExternalChatResponseWaitAuthorizationInTransaction( await authorizeChatConversationForBoundRun( tx, binding, - run.contextSnapshot, + authorizationContext, lockMode, ); return "authorized"; @@ -811,6 +837,49 @@ export async function resolveExternalChatResponseWaitAuthorization(input: { ); } +const publishedGitHubReviewWaitKey = "github-review-response:"; + +export function isPublishedGitHubReviewCompletionContinuation(value: unknown): boolean { + const continuation = record(value); + return continuation.kind === "response_wake" && typeof continuation.idempotencyKey === "string" && + continuation.idempotencyKey.startsWith(publishedGitHubReviewWaitKey); +} + +/** Project a wait from the same saved publication and current authority used + * by finalization. Inherited waits need an accepted tree resume or a confirmed + * reply from this run to its exact current source. An arbitrary old review does + * not assign an unanswered new turn a wait. */ +export async function readPublishedGitHubReviewCompletionContinuationInTransaction( + tx: Db, binding: ChatReuseBinding, lockMode: AuthorizationLockMode = "read", +) { + const [run] = await tx.select(responseWaitRun).from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, binding.runId), eq(heartbeatRuns.companyId, binding.companyId), + eq(heartbeatRuns.agentId, binding.agentId), eq(heartbeatRuns.nativeIssueId, binding.issueId), + eq(heartbeatRuns.runtimeMode, "native"), + )).limit(1); + if (!run || run.status !== "running") return null; + const wait = await readInheritedGitHubReviewWait(tx, run); + if (wait?.kind !== "published") return null; + if (wait.reviewRunId !== run.id) { + const [identity] = await tx.select({ cause: runIdentityContexts.cause, parentContextId: runIdentityContexts.parentContextId }) + .from(runIdentityContexts).where(and(eq(runIdentityContexts.id, run.activeIdentityContextId!), + eq(runIdentityContexts.companyId, run.companyId), eq(runIdentityContexts.runId, run.id), + eq(runIdentityContexts.status, "accepted"))).limit(1); + const resumed = run.contextSnapshot?.source === "issue.tree_resume" && identity?.cause === "issue_tree_resumed" && + Boolean(identity.parentContextId); + if (!resumed && !(run.contextSnapshot?.source === "chat:github" && + await githubRunReplyState(tx, { ...binding, endpointId: wait.endpointId, + sourceCommentId: wait.context.wakeCommentId, requireSettled: true }) === "confirmed")) return null; + } + if (await resolveExternalChatResponseWaitAuthorizationInTransaction(tx, binding, lockMode) !== "authorized") return null; + return { kind: "response_wake" as const, summary: "Wait for the next authorized GitHub response to the published review.", + idempotencyKey: `${publishedGitHubReviewWaitKey}${run.id}:${wait.reviewId}:${wait.headSha}` }; +} + +export async function readPublishedGitHubReviewCompletionContinuation(input: { db: Db; binding: ChatReuseBinding }) { + return input.db.transaction(tx => readPublishedGitHubReviewCompletionContinuationInTransaction(tx as unknown as Db, input.binding)); +} + async function sourceLineageExists( tx: Db, binding: ChatReuseBinding, diff --git a/server/src/services/native-runtime/external-chat-wait.integration.test.ts b/server/src/services/native-runtime/external-chat-wait.integration.test.ts index 70a6933e80a..50d7910bb96 100644 --- a/server/src/services/native-runtime/external-chat-wait.integration.test.ts +++ b/server/src/services/native-runtime/external-chat-wait.integration.test.ts @@ -1,3 +1,4 @@ +import { initializeRunIdentity } from "../run-identity.js"; import { createHash, randomUUID } from "node:crypto"; import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; @@ -36,12 +37,14 @@ import { issues, nativeRunFinalizations, nativeRunResults, + runIdentityContexts, statusDecisions, toolApplications, toolConnections, chatVoiceSessions, chatVoiceReplies, chatVoiceToolCalls, workspaceOperations, } from "@paperclipai/db"; +import { validatePrpStructuredRunResult } from "../../vendor/paperclip-runner/index.js"; import type { PrpStructuredRunResult, PrpTerminalState, @@ -50,7 +53,10 @@ import type { import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; import { finalizeNativeRun } from "./native-run-finalizer.js"; import { nativeCompletionFeedback } from "./native-completion-feedback.js"; -import { hasPublishedGitHubReview } from "../chat-channels/github/review-wait.js"; +import { hasPublishedGitHubReview, readInheritedGitHubReviewWait } from "../chat-channels/github/review-wait.js"; +import * as chatWaitAuthority from "./chat-attachment-reuse.js"; +import * as publishedReviewAuthority from "../chat-channels/github/review-wait.js"; +import { toolAccessService } from "../tool-access.js"; import { PaperclipControlPlanePort } from "./paperclip-control-plane-port.js"; import { authorizeNativeChatReviewPresentation, @@ -114,6 +120,8 @@ describe("native external-chat response wait", () => { | "slack" = "telegram", attentionRequests: PrpStructuredRunResult["attentionRequests"] = [], resultOverrides: Partial = {}, + deferCompletion = false, + contractOverride?: { objective: string; criteria: Array<{ id: string; requirement: string }> }, ) { const companyId = randomUUID(); const agentId = randomUUID(); @@ -177,6 +185,7 @@ describe("native external-chat response wait", () => { criteria: [ { id: "response", requirement: "Return the requested photo" }, ], + ...contractOverride, }, canonicalSha256: contractSha256, createdByActorType: "system", @@ -384,7 +393,7 @@ describe("native external-chat response wait", () => { workAssessmentId: randomUUID(), statusDecisionId: randomUUID(), }; - await port.completeRun({ + if (!deferCompletion) await port.completeRun({ result, terminal, callerResultId: `wait-result-${runId}`, @@ -401,6 +410,7 @@ describe("native external-chat response wait", () => { .where(eq(heartbeatRuns.id, runId)); return { + port, result, terminal, agentId, companyId, conversationId, @@ -4385,7 +4395,14 @@ describe("native external-chat response wait", () => { [5, "done", "done", null, false], [3, "yielded", "in_review", "monitor", false], [3, "yielded", "in_review", "response_wake", true], - ] as const)("settles a published %s/5 review reporting %s in %s with %s, separate approval: %s", async (score, disposition, status, continuationKind, separateApproval) => { + [3, "yielded", "in_review", "response_wake", false, "tree_resume"], + [3, "yielded", "in_review", "response_wake", false, "continued_review"], + [3, "yielded", "in_review", "monitor", false, "continued_review"], + [3, "yielded", "in_progress", "response_wake", false, "new_head"], + ] as const)("settles a published %s/5 review reporting %s in %s with %s, separate approval: %s, resume: %s", async (score, disposition, status, continuationKind, separateApproval, resumeMode?: "tree_resume" | "continued_review" | "new_head") => { + const resumed = resumeMode !== undefined; + const savedPublication = resumeMode === "continued_review" || resumeMode === "new_head"; + const unreviewedHead = resumeMode === "new_head"; const fixture = await seedWaitTurn("github", [], { reportedWorkDisposition: disposition, summary: disposition === "done" ? "Review is 5/5 and required checks passed." : "Review published; waiting for the next PR update.", @@ -4394,7 +4411,7 @@ describe("native external-chat response wait", () => { }); await db.update(issues).set({ title: "Review the pull request", originKind: "chat_channel", createdByUserId: fixture.userId, responsibleUserId: fixture.userId }).where(eq(issues.id, fixture.issueId)); - const oldRequest = score === 3 ? await seedPriorCompletionReview(fixture) : null; + const oldRequest = score === 3 && !resumed ? await seedPriorCompletionReview(fixture) : null; if (continuationKind === "monitor") { await new PaperclipRunnerToolAuthority(db, fixture).execute({ tool: "set_task_monitor", callId: randomUUID(), arguments: { idempotencyKey: "review-monitor", monitor: { nextCheckAt: new Date(Date.now() + 60_000).toISOString(), notes: "Check PR status." } } }); @@ -4417,7 +4434,7 @@ describe("native external-chat response wait", () => { }).returning(); await expect(hasPublishedGitHubReview(db, fixture)).resolves.toBe(false); // A completed but unpublished head must not hide another published head from this run. - const [published] = await db.insert(chatGitHubReviews).values({ ...review!, id: randomUUID(), deliveryId: randomUUID(), headSha: "c".repeat(40) }).returning(); + const [published] = await db.insert(chatGitHubReviews).values({ ...review!, id: randomUUID(), deliveryId: randomUUID(), headSha: "c".repeat(40), assessment: { ...review!.assessment!, reviewedCommit: "c".repeat(40) }, createdAt: new Date(Date.now() + 10) }).returning(); await db.insert(chatActions).values({ companyId: fixture.companyId, endpointId: fixture.endpointId, kind: "github_review_publication", providerActionId: `github_publication:${fixture.issueId}:assessment:${createHash("sha256").update(JSON.stringify(published!.id)).digest("hex")}`, @@ -4433,8 +4450,76 @@ describe("native external-chat response wait", () => { providerActionId: `github_publication:${fixture.issueId}:assessment:${createHash("sha256").update(JSON.stringify(older!.id)).digest("hex")}`, }); } + let inheritedOriginalRunId: string | null = null; + if (resumed) { + await db.update(chatEndpointResources).set({ metadata: { providerRepositoryId: "1" } }).where(eq(chatEndpointResources.id, fixture.resourceId)); + await db.update(chatConversations).set({ externalThreadId: "github:paperclip/test-repository:1" }).where(eq(chatConversations.id, fixture.conversationId)); + // A tree resume inherits its requester through the accepted identity + // ledger, while its immediate wake correctly has no chat envelope. + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + const originalRunId = randomUUID(); + inheritedOriginalRunId = originalRunId; + await db.insert(heartbeatRuns).values({ ...run!, id: originalRunId, activeIdentityContextId: null, status: "succeeded", finishedAt: new Date() }); + const [origin] = await db.insert(runIdentityContexts).values({ + companyId: fixture.companyId, runId: originalRunId, revision: 1, + messageId: fixture.commentId, responsibleUserId: fixture.userId, + cause: "chat_message", correlationId: "dispatch", status: "accepted", + }).returning(); + const [continuation] = await db.insert(runIdentityContexts).values({ + companyId: fixture.companyId, runId: fixture.runId, revision: 1, + parentContextId: origin!.id, responsibleUserId: fixture.userId, + cause: "issue_tree_resumed", correlationId: "dispatch", status: "accepted", + }).returning(); + await db.update(heartbeatRuns).set({ activeIdentityContextId: continuation!.id, + responsibleUserId: fixture.userId, contextSnapshot: { + source: "issue.tree_resume", wakeReason: "issue_tree_resumed", + issueId: fixture.issueId, paperclipHarnessCheckedOut: true, + paperclipWake: { checkedOutByHarness: true, issue: { id: fixture.issueId }, commentIds: [] }, + }, + }).where(eq(heartbeatRuns.id, fixture.runId)); + } await expect(hasPublishedGitHubReview(db, fixture)).resolves.toBe(true); await expect(hasPublishedGitHubReview(db, { ...fixture, runId: randomUUID() })).resolves.toBe(false); + if (resumed) { + const [inheritedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + await expect(readInheritedGitHubReviewWait(db, inheritedRun!)).resolves.toMatchObject({ kind: "published" }); + await expect(resolveExternalChatResponseWaitAuthorization({ db, binding: fixture })).resolves.toBe("authorized"); + for (const binding of [ + { ...fixture, companyId: randomUUID() }, { ...fixture, issueId: randomUUID() }, + { ...fixture, agentId: randomUUID() }, + ]) await expect(resolveExternalChatResponseWaitAuthorization({ db, binding })).resolves.not.toBe("authorized"); + await db.update(chatActions).set({ status: "pending" }).where(eq(chatActions.companyId, fixture.companyId)); + await expect(resolveExternalChatResponseWaitAuthorization({ db, binding: fixture })).resolves.not.toBe("authorized"); + await db.update(chatActions).set({ status: "processed" }).where(eq(chatActions.companyId, fixture.companyId)); + await db.update(chatEndpointResources).set({ enabled: false }).where(eq(chatEndpointResources.endpointId, fixture.endpointId)); + await expect(resolveExternalChatResponseWaitAuthorization({ db, binding: fixture })).resolves.toBe("revoked"); + await db.update(chatEndpointResources).set({ enabled: true }).where(eq(chatEndpointResources.endpointId, fixture.endpointId)); + await db.update(heartbeatRuns).set({ responsibleUserId: "different-requester" }).where(eq(heartbeatRuns.id, fixture.runId)); + await expect(resolveExternalChatResponseWaitAuthorization({ db, binding: fixture })).resolves.not.toBe("authorized"); + await db.update(heartbeatRuns).set({ responsibleUserId: fixture.userId }).where(eq(heartbeatRuns.id, fixture.runId)); + await db.update(chatConversations).set({ externalThreadId: "github:paperclip/other-repository:1" }).where(eq(chatConversations.id, fixture.conversationId)); + await expect(resolveExternalChatResponseWaitAuthorization({ db, binding: fixture })).resolves.not.toBe("authorized"); + await db.update(chatConversations).set({ externalThreadId: "github:paperclip/test-repository:1" }).where(eq(chatConversations.id, fixture.conversationId)); + await db.update(chatEndpointResources).set({ metadata: { providerRepositoryId: "2" } }).where(eq(chatEndpointResources.id, fixture.resourceId)); + await expect(resolveExternalChatResponseWaitAuthorization({ db, binding: fixture })).resolves.not.toBe("authorized"); + await db.update(chatEndpointResources).set({ metadata: { providerRepositoryId: "1" } }).where(eq(chatEndpointResources.id, fixture.resourceId)); + const [newHead] = await db.insert(chatGitHubReviews).values({ ...review!, id: randomUUID(), + deliveryId: randomUUID(), runId: null, headSha: "d".repeat(40), state: "queued", createdAt: new Date(Date.now() + 1000), + }).returning(); + await expect(resolveExternalChatResponseWaitAuthorization({ db, binding: fixture })).resolves.not.toBe("authorized"); + await db.delete(chatGitHubReviews).where(eq(chatGitHubReviews.id, newHead!.id)); + if (savedPublication) { + await db.update(chatGitHubReviews).set({ runId: inheritedOriginalRunId }).where(eq(chatGitHubReviews.issueId, fixture.issueId)); + await expect(hasPublishedGitHubReview(db, fixture)).resolves.toBe(false); + await expect(resolveExternalChatResponseWaitAuthorization({ db, binding: fixture })).resolves.toBe("authorized"); + } + } + if (unreviewedHead) { + await db.insert(chatGitHubReviews).values({ ...review!, id: randomUUID(), runId: null, + deliveryId: randomUUID(), headSha: "e".repeat(40), state: "queued", createdAt: new Date(Date.now() + 1000), + }); + await expect(resolveExternalChatResponseWaitAuthorization({ db, binding: fixture })).resolves.toBe("revoked"); + } const [stored] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, fixture.runId)); await expect(nativeCompletionFeedback(db, fixture.runId, stored!.resultJson.result as PrpStructuredRunResult)).resolves.toContain("accepted"); if (oldRequest) { @@ -4468,6 +4553,15 @@ describe("native external-chat response wait", () => { if (oldRequest) expect(await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, oldRequest.id))) .toEqual([expect.objectContaining({ status: "cancelled", result: expect.objectContaining({ outcome: "withdrawn" }) })]); expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, fixture.companyId))).toEqual([]); + if (resumed) { + await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, fixture.companyId))).toEqual([]); + expect(await db.select().from(chatActions).where(and(eq(chatActions.companyId, fixture.companyId), eq(chatActions.kind, "github_review_publication")))).toHaveLength(1); + if (unreviewedHead) { + expect((await db.select().from(statusDecisions).where(eq(statusDecisions.runId, fixture.runId)))[0]) + .toMatchObject({ reasonCode: "external_chat_response_wait_authorization_lost" }); + } + } if (status === "in_review" && continuationKind === "response_wake") { const attention = async () => (await issueService(db).listReviewAttention(fixture.companyId, [{ id: fixture.issueId, companyId: fixture.companyId, status }])).get(fixture.issueId)?.state; await db.update(chatEndpointResources).set({ enabled: false }).where(eq(chatEndpointResources.endpointId, fixture.endpointId)); @@ -4483,6 +4577,397 @@ describe("native external-chat response wait", () => { } }); + async function seedPublishedReviewCompletionTurn(contractOverride?: Parameters[4]) { + const fixture = await seedWaitTurn("github", [], {}, true, contractOverride); + await db.update(chatEndpointResources).set({ metadata: { providerRepositoryId: "1" } }) + .where(eq(chatEndpointResources.id, fixture.resourceId)); + await db.update(chatConversations).set({ externalThreadId: "github:paperclip/test-repository:1" }) + .where(eq(chatConversations.id, fixture.conversationId)); + const [identity] = await db.insert(runIdentityContexts).values({ companyId: fixture.companyId, + runId: fixture.runId, revision: 1, messageId: fixture.commentId, responsibleUserId: fixture.userId, + cause: "chat_message", correlationId: "dispatch", status: "accepted" }).returning(); + await db.update(heartbeatRuns).set({ activeIdentityContextId: identity!.id, responsibleUserId: fixture.userId }) + .where(eq(heartbeatRuns.id, fixture.runId)); + const [review] = await db.insert(chatGitHubReviews).values({ companyId: fixture.companyId, + endpointId: fixture.endpointId, issueId: fixture.issueId, runId: fixture.runId, + repositoryId: "1", repository: "paperclip/test-repository", pullNumber: 1, headSha: "a".repeat(40), + deliveryId: fixture.deliveryId, configurationRevision: 1, policySnapshot: defaultGitHubReviewPolicy(), + event: { event: "opened", deliveryId: fixture.deliveryId, repositoryId: "1", repository: "paperclip/test-repository", + pullNumber: 1, title: "Review", body: "", baseSha: "b".repeat(40), headSha: "a".repeat(40), + baseBranch: "master", author: { id: "1", login: "author", isBot: false }, sender: { id: "1", login: "author" }, draft: false, labels: [] }, + state: "completed", assessment: { complete: true, score: 3, reviewedCommit: "a".repeat(40), + summary: "Quantity is ignored", rationale: "Review complete", findings: [], + coverage: { reviewedPaths: [], omittedPaths: [], limitations: [] } }, checkId: "123" }).returning(); + await db.insert(chatActions).values({ companyId: fixture.companyId, endpointId: fixture.endpointId, + kind: "github_review_publication", status: "processed", payload: { operation: "assessment", reviewId: review!.id }, + providerActionId: `github_publication:${fixture.issueId}:assessment:${createHash("sha256").update(JSON.stringify(review!.id)).digest("hex")}` }); + return { ...fixture, review: review! }; + } + + it.each([false, true])("binds a low-score Retry wait to its accepted newer source after a predecessor failed before identity: %s", async acceptedSource => { + const objective = "Review the current head. A lower score stays In review with a passive response wake; only a merged PR or 5/5 with passing checks is Done."; + const fixture = await seedPublishedReviewCompletionTurn({ objective, + criteria: [{ id: "response", requirement: objective }] }); + await db.update(chatGitHubReviews).set({ policySnapshot: { ...fixture.review.policySnapshot, ratingThreshold: 5 }, + assessment: { ...fixture.review.assessment!, score: 2 } }).where(eq(chatGitHubReviews.id, fixture.review.id)); + const previousRunId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: previousRunId, companyId: fixture.companyId, agentId: fixture.agentId, + issueId: fixture.issueId, status: "failed", errorCode: "chat_failed_run_retry_not_authorized", + finishedAt: new Date(), contextSnapshot: { issueId: fixture.issueId, source: "chat:github" } }); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + await db.update(heartbeatRuns).set({ activeIdentityContextId: null, retryOfRunId: previousRunId, + contextSnapshot: { ...run!.contextSnapshot, retryOfRunId: previousRunId, wakeReason: "retry_failed_run" } }) + .where(eq(heartbeatRuns.id, fixture.runId)); + await db.delete(runIdentityContexts).where(eq(runIdentityContexts.id, run!.activeIdentityContextId!)); + await initializeRunIdentity(db, { companyId: fixture.companyId, issueId: fixture.issueId, + runId: fixture.runId, responsibleUserId: fixture.userId, parentRunId: previousRunId, + messageIds: acceptedSource ? [fixture.commentId] : [], cause: "retry_failed_run" }); + const authority = new PaperclipRunnerToolAuthority(db, fixture); + const read = () => authority.execute({ tool: "get_task_context", callId: randomUUID(), arguments: {} }); + const { resolveCompletionContinuation } = await import(new URL("../../../../packages/paperclip-runner/src/drivers/completion-continuation.ts", import.meta.url).href); + const report = { ...fixture.result, continuation: undefined, completionClaim: { + ...fixture.result.completionClaim, objectiveSatisfied: false, + criteria: [{ criterionId: "response", status: "unknown" as const, evidenceRefs: [] }], + remainingWork: [{ description: "Author must fix the quantity defect before this review can reach the passing score.", blocksCompletion: true }], + } }; + if (!acceptedSource) { + await expect(resolveCompletionContinuation(report, fixture.runId, read)).rejects.toThrow("no eligible saved monitor"); + expect((await db.select().from(issues).where(eq(issues.id, fixture.issueId)))[0].status).not.toBe("done"); + return; + } + const resolved = await resolveCompletionContinuation(report, fixture.runId, read) as PrpStructuredRunResult; + expect(resolved).toMatchObject({ reportedWorkDisposition: "yielded", continuation: { kind: "response_wake" } }); + await expect(nativeCompletionFeedback(db, fixture.runId, resolved)).resolves.toContain("accepted"); + await fixture.port.completeRun({ result: resolved, terminal: fixture.terminal, callerResultId: `retry-wait-${fixture.runId}` }); + await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + expect((await db.select().from(issues).where(eq(issues.id, fixture.issueId)))[0]).toMatchObject({ status: "in_review" }); + expect((await db.select().from(statusDecisions).where(eq(statusDecisions.runId, fixture.runId)))[0]) + .toMatchObject({ reasonCode: "external_chat_response_waiting", toStatus: "in_review", decisionJson: { effects: [] } }); + await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, fixture.companyId))).toHaveLength(0); + expect(await db.select().from(chatActions).where(and(eq(chatActions.companyId, fixture.companyId), + eq(chatActions.kind, "github_review_publication")))).toHaveLength(1); + }); + + async function seedInheritedReviewCompletionTurn(resume: boolean) { + const fixture = await seedPublishedReviewCompletionTurn(); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + const predecessorId = randomUUID(); + await db.insert(heartbeatRuns).values({ ...run!, id: predecessorId, activeIdentityContextId: null, + status: "succeeded", finishedAt: new Date() }); + await db.update(chatGitHubReviews).set({ runId: predecessorId }).where(eq(chatGitHubReviews.id, fixture.review.id)); + if (resume) { + const [origin] = await db.insert(runIdentityContexts).values({ companyId: fixture.companyId, + runId: predecessorId, revision: 1, messageId: fixture.commentId, responsibleUserId: fixture.userId, + cause: "chat_message", correlationId: "dispatch", status: "accepted" }).returning(); + await db.update(runIdentityContexts).set({ messageId: null, parentContextId: origin!.id, + cause: "issue_tree_resumed" }).where(eq(runIdentityContexts.id, run!.activeIdentityContextId!)); + await db.update(heartbeatRuns).set({ contextSnapshot: { source: "issue.tree_resume", wakeReason: "issue_tree_resumed", + issueId: fixture.issueId, paperclipHarnessCheckedOut: true, + paperclipWake: { checkedOutByHarness: true, issue: { id: fixture.issueId }, commentIds: [] } }, + }).where(eq(heartbeatRuns.id, fixture.runId)); + } + return { ...fixture, predecessorId }; + } + + it.each([undefined, null, "malformed-first"] as const)("derives an omitted inherited GitHub review wait after normal tree resume: %s", async absent => { + const fixture = await seedInheritedReviewCompletionTurn(true); + const authority = new PaperclipRunnerToolAuthority(db, fixture); + const read = () => authority.execute({ tool: "get_task_context", callId: randomUUID(), arguments: {} }); + const { resolveCompletionContinuation } = await import(new URL("../../../../packages/paperclip-runner/src/drivers/completion-continuation.ts", import.meta.url).href); + const report = { ...fixture.result, continuation: absent === "malformed-first" ? "response_wake" : absent }; + if (absent === "malformed-first") { + expect(await resolveCompletionContinuation(report, fixture.runId, read)).toEqual(report); + expect(validatePrpStructuredRunResult(report).ok).toBe(false); + delete report.continuation; + } + const resolved = await resolveCompletionContinuation(report, fixture.runId, read) as PrpStructuredRunResult; + expect(resolved).toMatchObject({ reportedWorkDisposition: "yielded", continuation: { kind: "response_wake" } }); + await expect(nativeCompletionFeedback(db, fixture.runId, resolved)).resolves.toContain("accepted"); + await fixture.port.completeRun({ result: resolved, terminal: fixture.terminal, callerResultId: `wait-result-${fixture.runId}` }); + await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + expect((await db.select().from(issues).where(eq(issues.id, fixture.issueId)))[0]).toMatchObject({ status: "in_review" }); + expect((await db.select().from(statusDecisions).where(eq(statusDecisions.runId, fixture.runId)))[0]) + .toMatchObject({ reasonCode: "external_chat_response_waiting", decisionJson: { effects: [] } }); + await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, fixture.companyId))).toHaveLength(0); + expect(await db.select().from(chatActions).where(and(eq(chatActions.companyId, fixture.companyId), + eq(chatActions.kind, "github_review_publication")))).toHaveLength(1); + }); + + async function seedCurrentAuthorReply(fixture: Awaited>) { + const [reply] = await db.insert(chatActions).values({ companyId: fixture.companyId, + endpointId: fixture.endpointId, conversationId: fixture.conversationId, deliveryId: fixture.deliveryId, + kind: "github_review_publication", status: "processed", providerActionId: `github_publication:${fixture.issueId}:comment:${randomUUID()}`, + payload: { operation: "comment", session: { companyId: fixture.companyId, issueId: fixture.issueId, + runId: fixture.runId, agentId: fixture.agentId }, body: "Quantity is ignored by the current total calculation." }, + result: { id: "123", url: "https://github.com/paperclip/test-repository/pull/1#issuecomment-123" }, + }).returning(); + return reply!; + } + + it.each([undefined, null] as const)("derives an omitted inherited review wait only after this author reply is delivered: %s", async continuation => { + const fixture = await seedInheritedReviewCompletionTurn(false); + await seedCurrentAuthorReply(fixture); + const authority = new PaperclipRunnerToolAuthority(db, fixture); + const { resolveCompletionContinuation } = await import(new URL("../../../../packages/paperclip-runner/src/drivers/completion-continuation.ts", import.meta.url).href); + const resolved = await resolveCompletionContinuation({ ...fixture.result, continuation }, fixture.runId, + () => authority.execute({ tool: "get_task_context", callId: randomUUID(), arguments: {} })) as PrpStructuredRunResult; + expect(resolved).toMatchObject({ continuation: { kind: "response_wake" } }); + await expect(nativeCompletionFeedback(db, fixture.runId, resolved)).resolves.toContain("accepted"); + await fixture.port.completeRun({ result: resolved, terminal: fixture.terminal, callerResultId: `wait-result-${fixture.runId}` }); + await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + expect((await db.select().from(issues).where(eq(issues.id, fixture.issueId)))[0]).toMatchObject({ status: "in_review" }); + expect((await db.select().from(statusDecisions).where(eq(statusDecisions.runId, fixture.runId)))[0]) + .toMatchObject({ reasonCode: "external_chat_response_waiting", decisionJson: { effects: [] } }); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, fixture.companyId))).toHaveLength(0); + }); + + it.each(["missing_parent", "wrong_cause", "principal_revoked", "new_head", "pending_publication"] as const)( + "does not derive an inherited tree resume wait after %s", async gate => { + const fixture = await seedInheritedReviewCompletionTurn(true); + if (gate === "missing_parent" || gate === "wrong_cause") await db.update(runIdentityContexts) + .set(gate === "missing_parent" ? { parentContextId: null } : { cause: "chat_message" }) + .where(eq(runIdentityContexts.runId, fixture.runId)); + if (gate === "principal_revoked") await db.update(companyMemberships).set({ status: "suspended" }) + .where(eq(companyMemberships.companyId, fixture.companyId)); + if (gate === "new_head") await db.insert(chatGitHubReviews).values({ ...fixture.review, id: randomUUID(), + runId: null, deliveryId: randomUUID(), headSha: "c".repeat(40), state: "queued", assessment: null, + createdAt: new Date(Date.now() + 1000) }); + if (gate === "pending_publication") await db.update(chatActions).set({ status: "received" }) + .where(eq(chatActions.companyId, fixture.companyId)); + const context = new PaperclipRunnerToolAuthority(db, fixture) + .execute({ tool: "get_task_context", callId: randomUUID(), arguments: {} }); + if (gate === "principal_revoked") await expect(context).rejects.toThrow("task is no longer available"); + else expect(await context).toMatchObject({ completionContinuation: null }); + }); + + it.each(["missing", "pending", "unknown", "mixed_unknown", "mixed_missing_assessment", "other_run", "other_source", "ack_only", "partial_assessment", "new_head", "revoked"] as const)( + "does not derive an inherited author reply wait after %s", async gate => { + const fixture = await seedInheritedReviewCompletionTurn(false); + const reply = await seedCurrentAuthorReply(fixture); + if (gate === "missing") await db.delete(chatActions).where(eq(chatActions.id, reply.id)); + if (gate === "pending" || gate === "unknown") await db.update(chatActions) + .set({ status: gate === "pending" ? "received" : "failed", result: { deliveryUnknown: true } }) + .where(eq(chatActions.id, reply.id)); + if (gate === "mixed_unknown") await db.insert(chatActions).values({ ...reply, id: randomUUID(), + providerActionId: `github_publication:${randomUUID()}`, status: "failed", result: { deliveryUnknown: true } }); + if (gate === "mixed_missing_assessment") await db.insert(chatActions).values({ ...reply, id: randomUUID(), + providerActionId: `github_publication:${randomUUID()}`, payload: { ...reply.payload, + operation: "assessment", reviewId: fixture.review.id }, result: { published: 1, total: 2 } }); + if (gate === "other_run") await db.update(chatActions).set({ payload: { ...reply.payload, + session: { companyId: fixture.companyId, issueId: fixture.issueId, runId: fixture.predecessorId, agentId: fixture.agentId } } }) + .where(eq(chatActions.id, reply.id)); + if (gate === "other_source") { + const [source] = await db.select().from(chatDeliveries).where(eq(chatDeliveries.id, fixture.deliveryId)); + const [other] = await db.insert(chatDeliveries).values({ ...source!, id: randomUUID(), providerEventId: randomUUID(), deduplicationKey: randomUUID() }).returning(); + await db.update(chatActions).set({ deliveryId: other!.id }).where(eq(chatActions.id, reply.id)); + } + if (gate === "partial_assessment") { + const [partial] = await db.insert(chatGitHubReviews).values({ ...fixture.review, id: randomUUID(), + runId: fixture.runId, deliveryId: randomUUID(), createdAt: new Date(Date.now() - 86_400_000), publicationReceipts: { + partial: { id: "123", url: "https://github.com/paperclip/test-repository/pull/1#issuecomment-123", digest: "partial" }, + } }).returning(); + await db.update(chatActions).set({ status: "failed", payload: { ...reply.payload, + operation: "assessment", reviewId: partial!.id }, result: { code: "publication_failed" } }) + .where(eq(chatActions.id, reply.id)); + } + if (gate === "ack_only") await db.update(chatActions).set({ kind: "github_response_comment" }) + .where(eq(chatActions.id, reply.id)); + if (gate === "new_head") await db.insert(chatGitHubReviews).values({ ...fixture.review, id: randomUUID(), + runId: null, deliveryId: randomUUID(), headSha: "c".repeat(40), state: "queued", assessment: null, + createdAt: new Date(Date.now() + 1000) }); + if (gate === "revoked") await db.update(toolConnections).set({ enabled: false }) + .where(eq(toolConnections.companyId, fixture.companyId)); + const context = await new PaperclipRunnerToolAuthority(db, fixture) + .execute({ tool: "get_task_context", callId: randomUUID(), arguments: {} }); + expect(context).toMatchObject({ completionContinuation: null }); + }); + + it.each(["current", "resume", "reply"] as const)("projects only bound authority while deriving a published GitHub completion wait: %s", async turn => { + const fixture = turn === "current" ? await seedPublishedReviewCompletionTurn() : await seedInheritedReviewCompletionTurn(turn === "resume"); + const reply = turn === "reply" ? await seedCurrentAuthorReply(fixture as Awaited>) : null; + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + const payload = "unused-saved-review-rendering".repeat(32768); + await db.update(heartbeatRuns).set({ runnerProfileJson: { retained: payload }, stdoutExcerpt: payload, stderrExcerpt: payload, + contextSnapshot: { ...run.contextSnapshot, executionContinuation: { savedInput: payload }, + paperclipTaskMarkdown: payload, paperclipWake: { ...(run.contextSnapshot!.paperclipWake as object), retainedPayload: payload } }, + }).where(eq(heartbeatRuns.id, fixture.runId)); + if (reply) await db.update(chatActions).set({ result: { ...reply.result, savedProviderOutput: payload } }).where(eq(chatActions.id, reply.id)); + const observed = createDb(temporary?.connectionString ?? externalTestDatabaseUrl!); + const events: Array> = [], now = Date.now(); + const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { heartbeatRuns, chatActions }, capture); + try { + const context = await new PaperclipRunnerToolAuthority(observed, fixture) + .execute({ tool: "get_task_context", callId: randomUUID(), arguments: {} }); + expect(context).toMatchObject({ completionContinuation: { kind: "response_wake" } }); + capture.flush(); + const reads = events.filter(event => event.table === "heartbeat_runs" && event.operation === "select"); + expect(reads.length).toBeGreaterThan(0); + expect(reads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(16 * 1024); + const actionReads = events.filter(event => event.table === "chat_actions" && event.operation === "select"); + expect(actionReads.reduce((bytes, event) => bytes + Number(event.responseValueBytes), 0)).toBeLessThan(16 * 1024); + expect(JSON.stringify(context)).not.toContain(payload); + } finally { capture.flush(); await observed.$client.end(); } + }); + + it.each([undefined, null, "malformed-first"] as const)("derives an omitted published GitHub review wait through current task context: %s", async absent => { + const fixture = await seedPublishedReviewCompletionTurn(); + const authority = new PaperclipRunnerToolAuthority(db, fixture); + const read = () => authority.execute({ tool: "get_task_context", callId: randomUUID(), arguments: {} }); + const { resolveCompletionContinuation } = await import(new URL("../../../../packages/paperclip-runner/src/drivers/completion-continuation.ts", import.meta.url).href); + const report = { ...fixture.result, continuation: absent === "malformed-first" ? "response_wake" : absent }; + if (absent === "malformed-first") { + // Explicit malformed input remains invalid; the next omitted report can + // recover from authoritative state without changing its disposition. + expect(await resolveCompletionContinuation(report, fixture.runId, read)).toEqual(report); + expect(validatePrpStructuredRunResult(report).ok).toBe(false); + delete report.continuation; + } + const resolved = await resolveCompletionContinuation(report, fixture.runId, read) as PrpStructuredRunResult; + expect(resolved).toMatchObject({ reportedWorkDisposition: "yielded", continuation: { kind: "response_wake" } }); + await expect(nativeCompletionFeedback(db, fixture.runId, resolved)).resolves.toContain("accepted"); + await fixture.port.completeRun({ result: resolved, terminal: fixture.terminal, callerResultId: `wait-result-${fixture.runId}` }); + await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + expect((await db.select().from(issues).where(eq(issues.id, fixture.issueId)))[0]).toMatchObject({ status: "in_review" }); + expect((await db.select().from(statusDecisions).where(eq(statusDecisions.runId, fixture.runId)))[0]) + .toMatchObject({ reasonCode: "external_chat_response_waiting", decisionJson: { effects: [] } }); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, fixture.companyId))).toHaveLength(0); + }); + + it.each(["missing_publication", "pending_publication", "unknown_publication", "new_head", "other_run", "review_mismatch", "principal_revoked", "connection_revoked", "endpoint_revoked", "ownership_changed"] as const)( + "does not derive a published GitHub completion wait after %s", async gate => { + const fixture = await seedPublishedReviewCompletionTurn(); + if (gate === "missing_publication") await db.delete(chatActions).where(eq(chatActions.companyId, fixture.companyId)); + if (gate === "pending_publication" || gate === "unknown_publication") await db.update(chatActions) + .set({ status: gate === "pending_publication" ? "received" : "failed", result: { deliveryUnknown: true } }) + .where(eq(chatActions.companyId, fixture.companyId)); + if (gate === "new_head") await db.insert(chatGitHubReviews).values({ ...fixture.review, id: randomUUID(), + runId: null, deliveryId: randomUUID(), headSha: "c".repeat(40), state: "queued", assessment: null, + createdAt: new Date(Date.now() + 1000) }); + if (gate === "other_run") { + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + const previousId = randomUUID(); + await db.insert(heartbeatRuns).values({ ...run!, id: previousId, activeIdentityContextId: null, + status: "succeeded", finishedAt: new Date() }); + await db.update(chatGitHubReviews).set({ runId: previousId }).where(eq(chatGitHubReviews.id, fixture.review.id)); + } + if (gate === "review_mismatch") await db.update(chatGitHubReviews).set({ assessment: { + ...fixture.review.assessment!, reviewedCommit: "f".repeat(40), + } }).where(eq(chatGitHubReviews.id, fixture.review.id)); + if (gate === "principal_revoked") await db.update(companyMemberships).set({ status: "suspended" }) + .where(eq(companyMemberships.companyId, fixture.companyId)); + if (gate === "connection_revoked") await db.update(toolConnections).set({ enabled: false }) + .where(eq(toolConnections.companyId, fixture.companyId)); + if (gate === "endpoint_revoked") await db.update(chatEndpoints).set({ status: "revoked" }) + .where(eq(chatEndpoints.id, fixture.endpointId)); + if (gate === "ownership_changed") await db.update(issues).set({ executionRunId: null }) + .where(eq(issues.id, fixture.issueId)); + const authority = new PaperclipRunnerToolAuthority(db, fixture); + const { resolveCompletionContinuation } = await import(new URL("../../../../packages/paperclip-runner/src/drivers/completion-continuation.ts", import.meta.url).href); + await expect(resolveCompletionContinuation({ ...fixture.result, continuation: undefined }, fixture.runId, + () => authority.execute({ tool: "get_task_context", callId: randomUUID(), arguments: {} }))).rejects.toThrow(); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, fixture.companyId))).toHaveLength(0); + }, + ); + + it("serializes an actual connection disable against the final status committer proof", async () => { + const fixture = await seedPublishedReviewCompletionTurn(); + const authority = new PaperclipRunnerToolAuthority(db, fixture); + const { resolveCompletionContinuation } = await import(new URL("../../../../packages/paperclip-runner/src/drivers/completion-continuation.ts", import.meta.url).href); + const result = await resolveCompletionContinuation({ ...fixture.result, continuation: null }, fixture.runId, + () => authority.execute({ tool: "get_task_context", callId: randomUUID(), arguments: {} })) as PrpStructuredRunResult; + await nativeCompletionFeedback(db, fixture.runId, result); + await fixture.port.completeRun({ result, terminal: fixture.terminal, callerResultId: `wait-result-${fixture.runId}` }); + const [connection] = await db.select().from(toolConnections).where(eq(toolConnections.companyId, fixture.companyId)); + const applicationName = `review-wait-race-${fixture.runId}`; + const writerDb = createDb(externalTestDatabaseUrl ?? temporary!.connectionString, { applicationName }); + let writer: Promise | undefined; + let writerFinished = false, serialized = false; + const readProof = publishedReviewAuthority.readInheritedGitHubReviewWait; + const readForCommit = chatWaitAuthority.readPublishedGitHubReviewCompletionContinuationInTransaction; + const commitGuard = vi.spyOn(chatWaitAuthority, "readPublishedGitHubReviewCompletionContinuationInTransaction") + .mockImplementationOnce(async (...args) => { + const [committer] = await args[0].execute(sql`select pg_backend_pid() as pid`); + const proofGuard = vi.spyOn(publishedReviewAuthority, "readInheritedGitHubReviewWait") + .mockImplementationOnce(async (...proofArgs) => { + const proof = await readProof(...proofArgs); + // This is the same maintained writer used by PATCH tool-connections. + // It does not borrow the task's issue lock to disable a chat connection. + writer = toolAccessService(writerDb).updateConnection(connection!.id, { enabled: false }) + .finally(() => { writerFinished = true; }); + await vi.waitFor(async () => { + const waiting = await db.execute(sql`select query from pg_stat_activity + where application_name = ${applicationName} and wait_event_type = 'Lock'`); + expect(writerFinished || waiting.length > 0).toBe(true); + if (waiting.length) expect(String(waiting[0]!.query)).toContain('update "tool_connections"'); + serialized = !writerFinished && waiting.length > 0; + if (serialized) { + const blockers = await db.execute(sql`select pg_blocking_pids(pid) as blockers from pg_stat_activity where application_name = ${applicationName} and wait_event_type = 'Lock'`); + expect(blockers[0]!.blockers).toContain(Number(committer!.pid)); + } + }); + return proof; + }); + try { return await readForCommit(...args); } + finally { proofGuard.mockRestore(); } + }); + try { + await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + await writer; + expect(serialized).toBe(true); + expect((await db.select().from(issues).where(eq(issues.id, fixture.issueId)))[0]).toMatchObject({ status: "in_review" }); + expect((await db.select().from(toolConnections).where(eq(toolConnections.id, connection!.id)))[0]).toMatchObject({ enabled: false }); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, fixture.companyId))).toHaveLength(0); + } finally { commitGuard.mockRestore(); await Promise.allSettled(writer ? [writer] : []); await writerDb.$client.end({ timeout: 1 }); } + }); + + it.each((["current", "resume", "reply"] as const).flatMap(turn => + (["before_feedback", "before_finalization", "during_commit"] as const).map(phase => ({ turn, phase }))))( + "rechecks a projected GitHub completion wait $turn $phase", async ({ turn, phase }) => { + const fixture = turn === "current" ? await seedPublishedReviewCompletionTurn() : await seedInheritedReviewCompletionTurn(turn === "resume"); + const reply = turn === "reply" ? await seedCurrentAuthorReply(fixture as Awaited>) : null; + const authority = new PaperclipRunnerToolAuthority(db, fixture); + const { resolveCompletionContinuation } = await import(new URL("../../../../packages/paperclip-runner/src/drivers/completion-continuation.ts", import.meta.url).href); + const result = await resolveCompletionContinuation({ ...fixture.result, continuation: null }, fixture.runId, + () => authority.execute({ tool: "get_task_context", callId: randomUUID(), arguments: {} })) as PrpStructuredRunResult; + const revoke = () => reply + ? db.update(chatActions).set({ status: "failed", result: { deliveryUnknown: true } }).where(eq(chatActions.id, reply.id)) + : turn === "resume" ? db.update(chatActions).set({ status: "received" }).where(eq(chatActions.companyId, fixture.companyId)) + : db.update(toolConnections).set({ enabled: false }).where(eq(toolConnections.companyId, fixture.companyId)); + if (phase === "before_feedback") { + await revoke(); + await expect(nativeCompletionFeedback(db, fixture.runId, result)).rejects.toThrow("no longer authorized"); + return; + } + await nativeCompletionFeedback(db, fixture.runId, result); + await fixture.port.completeRun({ result, terminal: fixture.terminal, callerResultId: `wait-result-${fixture.runId}` }); + let guard: ReturnType | undefined; + if (phase === "before_finalization") await revoke(); + if (phase === "during_commit") { + const read = chatWaitAuthority.readPublishedGitHubReviewCompletionContinuationInTransaction; + guard = vi.spyOn(chatWaitAuthority, "readPublishedGitHubReviewCompletionContinuationInTransaction") + .mockImplementation(async (...args) => { + if (reply) await args[0].update(chatActions).set({ status: "failed", result: { deliveryUnknown: true } }).where(eq(chatActions.id, reply.id)); + else if (turn === "resume") await args[0].update(chatActions).set({ status: "received" }).where(eq(chatActions.companyId, fixture.companyId)); + else await args[0].update(toolConnections).set({ enabled: false }).where(eq(toolConnections.companyId, fixture.companyId)); + return read(...args); + }); + } + try { + const finalized = await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + expect((await db.select().from(issues).where(eq(issues.id, fixture.issueId)))[0]).toMatchObject({ status: "in_progress" }); + const decisions = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, fixture.runId)); + if (phase === "during_commit") { + expect(finalized).toMatchObject({ phase: "retryable_failure" }); + expect(decisions).toHaveLength(0); + } else expect(decisions.at(-1)).toMatchObject({ reasonCode: "external_chat_response_wait_authorization_lost" }); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, fixture.companyId))).toHaveLength(0); + } finally { guard?.mockRestore(); } + }, + ); + it("admits a real repository-access request to Mine until it is resolved", async () => { const fixture = await seedWaitTurn("github", [{ kind: "approval", ownerClass: "human", summary: "Grant repository access so I can read the diff." }], { reportedWorkDisposition: "needs_review", continuation: undefined, diff --git a/server/src/services/native-runtime/external-review-wait.ts b/server/src/services/native-runtime/external-review-wait.ts index 4a63ad6510d..b59aed390c3 100644 --- a/server/src/services/native-runtime/external-review-wait.ts +++ b/server/src/services/native-runtime/external-review-wait.ts @@ -1,5 +1,6 @@ import { and, eq, inArray, sql } from "drizzle-orm"; import { chatConversations, chatEndpoints, heartbeatRuns, issues, statusDecisions, type Db } from "@paperclipai/db"; +import { readInheritedGitHubReviewWait } from "../chat-channels/github/review-wait.js"; import { authorizeChatConversationForBoundRun } from "./chat-attachment-reuse.js"; import { EXTERNAL_CHAT_QUESTION_RESPONSE_KEY } from "./external-chat-question-response.js"; @@ -30,6 +31,8 @@ export async function listExternalReviewWaits(db: Db, companyId: string, issueId runId: heartbeatRuns.id, agentId: heartbeatRuns.agentId, contextSnapshot: authorizationContext, + responsibleUserId: heartbeatRuns.responsibleUserId, + activeIdentityContextId: heartbeatRuns.activeIdentityContextId, }).from(issues) .innerJoin(statusDecisions, and( eq(statusDecisions.id, issues.lastStatusDecisionId), @@ -58,9 +61,14 @@ export async function listExternalReviewWaits(db: Db, companyId: string, issueId eq(chatEndpoints.status, "active"), )) .where(and(eq(issues.companyId, companyId), inArray(issues.id, issueIds), eq(issues.status, "in_review"))); - const waits = await Promise.all(candidates.map(async ({ runId, agentId, contextSnapshot, ...wait }) => { + const waits = await Promise.all(candidates.map(async ({ runId, agentId, contextSnapshot, responsibleUserId, activeIdentityContextId, ...wait }) => { try { - await authorizeChatConversationForBoundRun(db, { companyId, issueId: wait.issueId, runId, agentId }, contextSnapshot, "read"); + const inheritedWait = await readInheritedGitHubReviewWait(db, { + id: runId, companyId, agentId, responsibleUserId, activeIdentityContextId, + nativeIssueId: wait.issueId, issueId: wait.issueId, contextSnapshot, + }); + if (inheritedWait?.kind === "revoked") return null; + await authorizeChatConversationForBoundRun(db, { companyId, issueId: wait.issueId, runId, agentId }, inheritedWait?.context ?? contextSnapshot, "read"); return wait; } catch (error) { if (error instanceof Error && ["paperclip_runner_chat_attachment_binding_denied", diff --git a/server/src/services/native-runtime/native-completion-feedback.ts b/server/src/services/native-runtime/native-completion-feedback.ts index 407201931f3..9bb94b39c72 100644 --- a/server/src/services/native-runtime/native-completion-feedback.ts +++ b/server/src/services/native-runtime/native-completion-feedback.ts @@ -9,7 +9,7 @@ import { evaluateAgentInvokabilityFromDb } from "../agent-invokability.js"; import { issueService } from "../issues.js"; import { isConversation } from "../agent-conversations.js"; import { issueTreeControlService } from "../issue-tree-control.js"; -import { resolveExternalChatResponseWaitAuthorization } from "./chat-attachment-reuse.js"; +import { isPublishedGitHubReviewCompletionContinuation, readPublishedGitHubReviewCompletionContinuation, resolveExternalChatResponseWaitAuthorization } from "./chat-attachment-reuse.js"; import { getNativeReviewAssignment, readNativeReviewAssignmentContext } from "./native-review-participant.js"; import { and, eq, inArray, notInArray } from "drizzle-orm"; import { @@ -68,6 +68,12 @@ export async function nativeCompletionFeedback( ) .then((rows) => rows[0]); if (!issue) throw new Error("Completion task no longer exists."); + if (result.reportedWorkDisposition === "yielded" && isPublishedGitHubReviewCompletionContinuation(result.continuation)) { + const current = await readPublishedGitHubReviewCompletionContinuation({ db, + binding: { companyId: run.companyId, issueId: issue.id, runId, agentId: run.agentId } }); + if (!current || current.idempotencyKey !== result.continuation!.idempotencyKey) + throw new Error("The published GitHub review wait is no longer authorized for this run. Reassess current task state without repeating the published review."); + } const acceptedHandoff = await assertNativeCaseHandoffResult(db, runId, issue.id, result); if (acceptedHandoff) return "The persisted case handoff controls this turn. End the turn; the task remains open for its next agent."; const reviewContext = readNativeReviewAssignmentContext(run.contextSnapshot); diff --git a/server/src/services/native-runtime/native-run-finalizer.ts b/server/src/services/native-runtime/native-run-finalizer.ts index 5931662d7db..4ade80add27 100644 --- a/server/src/services/native-runtime/native-run-finalizer.ts +++ b/server/src/services/native-runtime/native-run-finalizer.ts @@ -3,7 +3,7 @@ import { notifyExecutionRunWork } from "../execution-control-notifications.js"; import { reconcileMuseNativeTerminalRun } from "./muse-terminal-reconciliation.js"; import { assertNativeCaseHandoffResult } from "./native-case-handoff.js"; import { scheduledIssueMonitorWait } from "../issue-monitors.js"; -import { hasPublishedGitHubReview } from "../chat-channels/github/review-wait.js"; +import { hasPublishedGitHubReview, readInheritedGitHubReviewWait } from "../chat-channels/github/review-wait.js"; import { readNativeMonitorDelivery, readNativePreservedMonitorWait } from "./native-monitor-delivery.js"; import { isNativePlanWaitResult, readNativePlanWait } from "./native-plan-wait.js"; import { hasPendingNativeChildCompletion } from "./native-child-completion-delivery.js"; @@ -57,7 +57,7 @@ import { } from "./native-board-response-wait.js"; import { emitAgentTaskRun } from "../agent-task-run-telemetry.js"; import { isNewTerminalRunFailure, reportRunFailure } from "../run-failure-report.js"; -import { resolveExternalChatResponseWaitAuthorization } from "./chat-attachment-reuse.js"; +import { isPublishedGitHubReviewCompletionContinuation, readPublishedGitHubReviewCompletionContinuation, resolveExternalChatResponseWaitAuthorization } from "./chat-attachment-reuse.js"; import { authorizeNativeChatReviewPresentation, hasMaterializedNativeReviewResponse, @@ -1317,10 +1317,17 @@ export async function finalizeNativeRun(input: { .then((rows) => rows[0] ?? null); if (!authoritativeIssue) throw new Error("native_finalization_issue_missing"); + const publishedReviewContinuation = assessment.reportedDisposition === "yielded" && + isPublishedGitHubReviewCompletionContinuation(assessment.continuation) + ? await readPublishedGitHubReviewCompletionContinuation({ db: input.db, + binding: { companyId: run.companyId, issueId: authoritativeIssue.id, runId: run.id, agentId: run.agentId } }) + : null; const externalChatResponseWaitAuthorization = assessment.reportedDisposition === "yielded" && assessment.continuation?.kind === "response_wake" - ? await resolveExternalChatResponseWaitAuthorization({ + ? isPublishedGitHubReviewCompletionContinuation(assessment.continuation) + ? publishedReviewContinuation?.idempotencyKey === assessment.continuation.idempotencyKey ? "authorized" : "revoked" + : await resolveExternalChatResponseWaitAuthorization({ db: input.db, binding: { companyId: run.companyId, @@ -1452,7 +1459,7 @@ export async function finalizeNativeRun(input: { (externalChatResponseWaitAuthorization === "authorized" || monitorWaitAt !== null) ? await hasPublishedGitHubReview(input.db, { companyId: run.companyId, issueId: authoritativeIssue.id, runId: run.id, - }) : false, + }) || (await readInheritedGitHubReviewWait(input.db, run))?.kind === "published" : false, boardResponseWaitAuthorized: boardResponseWait !== null, boardResponseWaitOrigin: boardResponseWaitOrigin !== null, isConversation: isConversation(authoritativeIssue), @@ -1554,7 +1561,7 @@ export async function finalizeNativeRun(input: { : undefined, requireExternalChatResponseWaitAuthorization: decision.reasonCode === "external_chat_response_waiting" - ? { agentId: run.agentId } + ? { agentId: run.agentId, ...(publishedReviewContinuation ? { completionContinuationKey: publishedReviewContinuation.idempotencyKey } : {}) } : undefined, reviewResponsePresentation: decision.reasonCode === "governed_response_waiting" && diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts index 31f65f12fcd..cc535813f77 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts @@ -103,6 +103,7 @@ import { type CurrentWakeCommentsBinding, } from "./current-wake-comments.js"; import { + readPublishedGitHubReviewCompletionContinuation, authorizeChatAttachmentReuse, authorizeChatConversationForBoundRun, LIST_CHAT_ATTACHMENTS_TOOL_DEFINITION, @@ -687,6 +688,7 @@ export class PaperclipRunnerToolAuthority { } : !this.binding.nativeReview && context.issue.workMode === "standard" && context.issue.executionRunId === this.binding.runId ? await pendingTaskRepositoryContinuation(this.db, { companyId: this.binding.companyId, issueId: this.binding.issueId, runId: this.binding.runId, workspaceId: context.issue.executionWorkspaceId }) + ?? await readPublishedGitHubReviewCompletionContinuation({ db: this.db, binding: this.binding }) : null, childTasks: visibleChildTasks.slice(0, 100).map(redactedTask), childTasksTruncated: childTasks.length > 100, diff --git a/server/src/services/native-runtime/status-decision-committer.ts b/server/src/services/native-runtime/status-decision-committer.ts index c7684448cbf..0e34904c5b3 100644 --- a/server/src/services/native-runtime/status-decision-committer.ts +++ b/server/src/services/native-runtime/status-decision-committer.ts @@ -1,5 +1,5 @@ import { scheduledIssueMonitorWait } from "../issue-monitors.js"; -import { hasPublishedGitHubReview } from "../chat-channels/github/review-wait.js"; +import { hasPublishedGitHubReview, readInheritedGitHubReviewWait } from "../chat-channels/github/review-wait.js"; import { pendingNativeGovernance } from "./native-governance.js"; import { findSupersededExternalReviewRequests, supersedeExternalReviewRequestsInTransaction } from "./superseded-external-review-requests.js"; import { normalizePrpResultSignals, type PrpStructuredRunResult } from "../../vendor/paperclip-runner/index.js"; @@ -46,6 +46,7 @@ import { restoreNativeChatReviewPresentationInTransaction, } from "./native-chat-review-presentation.js"; import { + readPublishedGitHubReviewCompletionContinuationInTransaction, isExternalChatWaitAuthorizationContention, resolveExternalChatResponseWaitAuthorizationInTransaction, } from "./chat-attachment-reuse.js"; @@ -1561,7 +1562,7 @@ export async function commitNativeStatusDecision(input: { preMaterializedEffects?: NativeMaterializedStatusEffect[]; supersedesCommittedDecisionId?: string; supersededExternalReviewRequests?: { agentId: string; resultId: string; interactionIds: string[] }; - requireExternalChatResponseWaitAuthorization?: { agentId: string }; + requireExternalChatResponseWaitAuthorization?: { agentId: string; completionContinuationKey?: string }; requirePlanWaitSource?: NativePlanWaitSource; requireMonitorWait?: { agentId: string; nextCheckAt: string }; requireMonitorDelivery?: NativeMonitorDelivery; @@ -1658,8 +1659,23 @@ export async function commitNativeStatusDecision(input: { } if (reasonCode === "scheduled_monitor_waiting") { const expected = input.requireMonitorWait; - const publishedReviewWait = input.decision.statusAction === "in_review" && input.decision.toStatus === "in_review" && - await hasPublishedGitHubReview(tx as unknown as Db, input); + const inReview = input.decision.statusAction === "in_review" && input.decision.toStatus === "in_review"; + let publishedReviewWait = inReview && await hasPublishedGitHubReview(tx as unknown as Db, input); + if (expected && inReview && !publishedReviewWait) { + // Arbitration may reuse a previous publication. Prove that same source + // here, while retaining the independently saved monitor and its owner. + const [run] = await tx.select({ + id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + responsibleUserId: heartbeatRuns.responsibleUserId, + activeIdentityContextId: heartbeatRuns.activeIdentityContextId, + nativeIssueId: heartbeatRuns.nativeIssueId, issueId: heartbeatRuns.issueId, + }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId), + eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.agentId, expected.agentId), + )).limit(1); + publishedReviewWait = !!run && (await readInheritedGitHubReviewWait(tx as unknown as Db, + { ...run, contextSnapshot: null }))?.kind === "published"; + } if (!expected || issue.workMode !== "standard" || issue.executionRunId !== input.runId || (input.decision.statusAction !== "preserve" && !publishedReviewWait) || input.decision.effects.length !== 1 || input.decision.effects[0]?.kind !== "release_checkout" || @@ -1731,6 +1747,13 @@ export async function commitNativeStatusDecision(input: { if (authorization !== "authorized") { throw new NativeStatusRaceError(); } + if (input.requireExternalChatResponseWaitAuthorization.completionContinuationKey) { + const current = await readPublishedGitHubReviewCompletionContinuationInTransaction(tx as unknown as Db, + { companyId: input.companyId, issueId: input.issueId, runId: input.runId, + agentId: input.requireExternalChatResponseWaitAuthorization.agentId }, "nonblocking"); + if (current?.idempotencyKey !== input.requireExternalChatResponseWaitAuthorization.completionContinuationKey) + throw new NativeStatusRaceError(); + } } const passiveBoardResponseWait = reasonCode === "board_response_waiting" || diff --git a/server/src/services/run-identity.ts b/server/src/services/run-identity.ts index 2f1e9e97d65..166b1ae8cac 100644 --- a/server/src/services/run-identity.ts +++ b/server/src/services/run-identity.ts @@ -1,3 +1,4 @@ +import { readTaskRetryIntent, readTaskRetryBoardMessageOrigin } from "./task-retry.js"; import { and, asc, desc, eq, getTableColumns, inArray, sql } from "drizzle-orm"; import { agentWakeupRequests, @@ -201,6 +202,39 @@ export async function initializeRunIdentity( return current!; } const operatorIdentity = await explicitOperatorRunIdentity(tx, run); + let retryOrigin: string | null = null; + const retryId = run.contextSnapshot?.taskRetryRequestId; + if (typeof retryId === "string") { + const [receipt] = await tx.select().from(agentWakeupRequests).where(and( + eq(agentWakeupRequests.id, retryId), eq(agentWakeupRequests.companyId, input.companyId), + eq(agentWakeupRequests.agentId, run.agentId))); + const retry = readTaskRetryIntent(receipt?.payload); + if (run.contextSnapshot?.boardMessageRetry === true || retry?.boardMessageOrigin) { + if (!retry?.boardMessageOrigin) throw forbidden("This saved message can no longer be retried safely. Send a new message on the task to continue from saved work."); + const [dispatch] = run.wakeupRequestId ? await tx.select().from(agentWakeupRequests).where(and( + eq(agentWakeupRequests.id, run.wakeupRequestId), eq(agentWakeupRequests.companyId, input.companyId), + eq(agentWakeupRequests.agentId, run.agentId), eq(agentWakeupRequests.runId, run.id), + eq(agentWakeupRequests.status, "claimed"), eq(agentWakeupRequests.reason, "task_retry"), + eq(agentWakeupRequests.idempotencyKey, `task-retry-dispatch:${receipt!.id}`), + eq(agentWakeupRequests.requestedByActorType, "user"), eq(agentWakeupRequests.requestedByActorId, retry.actorId))) : []; + const [task] = input.issueId ? await tx.select({ id: issues.id }).from(issues).where(and( + eq(issues.id, input.issueId), eq(issues.companyId, input.companyId), eq(issues.assigneeAgentId, run.agentId), + eq(issues.executionRunId, run.id))) : []; + const origin = input.issueId && run.status === "running" && task && dispatch && + [run.issueId, run.nativeIssueId, run.contextSnapshot?.issueId, run.contextSnapshot?.taskId] + .every(id => id === null || id === undefined || id === input.issueId) && receipt!.runId === run.id && receipt!.status === "coalesced" && + receipt!.reason === "task_retry" && receipt!.requestedByActorType === "user" && + run.contextSnapshot?.previousRunId === retry.sourceRunId && retry.generationId === receipt!.id && !retry.chatActionId && !retry.canonicalIntentId && + receipt!.payload?.issueId === input.issueId && retry.actorId === receipt!.requestedByActorId && + await readTaskRetryBoardMessageOrigin(tx as unknown as Db, { companyId: input.companyId, issueId: input.issueId, + agentId: run.agentId, sourceRunId: retry.sourceRunId, actorId: retry.actorId }); + if (!origin || origin.commentId !== retry.boardMessageOrigin.commentId || + origin.identityContextId !== retry.boardMessageOrigin.identityContextId) { + throw forbidden("This saved message can no longer be retried safely. Send a new message on the task to continue from saved work."); + } + retryOrigin = origin.identityContextId; + } + } const [parent] = input.parentRunId ? await tx .select() @@ -226,7 +260,7 @@ export async function initializeRunIdentity( ) : []; const parentId = operatorIdentity ? null : ( - interaction?.sourceIdentityContextId ?? input.parentContextId ?? parent?.activeIdentityContextId + retryOrigin ?? interaction?.sourceIdentityContextId ?? input.parentContextId ?? parent?.activeIdentityContextId ); const [origin] = parentId ? await tx diff --git a/server/src/services/task-retry.ts b/server/src/services/task-retry.ts index f1ec119386f..1d56d6d49b3 100644 --- a/server/src/services/task-retry.ts +++ b/server/src/services/task-retry.ts @@ -3,7 +3,7 @@ import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; import { z } from "zod"; import { randomUUID } from "node:crypto"; import { and, desc, eq, inArray, or, sql } from "drizzle-orm"; -import { agentWakeupRequests, chatActions, chatConversations, chatEndpoints, heartbeatRuns, issues, issueRecoveryActions, type Db } from "@paperclipai/db"; +import { agentWakeupRequests, chatActions, chatConversations, chatEndpoints, heartbeatRuns, issues, issueComments, runIdentityContexts, issueRecoveryActions, type Db } from "@paperclipai/db"; import { conflict, forbidden, notFound } from "../errors.js"; import { persistActivity } from "./activity-log.js"; import { createSystemRecovery, readSystemRecovery } from "./system-recovery-state.js"; @@ -13,6 +13,7 @@ import { notifyChatActionWork } from "./chat-work-notifications.js"; import { authorizationService } from "./authorization.js"; import { preserveNativeFailedWorkspace } from "./native-runtime/native-failed-workspace-recovery.js"; import { NativeWorkspaceFinalizationBusyError } from "./native-runtime/native-workspace-finalization-ownership.js"; +import { adapterExecutionControls } from "./adapter-execution-control.js"; import { hasRequiredWorkspaceRecovery } from "./workspace-restore-recovery-state.js"; export async function taskRetryMutationAllowed(db: Db, input: { @@ -42,6 +43,7 @@ const taskRetryIntentSchema = z.object({ schema: z.literal("paperclip.task-retry.v1"), generationId: z.string().uuid(), sourceRunId: z.string().uuid(), actorId: z.string().min(1), actorSource: z.enum(["local_implicit", "session", "board_key", "cloud_tenant"]).default("session"), strategy: z.enum(["evaluate", "continue", "fresh_isolated_workspace", "finalize_result"]), chatActionId: z.string().uuid().optional(), canonicalIntentId: z.string().uuid().optional(), + boardMessageOrigin: z.object({ commentId: z.string().uuid(), identityContextId: z.string().uuid() }).optional(), }); export type TaskRetryIntent = z.infer; export function readTaskRetryIntent(payload: Record | null | undefined): TaskRetryIntent | null { @@ -49,6 +51,126 @@ export function readTaskRetryIntent(payload: Record | null | un return result.success ? result.data : null; } +/** A board message has its own accepted user identity, not an external inbound + * receipt. Certify its exact unchanged source and retained pre-provider evidence; + * this selects the canonical Retry owner and never substitutes chat authority. + * Normal queue containment and current permissions still govern dispatch. */ +export async function readTaskRetryBoardMessageOrigin(db: Db, input: { + companyId: string; issueId: string; agentId: string; sourceRunId: string; actorId: string; +}) { + return readBoardMessageRetrySource(db, input, new Set()); +} + +async function readBoardMessageRetrySource(db: Db, input: { + companyId: string; issueId: string; agentId: string; sourceRunId: string; actorId: string; +}, seen: Set): Promise<{ commentId: string; identityContextId: string } | null> { + if (seen.size >= 20 || seen.has(input.sourceRunId) || adapterExecutionControls.has(input.sourceRunId)) return null; + seen.add(input.sourceRunId); + const [task] = await db.select({ id: issues.id }).from(issues).where(and( + eq(issues.id, input.issueId), eq(issues.companyId, input.companyId), eq(issues.assigneeAgentId, input.agentId), + sql`${issues.status} not in ('done', 'cancelled')`)); + if (!task) return null; + const sourceConditions = [ + eq(heartbeatRuns.id, input.sourceRunId), eq(heartbeatRuns.companyId, input.companyId), + eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.responsibleUserId, input.actorId), + eq(heartbeatRuns.status, "failed"), eq(heartbeatRuns.runtimeMode, "legacy"), eq(heartbeatRuns.errorCode, "setup_failed"), + sql`${heartbeatRuns.finishedAt} is not null`, + sql`(${heartbeatRuns.issueId} is null or ${heartbeatRuns.issueId} = ${input.issueId}::uuid)`, + sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${input.issueId}`, + sql`(${heartbeatRuns.contextSnapshot}->>'taskId' is null or ${heartbeatRuns.contextSnapshot}->>'taskId' = ${input.issueId})`, + sql`${heartbeatRuns.resultJson}#>>'{executionRecovery,kind}' = 'bootstrap'`, + sql`${heartbeatRuns.resultJson}#>'{executionRecovery,providerWorkStarted}' = 'false'::jsonb`, + sql`${heartbeatRuns.resultJson}->>'stopReason' = 'adapter_failed'`, + sql`not exists (select 1 from jsonb_object_keys(case when jsonb_typeof(${heartbeatRuns.resultJson}) = 'object' + then ${heartbeatRuns.resultJson} else '{}'::jsonb end) key + where key not in ('effectiveTimeoutSec', 'executionRecovery', 'stopReason', 'timeoutConfigured', 'timeoutFired', 'timeoutSource'))`, + sql`${heartbeatRuns.runnerProfileJson}->'adapterDispatch' = '{"adapterType":"paperclip_runner"}'::jsonb`, + sql`not exists (select 1 from jsonb_object_keys(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' + then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) key + where key not in ('adapterDispatch', 'aiConnectionNonAssigneeCommentWake', 'chatControlRecoveryAdmission'))`, + ...[heartbeatRuns.nativeIssueId, heartbeatRuns.nativeSessionId, heartbeatRuns.nativePhase, + heartbeatRuns.runtimeModeResolvedAt, heartbeatRuns.runtimeModeResolverVersion, heartbeatRuns.runtimeModeReason, heartbeatRuns.driverKind, heartbeatRuns.driverVersion, + heartbeatRuns.completionContractId, heartbeatRuns.runnerInstanceId, heartbeatRuns.sessionIdAfter, + heartbeatRuns.externalRunId, heartbeatRuns.processPid, heartbeatRuns.processGroupId, heartbeatRuns.processStartedAt, + heartbeatRuns.logStore, heartbeatRuns.logRef, heartbeatRuns.logBytes, heartbeatRuns.logSha256, heartbeatRuns.stdoutExcerpt, + heartbeatRuns.stderrExcerpt, heartbeatRuns.lastOutputAt, heartbeatRuns.lastOutputBytes, + heartbeatRuns.lastOutputStream, heartbeatRuns.exitCode, heartbeatRuns.signal].map(column => sql`${column} is null`), + sql`${heartbeatRuns.lastOutputSeq} = 0`, sql`not ${heartbeatRuns.logCompressed}`, + sql`(${heartbeatRuns.usageJson} is null or ${heartbeatRuns.usageJson} - 'ledgerScope' = '{}'::jsonb)`, + sql`(${heartbeatRuns.controllerLeaseExpiresAt} is null or ${heartbeatRuns.controllerLeaseExpiresAt} <= now())`, + sql`not exists (select 1 from native_run_results where company_id = ${input.companyId}::uuid and run_id = ${input.sourceRunId}::uuid)`, + sql`not exists (select 1 from native_run_finalizations where company_id = ${input.companyId}::uuid and run_id = ${input.sourceRunId}::uuid)`, + sql`not exists (select 1 from environment_leases where company_id = ${input.companyId}::uuid and heartbeat_run_id = ${input.sourceRunId}::uuid)`, + sql`exists (select 1 from heartbeat_run_events where company_id = ${input.companyId}::uuid and run_id = ${input.sourceRunId}::uuid + and event_type = 'error' and stream = 'system' and source_event_id is null and message = ${heartbeatRuns.error})`, + sql`exists (select 1 from heartbeat_run_events where company_id = ${input.companyId}::uuid and run_id = ${input.sourceRunId}::uuid + and event_type = 'lifecycle' and stream = 'system' and source_event_id is null and message = 'task.run.measured' + and payload->>'durableSettlement' = 'true' and payload->>'outcome' = 'failed')`, + sql`not exists (select 1 from heartbeat_run_events where company_id = ${input.companyId}::uuid and run_id = ${input.sourceRunId}::uuid + and (stream <> 'system' or source_event_id is not null or source_instance_id is not null or source_seq is not null + or source_payload_sha256 is not null or protocol_schema_version is not null or event_type not in ('error', 'agent_observability.unavailable', 'lifecycle') + or (event_type = 'lifecycle' and (message <> 'task.run.measured' or payload->>'durableSettlement' is distinct from 'true')) + or (event_type = 'error' and message is distinct from ${heartbeatRuns.error})))`, + ]; + const [origin] = await db.select({ commentId: issueComments.id, identityContextId: runIdentityContexts.id }) + .from(heartbeatRuns) + .innerJoin(runIdentityContexts, and(eq(runIdentityContexts.id, heartbeatRuns.activeIdentityContextId), + eq(runIdentityContexts.companyId, input.companyId), eq(runIdentityContexts.runId, heartbeatRuns.id), + eq(runIdentityContexts.status, "accepted"), eq(runIdentityContexts.responsibleUserId, input.actorId))) + .innerJoin(issueComments, and(eq(issueComments.id, runIdentityContexts.messageId), + eq(issueComments.companyId, input.companyId), eq(issueComments.issueId, input.issueId), + eq(issueComments.authorType, "user"), eq(issueComments.authorUserId, input.actorId), + sql`${issueComments.createdByRunId} is null`, sql`${issueComments.deletedAt} is null`, + eq(issueComments.createdAt, issueComments.updatedAt), sql`nullif(trim(${issueComments.body}), '') is not null`)) + .innerJoin(agentWakeupRequests, and(eq(agentWakeupRequests.id, heartbeatRuns.wakeupRequestId), + eq(agentWakeupRequests.companyId, input.companyId), eq(agentWakeupRequests.agentId, input.agentId), + eq(agentWakeupRequests.runId, heartbeatRuns.id), eq(agentWakeupRequests.requestedByActorType, "user"), + eq(agentWakeupRequests.requestedByActorId, input.actorId), eq(agentWakeupRequests.reason, "issue_reopened_via_comment"), eq(agentWakeupRequests.status, "failed"), + sql`${agentWakeupRequests.payload}->>'issueId' = ${input.issueId}`, + sql`${agentWakeupRequests.payload}->>'commentId' = ${issueComments.id}::text`)) + .where(and(...sourceConditions, + sql`${heartbeatRuns.contextSnapshot}->>'source' = 'issue.comment.reopen'`, + sql`${heartbeatRuns.contextSnapshot}->>'wakeCommentId' = ${issueComments.id}::text`, + sql`${heartbeatRuns.contextSnapshot}->'wakeCommentIds' = jsonb_build_array(${issueComments.id}::text)`, + )).limit(1); + if (origin) return origin; + // A repeated preparation failure must retain the same certified board + // source through its real coalesced intent and failed dispatch receipt. + // Every predecessor is checked again; cycles and long damaged chains fail closed. + const [successor] = await db.select({ identityContextId: heartbeatRuns.activeIdentityContextId, + retryId: sql`${heartbeatRuns.contextSnapshot}->'taskRetryRequestId'`, + previousRunId: sql`${heartbeatRuns.contextSnapshot}->'previousRunId'`, + dispatchKey: agentWakeupRequests.idempotencyKey }).from(heartbeatRuns) + .innerJoin(agentWakeupRequests, and(eq(agentWakeupRequests.id, heartbeatRuns.wakeupRequestId), + eq(agentWakeupRequests.companyId, input.companyId), eq(agentWakeupRequests.agentId, input.agentId), + eq(agentWakeupRequests.runId, heartbeatRuns.id), eq(agentWakeupRequests.status, "failed"), + eq(agentWakeupRequests.reason, "task_retry"), eq(agentWakeupRequests.requestedByActorType, "user"), + eq(agentWakeupRequests.requestedByActorId, input.actorId))) + .where(and(...sourceConditions, sql`${heartbeatRuns.contextSnapshot}->'boardMessageRetry' = 'true'::jsonb`)).limit(1); + if (!successor?.identityContextId || !z.string().uuid().safeParse(successor.retryId).success || + successor.dispatchKey !== `task-retry-dispatch:${successor.retryId}`) return null; + const [receipt] = await db.select().from(agentWakeupRequests).where(and( + eq(agentWakeupRequests.id, String(successor.retryId)), eq(agentWakeupRequests.companyId, input.companyId), + eq(agentWakeupRequests.agentId, input.agentId), eq(agentWakeupRequests.runId, input.sourceRunId), + eq(agentWakeupRequests.status, "coalesced"), eq(agentWakeupRequests.reason, "task_retry"), + eq(agentWakeupRequests.requestedByActorType, "user"), eq(agentWakeupRequests.requestedByActorId, input.actorId), + sql`${agentWakeupRequests.payload}->>'issueId' = ${input.issueId}`)); + const intent = readTaskRetryIntent(receipt?.payload); + if (!intent?.boardMessageOrigin || intent.generationId !== receipt!.id || intent.actorId !== input.actorId || + intent.chatActionId || intent.canonicalIntentId || intent.sourceRunId !== successor.previousRunId) return null; + const [identity] = await db.select({ id: runIdentityContexts.id }).from(runIdentityContexts).where(and( + eq(runIdentityContexts.id, successor.identityContextId), eq(runIdentityContexts.companyId, input.companyId), + eq(runIdentityContexts.runId, input.sourceRunId), eq(runIdentityContexts.responsibleUserId, input.actorId), + eq(runIdentityContexts.status, "accepted"), eq(runIdentityContexts.cause, "task_retry"), + eq(runIdentityContexts.parentContextId, intent.boardMessageOrigin.identityContextId), + sql`${runIdentityContexts.messageId} is null`)).limit(1); + if (!identity) return null; + const parent = await readBoardMessageRetrySource(db, { ...input, sourceRunId: intent.sourceRunId }, seen); + return parent && parent.commentId === intent.boardMessageOrigin.commentId && + parent.identityContextId === intent.boardMessageOrigin.identityContextId + ? { commentId: parent.commentId, identityContextId: identity.id } : null; +} + /** Caller holds the issue dispatch lock. Newer work supersedes an unadmitted * Retry; retiring its authority does not acknowledge or discard saved input. */ export async function retireSupersededTaskRetry(db: Db, input: { @@ -155,13 +277,22 @@ export async function requestTaskRetry(input: { .innerJoin(chatEndpoints, eq(chatEndpoints.id, chatConversations.endpointId)).where(and( eq(chatConversations.companyId, issue.companyId), eq(chatConversations.issueId, issue.id), eq(chatEndpoints.externalExecutionPolicy, "restricted"))).limit(1); + const boardMessageOrigin = chat && !active && !busy + ? await readTaskRetryBoardMessageOrigin(db, { companyId: issue.companyId, + issueId: issue.id, agentId: issue.assigneeAgentId, sourceRunId: latest.id, actorId: input.actorId }) : null; + if (boardMessageOrigin) intent.boardMessageOrigin = boardMessageOrigin; + if (chat && !active && !busy && !boardMessageOrigin && (latest.contextSnapshot?.source === "issue.comment.reopen" || latest.contextSnapshot?.boardMessageRetry === true) && + latest.responsibleUserId === input.actorId) { + throw conflict("This saved message cannot be retried safely. Send a new message on the task to continue from saved work.", + { code: "chat_failed_run_retry_not_authorized", reason: "saved_board_message_not_replayable" }); + } await tx.insert(agentWakeupRequests).values({ id, companyId: issue.companyId, agentId: issue.assigneeAgentId, source: "on_demand", triggerDetail: "manual", reason: "task_retry", idempotencyKey: key, requestedByActorType: "user", requestedByActorId: input.actorId, status: active || busy ? "coalesced" : "deferred_issue_execution", runId: active?.runId ?? (busy ? latest.id : null), payload: { issueId: issue.id, taskRetry: intent }, finishedAt: active || busy ? new Date() : null }); - if (chat && !active && !busy) { + if (chat && !boardMessageOrigin && !active && !busy) { if (!input.prepareChatRetry) throw conflict("Retry is unavailable for this connected conversation."); intent.chatActionId = (await input.prepareChatRetry(db, { companyId: issue.companyId, issueId: issue.id, agentId: issue.assigneeAgentId, failedRunId: latest.id, initiatedByUserId: input.actorId, retryIntentId: id })).actionId; diff --git a/ui/src/hooks/useTaskRetry.test.tsx b/ui/src/hooks/useTaskRetry.test.tsx index 1cbec6ee9fa..c47c9ce3ddc 100644 --- a/ui/src/hooks/useTaskRetry.test.tsx +++ b/ui/src/hooks/useTaskRetry.test.tsx @@ -5,12 +5,17 @@ import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import type { Issue } from "@paperclipai/shared"; import { useTaskRetry } from "./useTaskRetry"; +import { ExecutionBlockerNotice } from "../components/ExecutionBlockerNotice"; +import { ApiError } from "../api/client"; import { issuesApi } from "../api/issues"; vi.mock("../api/issues", () => ({ issuesApi: { retryTask: vi.fn() } })); const issue = { id: "task", interruption: { observedRunId: "failed-run" } } as Issue; function View({ task = issue }: { task?: Issue }) { const retry = useTaskRetry(task); - return <>{retry.optimistic && Thinking}{retry.takingLonger && Taking longer than usual…}; + return <>{retry.optimistic && Thinking}{retry.takingLonger && Taking longer than usual…}{retry.error && }; } describe("optimistic task Retry", () => { let container: HTMLDivElement, root: Root, client: QueryClient; @@ -20,6 +25,19 @@ describe("optimistic task Retry", () => { }); afterEach(async () => { await act(async () => root.unmount()); client.clear(); container.remove(); vi.useRealTimers(); }); async function render(task = issue) { await act(async () => root.render()); } + it("shows the saved-work continuation path after normal Retry is definitively denied", async () => { + const message = "This attempt already recorded a result. Send a new message on the task to continue from the saved work."; + vi.mocked(issuesApi.retryTask).mockRejectedValue(new ApiError(message, 409, + { error: message, details: { code: "chat_failed_run_retry_not_authorized", reason: "accepted_result_without_new_input" } })); + await render(); + await act(async () => container.querySelector("button")!.click()); + expect(issuesApi.retryTask).toHaveBeenCalledExactlyOnceWith("task", { + observedRunId: "failed-run", idempotencyKey: expect.any(String), + }); + expect(container.querySelector('[role="alert"]')?.textContent).toContain(message); + expect(container.textContent).not.toContain("Thinking"); + expect(container.textContent).not.toContain("Taking longer"); + }); it("paints Thinking before admission resolves and shares the same request across double clicks", async () => { let settle!: (value: Awaited>) => void; vi.mocked(issuesApi.retryTask).mockImplementation(() => new Promise(resolve => { settle = resolve; }));