From 85003a7ac43e4b67e75b0c0eebe883e866e4c14b Mon Sep 17 00:00:00 2001 From: Dotta Date: Sat, 24 Oct 2026 10:54:03 -0700 Subject: [PATCH] docs: record interrupted-tool recovery contracts and verification Co-Authored-By: Paperclip --- doc/idle-interrupted-tool-recovery.md | 75 +++++++++++++++++++ ...2026-10-24-recurring-server-sleep-stack.md | 50 +++++++++++++ 2 files changed, 125 insertions(+) create mode 100644 doc/idle-interrupted-tool-recovery.md diff --git a/doc/idle-interrupted-tool-recovery.md b/doc/idle-interrupted-tool-recovery.md new file mode 100644 index 00000000000..885c3cf4d44 --- /dev/null +++ b/doc/idle-interrupted-tool-recovery.md @@ -0,0 +1,75 @@ +# Interrupted tool-action recovery + +An executing tool-action request has an existing ten-minute stale deadline from +its `updatedAt` claim timestamp. Recovery now registers the earliest actual +claim deadline with the shared scheduler. It no longer scans executing requests +on every aggregate maintenance tick. + +## Ownership and producers + +The gateway owns execution claims, provider calls, and result settlement. The +new interruption service owns only the deadline query and stale-row settlement. +The worker uses the existing deadline registration and exposes no provider +protocol or signed argument representation to the scheduler. + +The review write helper signals interruption alongside approved recovery and +pending expiry. All three agent claim paths and the Test-tab claim use that +helper. Test-tab outcomes already commit both rows inside it. Agent pre-dispatch +and execution failure settlement uses the same notification transaction, and +successful or failed direct execution request updates now notify before writing. +Administrative review transitions continue to use the same existing boundary. + +Unrelated pending review changes can cause a bounded empty reconciliation. They +do not create a periodic deadline. Process-local signals cover the existing +single-active-writer database owner; independent writers require a wake contract. + +## Settlement and quiet states + +The query uses a partial executing-request index and reads one earliest row. +There is no deadline after all executing claims settle. Retained terminal +requests and configured-but-idle tools produce no recurring worker queries. + +At the deadline, settlement locks invocation then request, matching the existing +execution-failure lock order. It rechecks the claim status and timestamp after +locking. A changed or completed claim cannot be settled from an earlier scan. +Unknown outcomes atomically mark both rows failed with +`tool_execution_outcome_unknown`. Recovery never calls the external provider. + +A saved terminal invocation is authoritative. Recovery repairs its request to +executed or failed without replacing the saved result, failure reason, or +completion time with an unknown-outcome failure. This covers a crash between +legacy invocation completion and request settlement. Interaction projection +runs after the transaction through the existing gateway owner. A failed +projection still relies on the separate projection-repair duty. + +A database failure rolls back both outcome rows and retains a 30-second retry. +Startup reconstructs saved deadlines. Drain pauses dispatch without forgetting +the first claim or failed retry; resume processes overdue claims. Completion +notifications cancel unnecessary future scans. The scheduler-disabled path keeps +one startup pass. The public combined review sweep retains its existing duties. + +## Verification and limits + +Thirteen new real-PostgreSQL cases cover real foreground execution activation +and completion, configured idle and terminal history, restart/deadline timing, +five saved terminal invocation states, first-write drain/resume, outer commit and +rollback, transactional failure/retry, concurrent claim change, and multi-page +settlement. The service/startup selection passes 136 tests on the combined parent-review +head. Workspace typecheck/build and migration safety checks pass on that head. +Migration application and reapplication passed. The combined run also exercised +469 tool-access tests: 468 passed and an unchanged connection-archive case timed +out under concurrent build load. That case passes in isolation. The separate full tool-access suite passes all +469 tests and gateway acceptance passes all 92 (697 across the final selections). +Fresh full CI/review remains required. + +The first typecheck caught optional approval IDs losing narrowing in transaction +callbacks; capturing them before the callback fixes it. Two new fault/transaction +cases initially waited for idle before advancing their next fake-clock retry. +They now advance time while waiting for the authoritative outcome, and the full +selection passes. These are fixture corrections, not permission to clear an +unresolved work hold. + +The ten-minute policy is unchanged. Active provider execution, unknown external +outcomes, and other worker obligations still prevent unsafe sleep. No Cloud +eligibility or external-alarm coverage changes here. Worker quiet is not evidence +of whole-instance or database suspension. diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index ad4e1e3f1be..a529b4abe14 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -198,3 +198,53 @@ tool-access service tests pass after enabling the existing external database path independently of embedded support. Its initial browser shard failed during unrelated plan-task fixture company deletion with a concurrent activity-log FK violation. Inspect fresh-head CI before deciding whether that needs a repair. + + +## Slice 06: interrupted tool executions + +Branch `codex/sleep-06-interrupted-tool-recovery` is based on #17454 and includes +its one-shot outcome-write repair. The next deadline is the minimum executing +claim timestamp plus the existing ten minutes. The new service locks invocation +then request, rechecks claim identity, preserves saved terminal outcomes, and +atomically quarantines unknown outcomes without calling a provider. All claim +producers and request settlements use the existing review notification boundary. +A partial index supports the first deadline. The aggregate tick now excludes +interrupted claims; projection repair remains its own pending slice. + +Thirteen new worker cases and the existing service/startup selection pass +(136 total on the combined head). Workspace typecheck/build and migration checks +pass. Two fake-clock fixture ordering failures were corrected to wait for real +settlement while advancing retry time. The combined tool-access run had one +unchanged connection-archive timeout; it passed in isolation. The full suite +and gateway acceptance are running separately before readiness is claimed. +No PR is claimed ready until fresh CI and review pass. + +## Next owner audit: tool-review interaction projection + +The remaining aggregate tool-review duty selects terminal requests whose +interaction result does not match the authoritative request state. Recovery +currently reads up to 100 joined rows. Move this duty to its own transaction- +notified worker, with retries only while projection debt remains. Do not add +an outbox table merely to migrate the scan. + +Audit the final two direct request writers in the gateway as well as existing +review helpers, expiry, interruption, and Test-tab outcome settlement. Preserve +the existing reflection owner: it locks issue/interaction/request, checks the +current request state before writing the card, then publishes activity and +requests continuation. A no-op reflection can mean an ownership or lifecycle +condition, not successful reconciliation. Retained terminal requests with +already matching cards must be quiet. Unprojectable/mismatched durable state +must not be silently dropped. Paginate without starving later candidates. + +Continuation delivery already has its own coordinator. A projection write is +not proof that downstream delivery or activity publication completed. Identify +the exact existing recovery owner before removing any backstop, and keep +provider/private payload interpretation out of the scheduler. + +Final slice-06 local verification: the isolated tool-access rerun passes all +469 tests, gateway acceptance passes 92, and the combined service/startup +selection passed 136. All 697 selected tests pass on the combined head. The +initial concurrent-run timeout is retained above as evidence, not hidden. +Workspace typecheck/build and migration safety/numbering pass. The first four +PRs are mergeable and have all checks passed at their recorded heads. #17454 +is mergeable with fresh checks and review in progress at `7a1cf03e62`.