Retain workspace preservation deadlines after transient contention

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed 2026-10-30 21:25:34 -05:00
1 parent bb4d6b244c
commit c8a213f29b
4 files changed
+95 -3

No files matched your search

+7 -1
View File
@@ -8,7 +8,7 @@ This is the final recovery slice after run dispatch, issue state recovery, and d
| Native question cancellation | Cancellable run with committed `nativeQuestionCancellation` marker | Authorized task-completion interaction transaction | Immediate. Existing cancellation owner consumes exact company/run marker. Terminal status removes debt. |
| Active execution | Running run | Run lifecycle, controller renewal/revocation, native authority and process evidence writes | Fresh native/legacy controller expiry, saved five-minute preparation grace, or bounded physical-process probe. Local execution promises retain execution ownership. Explicit evidence/manual gates park until their owning producer changes truth. |
| Settled native process identity | Terminal native failure/cancellation with persisted process identity | Terminal execution writes, server process identity/stop receipts, environment settlement | Immediate inspection. Only matched local host/PID/group/birth or confirmed remote stop evidence clears identity. A live matched process or local execution owner keeps a bounded probe. Missing authority waits for committed evidence. Startup reconstructs rows independently of the old capped process Map. Successful release removes any obsolete compatibility Map entry. |
| Orphan active environment lease | Active lease with NULL run reference or terminal same-company run | Environment owning transactions and indexed physical-resource peer notifications | Saved lease updatedAt plus five minutes, re-read before due work. Existing export, shutdown, and shared-resource fences remain. Exact-lease dispatch does not mutate timestamps for broad-page fairness. Successful CAS hands work to the existing pending-cleanup owner. |
| Orphan active environment lease | Active lease with NULL run reference or terminal same-company run | Environment owning transactions and indexed physical-resource peer notifications | Saved lease updatedAt plus five minutes, re-read before due work. Existing export, shutdown, and shared-resource fences remain. Workspace preservation lock/statement timeouts retain a bounded thirty-second retry; claim/evidence gates still park. Exact-lease dispatch does not mutate timestamps for broad-page fairness. Successful CAS hands work to the existing pending-cleanup owner. |
| Legacy shutdown settlement | Exact lease with pending shutdown marker | Begin, renew, finish, expiry settlement owning transactions | Exact saved marker deadline. Existing owner revalidates boot/controller, source, competing allocations and stop/copyback authority. Expiry records preserved workspace recovery; it never infers file export from process loss. |
| Stranded agent status | Ready agent with status running | Agent configuration/lifecycle/instruction settlement, run/resource transitions, accounting acknowledgement, canonical wake status writes, and direct existing-receipt terminal writers | Saved updatedAt plus five minutes, then actual surviving controller expiry when present. Fresh settlement rechecks live execution, native controller, working copy, and lease truth. Settled status removes debt. |
| Active output silence | Running run and coalesced last output/process start/run start/created timestamp | Actual output progress transaction, terminal source mutation, same-run terminal activity commit, watchdog decisions, execution state | Five-minute threshold, exact saved snooze/continue expiry, or physical termination probe. Fresh check re-reads output and decision. Existing watchdog alone owns completion evidence, process cleanup and fold. Suppressed/manual/blocked history does not poll. |
@@ -37,3 +37,9 @@ A run terminal commit activates resource recovery even if cleanup stops before a
The stranded-agent predicate depends on final accounting and wake receipts as well as runs. `accountRunCost` publishes before acknowledgement, including stale acknowledgement cleanup, with no budget policy required. The canonical wake status writer publishes the known agent before mutation. Direct receipt writers in wake-queue cancellation/authorization, task Retry retirement/chat settlement, late chat authorization failure, Slack stop, tree cancellation and assignment cancellation do the same. Run-coupled queue/retry cancellation paths use execution-control notifications. Newly inserted already-terminal receipts do not remove a prior blocker and need no such signal. Assignment cancellation selects only actual unfinished receipts with the canonical idle proof; historical cancellation write semantics remain unchanged.
Each producer regression starts with a worker already parked and no unrelated budget/resource transition. The final receipt controls require durable agent status settlement, no scheduler deadline/idle hold, and twenty minutes with zero SQL. The native gate and terminal-resource controls captured the original failure before the missing topic fanout was added.
## Workspace preservation contention
The exact-lease owner forwards an optional deadline callback through the existing preservation helper. Only PostgreSQL lock-unavailable (`55P03`) and statement-cancelled (`57014`) outcomes retain a thirty-second retry. A rolled-back write intent may cause earlier bounded reconciliation. `NativeWorkspaceFinalizationBusyError` remains an ownership refusal, and wrong-company or manual/evidence gates do not acquire periodic retries. Legacy explicit callers keep the existing boolean contract.
The regression holds the real task row `FOR UPDATE` until the existing one-second lock timeout fires, then releases only the transaction, without a mutation or notification. The registered worker retains its deadline and records workspace preservation exactly once on retry, handing the original allocation to stop-only pending cleanup with `retain_on_failure`. A second fixture exceeds the actual fifteen-second statement timeout with a one-shot PostgreSQL trigger. Both require the twenty-minute zero-SQL idle proof after the known producer fanout settles; wrong-company, active-claim and manual-gate controls remain parked.