mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 13:24:55 +01:00
Merge commit '51ea62434413df7a0485e8b0c5f6700671b698a5' into codex/chat-catalog-master-integration
* commit '51ea62434413df7a0485e8b0c5f6700671b698a5': fix(server): reject a native run's own task status change (#18411) Make saved execution receipt recovery react to durable work (#18409) Drive issue recovery from committed changes and exact deadlines (#18402)
This commit is contained in:
commit
e7522f48bb
65 files changed
+2836
-305
No files matched your search
@@ -0,0 +1,54 @@
|
||||
# Deferred execution receipt ownership
|
||||
|
||||
This slice removes the periodic `resumeQueuedRuns` sweep after queue dispatch migrated separately. It also delegates native status wake intents out of the periodic orphan reaper. Existing queue admission, authorization, task recovery, cancellation, resource and controller checks remain the mutation authority.
|
||||
|
||||
| Duty | Durable pending truth | Committing producers | Deadline and exit |
|
||||
| --- | --- | --- | --- |
|
||||
| Task Retry | deferred task_retry wake with certified actor/generation | requestTaskRetry through execution control; execution, issue, agent, company and budget eligibility | immediate; current admission links canonical intent to one dispatch/run, supersession or terminal policy retires it; blocked input awaits its actual gate mutation |
|
||||
| Saved comments and interaction responses | deferred wake carrying saved comment IDs or interaction mutation | wake admission insert; merge only genuinely new comment IDs or interaction identity; issue/lock/assignment/dependency and execution eligibility | immediate after terminal source; original release owner revalidates comment/card authority, process/cleanup evidence and issue claim |
|
||||
| Execution wait | deferred wake with executionWait | original admission/restored source receipt; actual recovery action, execution and eligibility changes | original updatedAt + 30 seconds, then current owner revalidation; unchanged blocked receipt carries no recurring scan |
|
||||
| Interrupt retry | deferred wake with queuedCommentInterrupt | authorized route persists interrupt marker; cancellation lifecycle and cleanup/eligibility transitions | original updatedAt + 30 seconds; cancelled legacy run naming exact queue bypasses initial grace as before; owner retains explicit interrupt and stop evidence |
|
||||
| Cancelled legacy queue | cancelled legacy run result names exact still-deferred queue | run lifecycle terminal/control commits; original interrupt intent | immediate restart reconstruction; original release/admission owner settles or leaves saved input behind its actual blocker |
|
||||
| Task chat receipt settlement | deferred task_retry with taskRetry.chatActionId | failed chat retry action settlement | immediate settlement-only pass folds linked action result; it cannot restage its own processed-but-deferred chat action. Startup/actual eligibility retains normal re-admission policy |
|
||||
| Native status wake intent | system/native-status-committer wake queued/claimed without run | status committer transaction; actual execution/issue/agent/company/budget eligibility | immediate queued intent; claimedAt + 60 seconds for crash recovery; error updatedAt + 30 seconds. Existing deterministic dispatcher identity/claims preserve per-intent crash recovery; original intent becomes linked/coalesced/skipped while a separate blocked dispatch receipt owns saved admission |
|
||||
|
||||
Three scoped owners reconstruct only their actual durable receipt predicates, with keyset pages of 100. Normal signals contain normalized company/request identity. Instance scheduler settings alone may request instance-wide reconstruction. Repeated source changes coalesce reconstruction for 30 seconds; already-known stored deadlines preempt that cooldown. Pending checks read memory only. Shutdown uses the existing coordinator and physical execution tracking; task drain reconciles producer transactions and releases the settled intent hold while retaining work for resume. Warm standby and disabled registration do not execute domain work.
|
||||
|
||||
The generic run-dispatch notifier and broad issue review-state notifier deliberately do not fan out deferred eligibility. Queue bookkeeping may write them while processing its own saved input. Separate eligibility commits and settlement-only chat notifications avoid a self-restaging loop.
|
||||
|
||||
C2 remains separate and mandatory: active output silence deadlines, remaining orphan/process/controller/remote authority and resource debts, legacy shutdown settlements, stranded agent status, and final aggregate chain removal. Retained workspace and migrated native finalization workers remain their existing owners.
|
||||
|
||||
Review controls cover queue edits that move an already loaded grace deadline. Each due check re-reads current stored grace or claim time before admission. Subtree hold release, subtree restore, and member archival publish eligibility inside their own transactions. Member archival preserves the cancelled Retry tombstone created by an earlier user reassignment.
|
||||
|
||||
A resume that first records a new execution-wait reason arms its existing 30-second grace once. Writing the same closed-gate reason again does not publish another hint. A clock obligation, an eligibility change, and an infrastructure failure therefore retain separate exits: stored deadline, owning committed producer, and bounded failure backoff. Refreshing a diagnostic timestamp alone does not create permanent polling behind a deliberate pause.
|
||||
|
||||
Persisted budget and agent refusals are normalized by the existing deferred-wait owner. The wait writer reports whether its locked conditional update saved the receipt, and only that successful path can produce the internal `PersistedExecutionWaitRefusal` subtype. It remains an HTTP409 with the original message and details. Receipt recovery parks that outcome instead of creating failure debt; the committed first/changed reason still grants its one stored30-second grace. Unrelated409s, failed writes and unexpected provider/infrastructure errors retain failure backoff. This adds no admission policy and changes no native/task-chat ownership.
|
||||
|
||||
Production-path controls use actual task Retry, heartbeat, PostgreSQL budget/lifecycle services and a child process writing saved stdout. The lifecycle host completion boundary is controlled explicitly. One set of real budget/agent refusal fixtures consumes its finite grace and becomes quiet for25 minutes of timer advancement with zero driver SQL and no execution. Independent warm-worker fixtures keep the real PostgreSQL clock throughout admission, actual gate-producer release, controller authority and child-process exit, proving exactly one saved execution. Long-idle and provider-release evidence are separate; no clock reset or restart is used to claim warm recovery. A PostgreSQL trigger fault proves failed wait persistence retains recovery backoff, and an unrelated unbranded409 remains a failure.
|
||||
|
||||
Project budget admission locks and re-reads the company-scoped project before publishing pause intent. An already-paused project emits no new eligibility hint and preserves its pause reason and timestamp, including a manual pause. The actual transition still publishes dispatch and deferred eligibility intent before updating the project in the accounting transaction. Concurrent admission checks serialize through that authority. Project-scoped saved Retry controls cover its one grace followed by25 minutes with zero driver SQL, plus a separate real-clock budget increase through one succeeded process with saved stdout.
|
||||
|
||||
|
||||
Native receipt checks share an opaque dispatcher created by the existing queue owner for exactly one bounded worker pass. Its canonical company/agent/issue/review scope map is the same map used by the manual dispatcher; the native worker explicitly admits at most100due entries per pass, matching the existing default batch while retaining company fairness. The next pass creates a fresh map. This restores same-pass grouping without introducing global or cross-invocation deduplication, changing claim authority, or consuming later notices. Each receipt still re-reads its own moved claim/error deadline before dispatch.
|
||||
|
||||
Actual PostgreSQL controls reconstruct two saved native intents into one dispatch and shared run with real process stdout, cover the100-intent boundary, and keep another issue and company separate. A real issue-row lock holds the first admission while a separate transaction commits a new notice whose requestedAt is explicitly older than admission. The fixed current batch cannot consume that new identity; a fresh pass creates a distinct run, preserving the late notice even though its ordering metadata is old. Once all executions and stored deadlines settle,25virtual minutes produce zero driver SQL. The canonical dispatch body, including parsing, claims, deterministic receipt recovery and error handling, is unchanged apart from moving its map into the owner-created batch closure.
|
||||
|
||||
|
||||
### Browser fixture producer boundary
|
||||
|
||||
The legacy recovery browser fixture used to insert a deferred request through a
|
||||
separate SQL connection after server startup, then wait for the removed poll.
|
||||
That is neither saved startup history nor a supported in-process producer. The
|
||||
automatic browser case now seeds while the company is paused, verifies the saved
|
||||
request and sole failed source run, and reactivates through the ordinary company
|
||||
API. No Retry or duplicate message is sent. It requires exactly one new run,
|
||||
one persisted agent response, the original queued comment in the process prompt,
|
||||
and the response after page reload. Startup reconstruction remains covered by
|
||||
the PostgreSQL case that persists history before worker registration.
|
||||
|
||||
The corrected automatic path passes three consecutive built-UI/browser runs
|
||||
(2.1 minutes total). All six legacy continuation browser paths pass together
|
||||
(1.6 minutes), including task/inbox Retry, Retry after reload, ordinary message,
|
||||
queued Retry and automatic reactivation. The fixture also reads the configured
|
||||
external PostgreSQL connection when present, so local proof can reuse a dedicated
|
||||
test database instead of starting another database process.
|
||||
@@ -0,0 +1,32 @@
|
||||
# Issue recovery without periodic scans
|
||||
|
||||
The issue recovery owners reconstruct their current obligations once at startup. Ordinary comment writes invalidate only the affected issue for assignment and lock recovery. Lifecycle, graph, run, configuration and eligibility changes invalidate their company. The coordinator keeps scopes, per-issue deadlines, and per-entry failures in memory. Its pending check does not query the database.
|
||||
|
||||
| Owner | Durable truth and producer boundary | Deadline or event | Exit |
|
||||
| --- | --- | --- | --- |
|
||||
| Stranded assignment | Visible assigned todo/in-progress/review issues; issue create/update/checkout/release; run lifecycle and execution control; agent/company/budget/configuration; interaction, approval and subtree hold changes | A committed eligibility change. A transient continuation uses the latest finish plus the existing exponential delay. | An execution path, durable wait, escalation, or current ineligibility. |
|
||||
| Orphan blocker assignment | Agent-created unassigned todo/blocked blockers with a nonterminal dependent; issue, relation, agent and company changes | Once per changed company, independent of assignment candidates. | Assigned, no longer blocking, or creator cannot execute until its next eligibility change. |
|
||||
| Active recovery actions | Active/escalated action records, including actions on completed sources; action, issue, run and wait changes | Once per changed company, independent of assignment candidates. | Resolved, restored path, explicit hold or a separately owned retry. |
|
||||
| Resolved dependency | Blocked assigned issues and company-scoped blocker relations; blocker status or relation changes through issue service and import; run/control and pause gate changes | The committing change. A failed enqueue retains that issue with error backoff. | Existing idempotent wake, live path, unresolved dependency or deliberate self-block. |
|
||||
| Task watchdog | Active issue watchdogs and their current source subtree; normal watchdog upsert/disable; issue/comment/document/work-product and wait changes | Exact source creation plus the existing 15-second first-run grace. An incomplete scan retains the watchdog with error backoff. | Live path, open/reviewed reusable review, disabled configuration, or ineligible source. |
|
||||
| Stale issue lock | Existing checkout/execution run references; issue and run lifecycle changes | A committing ownership change, or the fresh owning controller's saved lease expiry. | No stale reference, a valid owner, or the existing guarded terminalization and lock clear. |
|
||||
|
||||
Local comment hints do not broaden into an assignment check for every open task. A real PostgreSQL regression starts with1,000 open assigned issues, writes one comment through `issueService.addComment`, and observes exactly one assignment check; a subsequent company eligibility notification still checks all1,000. Restoring the previous comment hint makes that same regression check all1,000 after the comment. The test substitutes the expensive assignment decision only to count its invocations; it uses the real transaction, notification, coordinator and paged SQL. Separate producer acceptance cases exercise real recovery decisions and process execution.
|
||||
|
||||
Parent/descendant disposition repair and shared execution references make status and run changes broader than a comment. Their existing notification boundaries deliberately retain company scope. Orphan blockers, active recovery actions, resolved dependencies and watchdogs also keep their company/subtree duties; an issue comment does not drop those independent obligations. The scope serializer selects known primitive fields and never serializes an entire insert or update object.
|
||||
|
||||
Reconstruction uses keyset pages of at most100rows. Repeated company hints coalesce reconstruction for30seconds. Already-known entry deadlines preempt that spacing without an early probe on each comment. Scheduling suppression is read once per pass. A failed read keeps scoped backoff debt. The two subsidiary company repairs do not run inside the per-issue loop. Legacy direct aggregate API calls retain both repairs.
|
||||
|
||||
The owner methods accept company and issue/watchdog identity. A narrowed lock check still reads other same-company references before using terminal-issue authority. A stale terminal alias cannot end an active issue's shared run. The controller lease owner supplies its fresh expiry; the existing write predicate revalidates the observed boot ID and database-clock expiry.
|
||||
|
||||
Producer coverage uses the existing run-dispatch, execution-control, issue-review-state, native status reassessment and external conversation settlement boundaries. Direct gaps have explicit signals inside their owning transaction: document writes/import, attachment removal, blocker import, recovery-action adoption/mutation, conversation pause release, Telegram failure comment replacement, native artifact title/metadata updates and company deletion. Metadata-only comment classification that does not change timestamp, visibility or execution eligibility needs no additional watchdog activation. Approval links and decisions use their existing issue-review/continuation effects. Work-product create/update/delete/import uses its existing native status boundary.
|
||||
|
||||
The new owners replace only these four startup and periodic calls. Deferred comments, silent active runs and residual orphan/resource recovery retain their existing backstop until the next slice. Every previously migrated native-owner skip remains unchanged.
|
||||
|
||||
The ordinary dependency and watchdog process controls request completion through a loopback test-operator boundary before exiting. That boundary validates the actual run/company/agent and uses issue service; the watchdog control also disables its source through watchdog service. These cases prove normal completion without creating new disposition debt. The separate incomplete-process control prints an execution proof but cannot apply a task disposition, so the test operator completes that task through issue service after the existing policy saves a retry. If the existing continuation policy already saved a disposition-repair retry, the test preserves its actual timer and stored deadline on the real clock. Cleanup joins both competing coordinators before inspecting cancellation. A separate acceptance case deliberately waits for a production-created retry and verifies that exact row becomes cancelled, then all registered deadlines settle before the twenty-minute virtual idle measurement. Its120-second budget covers the existing60-second retry,30-second reconstruction and real I/O; the ordinary dependency test retains90seconds. Installing fake timers after an owner armed a real timer left that timer waiting on wall time despite virtual advances, so cleanup does not switch clocks while any owner obligation remains. The final quiet proof requires no registered deadlines, no tracked in-flight work and no owner SQL. It does not claim a saved future retry is already drained.
|
||||
|
||||
Active action repair has no separate grace or lease countdown. Its delayed disposition repair first writes a scheduled-retry run, then mirrors the date into action wakePolicy.retryAt and timeoutAt. The dispatch owner owns that saved run deadline. Other skipped gates wait for their existing execution, wait, pause, budget, or finalizer producer; the company action worker does not add a polling fallback.
|
||||
|
||||
Agent Chat is an instance-wide eligibility gate for saved conversation comments. An effective disabled-to-enabled transition publishes the existing stranded-assignment topic before the settings write in the same transaction. The worker reconstructs eligible conversation assignments after commit; other issue owners are not activated by this flag. Effective unchanged values, unrelated flags and disabling do not publish assignment hints. A rolled-back enable leaves the accepted comment parked.
|
||||
|
||||
The real PostgreSQL control accepts a valid conversation comment while chat is enabled, disables chat before registering recovery, and proves the outbox stays quiet. Committing the ordinary settings enable alone launches one real process. That process calls a controlled loopback reply boundary which writes the agent comment through issue service; normal conversation completion consumes the message and returns to waiting. Rollback, unchanged/unrelated settings and disable-before-dispatch controls retain input without launching a run. Idle checks inspect actual PostgreSQL statements over twenty virtual minutes, including transactional queries.
|
||||
Reference in new issue
Block a user