df6156c2e3
## Thinking Path > - Paperclip manages agents, tasks and their durable native results. > - Changed evidence, task authority or review retirement can require a new status assessment. > - Periodic orphan recovery repeatedly reads history to find these changes. > - This change gives reassessment an owner driven by committed changes and saved deadlines. > - Existing result, assessment, decision and task rows remain durable truth. > - The status committer still verifies current sources and authority before writing. > - Settled or insufficient evidence becomes quiet until a supported producer changes it. ## Linked Issues or Issue Description Refs #17702 (merged), #17679 and #17680. This is slice 50, now based on master. It owns native status reassessment; workspace settlement, retained sessions and other orphan duties remain separate. **What happened?** Periodic recovery repeatedly inspects completed native results. Removing that scan without complete writer coverage or current authority checks could strand new evidence or commit an old result over newer work. **Expected behavior** Reassess from existing durable facts after relevant outer transactions commit. Retain real retry/lease deadlines and restart/drain recovery. Preserve current authority, quality-review and supersession checks. Make no recurring SQL when work is settled or dormant. **Steps to reproduce** Complete a native result awaiting evidence or review, leave it idle, then approve a real work product or retire the selected automatic review. Its saved result should advance once when authorized, or record the existing nonterminal business gate, without a periodic scan. ## What Changed - Extract reassessment and its pure status policy from finalization recovery into their owning modules. Add a scoped worker using the existing delivery coordinator and scheduler; no queue table or migration is added. - Retain the three durable discovery triggers: changed work-product history, authoritative task status/version, or retirement of the selected automatic completion review. Preserve selected-source, ownership, effects, retry and lease fences under the existing committer. - Notify from actual task/work-product, claim release, approval, interaction, OAuth, event evidence and workspace-settlement owners before their transaction writes. Keep provider calls outside short transactions and nested writes behind outer commit. - Scope ordinary milestone/finalization/coalesced-input wakes to an issue/run. Keep genuine company authority restoration explicit. Real Photon and coalescer regressions cover all three response/status queues. - Move the current parent's required quality policy into the owner and apply it to every reassessment that could complete a task. An unmet review records the existing in-progress decision and next action, avoiding an operational retry loop. Preserve conversation/session reset locks and the workspace omission-barrier signal. - Skip reassessment in periodic recovery while preserving explicit/startup recovery and the historical supersession guard before independent workspace work. Document the source contract, producer inventory, failures and qualifications in the linked audit. ## Verification - Broad source `9fddb3f0b1157df32b64fb1f22e9d6e9624470e8` passes **2731 distinct cases in 28 complete files**, with no skips: 2458 producer/owner/authority/workspace/quality cases in 394.43 seconds and all 273 governed-response cases in 53.74 seconds. Nineteen dedicated fresh PostgreSQL 17 databases migrate through 0413; eighteen embedded-only invocations preserve fresh-per-call isolation through the existing recovery helper. All 22 temporary imports restore exactly and all 28 hashes match. - Final runtime source `5ea3b0fd016bf7e63ba82a7044825a46a0cd2082` passes **1211 cases in five complete affected files** in 108.23 seconds after the narrowly reproduced evidence-trigger fix: all 43 worker, 145 corpus, 32 finalization-recovery, 968 native-session and 23 quality-review cases. Four fresh databases plus the quality helper's own fresh fixture use PostgreSQL 17. Its one adapted import restores exactly; all five hashes match. The other 27 broad-run test files are unchanged. These qualifications are source-scoped and not cumulatively double-counted. - Required/optional real regressions reproduce retry loops on retired-review and evidence-approval paths before repair; both become SQL quiet afterward with unchanged assertions/time limits. Controlled removal of only ordinary scope guards reproduces two coarse-hint SQL failures; exact guard restoration passes. Real producer, outer transaction, claim, authority, 103-item backlog, retry and drain cases pass. Photon transport is mocked; callbacks, locks and observed SQL are real. - Normal merges preserve 12661 parent/unrelated entries, 35 owned entries and 21 automatic shared results, with five original conflicts reviewed explicitly. The parent's actual squash reconciliation preserves the entire held child tree. The final quality fix changes only its owner and tests. Fourteen added comments are removed with identical nontrivia tokens; no directives are removed and the final comment check is empty. - Database build, server types/build, boundaries and exact-head migration order pass. The database tree is identical to merged #17702: no new SQL, queue table or snapshot is generated. Earlier benchmarks, host limitations, failed assertions and their repairs remain in `doc/plans/2026-10-26-native-status-reassessment-sleep-audit.md` and `doc/idle-native-status-reassessment.md`. - This qualifies targeted external PostgreSQL behavior, not embedded bootstrap on the semaphore-exhausted host, a full local repository suite, deployed sleep or measured fleet savings. Fresh exact-head CI and review remain required. ## Risks - Notifications retain the process-local supported-writer contract. Independent SQL writers and uncoordinated replicas are not covered by a new transport. - Reassessment competes with current authority changes. Existing row/source locks and retries remain required; arbitrary missing identities have no row to lock, so this is not a general predicate-lock guarantee. - Discovery pages 100 rows but retains outstanding candidates in memory. It remains proportional to backlog size, and cancellation occurs between queries. - Other orphan/workspace/session duties and Cloud sleep eligibility remain unchanged. Missing supported producer coverage could strand dormant work, so the producer audit and real callback/transaction tests remain part of acceptance. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell execution and PostgreSQL tests. The runtime does not expose a more specific serving snapshot or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted scope is stated above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>
6.2 KiB
Native status reassessment without idle polling
Native status reassessment uses committed changes and the shared delivery scheduler. It no longer runs on every periodic orphan-recovery pass.
What creates work
Existing native result, assessment, decision, task and review rows remain the source of truth. No new queue table or database migration is required.
| Change | Existing producer boundary |
|---|---|
| Work-product creation, update, import, removal, attachment or trusted promotion | Work-product/task services and the promotion transaction |
| Task status/version, assignment, execution state or blocker configuration | Task mutation transaction |
| Direct claim release, terminal/stale lock cleanup, queue release, recovery or tree restoration | Existing task-scoped claim mutation transaction |
| Automatic review retirement | Automatic completion review owner |
| Approval/link or interaction outcome, including OAuth card creation/reopening | Approval/review-state notification boundaries |
| Accepted/rejected control-plane run evidence | Run-event append transaction |
| Workspace barrier or physical ownership settlement | Workspace operation, finalizer and verified recovery owners |
| Coordinator claim/retry, effect settlement or run projection | Native finalization and run-milestone boundaries |
These signals request inspection. They do not grant permission to complete a task. Discovery retains the three existing conditions: changed work-product history, changed authoritative task status/version, or a retired selected completion review. The owner still classifies the selected result and appends assessments and decisions through the existing status committer.
A settled worker makes no recurring queries. Insufficient evidence is dormant. Real contention or operational failure keeps a retry. A saved retry or lease expiry keeps its actual future deadline. Startup reconstructs candidates in 100-row pages; the coordinator processes bounded batches and retains changes through drain/resume and concurrent dispatch. Shutdown uses its existing owner.
Authority and separate responsibilities
Scheduled commit checks the selected source, current task/run, retry and lease, pending effects and native process ownership inside the status transaction. Retired-review completion also checks the latest contract, workspace barrier, physical workspace ownership, governance and dependencies. Evidence is reclassified under source locks. Contention defers instead of blocking an idle worker indefinitely or committing stale completion.
The legacy reconciler retains explicit recovery and a one-time startup pass. Its periodic call skips reassessment but preserves the superseded historical source guard before independent workspace recovery. Workspace/result settlement, retained session cleanup, native wake delivery and generic orphan recovery are separate duties; their recurring scans have not been removed by this slice.
Case-handoff delivery and lease changes do not improve evidence: every persisted non-cancelled state already counts as accepted. Cancellation is negative evidence and is fenced by the source lock at scheduled commit. Creation generates the handoff identity inside its transaction before returning it. No reassessment wake is added to every lease renewal. Target-agent status is also not a classification trigger: attention validation checks existence, not whether an agent is invokable. Creating an agent does not satisfy result criteria or verification. Missing arbitrary target identities have no row to lock; this change does not provide a general predicate-lock guarantee for independent identity insertion.
Pending child completion is a guard and a fresh parent-turn obligation. It is not silently consumed by completing the old result. New contracts supersede the old result's eligibility; creating one does not revive that old result. Direct SQL writers and independent replicas do not receive process-local notifications. The supported producer model remains the active server and integrated transaction owners; this change does not add a distributed notification transport.
Evidence and limits
Current qualification is recorded in the audit. The broad 9fddb source passes 2731 cases in 28 complete files. The final 5ea3 runtime source, after the evidence-trigger quality fix, passes 1211 cases in five complete affected files. Their overlapping counts are not added together. Real producer/transaction coverage includes first writes, rollback/outer commit, authority races, saved deadlines, drain/resume, claim release and retained history. The final worker suite includes all 43 cases.
Required quality review remains mandatory on both retired-review and evidence reassessment. An unmet review produces its existing nonterminal decision and review recovery action, rather than repeatedly attempting completion as an operational retry. Real required/optional regressions and controlled coarse-hint negatives fail before repair and pass afterward without changing time limits.
Historical discovery measurements in the audit retain their original revision: 10,000 selected-history tasks return no company candidates in about 35 ms; 10,000 real candidates require 101 queries and about 1.56 seconds to discover. This is discovery cost, not complete backlog execution. The current worker still retains candidates in memory proportional to backlog size. Its permanent 103-item lifecycle case verifies settlement across batches and SQL quiet. Cancellation is checked between pages, not during an already-running SQL call.
Database build, server typecheck/build, module boundaries and migration-order checks pass. The host's embedded PostgreSQL bootstrap remains limited by exhausted System V semaphores; acceptance uses fresh owned PostgreSQL 17 databases and preserves embedded-only fresh-per-call isolation through the existing recovery helper. Adapted test imports restore exactly; assertions are unchanged. No full local repository-suite or live external-provider qualification is claimed.
Remote exact-head CI and review still gate merging. The stack ledger tracks other duties. Worker quiet does not establish database suspension, Cloud alarm coverage, host sleep or fleet cost savings.