From c17a01e9490bb8f63b4fa5902a47ace06272e5fa Mon Sep 17 00:00:00 2001 From: Dotta Date: Tue, 21 Apr 2026 13:00:17 -0500 Subject: [PATCH] chore(docs): remove planning docs from adapter parser PR --- ...2026-04-20-background-jobs-mental-model.md | 300 ------------- .../2026-04-20-memory-extraction-job-state.md | 423 ------------------ 2 files changed, 723 deletions(-) delete mode 100644 doc/plans/2026-04-20-background-jobs-mental-model.md delete mode 100644 doc/plans/2026-04-20-memory-extraction-job-state.md diff --git a/doc/plans/2026-04-20-background-jobs-mental-model.md b/doc/plans/2026-04-20-background-jobs-mental-model.md deleted file mode 100644 index 65643cd0c6f..00000000000 --- a/doc/plans/2026-04-20-background-jobs-mental-model.md +++ /dev/null @@ -1,300 +0,0 @@ -# Background Jobs Mental Model - -## Goal - -Define how Paperclip should think about "jobs that run" without introducing a second orchestration system beside issues, routines, and heartbeat runs. - -The immediate design target is memory maintenance and similar background work that must be: - -- framework-owned -- auditable -- deployable outside a single local machine -- compatible with plugin-provided behavior - -## Recommendation In One Sentence - -Paperclip should treat background automation as a small taxonomy of existing control-plane primitives, not as one new generic queue abstraction: use agent timer heartbeats to wake agents, routines to create recurring issue work, plugin or system jobs for deterministic host-side maintenance, and explicit domain job tables when a subsystem like memory needs auditable async work that is not itself an issue. - -## Current Reality In The Repo - -Paperclip already has four adjacent mechanisms: - -1. **Agent timer heartbeats** - - Defined in agent adapter config. - - Purpose: wake an agent on a schedule. - - Output: `heartbeat_runs`. - - This is a wake policy, not a business-work model. - -2. **Routines** - - Company-scoped and project/goal/issue-aware. - - Purpose: recurring automation that should create or continue issue work. - - Output: `routine_runs`, linked issues, then heartbeat runs. - - This is the auditable recurring-task path. - -3. **Plugin scheduled jobs** - - Plugin-owned host-side handlers scheduled by the host. - - Purpose: deterministic plugin maintenance, sync, cleanup, polling. - - Output: `plugin_job_runs`. - - This is infrastructure automation, not issue orchestration. - -4. **Server-owned maintenance loops** - - Examples: heartbeat recovery, routine ticking, database backups. - - Purpose: keep the Paperclip control plane healthy. - - Output today is mostly logs plus subsystem-specific records. - - This is framework maintenance, not user-facing work management. - -The problem is not that Paperclip has no job model. The problem is that these mechanisms need an explicit boundary so new systems like memory do not accidentally invent a fifth orchestration surface. - -## Core Product Decision - -Paperclip should not add a generic "jobs engine" that tries to flatten all background work into one abstraction. - -Why: - -- V1 already explicitly allows a lightweight in-process scheduler/worker and does not require queue infrastructure. -- Paperclip's differentiator is company-scoped orchestration, not general cron/queue infrastructure. -- A generic Bull or Redis queue layer would duplicate ownership with routines, issues, heartbeat runs, and plugin jobs. -- The repo already has a documented bias against importing runtime-level workflow or queue abstractions into the control plane without mapping them back to Paperclip entities. - -The right move is to sharpen the taxonomy and add only the missing domain-specific records where needed. - -## The Taxonomy - -### 1. Agent timer heartbeat - -Use when the question is: - -- "When should this agent wake up and look for work?" - -Characteristics: - -- executor is an agent adapter -- unit of execution is a `heartbeat_run` -- schedule belongs to the agent runtime config -- may or may not result in issue activity - -Do not use this as the default model for deterministic maintenance like compaction, syncing, or shell scripts. - -### 2. Routine - -Use when the question is: - -- "What recurring work should become an issue in the company?" - -Characteristics: - -- company-scoped -- goal/project/parent-issue aware -- human-visible and auditable as work -- creates or coalesces into issue execution -- hands off to the existing issue and heartbeat execution path - -Examples: - -- weekly CEO review -- nightly "summarize customer feedback" task -- recurring research sweep -- scheduled agent-authored report - -If the output should be a task someone can inspect, comment on, block, review, or approve, it should usually be a routine. - -### 3. Plugin job or system job - -Use when the question is: - -- "What host-side maintenance should run without becoming an issue by default?" - -Characteristics: - -- deterministic host-side execution -- run by plugin worker or server-owned code -- recorded as job-run history, not as issue work -- appropriate for cleanup, polling, reconciliation, compaction, indexing, and sync - -Examples: - -- refresh connector metadata -- poll an external API and cache results -- compact a local index -- database backups -- plugin-specific cleanup or reconciliation - -These jobs are real framework-owned automation, but they are not the same thing as company task execution. - -### 4. Domain-specific async job record - -Use when the question is: - -- "This subsystem needs async background work with auditability, provenance, and operator visibility, but it should not become an issue every time." - -This is the missing piece for memory. - -Examples: - -- memory extraction -- memory compaction -- dedupe / correction pipelines -- provider-managed async indexing callbacks - -For these cases, the right answer is not "pretend everything is an issue" and not "hide everything inside a plugin job." - -The right answer is: - -- keep the scheduler/executor simple -- add subsystem-specific run records that carry Paperclip provenance -- escalate into issue work only when human-facing follow-up is required - -This matches the existing memory plan's direction toward explicit extraction jobs. - -## Decision Rules - -When a new background process is proposed, ask these questions in order: - -1. **Should this show up as company work?** - - If yes, it should be a routine that creates issue work. - - If no, continue. - -2. **Does this require language-model reasoning or agent tool use?** - - If yes, dispatch to an agent through the normal issue/heartbeat path. - - If no, continue. - -3. **Is this deterministic host-side maintenance owned by the framework or a plugin?** - - If yes, use a plugin job or server-owned maintenance loop. - - If no, continue. - -4. **Does the subsystem still need first-class auditability and provenance even though it is not issue work?** - - If yes, add a domain-specific async job record. - -This keeps the control plane authoritative without forcing one storage shape onto every kind of automation. - -## What This Means For Memory - -Memory should use three different execution shapes depending on the operation. - -### A. Pre-run hydrate - -Use the heartbeat path directly. - -- Trigger: before an agent heartbeat starts -- Executor: memory provider call in the run path -- Attribution: included in the run or directly metered if the provider bills separately -- Why: this is part of prompt assembly, not a background scheduler concern - -### B. Post-run capture - -Use one of two paths: - -- synchronous capture in the run path for cheap/local providers -- asynchronous memory extraction job for heavier provider-managed or delayed work - -Why: - -- some providers can write immediately -- others need extraction, chunking, or delayed indexing -- Paperclip should not hide that async work if it matters for auditability or failure visibility - -### C. Periodic memory "sleep cycle" / dreaming / compaction - -Default to a memory-domain background job, not a system agent and not a routine. - -Why: - -- this is maintenance on the memory subsystem -- it may need to scan or compact provider state -- it often does not map to a user-facing issue -- forcing it through an agent adds reasoning cost and muddles ownership - -If that maintenance discovers something that should become company work, it should then escalate by: - -- creating an issue directly, or -- triggering a routine that creates issue work - -Examples: - -- "memory compaction completed" stays a memory job record -- "memory system found duplicate identities needing review" becomes an issue -- "nightly memory digest for leadership" becomes a routine because the output is work product - -## Plugin Recommendation - -Plugins should be able to participate in both layers, but the layers should stay distinct. - -### Plugin maintenance - -Plugins may continue to register scheduled jobs for host-side deterministic work. - -Good fit: - -- sync -- compaction -- polling -- cleanup -- provider callbacks - -### Plugin-created recurring company work - -If a plugin wants recurring work to enter the company task system, it should register or provision a routine, and the routine should record that provenance. - -Recommended future addition: - -- add routine provenance metadata such as `registeredByPluginId` or equivalent source metadata - -That gives us the useful semantics behind "a plugin registered this routine" without collapsing routines into plugin jobs. - -### Why not use plugin jobs alone for memory maintenance? - -Because memory is company-scoped in Paperclip's product model, while plugin jobs today are plugin-owned host jobs with no first-class company targeting in their core record shape. - -That makes plugin jobs a good execution mechanism, but not the full product model for company-auditable memory operations. - -## System Agents - -Paperclip may eventually want system agents, but they should be optional and narrow. - -Good reasons to use a system agent: - -- the work genuinely needs reasoning -- the output should look like authored task work -- budget attribution should land on an agent -- approvals, comments, or review are part of the loop - -Bad reason to use a system agent: - -- "we need to run a shell script every few hours" - -That is deterministic maintenance and should stay a job, not become fake agent labor. - -## Proposed Product Shape - -Paperclip should describe background automation using three top-level concepts: - -1. **Wakeups** - - wake an existing executor, usually an agent - -2. **Recurring work** - - routines that create or continue issue execution - -3. **Background maintenance** - - plugin jobs, server jobs, and domain-specific async jobs - -These can later share a UI grouping like "Automation" or "Background Activity" without requiring a single underlying table. - -## Recommended Near-Term Follow-Up - -1. Document this taxonomy in product and engineering docs where routines, plugins, and memory are discussed. -2. Add provenance metadata for routines created or managed by plugins. -3. When memory implementation begins, add explicit memory extraction job state instead of hiding async maintenance inside generic logs. -4. Keep the scheduler in-process for now; do not add Redis/Bull-style queue infrastructure unless scale or deployment evidence forces it. - -## Bottom Line - -Paperclip should not think of "jobs" as one new generic abstraction. - -It should think of them as: - -- agent wakeups when we need an agent to run -- routines when recurring automation should become issue work -- host-side jobs when the framework or a plugin needs deterministic maintenance -- domain-specific async records when a subsystem like memory needs auditable background execution without becoming a task every time - -That preserves Paperclip's control-plane model, keeps deployment complexity low, and gives memory a clean place to live. diff --git a/doc/plans/2026-04-20-memory-extraction-job-state.md b/doc/plans/2026-04-20-memory-extraction-job-state.md deleted file mode 100644 index 7ebd23051d3..00000000000 --- a/doc/plans/2026-04-20-memory-extraction-job-state.md +++ /dev/null @@ -1,423 +0,0 @@ -# Memory Extraction Job State - -Status: proposed design -Date: 2026-04-20 -Owner issue: `PAP-1702` -Related docs: - -- `doc/plans/2026-04-20-background-jobs-mental-model.md` -- `doc/plans/2026-03-17-memory-service-surface-api.md` - -## Goal - -Define the first domain-specific async job record in Paperclip: a company-scoped memory job state model for asynchronous capture, compaction, and dreaming work that must be auditable without becoming issue work by default. - -This design keeps the background-jobs taxonomy from `2026-04-20-background-jobs-mental-model.md` intact: - -- no generic queue engine -- no reuse of plugin job tables as the product model -- no requirement to invent a system agent just to run deterministic maintenance - -## Decision Summary - -Paperclip should add a `memory_extraction_jobs` table whose rows represent **job attempts**, not recurring definitions and not a generic workflow engine. - -The row is the durable audit object for async memory work that sits beside, but does not replace: - -- `heartbeat_runs` for agent execution -- `routine_runs` for recurring issue work -- `plugin_job_runs` for plugin-owned host maintenance -- `cost_events` for canonical spend accounting - -The row should carry enough provenance to answer: - -- what memory work happened -- what company, agent, issue, and run it came from -- which memory binding handled it -- what it cost -- whether it succeeded, failed, or needs operator attention - -Retries should create a **new** job row with `retryOfJobId` instead of mutating a terminal row back to `queued`. - -## Why A Domain Job Record Is The Right Layer - -Memory is the first concrete subsystem that needs background work with all of the following: - -- company scope -- auditability -- operator visibility -- attribution back to issue and run context -- optional provider-managed asynchronous completion - -That makes it a poor fit for: - -- issues, because most memory capture/compaction work is not human-facing task work -- plugin jobs, because those are plugin-owned host jobs rather than company-auditable memory records -- system agents, because capture/compaction/dreaming are deterministic maintenance unless they explicitly need reasoning - -## Proposed Schema - -### Minimum persisted row - -The minimum record should be a new `memory_extraction_jobs` table with these fields. - -| Field | Purpose | -| --- | --- | -| `id` | Stable job-attempt id | -| `company_id` | Company scope boundary | -| `binding_id` | FK to the resolved memory binding | -| `binding_key` | Stable operator-facing binding identifier snapshot | -| `operation_type` | `capture | compaction | dreaming` | -| `status` | `queued | running | succeeded | failed | cancelled` | -| `source_agent_id` | Agent provenance when present | -| `source_issue_id` | Issue provenance when present | -| `source_project_id` | Project provenance when present | -| `source_goal_id` | Goal provenance when present | -| `source_heartbeat_run_id` | Run provenance when present | -| `hook_kind` | `post_run_capture`, `issue_comment_capture`, `issue_document_capture`, or null for pure maintenance jobs | -| `provider_job_id` | Provider-native async job id when one exists | -| `submitted_at` | When Paperclip enqueued the job | -| `started_at` | When a worker claimed execution | -| `finished_at` | Terminal timestamp | -| `attribution_mode` | `included_in_run | billed_directly | external_invoice | untracked` | -| `cost_cents` | Job-level billed cost summary for operator visibility | -| `result_summary` | Human-readable summary suitable for a list row | -| `error_code` | Stable failure classifier | -| `error` | Human-readable failure detail | -| `created_at` / `updated_at` | Audit timestamps | - -### Supporting fields needed for safe dispatch and rerun - -The following fields are not part of the problem statement's minimum list, but they make the V1 design operationally sound: - -| Field | Why it exists | -| --- | --- | -| `source_kind` | Distinguish run/comment/document/manual/maintenance origins | -| `source_ref_json` | Hold comment id, document key, subject id, namespace, and other source-specific identifiers without over-normalizing the first slice | -| `retry_of_job_id` | Preserve attempt history when a job is rerun | -| `attempt_number` | Make retries visible in the list and detail surfaces | -| `dispatcher_kind` | Start with `in_process`, preserve a clean seam for later host-side workers | -| `lease_expires_at` | Lets the scheduler detect stale `running` work without adding a new status | -| `usage_json` | Store token/latency/provider usage summary without overextending `cost_events` in the first slice | -| `result_json` | Structured output such as record counts, callback payload, or escalation metadata | - -### Notes on normalization - -- `binding_id` should be the authoritative FK. -- `binding_key` should still be stored on the row so operators can read history even if the binding is later renamed or deleted. -- `source_ref_json` should stay lightweight. Do not snapshot full transcript bodies here. -- `result_summary` should be short and readable, for example `Captured 14 records from run residue` or `Compaction found 3 duplicate record groups`. - -## Relationship To `memory_operations` - -The earlier memory plan introduced `memory_operations` as the broader audit trail for all memory actions. This design does not replace that. - -The cleanest end state is: - -- `memory_operations` logs every memory action, synchronous or asynchronous -- `memory_extraction_jobs` tracks async job-attempt state for the subset of operations that run outside the initiating request path - -For the first implementation slice, the job record should include enough provenance and usage fields to stand on its own even before a full memory operation explorer lands. - -When `memory_operations` is added, `memory_extraction_jobs` should gain an optional `operation_id` FK rather than forcing operators to infer the relationship. - -## Lifecycle - -### States - -The persisted lifecycle is intentionally small: - -- `queued` -- `running` -- `succeeded` -- `failed` -- `cancelled` - -`stuck` is **not** a persisted status. It is a derived operator state: - -- `status = running` -- `lease_expires_at < now()` - -That keeps the state machine simple while still giving the board a meaningful "this needs attention" filter. - -### State transitions - -1. `queued -> running` - The in-process dispatcher claims the job transactionally, sets `started_at`, and sets `lease_expires_at`. - -2. `running -> succeeded` - The provider or worker completes successfully. Persist `finished_at`, `result_summary`, optional `result_json`, and usage/cost summary. - -3. `running -> failed` - The provider errors, a callback/poll path reports failure, or the lease expires and the recovery sweep marks the job failed with a stable `error_code`. - -4. `queued -> cancelled` - Operator or system cancellation before execution starts. - -5. `running -> cancelled` - Allowed only when the provider or host-side runner can stop safely. If cancellation is best-effort only, the row should still be marked cancelled and late provider callbacks should be ignored. - -6. `failed -> rerun` - Not an in-place state transition. Create a new `queued` row with `retry_of_job_id` pointing to the failed attempt. - -### Failure and stuck handling - -V1 should use conservative recovery: - -- expired `running` jobs are surfaced as derived `stuck` -- the scheduler recovery sweep marks them `failed` with `error_code = "lease_expired"` or `error_code = "worker_lost"` -- operators can rerun the job from the detail view - -This avoids silent duplicate compaction/capture work after a server restart. - -## Run And Cost Attribution - -Memory jobs should integrate with the current Paperclip run/cost model without pretending to be heartbeat runs. - -### Run model - -- A memory job does **not** create a `heartbeat_run`. -- If the job came from a heartbeat hook, the job stores `source_heartbeat_run_id` and links back to the originating run. -- For maintenance jobs like dreaming or compaction, `source_heartbeat_run_id` may be null while `source_agent_id` or only company scope is present. - -### Cost model - -`cost_events` remains the canonical spend ledger. - -The job row should store a summarized view of usage and cost so the operator can inspect one row without joining multiple financial tables. - -Rules: - -1. If the memory work was already billed inside the originating heartbeat run, store: - - `attribution_mode = "included_in_run"` - - `cost_cents` as the summarized included amount if the provider reports it, otherwise `0` - - `source_heartbeat_run_id` linking back to the run ledger - -2. If the provider makes direct metered calls outside the run accounting path, store: - - `attribution_mode = "billed_directly"` - - summarized usage on the job row - - one or more normal `cost_events` with the same company/agent/issue/project/goal provenance when available - -3. Flat provider subscription costs stay in `finance_events`, not on job rows except as attribution metadata. - -The memory job list is for operational visibility. Finance surfaces still aggregate from `cost_events` and `finance_events`. - -## Dispatch Model - -### V1 dispatcher - -V1 should use an in-process server-owned dispatcher with the same product posture as the rest of the current scheduler work. - -Recommended service split: - -- `MemoryJobStore` - - enqueue job - - claim next queued job - - complete/fail/cancel job - - list/filter/get jobs - -- `MemoryJobDispatcher` - - polls for queued jobs - - claims a job transactionally - - resolves the memory binding/provider - - executes the appropriate capture/compaction/dreaming handler - - records result, usage, and failure state - -- `MemoryJobRecovery` - - sweeps expired leases - - marks stale `running` jobs failed - - leaves rerun as an explicit operator action in V1 - -### Why a lease matters in V1 - -Even with one in-process dispatcher, the record shape should assume that workers can disappear between `running` and terminal state. - -`lease_expires_at` gives V1: - -- a clean way to derive `stuck` -- deterministic startup recovery -- a seamless transition to future host-side workers that use the same claim protocol - -### Seam for host-side workers later - -The host-side worker version should reuse the same table and lifecycle: - -- workers still claim `queued` rows -- workers still extend or complete leases -- callbacks or pollers still update the same job row - -The seam is **where the claim/execute loop lives**, not a different product model. - -## Which Work Enqueues Memory Jobs - -### Capture - -Use `memory_extraction_jobs` when post-run or post-comment capture cannot finish synchronously in the originating request path. - -Examples: - -- provider-managed extraction that returns a provider job id -- delayed embedding/indexing -- large comment/document capture that should not block the request - -### Compaction - -Use the job table for deterministic maintenance like: - -- dedupe -- merge -- index rebuild -- cleanup of stale records - -These are maintenance tasks, not issue work by default. - -### Dreaming - -Use the same table for periodic higher-level memory maintenance that still stays inside the memory subsystem: - -- nightly synthesis -- relationship consolidation -- profile refresh - -If the work needs real reasoning, authored output, or review, that is the point where it stops being "just a memory job" and should escalate. - -## Escalation Rules - -Most memory jobs should remain job records. Escalation is for human-facing follow-up, not for ordinary maintenance noise. - -### Stay as a job record - -Keep the result inside `memory_extraction_jobs` when: - -- capture succeeded -- compaction succeeded -- dreaming succeeded -- a job failed once but the next action is simply rerun or inspect in the memory UI -- the output is operational metadata, not company work - -### Create an issue directly - -Create an issue when: - -- a binding is misconfigured and future jobs will keep failing -- the same job class fails repeatedly and needs product/engineering intervention -- compaction or dreaming finds data quality anomalies that require review -- a rerun cannot proceed because the required source is gone or corrupted - -The issue should link back to the originating memory job id and source issue/run where available. - -### Trigger a routine - -Use a routine only when the desired outcome is recurring company work rather than maintenance. - -Examples: - -- nightly leadership digest built from memory state -- recurring agent task to review suspicious memory merges -- periodic audit report that should appear as issue work - -The key rule is: - -- memory maintenance stays a memory job -- recurring deliverable-producing work becomes a routine - -## Operator Visibility - -### Minimum API surface - -Add company-scoped memory job endpoints: - -- `GET /api/companies/:companyId/memory/jobs` -- `GET /api/companies/:companyId/memory/jobs/:jobId` -- `POST /api/companies/:companyId/memory/jobs/:jobId/rerun` - -Initial list filters should include: - -- `status` -- `effectiveState=stuck` -- `bindingKey` -- `operationType` -- `agentId` -- `issueId` -- `runId` -- date range - -The list response should include enough summary data for a table row: - -- status / effective state -- operation type -- binding key -- source issue/run links -- started/finished timestamps -- cost summary -- result summary or error -- retry count - -### Minimum board UI - -The first UI does not need a full "memory dashboard". It does need an operator-visible failure surface. - -Minimum board behavior: - -- a memory jobs list reachable from the future memory settings/operations area -- filters for failed and stuck work -- list rows that link back to issue and run provenance -- detail view with error, usage, provider job id, and rerun action -- optional dashboard badge/count for failed or stuck jobs - -### Rerun semantics - -`rerun` should: - -- validate the source can still be resolved or that the job is self-contained -- create a new `queued` row -- set `retry_of_job_id` -- preserve original provenance - -It should not mutate the old row, clear its error, or hide prior operator history. - -## Explicit Non-Goals - -This design deliberately does **not** do the following: - -- no generic Paperclip-wide job engine -- no Bull/Redis queue infrastructure -- no coupling memory jobs to `plugin_jobs` / `plugin_job_runs` -- no requirement for a system agent to perform deterministic memory maintenance -- no silent conversion of every memory failure into an issue -- no full provider corpus mirroring in Postgres - -## Recommended Implementation Follow-Ups - -The next implementation work should be split into separate child issues: - -1. **Schema and contracts** - - add `memory_extraction_jobs` schema - - add shared types/constants/validators - - add list/detail/rerun API contracts - -2. **Dispatcher and recovery** - - implement `MemoryJobStore` - - implement in-process dispatcher claim/run/complete flow - - add lease-expiry recovery and derived stuck detection - -3. **First async memory path** - - wire async post-run capture to enqueue memory jobs - - persist provenance, result summaries, and cost attribution - - validate rerun behavior for capture jobs - -4. **Board visibility** - - add memory jobs list/detail UI - - expose failed/stuck filters - - add rerun action for eligible jobs - -## Bottom Line - -The first memory job state should be a narrow, company-scoped, domain-specific async job record. - -It should: - -- feel native to Paperclip's existing run/cost/audit model -- keep deterministic memory maintenance out of the issue system by default -- preserve enough provenance for operators to debug failures -- stay simple enough that V1 can run it in-process today and move it to host-side workers later without changing the product model