diff --git a/doc/plans/2026-04-20-background-jobs-mental-model.md b/doc/plans/2026-04-20-background-jobs-mental-model.md new file mode 100644 index 00000000000..65643cd0c6f --- /dev/null +++ b/doc/plans/2026-04-20-background-jobs-mental-model.md @@ -0,0 +1,300 @@ +# Background Jobs Mental Model + +## Goal + +Define how Paperclip should think about "jobs that run" without introducing a second orchestration system beside issues, routines, and heartbeat runs. + +The immediate design target is memory maintenance and similar background work that must be: + +- framework-owned +- auditable +- deployable outside a single local machine +- compatible with plugin-provided behavior + +## Recommendation In One Sentence + +Paperclip should treat background automation as a small taxonomy of existing control-plane primitives, not as one new generic queue abstraction: use agent timer heartbeats to wake agents, routines to create recurring issue work, plugin or system jobs for deterministic host-side maintenance, and explicit domain job tables when a subsystem like memory needs auditable async work that is not itself an issue. + +## Current Reality In The Repo + +Paperclip already has four adjacent mechanisms: + +1. **Agent timer heartbeats** + - Defined in agent adapter config. + - Purpose: wake an agent on a schedule. + - Output: `heartbeat_runs`. + - This is a wake policy, not a business-work model. + +2. **Routines** + - Company-scoped and project/goal/issue-aware. + - Purpose: recurring automation that should create or continue issue work. + - Output: `routine_runs`, linked issues, then heartbeat runs. + - This is the auditable recurring-task path. + +3. **Plugin scheduled jobs** + - Plugin-owned host-side handlers scheduled by the host. + - Purpose: deterministic plugin maintenance, sync, cleanup, polling. + - Output: `plugin_job_runs`. + - This is infrastructure automation, not issue orchestration. + +4. **Server-owned maintenance loops** + - Examples: heartbeat recovery, routine ticking, database backups. + - Purpose: keep the Paperclip control plane healthy. + - Output today is mostly logs plus subsystem-specific records. + - This is framework maintenance, not user-facing work management. + +The problem is not that Paperclip has no job model. The problem is that these mechanisms need an explicit boundary so new systems like memory do not accidentally invent a fifth orchestration surface. + +## Core Product Decision + +Paperclip should not add a generic "jobs engine" that tries to flatten all background work into one abstraction. + +Why: + +- V1 already explicitly allows a lightweight in-process scheduler/worker and does not require queue infrastructure. +- Paperclip's differentiator is company-scoped orchestration, not general cron/queue infrastructure. +- A generic Bull or Redis queue layer would duplicate ownership with routines, issues, heartbeat runs, and plugin jobs. +- The repo already has a documented bias against importing runtime-level workflow or queue abstractions into the control plane without mapping them back to Paperclip entities. + +The right move is to sharpen the taxonomy and add only the missing domain-specific records where needed. + +## The Taxonomy + +### 1. Agent timer heartbeat + +Use when the question is: + +- "When should this agent wake up and look for work?" + +Characteristics: + +- executor is an agent adapter +- unit of execution is a `heartbeat_run` +- schedule belongs to the agent runtime config +- may or may not result in issue activity + +Do not use this as the default model for deterministic maintenance like compaction, syncing, or shell scripts. + +### 2. Routine + +Use when the question is: + +- "What recurring work should become an issue in the company?" + +Characteristics: + +- company-scoped +- goal/project/parent-issue aware +- human-visible and auditable as work +- creates or coalesces into issue execution +- hands off to the existing issue and heartbeat execution path + +Examples: + +- weekly CEO review +- nightly "summarize customer feedback" task +- recurring research sweep +- scheduled agent-authored report + +If the output should be a task someone can inspect, comment on, block, review, or approve, it should usually be a routine. + +### 3. Plugin job or system job + +Use when the question is: + +- "What host-side maintenance should run without becoming an issue by default?" + +Characteristics: + +- deterministic host-side execution +- run by plugin worker or server-owned code +- recorded as job-run history, not as issue work +- appropriate for cleanup, polling, reconciliation, compaction, indexing, and sync + +Examples: + +- refresh connector metadata +- poll an external API and cache results +- compact a local index +- database backups +- plugin-specific cleanup or reconciliation + +These jobs are real framework-owned automation, but they are not the same thing as company task execution. + +### 4. Domain-specific async job record + +Use when the question is: + +- "This subsystem needs async background work with auditability, provenance, and operator visibility, but it should not become an issue every time." + +This is the missing piece for memory. + +Examples: + +- memory extraction +- memory compaction +- dedupe / correction pipelines +- provider-managed async indexing callbacks + +For these cases, the right answer is not "pretend everything is an issue" and not "hide everything inside a plugin job." + +The right answer is: + +- keep the scheduler/executor simple +- add subsystem-specific run records that carry Paperclip provenance +- escalate into issue work only when human-facing follow-up is required + +This matches the existing memory plan's direction toward explicit extraction jobs. + +## Decision Rules + +When a new background process is proposed, ask these questions in order: + +1. **Should this show up as company work?** + - If yes, it should be a routine that creates issue work. + - If no, continue. + +2. **Does this require language-model reasoning or agent tool use?** + - If yes, dispatch to an agent through the normal issue/heartbeat path. + - If no, continue. + +3. **Is this deterministic host-side maintenance owned by the framework or a plugin?** + - If yes, use a plugin job or server-owned maintenance loop. + - If no, continue. + +4. **Does the subsystem still need first-class auditability and provenance even though it is not issue work?** + - If yes, add a domain-specific async job record. + +This keeps the control plane authoritative without forcing one storage shape onto every kind of automation. + +## What This Means For Memory + +Memory should use three different execution shapes depending on the operation. + +### A. Pre-run hydrate + +Use the heartbeat path directly. + +- Trigger: before an agent heartbeat starts +- Executor: memory provider call in the run path +- Attribution: included in the run or directly metered if the provider bills separately +- Why: this is part of prompt assembly, not a background scheduler concern + +### B. Post-run capture + +Use one of two paths: + +- synchronous capture in the run path for cheap/local providers +- asynchronous memory extraction job for heavier provider-managed or delayed work + +Why: + +- some providers can write immediately +- others need extraction, chunking, or delayed indexing +- Paperclip should not hide that async work if it matters for auditability or failure visibility + +### C. Periodic memory "sleep cycle" / dreaming / compaction + +Default to a memory-domain background job, not a system agent and not a routine. + +Why: + +- this is maintenance on the memory subsystem +- it may need to scan or compact provider state +- it often does not map to a user-facing issue +- forcing it through an agent adds reasoning cost and muddles ownership + +If that maintenance discovers something that should become company work, it should then escalate by: + +- creating an issue directly, or +- triggering a routine that creates issue work + +Examples: + +- "memory compaction completed" stays a memory job record +- "memory system found duplicate identities needing review" becomes an issue +- "nightly memory digest for leadership" becomes a routine because the output is work product + +## Plugin Recommendation + +Plugins should be able to participate in both layers, but the layers should stay distinct. + +### Plugin maintenance + +Plugins may continue to register scheduled jobs for host-side deterministic work. + +Good fit: + +- sync +- compaction +- polling +- cleanup +- provider callbacks + +### Plugin-created recurring company work + +If a plugin wants recurring work to enter the company task system, it should register or provision a routine, and the routine should record that provenance. + +Recommended future addition: + +- add routine provenance metadata such as `registeredByPluginId` or equivalent source metadata + +That gives us the useful semantics behind "a plugin registered this routine" without collapsing routines into plugin jobs. + +### Why not use plugin jobs alone for memory maintenance? + +Because memory is company-scoped in Paperclip's product model, while plugin jobs today are plugin-owned host jobs with no first-class company targeting in their core record shape. + +That makes plugin jobs a good execution mechanism, but not the full product model for company-auditable memory operations. + +## System Agents + +Paperclip may eventually want system agents, but they should be optional and narrow. + +Good reasons to use a system agent: + +- the work genuinely needs reasoning +- the output should look like authored task work +- budget attribution should land on an agent +- approvals, comments, or review are part of the loop + +Bad reason to use a system agent: + +- "we need to run a shell script every few hours" + +That is deterministic maintenance and should stay a job, not become fake agent labor. + +## Proposed Product Shape + +Paperclip should describe background automation using three top-level concepts: + +1. **Wakeups** + - wake an existing executor, usually an agent + +2. **Recurring work** + - routines that create or continue issue execution + +3. **Background maintenance** + - plugin jobs, server jobs, and domain-specific async jobs + +These can later share a UI grouping like "Automation" or "Background Activity" without requiring a single underlying table. + +## Recommended Near-Term Follow-Up + +1. Document this taxonomy in product and engineering docs where routines, plugins, and memory are discussed. +2. Add provenance metadata for routines created or managed by plugins. +3. When memory implementation begins, add explicit memory extraction job state instead of hiding async maintenance inside generic logs. +4. Keep the scheduler in-process for now; do not add Redis/Bull-style queue infrastructure unless scale or deployment evidence forces it. + +## Bottom Line + +Paperclip should not think of "jobs" as one new generic abstraction. + +It should think of them as: + +- agent wakeups when we need an agent to run +- routines when recurring automation should become issue work +- host-side jobs when the framework or a plugin needs deterministic maintenance +- domain-specific async records when a subsystem like memory needs auditable background execution without becoming a task every time + +That preserves Paperclip's control-plane model, keeps deployment complexity low, and gives memory a clean place to live. diff --git a/doc/plans/2026-04-20-memory-extraction-job-state.md b/doc/plans/2026-04-20-memory-extraction-job-state.md new file mode 100644 index 00000000000..7ebd23051d3 --- /dev/null +++ b/doc/plans/2026-04-20-memory-extraction-job-state.md @@ -0,0 +1,423 @@ +# Memory Extraction Job State + +Status: proposed design +Date: 2026-04-20 +Owner issue: `PAP-1702` +Related docs: + +- `doc/plans/2026-04-20-background-jobs-mental-model.md` +- `doc/plans/2026-03-17-memory-service-surface-api.md` + +## Goal + +Define the first domain-specific async job record in Paperclip: a company-scoped memory job state model for asynchronous capture, compaction, and dreaming work that must be auditable without becoming issue work by default. + +This design keeps the background-jobs taxonomy from `2026-04-20-background-jobs-mental-model.md` intact: + +- no generic queue engine +- no reuse of plugin job tables as the product model +- no requirement to invent a system agent just to run deterministic maintenance + +## Decision Summary + +Paperclip should add a `memory_extraction_jobs` table whose rows represent **job attempts**, not recurring definitions and not a generic workflow engine. + +The row is the durable audit object for async memory work that sits beside, but does not replace: + +- `heartbeat_runs` for agent execution +- `routine_runs` for recurring issue work +- `plugin_job_runs` for plugin-owned host maintenance +- `cost_events` for canonical spend accounting + +The row should carry enough provenance to answer: + +- what memory work happened +- what company, agent, issue, and run it came from +- which memory binding handled it +- what it cost +- whether it succeeded, failed, or needs operator attention + +Retries should create a **new** job row with `retryOfJobId` instead of mutating a terminal row back to `queued`. + +## Why A Domain Job Record Is The Right Layer + +Memory is the first concrete subsystem that needs background work with all of the following: + +- company scope +- auditability +- operator visibility +- attribution back to issue and run context +- optional provider-managed asynchronous completion + +That makes it a poor fit for: + +- issues, because most memory capture/compaction work is not human-facing task work +- plugin jobs, because those are plugin-owned host jobs rather than company-auditable memory records +- system agents, because capture/compaction/dreaming are deterministic maintenance unless they explicitly need reasoning + +## Proposed Schema + +### Minimum persisted row + +The minimum record should be a new `memory_extraction_jobs` table with these fields. + +| Field | Purpose | +| --- | --- | +| `id` | Stable job-attempt id | +| `company_id` | Company scope boundary | +| `binding_id` | FK to the resolved memory binding | +| `binding_key` | Stable operator-facing binding identifier snapshot | +| `operation_type` | `capture | compaction | dreaming` | +| `status` | `queued | running | succeeded | failed | cancelled` | +| `source_agent_id` | Agent provenance when present | +| `source_issue_id` | Issue provenance when present | +| `source_project_id` | Project provenance when present | +| `source_goal_id` | Goal provenance when present | +| `source_heartbeat_run_id` | Run provenance when present | +| `hook_kind` | `post_run_capture`, `issue_comment_capture`, `issue_document_capture`, or null for pure maintenance jobs | +| `provider_job_id` | Provider-native async job id when one exists | +| `submitted_at` | When Paperclip enqueued the job | +| `started_at` | When a worker claimed execution | +| `finished_at` | Terminal timestamp | +| `attribution_mode` | `included_in_run | billed_directly | external_invoice | untracked` | +| `cost_cents` | Job-level billed cost summary for operator visibility | +| `result_summary` | Human-readable summary suitable for a list row | +| `error_code` | Stable failure classifier | +| `error` | Human-readable failure detail | +| `created_at` / `updated_at` | Audit timestamps | + +### Supporting fields needed for safe dispatch and rerun + +The following fields are not part of the problem statement's minimum list, but they make the V1 design operationally sound: + +| Field | Why it exists | +| --- | --- | +| `source_kind` | Distinguish run/comment/document/manual/maintenance origins | +| `source_ref_json` | Hold comment id, document key, subject id, namespace, and other source-specific identifiers without over-normalizing the first slice | +| `retry_of_job_id` | Preserve attempt history when a job is rerun | +| `attempt_number` | Make retries visible in the list and detail surfaces | +| `dispatcher_kind` | Start with `in_process`, preserve a clean seam for later host-side workers | +| `lease_expires_at` | Lets the scheduler detect stale `running` work without adding a new status | +| `usage_json` | Store token/latency/provider usage summary without overextending `cost_events` in the first slice | +| `result_json` | Structured output such as record counts, callback payload, or escalation metadata | + +### Notes on normalization + +- `binding_id` should be the authoritative FK. +- `binding_key` should still be stored on the row so operators can read history even if the binding is later renamed or deleted. +- `source_ref_json` should stay lightweight. Do not snapshot full transcript bodies here. +- `result_summary` should be short and readable, for example `Captured 14 records from run residue` or `Compaction found 3 duplicate record groups`. + +## Relationship To `memory_operations` + +The earlier memory plan introduced `memory_operations` as the broader audit trail for all memory actions. This design does not replace that. + +The cleanest end state is: + +- `memory_operations` logs every memory action, synchronous or asynchronous +- `memory_extraction_jobs` tracks async job-attempt state for the subset of operations that run outside the initiating request path + +For the first implementation slice, the job record should include enough provenance and usage fields to stand on its own even before a full memory operation explorer lands. + +When `memory_operations` is added, `memory_extraction_jobs` should gain an optional `operation_id` FK rather than forcing operators to infer the relationship. + +## Lifecycle + +### States + +The persisted lifecycle is intentionally small: + +- `queued` +- `running` +- `succeeded` +- `failed` +- `cancelled` + +`stuck` is **not** a persisted status. It is a derived operator state: + +- `status = running` +- `lease_expires_at < now()` + +That keeps the state machine simple while still giving the board a meaningful "this needs attention" filter. + +### State transitions + +1. `queued -> running` + The in-process dispatcher claims the job transactionally, sets `started_at`, and sets `lease_expires_at`. + +2. `running -> succeeded` + The provider or worker completes successfully. Persist `finished_at`, `result_summary`, optional `result_json`, and usage/cost summary. + +3. `running -> failed` + The provider errors, a callback/poll path reports failure, or the lease expires and the recovery sweep marks the job failed with a stable `error_code`. + +4. `queued -> cancelled` + Operator or system cancellation before execution starts. + +5. `running -> cancelled` + Allowed only when the provider or host-side runner can stop safely. If cancellation is best-effort only, the row should still be marked cancelled and late provider callbacks should be ignored. + +6. `failed -> rerun` + Not an in-place state transition. Create a new `queued` row with `retry_of_job_id` pointing to the failed attempt. + +### Failure and stuck handling + +V1 should use conservative recovery: + +- expired `running` jobs are surfaced as derived `stuck` +- the scheduler recovery sweep marks them `failed` with `error_code = "lease_expired"` or `error_code = "worker_lost"` +- operators can rerun the job from the detail view + +This avoids silent duplicate compaction/capture work after a server restart. + +## Run And Cost Attribution + +Memory jobs should integrate with the current Paperclip run/cost model without pretending to be heartbeat runs. + +### Run model + +- A memory job does **not** create a `heartbeat_run`. +- If the job came from a heartbeat hook, the job stores `source_heartbeat_run_id` and links back to the originating run. +- For maintenance jobs like dreaming or compaction, `source_heartbeat_run_id` may be null while `source_agent_id` or only company scope is present. + +### Cost model + +`cost_events` remains the canonical spend ledger. + +The job row should store a summarized view of usage and cost so the operator can inspect one row without joining multiple financial tables. + +Rules: + +1. If the memory work was already billed inside the originating heartbeat run, store: + - `attribution_mode = "included_in_run"` + - `cost_cents` as the summarized included amount if the provider reports it, otherwise `0` + - `source_heartbeat_run_id` linking back to the run ledger + +2. If the provider makes direct metered calls outside the run accounting path, store: + - `attribution_mode = "billed_directly"` + - summarized usage on the job row + - one or more normal `cost_events` with the same company/agent/issue/project/goal provenance when available + +3. Flat provider subscription costs stay in `finance_events`, not on job rows except as attribution metadata. + +The memory job list is for operational visibility. Finance surfaces still aggregate from `cost_events` and `finance_events`. + +## Dispatch Model + +### V1 dispatcher + +V1 should use an in-process server-owned dispatcher with the same product posture as the rest of the current scheduler work. + +Recommended service split: + +- `MemoryJobStore` + - enqueue job + - claim next queued job + - complete/fail/cancel job + - list/filter/get jobs + +- `MemoryJobDispatcher` + - polls for queued jobs + - claims a job transactionally + - resolves the memory binding/provider + - executes the appropriate capture/compaction/dreaming handler + - records result, usage, and failure state + +- `MemoryJobRecovery` + - sweeps expired leases + - marks stale `running` jobs failed + - leaves rerun as an explicit operator action in V1 + +### Why a lease matters in V1 + +Even with one in-process dispatcher, the record shape should assume that workers can disappear between `running` and terminal state. + +`lease_expires_at` gives V1: + +- a clean way to derive `stuck` +- deterministic startup recovery +- a seamless transition to future host-side workers that use the same claim protocol + +### Seam for host-side workers later + +The host-side worker version should reuse the same table and lifecycle: + +- workers still claim `queued` rows +- workers still extend or complete leases +- callbacks or pollers still update the same job row + +The seam is **where the claim/execute loop lives**, not a different product model. + +## Which Work Enqueues Memory Jobs + +### Capture + +Use `memory_extraction_jobs` when post-run or post-comment capture cannot finish synchronously in the originating request path. + +Examples: + +- provider-managed extraction that returns a provider job id +- delayed embedding/indexing +- large comment/document capture that should not block the request + +### Compaction + +Use the job table for deterministic maintenance like: + +- dedupe +- merge +- index rebuild +- cleanup of stale records + +These are maintenance tasks, not issue work by default. + +### Dreaming + +Use the same table for periodic higher-level memory maintenance that still stays inside the memory subsystem: + +- nightly synthesis +- relationship consolidation +- profile refresh + +If the work needs real reasoning, authored output, or review, that is the point where it stops being "just a memory job" and should escalate. + +## Escalation Rules + +Most memory jobs should remain job records. Escalation is for human-facing follow-up, not for ordinary maintenance noise. + +### Stay as a job record + +Keep the result inside `memory_extraction_jobs` when: + +- capture succeeded +- compaction succeeded +- dreaming succeeded +- a job failed once but the next action is simply rerun or inspect in the memory UI +- the output is operational metadata, not company work + +### Create an issue directly + +Create an issue when: + +- a binding is misconfigured and future jobs will keep failing +- the same job class fails repeatedly and needs product/engineering intervention +- compaction or dreaming finds data quality anomalies that require review +- a rerun cannot proceed because the required source is gone or corrupted + +The issue should link back to the originating memory job id and source issue/run where available. + +### Trigger a routine + +Use a routine only when the desired outcome is recurring company work rather than maintenance. + +Examples: + +- nightly leadership digest built from memory state +- recurring agent task to review suspicious memory merges +- periodic audit report that should appear as issue work + +The key rule is: + +- memory maintenance stays a memory job +- recurring deliverable-producing work becomes a routine + +## Operator Visibility + +### Minimum API surface + +Add company-scoped memory job endpoints: + +- `GET /api/companies/:companyId/memory/jobs` +- `GET /api/companies/:companyId/memory/jobs/:jobId` +- `POST /api/companies/:companyId/memory/jobs/:jobId/rerun` + +Initial list filters should include: + +- `status` +- `effectiveState=stuck` +- `bindingKey` +- `operationType` +- `agentId` +- `issueId` +- `runId` +- date range + +The list response should include enough summary data for a table row: + +- status / effective state +- operation type +- binding key +- source issue/run links +- started/finished timestamps +- cost summary +- result summary or error +- retry count + +### Minimum board UI + +The first UI does not need a full "memory dashboard". It does need an operator-visible failure surface. + +Minimum board behavior: + +- a memory jobs list reachable from the future memory settings/operations area +- filters for failed and stuck work +- list rows that link back to issue and run provenance +- detail view with error, usage, provider job id, and rerun action +- optional dashboard badge/count for failed or stuck jobs + +### Rerun semantics + +`rerun` should: + +- validate the source can still be resolved or that the job is self-contained +- create a new `queued` row +- set `retry_of_job_id` +- preserve original provenance + +It should not mutate the old row, clear its error, or hide prior operator history. + +## Explicit Non-Goals + +This design deliberately does **not** do the following: + +- no generic Paperclip-wide job engine +- no Bull/Redis queue infrastructure +- no coupling memory jobs to `plugin_jobs` / `plugin_job_runs` +- no requirement for a system agent to perform deterministic memory maintenance +- no silent conversion of every memory failure into an issue +- no full provider corpus mirroring in Postgres + +## Recommended Implementation Follow-Ups + +The next implementation work should be split into separate child issues: + +1. **Schema and contracts** + - add `memory_extraction_jobs` schema + - add shared types/constants/validators + - add list/detail/rerun API contracts + +2. **Dispatcher and recovery** + - implement `MemoryJobStore` + - implement in-process dispatcher claim/run/complete flow + - add lease-expiry recovery and derived stuck detection + +3. **First async memory path** + - wire async post-run capture to enqueue memory jobs + - persist provenance, result summaries, and cost attribution + - validate rerun behavior for capture jobs + +4. **Board visibility** + - add memory jobs list/detail UI + - expose failed/stuck filters + - add rerun action for eligible jobs + +## Bottom Line + +The first memory job state should be a narrow, company-scoped, domain-specific async job record. + +It should: + +- feel native to Paperclip's existing run/cost/audit model +- keep deterministic memory maintenance out of the issue system by default +- preserve enough provenance for operators to debug failures +- stay simple enough that V1 can run it in-process today and move it to host-side workers later without changing the product model