mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 08:22:20 +02:00
chore(docs): remove planning docs from adapter parser PR
This commit is contained in:
Dotta committed
1 parent
b554542c75
commit
c17a01e949
2 files changed
-723
No files matched your search
@@ -1,300 +0,0 @@
|
||||
# Background Jobs Mental Model
|
||||
|
||||
## Goal
|
||||
|
||||
Define how Paperclip should think about "jobs that run" without introducing a second orchestration system beside issues, routines, and heartbeat runs.
|
||||
|
||||
The immediate design target is memory maintenance and similar background work that must be:
|
||||
|
||||
- framework-owned
|
||||
- auditable
|
||||
- deployable outside a single local machine
|
||||
- compatible with plugin-provided behavior
|
||||
|
||||
## Recommendation In One Sentence
|
||||
|
||||
Paperclip should treat background automation as a small taxonomy of existing control-plane primitives, not as one new generic queue abstraction: use agent timer heartbeats to wake agents, routines to create recurring issue work, plugin or system jobs for deterministic host-side maintenance, and explicit domain job tables when a subsystem like memory needs auditable async work that is not itself an issue.
|
||||
|
||||
## Current Reality In The Repo
|
||||
|
||||
Paperclip already has four adjacent mechanisms:
|
||||
|
||||
1. **Agent timer heartbeats**
|
||||
- Defined in agent adapter config.
|
||||
- Purpose: wake an agent on a schedule.
|
||||
- Output: `heartbeat_runs`.
|
||||
- This is a wake policy, not a business-work model.
|
||||
|
||||
2. **Routines**
|
||||
- Company-scoped and project/goal/issue-aware.
|
||||
- Purpose: recurring automation that should create or continue issue work.
|
||||
- Output: `routine_runs`, linked issues, then heartbeat runs.
|
||||
- This is the auditable recurring-task path.
|
||||
|
||||
3. **Plugin scheduled jobs**
|
||||
- Plugin-owned host-side handlers scheduled by the host.
|
||||
- Purpose: deterministic plugin maintenance, sync, cleanup, polling.
|
||||
- Output: `plugin_job_runs`.
|
||||
- This is infrastructure automation, not issue orchestration.
|
||||
|
||||
4. **Server-owned maintenance loops**
|
||||
- Examples: heartbeat recovery, routine ticking, database backups.
|
||||
- Purpose: keep the Paperclip control plane healthy.
|
||||
- Output today is mostly logs plus subsystem-specific records.
|
||||
- This is framework maintenance, not user-facing work management.
|
||||
|
||||
The problem is not that Paperclip has no job model. The problem is that these mechanisms need an explicit boundary so new systems like memory do not accidentally invent a fifth orchestration surface.
|
||||
|
||||
## Core Product Decision
|
||||
|
||||
Paperclip should not add a generic "jobs engine" that tries to flatten all background work into one abstraction.
|
||||
|
||||
Why:
|
||||
|
||||
- V1 already explicitly allows a lightweight in-process scheduler/worker and does not require queue infrastructure.
|
||||
- Paperclip's differentiator is company-scoped orchestration, not general cron/queue infrastructure.
|
||||
- A generic Bull or Redis queue layer would duplicate ownership with routines, issues, heartbeat runs, and plugin jobs.
|
||||
- The repo already has a documented bias against importing runtime-level workflow or queue abstractions into the control plane without mapping them back to Paperclip entities.
|
||||
|
||||
The right move is to sharpen the taxonomy and add only the missing domain-specific records where needed.
|
||||
|
||||
## The Taxonomy
|
||||
|
||||
### 1. Agent timer heartbeat
|
||||
|
||||
Use when the question is:
|
||||
|
||||
- "When should this agent wake up and look for work?"
|
||||
|
||||
Characteristics:
|
||||
|
||||
- executor is an agent adapter
|
||||
- unit of execution is a `heartbeat_run`
|
||||
- schedule belongs to the agent runtime config
|
||||
- may or may not result in issue activity
|
||||
|
||||
Do not use this as the default model for deterministic maintenance like compaction, syncing, or shell scripts.
|
||||
|
||||
### 2. Routine
|
||||
|
||||
Use when the question is:
|
||||
|
||||
- "What recurring work should become an issue in the company?"
|
||||
|
||||
Characteristics:
|
||||
|
||||
- company-scoped
|
||||
- goal/project/parent-issue aware
|
||||
- human-visible and auditable as work
|
||||
- creates or coalesces into issue execution
|
||||
- hands off to the existing issue and heartbeat execution path
|
||||
|
||||
Examples:
|
||||
|
||||
- weekly CEO review
|
||||
- nightly "summarize customer feedback" task
|
||||
- recurring research sweep
|
||||
- scheduled agent-authored report
|
||||
|
||||
If the output should be a task someone can inspect, comment on, block, review, or approve, it should usually be a routine.
|
||||
|
||||
### 3. Plugin job or system job
|
||||
|
||||
Use when the question is:
|
||||
|
||||
- "What host-side maintenance should run without becoming an issue by default?"
|
||||
|
||||
Characteristics:
|
||||
|
||||
- deterministic host-side execution
|
||||
- run by plugin worker or server-owned code
|
||||
- recorded as job-run history, not as issue work
|
||||
- appropriate for cleanup, polling, reconciliation, compaction, indexing, and sync
|
||||
|
||||
Examples:
|
||||
|
||||
- refresh connector metadata
|
||||
- poll an external API and cache results
|
||||
- compact a local index
|
||||
- database backups
|
||||
- plugin-specific cleanup or reconciliation
|
||||
|
||||
These jobs are real framework-owned automation, but they are not the same thing as company task execution.
|
||||
|
||||
### 4. Domain-specific async job record
|
||||
|
||||
Use when the question is:
|
||||
|
||||
- "This subsystem needs async background work with auditability, provenance, and operator visibility, but it should not become an issue every time."
|
||||
|
||||
This is the missing piece for memory.
|
||||
|
||||
Examples:
|
||||
|
||||
- memory extraction
|
||||
- memory compaction
|
||||
- dedupe / correction pipelines
|
||||
- provider-managed async indexing callbacks
|
||||
|
||||
For these cases, the right answer is not "pretend everything is an issue" and not "hide everything inside a plugin job."
|
||||
|
||||
The right answer is:
|
||||
|
||||
- keep the scheduler/executor simple
|
||||
- add subsystem-specific run records that carry Paperclip provenance
|
||||
- escalate into issue work only when human-facing follow-up is required
|
||||
|
||||
This matches the existing memory plan's direction toward explicit extraction jobs.
|
||||
|
||||
## Decision Rules
|
||||
|
||||
When a new background process is proposed, ask these questions in order:
|
||||
|
||||
1. **Should this show up as company work?**
|
||||
- If yes, it should be a routine that creates issue work.
|
||||
- If no, continue.
|
||||
|
||||
2. **Does this require language-model reasoning or agent tool use?**
|
||||
- If yes, dispatch to an agent through the normal issue/heartbeat path.
|
||||
- If no, continue.
|
||||
|
||||
3. **Is this deterministic host-side maintenance owned by the framework or a plugin?**
|
||||
- If yes, use a plugin job or server-owned maintenance loop.
|
||||
- If no, continue.
|
||||
|
||||
4. **Does the subsystem still need first-class auditability and provenance even though it is not issue work?**
|
||||
- If yes, add a domain-specific async job record.
|
||||
|
||||
This keeps the control plane authoritative without forcing one storage shape onto every kind of automation.
|
||||
|
||||
## What This Means For Memory
|
||||
|
||||
Memory should use three different execution shapes depending on the operation.
|
||||
|
||||
### A. Pre-run hydrate
|
||||
|
||||
Use the heartbeat path directly.
|
||||
|
||||
- Trigger: before an agent heartbeat starts
|
||||
- Executor: memory provider call in the run path
|
||||
- Attribution: included in the run or directly metered if the provider bills separately
|
||||
- Why: this is part of prompt assembly, not a background scheduler concern
|
||||
|
||||
### B. Post-run capture
|
||||
|
||||
Use one of two paths:
|
||||
|
||||
- synchronous capture in the run path for cheap/local providers
|
||||
- asynchronous memory extraction job for heavier provider-managed or delayed work
|
||||
|
||||
Why:
|
||||
|
||||
- some providers can write immediately
|
||||
- others need extraction, chunking, or delayed indexing
|
||||
- Paperclip should not hide that async work if it matters for auditability or failure visibility
|
||||
|
||||
### C. Periodic memory "sleep cycle" / dreaming / compaction
|
||||
|
||||
Default to a memory-domain background job, not a system agent and not a routine.
|
||||
|
||||
Why:
|
||||
|
||||
- this is maintenance on the memory subsystem
|
||||
- it may need to scan or compact provider state
|
||||
- it often does not map to a user-facing issue
|
||||
- forcing it through an agent adds reasoning cost and muddles ownership
|
||||
|
||||
If that maintenance discovers something that should become company work, it should then escalate by:
|
||||
|
||||
- creating an issue directly, or
|
||||
- triggering a routine that creates issue work
|
||||
|
||||
Examples:
|
||||
|
||||
- "memory compaction completed" stays a memory job record
|
||||
- "memory system found duplicate identities needing review" becomes an issue
|
||||
- "nightly memory digest for leadership" becomes a routine because the output is work product
|
||||
|
||||
## Plugin Recommendation
|
||||
|
||||
Plugins should be able to participate in both layers, but the layers should stay distinct.
|
||||
|
||||
### Plugin maintenance
|
||||
|
||||
Plugins may continue to register scheduled jobs for host-side deterministic work.
|
||||
|
||||
Good fit:
|
||||
|
||||
- sync
|
||||
- compaction
|
||||
- polling
|
||||
- cleanup
|
||||
- provider callbacks
|
||||
|
||||
### Plugin-created recurring company work
|
||||
|
||||
If a plugin wants recurring work to enter the company task system, it should register or provision a routine, and the routine should record that provenance.
|
||||
|
||||
Recommended future addition:
|
||||
|
||||
- add routine provenance metadata such as `registeredByPluginId` or equivalent source metadata
|
||||
|
||||
That gives us the useful semantics behind "a plugin registered this routine" without collapsing routines into plugin jobs.
|
||||
|
||||
### Why not use plugin jobs alone for memory maintenance?
|
||||
|
||||
Because memory is company-scoped in Paperclip's product model, while plugin jobs today are plugin-owned host jobs with no first-class company targeting in their core record shape.
|
||||
|
||||
That makes plugin jobs a good execution mechanism, but not the full product model for company-auditable memory operations.
|
||||
|
||||
## System Agents
|
||||
|
||||
Paperclip may eventually want system agents, but they should be optional and narrow.
|
||||
|
||||
Good reasons to use a system agent:
|
||||
|
||||
- the work genuinely needs reasoning
|
||||
- the output should look like authored task work
|
||||
- budget attribution should land on an agent
|
||||
- approvals, comments, or review are part of the loop
|
||||
|
||||
Bad reason to use a system agent:
|
||||
|
||||
- "we need to run a shell script every few hours"
|
||||
|
||||
That is deterministic maintenance and should stay a job, not become fake agent labor.
|
||||
|
||||
## Proposed Product Shape
|
||||
|
||||
Paperclip should describe background automation using three top-level concepts:
|
||||
|
||||
1. **Wakeups**
|
||||
- wake an existing executor, usually an agent
|
||||
|
||||
2. **Recurring work**
|
||||
- routines that create or continue issue execution
|
||||
|
||||
3. **Background maintenance**
|
||||
- plugin jobs, server jobs, and domain-specific async jobs
|
||||
|
||||
These can later share a UI grouping like "Automation" or "Background Activity" without requiring a single underlying table.
|
||||
|
||||
## Recommended Near-Term Follow-Up
|
||||
|
||||
1. Document this taxonomy in product and engineering docs where routines, plugins, and memory are discussed.
|
||||
2. Add provenance metadata for routines created or managed by plugins.
|
||||
3. When memory implementation begins, add explicit memory extraction job state instead of hiding async maintenance inside generic logs.
|
||||
4. Keep the scheduler in-process for now; do not add Redis/Bull-style queue infrastructure unless scale or deployment evidence forces it.
|
||||
|
||||
## Bottom Line
|
||||
|
||||
Paperclip should not think of "jobs" as one new generic abstraction.
|
||||
|
||||
It should think of them as:
|
||||
|
||||
- agent wakeups when we need an agent to run
|
||||
- routines when recurring automation should become issue work
|
||||
- host-side jobs when the framework or a plugin needs deterministic maintenance
|
||||
- domain-specific async records when a subsystem like memory needs auditable background execution without becoming a task every time
|
||||
|
||||
That preserves Paperclip's control-plane model, keeps deployment complexity low, and gives memory a clean place to live.
|
||||
@@ -1,423 +0,0 @@
|
||||
# Memory Extraction Job State
|
||||
|
||||
Status: proposed design
|
||||
Date: 2026-04-20
|
||||
Owner issue: `PAP-1702`
|
||||
Related docs:
|
||||
|
||||
- `doc/plans/2026-04-20-background-jobs-mental-model.md`
|
||||
- `doc/plans/2026-03-17-memory-service-surface-api.md`
|
||||
|
||||
## Goal
|
||||
|
||||
Define the first domain-specific async job record in Paperclip: a company-scoped memory job state model for asynchronous capture, compaction, and dreaming work that must be auditable without becoming issue work by default.
|
||||
|
||||
This design keeps the background-jobs taxonomy from `2026-04-20-background-jobs-mental-model.md` intact:
|
||||
|
||||
- no generic queue engine
|
||||
- no reuse of plugin job tables as the product model
|
||||
- no requirement to invent a system agent just to run deterministic maintenance
|
||||
|
||||
## Decision Summary
|
||||
|
||||
Paperclip should add a `memory_extraction_jobs` table whose rows represent **job attempts**, not recurring definitions and not a generic workflow engine.
|
||||
|
||||
The row is the durable audit object for async memory work that sits beside, but does not replace:
|
||||
|
||||
- `heartbeat_runs` for agent execution
|
||||
- `routine_runs` for recurring issue work
|
||||
- `plugin_job_runs` for plugin-owned host maintenance
|
||||
- `cost_events` for canonical spend accounting
|
||||
|
||||
The row should carry enough provenance to answer:
|
||||
|
||||
- what memory work happened
|
||||
- what company, agent, issue, and run it came from
|
||||
- which memory binding handled it
|
||||
- what it cost
|
||||
- whether it succeeded, failed, or needs operator attention
|
||||
|
||||
Retries should create a **new** job row with `retryOfJobId` instead of mutating a terminal row back to `queued`.
|
||||
|
||||
## Why A Domain Job Record Is The Right Layer
|
||||
|
||||
Memory is the first concrete subsystem that needs background work with all of the following:
|
||||
|
||||
- company scope
|
||||
- auditability
|
||||
- operator visibility
|
||||
- attribution back to issue and run context
|
||||
- optional provider-managed asynchronous completion
|
||||
|
||||
That makes it a poor fit for:
|
||||
|
||||
- issues, because most memory capture/compaction work is not human-facing task work
|
||||
- plugin jobs, because those are plugin-owned host jobs rather than company-auditable memory records
|
||||
- system agents, because capture/compaction/dreaming are deterministic maintenance unless they explicitly need reasoning
|
||||
|
||||
## Proposed Schema
|
||||
|
||||
### Minimum persisted row
|
||||
|
||||
The minimum record should be a new `memory_extraction_jobs` table with these fields.
|
||||
|
||||
| Field | Purpose |
|
||||
| --- | --- |
|
||||
| `id` | Stable job-attempt id |
|
||||
| `company_id` | Company scope boundary |
|
||||
| `binding_id` | FK to the resolved memory binding |
|
||||
| `binding_key` | Stable operator-facing binding identifier snapshot |
|
||||
| `operation_type` | `capture | compaction | dreaming` |
|
||||
| `status` | `queued | running | succeeded | failed | cancelled` |
|
||||
| `source_agent_id` | Agent provenance when present |
|
||||
| `source_issue_id` | Issue provenance when present |
|
||||
| `source_project_id` | Project provenance when present |
|
||||
| `source_goal_id` | Goal provenance when present |
|
||||
| `source_heartbeat_run_id` | Run provenance when present |
|
||||
| `hook_kind` | `post_run_capture`, `issue_comment_capture`, `issue_document_capture`, or null for pure maintenance jobs |
|
||||
| `provider_job_id` | Provider-native async job id when one exists |
|
||||
| `submitted_at` | When Paperclip enqueued the job |
|
||||
| `started_at` | When a worker claimed execution |
|
||||
| `finished_at` | Terminal timestamp |
|
||||
| `attribution_mode` | `included_in_run | billed_directly | external_invoice | untracked` |
|
||||
| `cost_cents` | Job-level billed cost summary for operator visibility |
|
||||
| `result_summary` | Human-readable summary suitable for a list row |
|
||||
| `error_code` | Stable failure classifier |
|
||||
| `error` | Human-readable failure detail |
|
||||
| `created_at` / `updated_at` | Audit timestamps |
|
||||
|
||||
### Supporting fields needed for safe dispatch and rerun
|
||||
|
||||
The following fields are not part of the problem statement's minimum list, but they make the V1 design operationally sound:
|
||||
|
||||
| Field | Why it exists |
|
||||
| --- | --- |
|
||||
| `source_kind` | Distinguish run/comment/document/manual/maintenance origins |
|
||||
| `source_ref_json` | Hold comment id, document key, subject id, namespace, and other source-specific identifiers without over-normalizing the first slice |
|
||||
| `retry_of_job_id` | Preserve attempt history when a job is rerun |
|
||||
| `attempt_number` | Make retries visible in the list and detail surfaces |
|
||||
| `dispatcher_kind` | Start with `in_process`, preserve a clean seam for later host-side workers |
|
||||
| `lease_expires_at` | Lets the scheduler detect stale `running` work without adding a new status |
|
||||
| `usage_json` | Store token/latency/provider usage summary without overextending `cost_events` in the first slice |
|
||||
| `result_json` | Structured output such as record counts, callback payload, or escalation metadata |
|
||||
|
||||
### Notes on normalization
|
||||
|
||||
- `binding_id` should be the authoritative FK.
|
||||
- `binding_key` should still be stored on the row so operators can read history even if the binding is later renamed or deleted.
|
||||
- `source_ref_json` should stay lightweight. Do not snapshot full transcript bodies here.
|
||||
- `result_summary` should be short and readable, for example `Captured 14 records from run residue` or `Compaction found 3 duplicate record groups`.
|
||||
|
||||
## Relationship To `memory_operations`
|
||||
|
||||
The earlier memory plan introduced `memory_operations` as the broader audit trail for all memory actions. This design does not replace that.
|
||||
|
||||
The cleanest end state is:
|
||||
|
||||
- `memory_operations` logs every memory action, synchronous or asynchronous
|
||||
- `memory_extraction_jobs` tracks async job-attempt state for the subset of operations that run outside the initiating request path
|
||||
|
||||
For the first implementation slice, the job record should include enough provenance and usage fields to stand on its own even before a full memory operation explorer lands.
|
||||
|
||||
When `memory_operations` is added, `memory_extraction_jobs` should gain an optional `operation_id` FK rather than forcing operators to infer the relationship.
|
||||
|
||||
## Lifecycle
|
||||
|
||||
### States
|
||||
|
||||
The persisted lifecycle is intentionally small:
|
||||
|
||||
- `queued`
|
||||
- `running`
|
||||
- `succeeded`
|
||||
- `failed`
|
||||
- `cancelled`
|
||||
|
||||
`stuck` is **not** a persisted status. It is a derived operator state:
|
||||
|
||||
- `status = running`
|
||||
- `lease_expires_at < now()`
|
||||
|
||||
That keeps the state machine simple while still giving the board a meaningful "this needs attention" filter.
|
||||
|
||||
### State transitions
|
||||
|
||||
1. `queued -> running`
|
||||
The in-process dispatcher claims the job transactionally, sets `started_at`, and sets `lease_expires_at`.
|
||||
|
||||
2. `running -> succeeded`
|
||||
The provider or worker completes successfully. Persist `finished_at`, `result_summary`, optional `result_json`, and usage/cost summary.
|
||||
|
||||
3. `running -> failed`
|
||||
The provider errors, a callback/poll path reports failure, or the lease expires and the recovery sweep marks the job failed with a stable `error_code`.
|
||||
|
||||
4. `queued -> cancelled`
|
||||
Operator or system cancellation before execution starts.
|
||||
|
||||
5. `running -> cancelled`
|
||||
Allowed only when the provider or host-side runner can stop safely. If cancellation is best-effort only, the row should still be marked cancelled and late provider callbacks should be ignored.
|
||||
|
||||
6. `failed -> rerun`
|
||||
Not an in-place state transition. Create a new `queued` row with `retry_of_job_id` pointing to the failed attempt.
|
||||
|
||||
### Failure and stuck handling
|
||||
|
||||
V1 should use conservative recovery:
|
||||
|
||||
- expired `running` jobs are surfaced as derived `stuck`
|
||||
- the scheduler recovery sweep marks them `failed` with `error_code = "lease_expired"` or `error_code = "worker_lost"`
|
||||
- operators can rerun the job from the detail view
|
||||
|
||||
This avoids silent duplicate compaction/capture work after a server restart.
|
||||
|
||||
## Run And Cost Attribution
|
||||
|
||||
Memory jobs should integrate with the current Paperclip run/cost model without pretending to be heartbeat runs.
|
||||
|
||||
### Run model
|
||||
|
||||
- A memory job does **not** create a `heartbeat_run`.
|
||||
- If the job came from a heartbeat hook, the job stores `source_heartbeat_run_id` and links back to the originating run.
|
||||
- For maintenance jobs like dreaming or compaction, `source_heartbeat_run_id` may be null while `source_agent_id` or only company scope is present.
|
||||
|
||||
### Cost model
|
||||
|
||||
`cost_events` remains the canonical spend ledger.
|
||||
|
||||
The job row should store a summarized view of usage and cost so the operator can inspect one row without joining multiple financial tables.
|
||||
|
||||
Rules:
|
||||
|
||||
1. If the memory work was already billed inside the originating heartbeat run, store:
|
||||
- `attribution_mode = "included_in_run"`
|
||||
- `cost_cents` as the summarized included amount if the provider reports it, otherwise `0`
|
||||
- `source_heartbeat_run_id` linking back to the run ledger
|
||||
|
||||
2. If the provider makes direct metered calls outside the run accounting path, store:
|
||||
- `attribution_mode = "billed_directly"`
|
||||
- summarized usage on the job row
|
||||
- one or more normal `cost_events` with the same company/agent/issue/project/goal provenance when available
|
||||
|
||||
3. Flat provider subscription costs stay in `finance_events`, not on job rows except as attribution metadata.
|
||||
|
||||
The memory job list is for operational visibility. Finance surfaces still aggregate from `cost_events` and `finance_events`.
|
||||
|
||||
## Dispatch Model
|
||||
|
||||
### V1 dispatcher
|
||||
|
||||
V1 should use an in-process server-owned dispatcher with the same product posture as the rest of the current scheduler work.
|
||||
|
||||
Recommended service split:
|
||||
|
||||
- `MemoryJobStore`
|
||||
- enqueue job
|
||||
- claim next queued job
|
||||
- complete/fail/cancel job
|
||||
- list/filter/get jobs
|
||||
|
||||
- `MemoryJobDispatcher`
|
||||
- polls for queued jobs
|
||||
- claims a job transactionally
|
||||
- resolves the memory binding/provider
|
||||
- executes the appropriate capture/compaction/dreaming handler
|
||||
- records result, usage, and failure state
|
||||
|
||||
- `MemoryJobRecovery`
|
||||
- sweeps expired leases
|
||||
- marks stale `running` jobs failed
|
||||
- leaves rerun as an explicit operator action in V1
|
||||
|
||||
### Why a lease matters in V1
|
||||
|
||||
Even with one in-process dispatcher, the record shape should assume that workers can disappear between `running` and terminal state.
|
||||
|
||||
`lease_expires_at` gives V1:
|
||||
|
||||
- a clean way to derive `stuck`
|
||||
- deterministic startup recovery
|
||||
- a seamless transition to future host-side workers that use the same claim protocol
|
||||
|
||||
### Seam for host-side workers later
|
||||
|
||||
The host-side worker version should reuse the same table and lifecycle:
|
||||
|
||||
- workers still claim `queued` rows
|
||||
- workers still extend or complete leases
|
||||
- callbacks or pollers still update the same job row
|
||||
|
||||
The seam is **where the claim/execute loop lives**, not a different product model.
|
||||
|
||||
## Which Work Enqueues Memory Jobs
|
||||
|
||||
### Capture
|
||||
|
||||
Use `memory_extraction_jobs` when post-run or post-comment capture cannot finish synchronously in the originating request path.
|
||||
|
||||
Examples:
|
||||
|
||||
- provider-managed extraction that returns a provider job id
|
||||
- delayed embedding/indexing
|
||||
- large comment/document capture that should not block the request
|
||||
|
||||
### Compaction
|
||||
|
||||
Use the job table for deterministic maintenance like:
|
||||
|
||||
- dedupe
|
||||
- merge
|
||||
- index rebuild
|
||||
- cleanup of stale records
|
||||
|
||||
These are maintenance tasks, not issue work by default.
|
||||
|
||||
### Dreaming
|
||||
|
||||
Use the same table for periodic higher-level memory maintenance that still stays inside the memory subsystem:
|
||||
|
||||
- nightly synthesis
|
||||
- relationship consolidation
|
||||
- profile refresh
|
||||
|
||||
If the work needs real reasoning, authored output, or review, that is the point where it stops being "just a memory job" and should escalate.
|
||||
|
||||
## Escalation Rules
|
||||
|
||||
Most memory jobs should remain job records. Escalation is for human-facing follow-up, not for ordinary maintenance noise.
|
||||
|
||||
### Stay as a job record
|
||||
|
||||
Keep the result inside `memory_extraction_jobs` when:
|
||||
|
||||
- capture succeeded
|
||||
- compaction succeeded
|
||||
- dreaming succeeded
|
||||
- a job failed once but the next action is simply rerun or inspect in the memory UI
|
||||
- the output is operational metadata, not company work
|
||||
|
||||
### Create an issue directly
|
||||
|
||||
Create an issue when:
|
||||
|
||||
- a binding is misconfigured and future jobs will keep failing
|
||||
- the same job class fails repeatedly and needs product/engineering intervention
|
||||
- compaction or dreaming finds data quality anomalies that require review
|
||||
- a rerun cannot proceed because the required source is gone or corrupted
|
||||
|
||||
The issue should link back to the originating memory job id and source issue/run where available.
|
||||
|
||||
### Trigger a routine
|
||||
|
||||
Use a routine only when the desired outcome is recurring company work rather than maintenance.
|
||||
|
||||
Examples:
|
||||
|
||||
- nightly leadership digest built from memory state
|
||||
- recurring agent task to review suspicious memory merges
|
||||
- periodic audit report that should appear as issue work
|
||||
|
||||
The key rule is:
|
||||
|
||||
- memory maintenance stays a memory job
|
||||
- recurring deliverable-producing work becomes a routine
|
||||
|
||||
## Operator Visibility
|
||||
|
||||
### Minimum API surface
|
||||
|
||||
Add company-scoped memory job endpoints:
|
||||
|
||||
- `GET /api/companies/:companyId/memory/jobs`
|
||||
- `GET /api/companies/:companyId/memory/jobs/:jobId`
|
||||
- `POST /api/companies/:companyId/memory/jobs/:jobId/rerun`
|
||||
|
||||
Initial list filters should include:
|
||||
|
||||
- `status`
|
||||
- `effectiveState=stuck`
|
||||
- `bindingKey`
|
||||
- `operationType`
|
||||
- `agentId`
|
||||
- `issueId`
|
||||
- `runId`
|
||||
- date range
|
||||
|
||||
The list response should include enough summary data for a table row:
|
||||
|
||||
- status / effective state
|
||||
- operation type
|
||||
- binding key
|
||||
- source issue/run links
|
||||
- started/finished timestamps
|
||||
- cost summary
|
||||
- result summary or error
|
||||
- retry count
|
||||
|
||||
### Minimum board UI
|
||||
|
||||
The first UI does not need a full "memory dashboard". It does need an operator-visible failure surface.
|
||||
|
||||
Minimum board behavior:
|
||||
|
||||
- a memory jobs list reachable from the future memory settings/operations area
|
||||
- filters for failed and stuck work
|
||||
- list rows that link back to issue and run provenance
|
||||
- detail view with error, usage, provider job id, and rerun action
|
||||
- optional dashboard badge/count for failed or stuck jobs
|
||||
|
||||
### Rerun semantics
|
||||
|
||||
`rerun` should:
|
||||
|
||||
- validate the source can still be resolved or that the job is self-contained
|
||||
- create a new `queued` row
|
||||
- set `retry_of_job_id`
|
||||
- preserve original provenance
|
||||
|
||||
It should not mutate the old row, clear its error, or hide prior operator history.
|
||||
|
||||
## Explicit Non-Goals
|
||||
|
||||
This design deliberately does **not** do the following:
|
||||
|
||||
- no generic Paperclip-wide job engine
|
||||
- no Bull/Redis queue infrastructure
|
||||
- no coupling memory jobs to `plugin_jobs` / `plugin_job_runs`
|
||||
- no requirement for a system agent to perform deterministic memory maintenance
|
||||
- no silent conversion of every memory failure into an issue
|
||||
- no full provider corpus mirroring in Postgres
|
||||
|
||||
## Recommended Implementation Follow-Ups
|
||||
|
||||
The next implementation work should be split into separate child issues:
|
||||
|
||||
1. **Schema and contracts**
|
||||
- add `memory_extraction_jobs` schema
|
||||
- add shared types/constants/validators
|
||||
- add list/detail/rerun API contracts
|
||||
|
||||
2. **Dispatcher and recovery**
|
||||
- implement `MemoryJobStore`
|
||||
- implement in-process dispatcher claim/run/complete flow
|
||||
- add lease-expiry recovery and derived stuck detection
|
||||
|
||||
3. **First async memory path**
|
||||
- wire async post-run capture to enqueue memory jobs
|
||||
- persist provenance, result summaries, and cost attribution
|
||||
- validate rerun behavior for capture jobs
|
||||
|
||||
4. **Board visibility**
|
||||
- add memory jobs list/detail UI
|
||||
- expose failed/stuck filters
|
||||
- add rerun action for eligible jobs
|
||||
|
||||
## Bottom Line
|
||||
|
||||
The first memory job state should be a narrow, company-scoped, domain-specific async job record.
|
||||
|
||||
It should:
|
||||
|
||||
- feel native to Paperclip's existing run/cost/audit model
|
||||
- keep deterministic memory maintenance out of the issue system by default
|
||||
- preserve enough provenance for operators to debug failures
|
||||
- stay simple enough that V1 can run it in-process today and move it to host-side workers later without changing the product model
|
||||
Reference in new issue
Block a user