## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `AGENTS.md` tells every contributor and agent to read `doc/DEVELOPING.md` for setup, commands, testing, and worktrees > - That file grew to 5,495 lines. Most of it was notes on how one subsystem behaves, added one pull request at a time > - Readers could not find the commands. Agents also read behavior notes as rules > - This pull request cuts `doc/DEVELOPING.md` to a 660-line setup and workflow guide. It moves the subsystem notes to focused docs > - The benefit is a guide that a person or an agent can read in full, and a clear place for each kind of note ## Linked Issues or Issue Description **Issue type** Outdated or confusing documentation. **Where is the issue?** `doc/DEVELOPING.md` **What's wrong?** The file was 5,495 lines. It opened with 112 lines about case handoffs before it reached Prerequisites. Unrelated sections sat below the "OpenClaw Docker UI One-Command Script" heading. Statements about current behavior read like policy. **Suggested fix** Keep setup, commands, and workflow in `doc/DEVELOPING.md`. Move each subsystem note to a doc for that subsystem. Add an index. ## What Changed - `doc/DEVELOPING.md` is now 660 lines: prerequisites, start dev, health checks, local run, test drive, Docker, instance layout, database, storage, secrets, worktrees, test commands, Storybook, CLI, and an index of the other docs. - 15 new docs hold the moved text without wording changes: `ci-workflows`, `test-infrastructure`, `worktrees`, `hot-restart-deploys`, `heartbeat-internals`, `runner-internals`, `workspace-repositories`, `workspace-storage`, `skills-and-teams-catalog`, `ui-internals`, `case-handoffs`, `chat-connector-development`, `instance-configuration`, `integration-api-notes`, `database-backups`. - "Publish a branch Storybook" moved to `doc/STORYBOOK-DEPLOYMENT.md`. The privacy rollout notes moved to `doc/ISSUE-PRIVACY.md`. - Deleted about 280 lines: the story-by-story Storybook notes, "Agent Artifact Uploads" (a copy of `doc/AGENT-ARTIFACTS.md`), two pointer-only sections, "Workspace feature checks (2026-10-09)", and "Chat and task navigation caching". The test-drive build-lock paragraph is now four lines. - Repointed every inbound link and every script or workflow comment that named a moved section. - `AGENTS.md` now says where a behavior note goes, so the guide does not grow again. ## Verification - A script split the file by line range. It asserts that each of the 5,495 source lines is kept, moved, or dropped exactly one time. - A link check on every changed Markdown file found 0 missing files and 0 missing anchors. - `node scripts/check-maps.mjs` passes. - `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail. - `node --check` passes on the three scripts with changed comments. - `pnpm verify` was not run. This worktree has no installed dependencies, and the change has no runtime code. ## Risks - Low risk. The change is documentation and comments only. - A link to a moved section from outside this repository will land on the top of `doc/DEVELOPING.md`. - The new docs are not in the audited prose list in `.github/scripts/ci-inputs.mjs`. An edit to one of them runs full CI. An edit to `doc/DEVELOPING.md` does not. This PR does not change that list. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Claude Code desktop, with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [ ] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
41 KiB
Heartbeat and background work internals
Module map and behavior notes for the heartbeat service and the server's background workers. The rules are in Heartbeat execution boundaries.
Agent status recovery
Lifecycle verification stops automatic attempts after missing authentication,
an unavailable model or command, an invalid working directory, unexpected
probe output, or an owned AI account selection refusal. Account selection guidance
names missing defaults, unavailable accounts, incompatible bindings, missing
responsible users, or credential sharing requirements without provider error text. A generic failed test gets up to three failures before automatic
checks stop. The saved verificationFailures count only advances when the host
verification test fails; claims deferred before the test do not spend that budget.
An explicit retry or execution configuration change resets that count.
The agent stays in verifying
with execution paused and a safe error message. Its saved lifecycle operation
records retryRequired; process restarts and queue scans retain that wait.
Changing execution configuration or using Retry now clears it and runs the
normal verification again. Name and title edits do not restart verification.
Timeouts, unavailable test environments, plugin outages and unclassified runtime
failures retain automatic retry. Pause, termination, resource cleanup and idle
sleep checks keep their existing safety gates.
Remote directory and command checks preserve timeout, interruption and transport
failures as typed execution-target failures. Adapters carry that signal in the
optional AdapterEnvironmentCheck.retryable field. A retryable error becomes
test_environment_unavailable before configuration classification, so it neither
parks verification nor consumes the generic failed-test budget. A completed check
that confirms a missing directory or command still waits for configuration repair.
The retry persistence tests can use a dedicated local PostgreSQL database ending
in _test through PAPERCLIP_LIFECYCLE_VERIFICATION_RETRY_TEST_DATABASE_URL when
embedded PostgreSQL is unavailable. The test validates that URL before use.
Heartbeat completion locks the agent row before it reads remaining running work and writes the agent status. A concurrent execution start cannot be overwritten by a completion that read older run state. Run terminalization and agent status finalization remain separate steps. If the process exits between them, the existing startup and periodic orphan-run recovery pass repairs stranded agent statuses after a five-minute grace period, at most 25 agents per pass.
The repair requires finished runs and wake requests, expired controller ownership, settled accounting and retries, and no pending resource or instruction cleanup. It also checks the shared local execution sets. It rechecks under an agent row lock, preserves pause and retirement, and records the correction in the same transaction. It preserves run history and does not invent a heartbeat timestamp. The latest failed or timed-out run produces an error status. This adds no timer, provider call, or automatic workspace wake. Idle sleep checks remain independent. Failed lease history uses the same exact cleanup proof as idle sleep. Cleanup queues are checked separately from retained run metadata. Recovery queries have a two-second statement timeout, and the repair stops starting new candidates after five seconds. A repair failure is logged without rejecting the rest of orphan recovery.
pnpm exec vitest run server/src/services/heartbeat/agent-status.test.ts server/src/__tests__/heartbeat-agent-status-recovery.test.ts
Run history authorization reads
Run history reuses the authorization service's batch read API for distinct task
bindings. Existence checks remain company-scoped and use batches of at most 500
IDs. Missing or foreign tasks stay unreadable in every privacy mode. The batch
snapshot lasts for one request only; the next request checks current grants,
membership, and run restrictions again. Adapters that expose only decide
retain the individual-read path with duplicate bindings shared within the request.
Config Freshness
Agent, project, environment, secret, skill, and workspace config edits are sampled at the next run boundary. A heartbeat that is already running finishes with the config it started with. Native runners project explicitly configured task environment variables into provider processes and tool commands, including custom PAPERCLIP_* names such as PAPERCLIP_PAGE_BUCKET. Adding, changing, or removing a projected variable replaces a retained provider process at the next run boundary. Unchanged variables permit process reuse. The projection contains names only; secret values stay in the child environment. Ambient host secrets are not projected. Native projections allow up to 128 variable names, 64 KiB per entry, and 256 KiB of values in total. These native launch limits do not apply to legacy adapter configuration. Runtime authority, provider login credentials, and process-loader settings use their existing restricted paths.
Native OpenCode applies that projection when starting opencode serve, so its
shell tools receive both plain bindings and resolved company secrets from
adapterConfig.env. Forwarded values participate in diagnostic and provider-trace
redaction. The isolated HOME, provider credential proxy, and GitHub broker retain
their existing launch rules. OpenCode owns the OPENCODE_* and XDG_*
configuration variables; task bindings cannot override them.
When effective run config changes, Paperclip may intentionally skip a saved adapter session, refresh persisted workspace runtime config, replace a reused execution workspace, or avoid reusing a sandbox/environment lease. Fresh execution can lose adapter-specific session, workspace, or sandbox state; correctness of the next run's config takes priority over continuity. Plain environment values affect freshness through value hashes; run result JSON and workspace operation logs expose only the non-sensitive freshness decision categories, without storing secret values, full env maps, provider credentials, or private path details.
An explicit reuse_existing Git workspace keeps its recorded source repository, including after its project-workspace row is deleted or the project primary changes. The server verifies the original company, project, repository origin, and worktree registration before reuse. It does not attach a deleted association to the new primary. A conflicting explicit project-workspace selection, missing source identity, or an unverified legacy local checkout blocks the run. Restore the original source. Before intentionally clearing the existing-workspace binding to choose a new repository, review and preserve its retained work. Missing worktrees may be reconstructed only from the verified original source; current project defaults are not a fallback.
Heartbeat Service Extractions
Routine and issue-monitor dispatch uses saved future deadlines through
server/src/services/scheduled-work.ts and the existing delivery coordinator.
Startup restores the next deadline. Routine edits and monitor changes notify
their worker after the enclosing database transaction settles, including native
set_task_monitor writes. Deadlines crossed during a scan receive an immediate
follow-up scan. Work already overdue at scan start retains heartbeat recovery.
Each worker runs
independently and retains the existing atomic claims and execution checks.
The heartbeat interval still requests a recovery scan; it does not determine
the dispatch time of a known future deadline. Failed scans retry at that same
interval. Future timers do not hold idle work open, and standby, idle drain and
shutdown retain the coordinator's admission gates. During idle drain, expired
deadlines retry at the recovery interval while dispatch is paused; future
deadlines keep their original timer so a canceled drain does not delay them. External database writers
rely on the periodic recovery scan. Sparse cron searches reuse one parsed
expression and skip impossible UTC dates before localized formatting.
Keep relevant heartbeat extractions and their focused tests in server/src/services/heartbeat/.
Follow Heartbeat execution boundaries
for state lifetime, admission, recovery ownership, cleanup ordering, and verification.
Runtime-service coordination is in server/src/services/heartbeat/runtime-services.ts.
It validates service intents, starts configured services, records adapter-managed
services and primary URLs, and posts workspace-ready notices. Process registries
and leases stay in workspace-runtime.ts; the executor keeps release ordering in
its existing cleanup path. Public-path tests use real local service processes.
Instruction working-copy coordination is in
server/src/services/heartbeat/instruction-working-copy.ts. Its per-run controller
prepares copies, tracks save receipts, and coordinates warm-directory handoffs.
Live getters preserve updated copies and expose acquired reservations even if
preparation fails. The executor keeps release and environment-cleanup ordering;
a separate release closure defers cleanup failures without replacing run results.
See tests/e2e/instruction-working-copy/README.md for isolated browser acceptance.
Task assignment Markdown is rendered by server/src/services/heartbeat/task-markdown.ts.
heartbeat.ts re-exports it for existing callers. Keep prompt
formatting changes in the renderer and its tests, separate from run orchestration.
Task context assembly is in server/src/services/heartbeat/task-context.ts.
prepareHeartbeatTaskContext rebuilds prompt fields on the current run context:
continuation summaries, wake comments, task plans, communication guidance, and
full/compact assignment Markdown. It preserves quarantine and secret redaction
before computing turn-context revisions. The executor supplies the already-loaded
issue, session, and trust policy, and retains admission, resource ownership, and
cleanup. Public-path characterization tests are in
server/src/__tests__/heartbeat-task-context.test.ts; the workspace-bootstrap
browser fixture also checks the persisted prompt variants and description revision
from a real process adapter before completing its task.
Execution configuration is assembled in
server/src/services/heartbeat/execution-configuration.ts: workspace and issue
overrides, scoped credentials, managed AI identity, and runtime skill delivery.
The executor receives credential ownership and failure-redaction updates as soon
as they are available, so later setup failures still clean up acquired resources.
Busy-account deferral returns explicitly before session/workspace preparation.
On Cloud-managed sandbox connections, an ACPX platform check can defer runtime
preparation to the normal Runner launch. After the selected account completes
its provider hello probe or live API-key check, that specific deferred check is
informational. Authentication failures, unsupported platforms, other warnings,
and self-hosted or SSH runtime checks retain their status. A passed connection
test does not replace Runner package verification before execution.
Public-path tests are in server/src/__tests__/heartbeat-execution-configuration.test.ts.
Concurrent skill preparations join the company inventory refresh before taking the version-cache lock. Each preparation then reads its actor's current skill visibility under that lock. Snapshot materialization and the durable run version pins stay in the same locked callback. Only the in-flight inventory work is shared; completed refreshes and permission results are not cached.
Browser Use skill assignment first checks for an enabled, active Browser Use connection in the run's company. With none, it skips full tool discovery. With one, the existing task, tool-policy and credential-grant checks still apply.
Session preparation is in server/src/services/heartbeat/session-preparation.ts.
It selects fresh or resumable sessions from effective config and managed-account
identity, then resolves adapter session identity and compaction after workspace
preparation. Both phases keep their original ordering in heartbeat.ts; they
acquire no runtime resources. Public-path characterization tests live in
server/src/__tests__/heartbeat-session-preparation.test.ts.
Session compatibility uses effective workspace policy and settings. Generic issue and project row timestamps do not reset a session. This includes the first project-owned directory binding. Changes to effective execution configuration still require a fresh session.
Run-log formatting is in server/src/services/heartbeat/run-log.ts. It bounds
stored event payloads, redacts and shortens log chunks, and caps stdout/stderr
excerpts. run-output.ts owns per-attempt log handles, current-user redaction,
output progress writes, live log delivery, and adapter event callbacks. The log
store remains service-scoped. Completion reads live excerpt getters so output
that arrives during finalization is retained. Existing public helpers and flush
interval constants remain available from heartbeat.ts.
Workspace preparation is in server/src/services/heartbeat/workspaces.ts. It
owns managed checkout materialization, workspace validation and reuse, referenced
project resolution, and session/workspace configuration freshness. Its
createHeartbeatWorkspaceResolver(db) factory binds the run workspace resolvers
to a service's database. The checkout single-flight map stays at module scope so
all service instances share in-flight materialization. Existing public helpers
and the workspace validation error class remain available from heartbeat.ts.
Keep workspace policy changes separate from scheduling and run execution changes.
Run preparation is in server/src/services/heartbeat/run-preparation.ts. It owns
issue and wake context, responsible-user resolution, routine environment snapshots,
skill mentions, adapter environment configuration, and MCP/tool access setup.
createHeartbeatRunPreparation(db) binds the context loaders to a service's database
without doing database work during construction. heartbeat.ts composes queue
dispatch with cancellation and execution, and re-exports the existing
public helpers and configuration-incomplete error class. Keep preparation policy
changes in this module and its tests.
Queued-comment display reads select only wake identity, queue state, typed comment and interaction bindings, the displayed wait reason, and fresh-session flags. They read only the queued run's runtime mode. Mutation locks retain complete wake and run rows, so edits, ordering, interruption, and saved execution inputs keep their existing write contract.
The task composer offers Steer when the native session has an active provider turn that accepts steering. If steering is unavailable or unsupported, it offers an explicit Interrupt action instead. Interrupt records the board user's intent, requests cancellation, and delivers the saved queue through normal admission after the previous execution stops. Preparation and cleanup can keep messages queued; the action does not bypass Stop confirmation or execution gates.
Run retrieval and session state are in server/src/services/heartbeat/run-state.ts.
It owns bounded run projections, database encoding checks, task session reads and
writes, explicit resumes, session compaction, and usage/billing helpers.
createHeartbeatRunState(db) binds these operations without doing database work
during construction. The encoding-check cache belongs to each factory instance.
Queued-comment polling reads current issue control fields, the typed issue binding,
and the selected adapter type. Mutation and execution paths retain their full run reads.
Workspace-operation polling reads the current run access scope and typed workspace
binding, then applies the existing run and operation permission checks.
heartbeat.ts keeps run execution and re-exports the
existing public helpers. Keep session policy changes separate from run
orchestration changes.
Explicit continuation admission reads only task/comment identity and dispatch evidence from the saved context. Its preflight reads the historical adapter type and workspace sync receipt. The coordinator and run locks retain their order. The locked reread retains the complete Runner profile because suspended-session and quarantined-cleanup proofs parse the exact saved execution input. Other run and dispatch paths retain their existing reads.
Measure these two admission queries with
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-continuation-run-reads.ts --baseline-ref=<pre-fix-commit>.
The synthetic PostgreSQL fixture compares the admission result and reports exact
text-format DataRow frame bytes, SQL samples, and execution plans. It excludes
TLS and protocol metadata. To use an isolated external database, set
PAPERCLIP_CONTINUATION_BENCHMARK_DATABASE_URL to a disposable database ending
in _test. The harness applies migrations there and adds synthetic fixtures.
Continuation history reads select completed tool receipts and the typed
conversation/workspace recovery fields used to choose the prior terminal run.
Every completed receipt keeps its full result. Receipt order, company/agent/task
scope, and the complete retained history stay unchanged. Saved results remain
available to detail, recovery, and execution paths.
Use pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-continuation-history.ts --baseline-ref=<pre-fix-commit>
to compare generated SQL, complete continuation envelopes, and DataRow bytes.
The benchmark includes a large required-receipt control. Its external database
option is PAPERCLIP_CONTINUATION_HISTORY_BENCHMARK_DATABASE_URL, with the same
disposable _test database restriction.
Retry scheduling is in server/src/services/heartbeat/retries.ts. It owns bounded
retry schedules, connection and workspace contention deferrals, shared-workspace
holder checks, due retry promotion, and retry-now requests. createHeartbeatRetries
binds these operations to the service database and explicit lifecycle callbacks
without doing work during construction. heartbeat.ts supplies status writes,
run events, issue-lock release, plan-resume reporting, and worktree cutoffs. It
re-exports the existing retry helpers and workspace-busy error class. Keep retry
policy changes separate from this extraction.
Restart recovery and lease cleanup are in server/src/services/heartbeat/recovery.ts.
It owns hot-restart snapshots and adoption, native restart recovery, shutdown
draining, orphaned-run reaping, and active/pending-cleanup lease sweeps.
createHeartbeatRecovery binds the service database and explicit lifecycle
callbacks without starting work. The service supplies its shutdown flag callback
and shared execution sets so separate service instances keep the same ownership
and shutdown barriers. Cleanup single-flight state stays at module scope.
Keep recovery policy changes separate from retry scheduling and execution changes.
Queue admission and wakeup dispatch are in server/src/services/heartbeat/queue.ts.
It owns wake coalescing and batching, queued-run claims, daily heartbeat caps,
concurrency and priority checks, timer admission, and native status wake intents.
createHeartbeatQueue binds these operations without querying or starting work.
The service supplies lifecycle effects, execution callbacks, worktree gates, and
its process-wide execution and wakeup promise sets. Forwarding callbacks preserve
construction order for retry and recovery services. Keep queue policy changes
separate from this extraction and from adapter execution changes.
Run lifecycle handling is in server/src/services/heartbeat/run-lifecycle.ts.
It owns status transitions, run events and progress, liveness classification,
completion handoffs, issue-comment finalization, and runtime/cost settlement.
createHeartbeatLifecycle binds the database and explicit reporting, recovery,
and wakeup callbacks without starting work. The service supplies terminal
reporting and forwarding callbacks preserve construction order for the other
heartbeat modules. Existing public helpers remain re-exported by heartbeat.ts.
Keep lifecycle policy changes separate from this extraction and adapter execution.
Run cancellation and cleanup are in server/src/services/heartbeat/run-control.ts.
It owns Stop, agent and budget cancellation, environment lease release, and
resumption of saved comments after execution stops. createHeartbeatRunControl
binds the database, lifecycle and admission callbacks, and shared executor
barriers without starting work. The service retains the process-wide execution
and cancellation maps so scheduler and route instances observe the same owners.
Forwarding callbacks preserve construction order for budget, retry, recovery,
and queue services. Existing public helpers remain re-exported by heartbeat.ts.
Keep cancellation policy changes separate from this extraction and execution.
Scheduling is in server/src/services/heartbeat/scheduling.ts. It owns timer
ticks, issue-monitor claims and dispatch, exhausted-monitor recovery, and
session-goal recovery. createHeartbeatScheduling binds explicit dependencies
without querying or starting timers. Queue admission, heartbeat policy parsing,
and atomic timer claims stay in queue.ts; the service supplies those operations
and the current scheduling-suppression and worktree-cutoff callbacks. Monitor
claims remain database-scoped so separate scheduler instances cannot dispatch
the same claim. The public heartbeat methods keep their existing signatures.
Keep scheduling policy changes separate from this extraction and execution.
Run completion is in server/src/services/heartbeat/run-completion.ts.
createHeartbeatRunCompletion binds terminal result persistence, accounting,
response presentation, task-session updates, and execution/setup failure handling.
The executor calls these handlers from the original try/catch boundaries. It
retains native recovery exceptions, dispatch ownership, and the outer cleanup.
Completion reads the shared Stop barriers, uses live output getters, and reports
trace finalization immediately so a later error cannot lose cleanup state.
Existing public failure helpers remain re-exported by heartbeat.ts.
Keep completion policy changes separate from this extraction and dispatch.
Runtime selection is in server/src/services/heartbeat/runtime-selection.ts.
selectHeartbeatRuntime constructs or restores native execution inputs, selects
completion contracts and sessions, and persists native or legacy runtime choices.
Native selection retains the row lock that serializes it with cancellation and
controller lease renewal. The executor receives an explicit selection result;
lifecycle and ownership callbacks update its teardown state immediately, including
when preparation later throws. Public sandbox lifecycle and model fallback helpers
remain re-exported by heartbeat.ts.
Environment selection is in server/src/services/heartbeat/environment-selection.ts.
selectHeartbeatEnvironment resolves environment policy and workspace reuse,
including immutable native recovery bindings, managed sandbox enforcement, and
Kubernetes provisioning. It validates native chat isolation and checks shared
workspace contention before acquisition. It can defer a sandbox run or add a
concurrency note to the task context. The executor retains lease ownership,
dispatch gates, and terminal cleanup.
Workspace preparation is in server/src/services/heartbeat/workspace-preparation.ts.
prepareHeartbeatWorkspace resolves and provisions the workspace, persists its
metadata and issue binding, then acquires and realizes the execution environment.
It preserves the immutable workspace binding used by native recovery and records
preparation spans even when acquisition or realization fails. The executor retains
dispatch gates and terminal lease cleanup. Persistence
failures still clean newly created workspace artifacts before propagating.
Runtime execution is in server/src/services/heartbeat/runtime-execution.ts.
executeHeartbeatRuntime dispatches a prepared native session or legacy adapter,
captures usage, finalizes workspace operations, and releases gateway tokens and
instruction copies. The executor supplies the atomic dispatch gate and retains
admission, terminal run status, retry handling, and outer lease cleanup. An explicit
dispatch result preserves early cancellation returns; ownership callbacks update
teardown state before errors propagate. Instruction getters retain updates made
during provider execution. Keep policy changes separate from this extraction.
Investigating polling load
Deferred-message recovery reads run admission fields before calling the release policy. The release coordinator reads authentication evidence and then refreshes completion identity after post-commit effects. These checks do not load stored prompts, Runner profiles, or output excerpts. Full execution and detail reads retain those fields. Queue cadence, delivery admission, and cleanup ownership remain unchanged.
Runtime progress updates recheck the persisted run status before publication.
This check selects only the run ID, company ID, agent ID, and status through
createHeartbeatRunState().getRunStatus. It does not fetch prompts, runner
configuration, or provider results. Keep this read fresh so terminal or deleted
runs cannot publish late progress from an old executor snapshot.
Native event and retry-exhaustion ingestion lock the run row to verify its company and agent bindings. That lock reads only those two fields. Sequence allocation, replay checks, and event insertion remain in the same transaction.
Sandbox-provider driver lookup filters plugin manifests by driver key and kind in Postgres before returning rows. Install order, status and worker checks still select the provider. Each lookup reads current registry state without a cache.
Run-to-issue links authorize through getRunReadAccess, which reads only the run
ID, company ID, scope kind, and issue ID. The existing visibility policy then
filters linked issues without loading the run's prompt, output, or result.
Run the deterministic native event replay with the normal server test runner:
pnpm --filter @paperclipai/server exec vitest run src/__tests__/native-observer-replay.test.ts
The test uses disposable PostgreSQL and the production event persistence and
authenticated WebSocket paths. Its OBSERVER_REPLAY JSON line follows the
existing benchmark stdout convention. Each case records the measured interval,
active viewers and subscribers, polling cadence and actual read offsets, timing,
query counts, and event/log/delivery/duplicate-replay assertions. Push-only cases
mark polling as not applicable. Subscriber counts include one diagnostic listener
that checks duplicate publication synchronously and does no database work.
The polling fixture reads the production replay
port; it does not model the full browser query graph. Query counts include
Drizzle statement attempts on the measured connections, including authorization,
duplicate replay, and separately counted delivery-time visibility proof reads.
Seed, handshake, post-interval proof reads, and driver SQL are outside that count.
interval.workMs stops after event ingestion, delivery, duplicate replay, and
visibility proofs. Performance comparisons use this elapsed time. The separate
interval.wallMs includes the minimum observation window and polling drain, so
that deliberate wait cannot hide work-time differences. Polling cases wait for
two completed read batches with a bounded deadline before stopping observation.
The slow-first-poll regression delays its first batch until 100 ms after work
completion and still requires two completed real reads per viewer. That injected
delay is explicit in the polling evidence and comparison matching rules.
Every viewer-sensitive case names its zero-viewer control. The report shows a matched push pair, unmatched viewer-count and polling pairs, and explicit observation-load experiments. Unmatched pairs have no timing ratio. A different fixture or observer churn stays unmatched even in an observation experiment. These local diagnostics do not establish hosted latency or provider performance. The output is test evidence; there is no product API or dashboard consumer to migrate. Keep future fields additive and check the schema and comparison status before interpreting timings.
The company heartbeat-run and live-run lists load secret registries in one
company-scoped query per response. Registry reads project only
paperclipSecretRedactions from the run context. They do not load the full
prompt/context JSON. Decrypted values live only for that request and each run
uses its own registry.
Hidden browser tabs suspend the company live-events connection and transcript log reads. Returning to a visible tab refreshes active queries once and resumes transcript reads from their retained offsets. A queued live-event invalidation that flushes after the tab hides marks data stale without starting a refetch.
Live-event delivery and workspace-operation run checks select only the stored company, run scope, and task binding. Event-list access also selects the run ID. These checks do not load run context, runner configuration, results, or output. Every delivery and request still checks current permissions. Grant revocation, deleted tasks, missing runs, and cross-company run IDs remain fail-closed. The developer-server health poll also stops in hidden tabs.
Workspace detail responses return stored metadata and runtime service information
without inspecting Git, resolving delivery PRs, or reconciling remote storage.
This also keeps operation-history reads, login handoffs, runtime controls and
metadata updates from inspecting storage through their common workspace lookup.
Delivery state stays unknown on these reads. Explicit close-readiness and final
cleanup validation still inspect current Git state; neither uses a display result
to authorize removal.
The terminal-workspace reaper uses the database work coordinator. Startup scans
recover existing workspace rows in bounded pages. The first readiness sweep
defers the new remote delivery-proof checks while preserving candidate deadlines;
later scheduled work performs them. GitHub latency does not block readiness
through this path. Issue status and tree changes,
workspace lifecycle writes, finalization receipts, and pull-request work-product writes signal work on
their transaction before writing. The coordinator wakes after the outer
transaction settles and holds idle admission while writes or cleanup are active.
Empty and nonterminal-only lanes have no recurring timer or query. Terminal trees
schedule their actual cooldown deadline; unresolved delivery or cleanup eligibility
is checked again after five minutes. Providers that explicitly declare
supportsMergeAncestry: false use the latest successful finalization receipt as a
candidate HEAD. Core checks GitHub before inspecting storage: the exact HEAD must
be an ancestor of the intended destination tip, or an associated merged PR must
have that exact final HEAD and destination, with its integration commit still
reachable from the destination. Recorded PR work products are optional lookup
hints. This supports squash merges without equating the squash commit with the
original workspace commit.
Preparation captures the resolved destination branch when the provider supports it. Cleanup never silently adopts a changed project source or today's default branch for an ambiguous old workspace. Missing receipts or destinations park the workspace; finalization, workspace, issue, provider and work-product changes and startup reconsider it. A known commit awaiting delivery, or an unavailable remote lookup, keeps a five-minute remote-only retry. GitHub merges have no guaranteed local ingress for this lane, so this concrete unresolved delivery obligation can keep the worker scheduled until delivery is proved. There is no new polling loop. Background GitHub delivery checks use the source project's explicit GitHub authorization, with the same consent checks as provider inspection and removal. The stored source must still match the workspace repository. Each proof, including the final removal check, revalidates the selected grant, project access and trust. Public sources use anonymous access; missing or revoked private-source consent retains storage without falling back to another identity. No task identity is borrowed.
Only positive delivery evidence triggers a fresh storage inspection. The actual
HEAD and branch must match the receipt. Cleanup revalidates delivery after its
last inspection before recording removal intent. A provider can opt into
recoveryPullRequestVersion: 1 to receive one specific PR number and original
HEAD for recovery checks. A merged PR alone does not prove that the original
commit remains recoverable after source-branch deletion. The provider must
independently prove recovery of that exact commit and reject any local data.
Missing capability declarations preserve existing provider protocols.
Nonterminal trees, cooldowns, and active runs are checked before Git. The worker
shares readiness across compatible rows within a bounded page. Each row retains
its own task and delivery assessment. Scoped task/workspace hints invalidate
backoff; changed tree or workspace revisions invalidate it too. Deadline selection
reuses task trees read by those pages without a separate full-inventory query.
Commits during a paged rotation request another bounded rotation so changes behind
its cursor are recovered.
The backoff retains at most 256 candidates and only applies to the background
worker. Final cleanup still checks the current task tree,
delivery evidence, ownership, lifecycle generation, and Git state under the
existing cleanup locks.
These wakes cover the owning createDb scope in this server process. Direct SQL
and other server processes require startup recovery. Local deadlines do not
establish external wake-alarm coverage or whole-instance sleep safety. The local
sleep inventory blocks host suspension until bounded recovery succeeds. Cooldown
eligibility is an earliest allowed cleanup time; passive retained directories can
wait for the next wake, when startup restores their rows. Existing idle-sleep
blockers and cleanup admission holds remain authoritative.
The connection-health sweep selects only due IDs in SQL before applying its
limit. Legacy paperclip_plugin placeholder connections are excluded: their
tools run in plugin workers and do not have remote MCP endpoints. These rows
remain available; the sweep does not disable or delete plugin connections.
When investigating an overloaded instance, distinguish request amplification from stored configuration problems. Verify connection transport and endpoint fields before disabling a connection. Verify workspace ownership, active runs, Git state, and runtime-service readiness before closing a workspace. A missing URL or old workspace timestamp alone does not prove that a row is disposable.
Budget accounting performance
Budget admission and settlement share the company accounting lock. They evaluate only the company, responsible agent, and applicable project. Running usage is an internal exposure projection, not a provisional ledger charge. Cancellation and lifecycle effects run through the delivery coordinator after accounting commits.
The final provider gate commits a dispatch claim before entering provider code. The reservation and native recovery lease generation fence duplicate handoffs; neither provider startup nor termination holds the company accounting lock. After accounting commits, a short ownership transaction rechecks the current task/run fences and covers the synchronous adapter handoff. It commits without awaiting provider bootstrap, results, or finalization. Hosted browser operations retain their own hold and admitted project in existing run metadata. Unknown provider outcomes retain that capacity until accounting establishes the charge. Browser holds are separate from their parent run's usage. Unfinished uncapped browsers and legacy rows without a reliable exposure record remain unknown usage when a new budget is introduced. The existing unpriced-usage policy controls admission until their terminal charge is known.
Use a disposable real PostgreSQL database to compare cost evaluation and complete admission:
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-budget-enforcement.ts --baseline-ref=6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837 > /tmp/budget-forward.json
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-budget-enforcement.ts --baseline-ref=6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837 --reverse > /tmp/budget-reverse.json
The cost comparison keeps one company policy and no active runs while adding 0,
17, and 1,000 unrelated idle agents. It records 20 warm samples, SQL statement counts,
company accounting locks, agent reads, and an EXPLAIN (ANALYZE, BUFFERS) plan.
It asserts unchanged agent state and no agent reads in the new evaluation path.
The harness also measures reservation, the final ownership gate, and an immediate
offline provider result with one responsible agent and one active operation. It
records transaction counts and asserts no reads of unrelated agents. Baseline
service, lifecycle, reservation, and dispatch modules come from the supplied Git
revision.
Provider inference, network latency, stop acknowledgement, and hosted database
latency need separate measurements. Do not run this benchmark beside test suites
or builds when reporting timing results.
Public MCP and Dot event delivery
Public MCP task events and Dot mailbox events use separate topics in the shared transaction-aware delivery coordinator. They no longer scan SQL every two seconds. An enabled subscription without new events or unfinished deliveries is quiet, including when delivery history remains in the database.
| Obligation | Source and wake path |
|---|---|
| Task status, comments, documents, prototype work | Existing activity rows; issue activity writes in persistActivity, the activity service, and wake recovery notify before their transaction commits |
| Dot task follow-up discovery | Existing comments and assignments; insertIssueComment centralizes committed comment notifications, and assignment acceptance wakes the scanner |
| Dot mailbox delivery | Existing mailbox rows; native broker events, explicit challenges, fencing, follow-up publication/consumption, and subscription reconnection notify through their owning transactions |
| Subscription creation, renewal, removal | The subscription transaction wakes its worker; renewal can recover unreceived Dot assignment references |
| Feature disable/reenable | The experimental settings transaction wakes both workers; disabled workers remain quiet until another change |
| Delivery retries and abandoned claims | Existing mcp_event_deliveries.next_attempt_at drives the next deadline, reconstructed on startup |
| Admission and old subscription cleanup | Opportunistic on startup and subsequent enabled sweeps; expiry remains enforced on use |
The scanner preserves durable delivery receipts instead of advancing a timestamp cursor that could skip a late commit. Subscription scans use pages of 20; full source/delivery batches schedule another sweep until caught up. Subscription creation, refresh, and removal target their subscription ID (or the shared Dot binding for reconnect mailbox entries). Issue activity targets its task; comment discovery targets its company; mailbox and assignment changes target their Dot binding. Delivery retries do not rescan unchanged subscriptions. Only startup and instance-wide settings changes request a full scan. Settings changes retain an explicit wildcard when a transaction also contains scoped hints. A fresh request or failed sweep survives warm standby and idle drain; skipped dispatch never marks it complete. Shutdown stops admission of new batches and waits for the current bounded webhook attempt to settle.
Automatic Dot readiness challenges are issued once for a connected subscription. A pending or expired challenge does not imply periodic renewal; an explicit retest creates and signals another challenge. The worker keeps the existing challenge and authorization checks. No schema, OAuth, or webhook contract changes are required.
Notifications require supported writers in the same live server/database scope. Independent SQL writers and separate live server replicas do not wake this process; startup reconstructs their durable records, but continuous multi-writer discovery would require an ingress or notification contract before removing its backstop. The host sleep-safety checks remain conservative for MCP/Dot state. These retry deadlines are not yet exported to Paperclip Cloud's scheduled-wake contract.
events-work.test.ts exercises real PostgreSQL commits, rollback, startup/drain,
retry restoration, producer reactivation, large batches, and SQL/network quiet.
When embedded PostgreSQL is unavailable, use separate disposable databases ending
in _test for PAPERCLIP_MCP_SLEEP_TEST_DATABASE_URL,
PAPERCLIP_PUBLIC_MCP_TEST_DATABASE_URL, and
PAPERCLIP_DOT_RUNNER_TEST_DATABASE_URL. The interaction-service regression suite
also accepts PAPERCLIP_INTERACTIONS_TEST_DATABASE_URL. The suites migrate and write to those
URLs; the sleep suite truncates company data between tests.