Files
PaperClipAI/doc/heartbeat-internals.md
T
DottaandPaperclip c8dd2e400f fix(tasks): confirm Stop after preparation settles (#21906)
## Thinking Path

> - Paperclip lets people start and stop work by AI agents.
> - Stop must end the current turn and preserve the task for a later
message.
> - A person can press Stop while the server still prepares the turn.
> - The preparation owner can prove that no model work started.
> - That path saved the proof but left the Stop receipt at `requested`.
> - The page then reported an unverified Stop and a failed attempt.
> - This PR completes the receipt after proof and shows the confirmed
Stop without a failure notice.

## Linked Issues or Issue Description

**What happened?**
A person submits a task by mistake and presses Stop during preparation.
The run ends before the adapter starts. The server records that no
provider work started, but the Stop receipt stays at `requested`. The
page waits for a receipt that will never change. It then reports that it
cannot verify Stop. The task also shows a failed-attempt notice.

**Expected behavior**
Stop ends the current turn without a failure notice. A later message
starts one new turn through normal admission. Stop does not restart the
task by itself.

**Steps to reproduce**
Hold session preparation before adapter dispatch. Press Stop through the
run API. Release preparation and wait for the owner to finish. Read the
run and the task's interruption state. Send a new task comment.

Related: #20644 covers saved Retry after a committed native
cancellation. #21382 covers a provider failure after native Stop. This
PR covers Stop before adapter dispatch and its page presentation.

## What Changed

- Complete the Stop receipt in `settleCancelledLegacyPreparation`. Keep
its existing checks for process, session, events, leases and saved
files.
- Require no saved `budgetDispatch.handedOffAt` marker under the run
lock. This marker can exist before an adapter records its start. A stale
controller must keep the recovery hold after this handoff.
- Write and require the `preparation_settled` proof before clearing the
failed-attempt notice. Both page queries use one bounded SQL projection.
Unknown and foreign proof labels keep the notice.
- Clear an obsolete automatic-recovery marker only for a user's operator
Stop. Keep system cancellation delivery metadata so saved comments still
reach admission. Cover both cases with persisted recovery records and
the stale-comment journey.
- Give the bounded SQL result an explicit column name. Keep its
real-query tests for typed and bounded Stop evidence.
- Give the Retry test an active company membership and remove it during
cleanup. The test uses normal authorization.
- Record an expected cancellation as information in the run log.
- Add a regression through the real Stop, run-read and comment routes
with a real test database. Only the adapter boundary is simulated. Check
that preparation cannot confirm Stop early, Stop does not restart work,
and one later message starts one turn.
- Document preparation Stop settlement in the heartbeat module notes.

## Verification

- Head: `372ad381f4b60258e17b6b05eda133ed875e7de3`. This revision
includes master `20b0d4e5a5`. [Full CI attempt
2](https://github.com/paperclipai/paperclip/actions/runs/43910771322/attempts/2)
completed successfully: all 52 executed jobs passed, including Build,
Typecheck, all Runner lanes, all nine serialized suites, all twelve
server shards and all eight browser shards. Greptile and Paperclip
Review are both 5/5, and Superagent passed on this exact head. All three
inline threads are resolved. The final commit inventory has 57
successful checks, one neutral review and three expected skips, with no
pending or failing checks; Snyk also passed.
- All 339 focused tests pass at this head across 11 suites. The Stop,
saved-comment and fresh-continuation suites pass 132 tests. Eight other
Stop contract suites pass 207 tests.
- Red: the preparation regression receives `executionCancellation.state
= requested` after the owner finishes. The presentation regressions
report a failed attempt after a confirmed Stop.
- Red: automatic recovery resolves the hold first, then preparation
confirms Stop, but the run still shows a recovery notice. Green: the
confirmed operator Stop clears the obsolete marker and projects
`completed` / `Stopped`.
- Red: the previous repair also changed system cancellations. Both
stale-comment and Retry cases pass on the base and fail on the previous
PR head. A direct regression also catches a preparation Stop receipt
written for a system cancellation. Green: only a user's operator Stop
gets this receipt; system cancellations keep their recovery delivery
metadata.
- Red: the real-query tests read PostgreSQL's implicit expression-column
name after the expression changed. Green: the query has the explicit
`result_json` alias, and all typed and bounded evidence assertions pass.
- Red: a saved dispatch handoff marker was present before any adapter
event. Both new cases returned an acknowledged Stop instead of keeping
the hold. Green: the current locked row must have no marker, including a
marker with JSON null. The receipt stays requested and the hold stays
active. The tests use a stale owner snapshot.
- The HTTP Stop/read/comment test uses real routes and PostgreSQL. It
proves that pending preparation cannot confirm Stop, the owner confirms
it after cleanup, Stop does not restart work, and a later message starts
exactly one turn. Only the adapter boundary is simulated.
- All source typecheck layers and the changed `test-tree-0` layer pass
in low-memory mode. The first full typecheck attempt lost the compiler
to the workspace memory limit while tests ran in parallel. The affected
layers then passed alone. Other test layers passed on the previous head.
The full current-head CI Typecheck + Release Registry job passed. `git
diff --check` passes.
- `pnpm verify` returned exit 2 and requested broader CI. No full local
suite was started.
- Environment: Linux x64, Node 25.9.0, fresh embedded PostgreSQL. This
does not execute a live model or change staging. CI supplies Rust and
full-suite coverage.
- Previous-head CI found the SQL assertion and continuation regressions
above. Its cancelled workspace lane lost its runner before a test result
and was retried once. Previous-head results do not count as final-head
passes.

## Risks

- Only a requested, expected operator Stop with an acknowledged receipt
gets neutral presentation. Unknown termination and file recovery remain
visible.
- The preparation owner writes acknowledgement only after all existing
evidence checks pass. Terminal status alone is insufficient. A retained
local lease test checks both refusal before release and confirmation
after release.
- The receipt does not grant replay authority. Budget, approval,
dependency and cleanup gates still control later messages.
- No schema or migration changes are required.

## Model Used

OpenAI Codex, GPT-6, with reasoning and code tools. The exact served
model build and context size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-16 14:38:41 -06:00

46 KiB

Heartbeat and background work internals

Module map and behavior notes for the heartbeat service and the server's background workers. The rules are in Heartbeat execution boundaries.

Agent status recovery

Lifecycle verification stops automatic attempts after missing authentication, an unavailable model or command, an invalid working directory, unexpected probe output, or an owned AI account selection refusal. Account selection guidance names missing defaults, unavailable accounts, incompatible bindings, missing responsible users, or credential sharing requirements without provider error text. A generic failed test gets up to three failures before automatic checks stop. The saved verificationFailures count only advances when the host verification test fails; claims deferred before the test do not spend that budget. An explicit retry or execution configuration change resets that count. The agent stays in verifying with execution paused and a safe error message. Its saved lifecycle operation records retryRequired; process restarts and queue scans retain that wait. Changing execution configuration or using Retry now clears it and runs the normal verification again. Name and title edits do not restart verification. Timeouts, unavailable test environments, plugin outages and unclassified runtime failures retain automatic retry. Pause, termination, resource cleanup and idle sleep checks keep their existing safety gates.

Remote directory and command checks preserve timeout, interruption and transport failures as typed execution-target failures. Adapters carry that signal in the optional AdapterEnvironmentCheck.retryable field. A retryable error becomes test_environment_unavailable before configuration classification, so it neither parks verification nor consumes the generic failed-test budget. A completed check that confirms a missing directory or command still waits for configuration repair.

The retry persistence tests can use a dedicated local PostgreSQL database ending in _test through PAPERCLIP_LIFECYCLE_VERIFICATION_RETRY_TEST_DATABASE_URL when embedded PostgreSQL is unavailable. The test validates that URL before use.

Heartbeat completion locks the agent row before it reads remaining running work and writes the agent status. A concurrent execution start cannot be overwritten by a completion that read older run state. Run terminalization and agent status finalization remain separate steps. If the process exits between them, the existing startup and periodic orphan-run recovery pass repairs stranded agent statuses after a five-minute grace period, at most 25 agents per pass.

The repair requires finished runs and wake requests, expired controller ownership, settled accounting and retries, and no pending resource or instruction cleanup. It also checks the shared local execution sets. It rechecks under an agent row lock, preserves pause and retirement, and records the correction in the same transaction. It preserves run history and does not invent a heartbeat timestamp. The latest failed or timed-out run produces an error status. This adds no timer, provider call, or automatic workspace wake. Idle sleep checks remain independent. Failed lease history uses the same exact cleanup proof as idle sleep. Cleanup queues are checked separately from retained run metadata. Recovery queries have a two-second statement timeout, and the repair stops starting new candidates after five seconds. A repair failure is logged without rejecting the rest of orphan recovery.

pnpm exec vitest run server/src/services/heartbeat/agent-status.test.ts server/src/__tests__/heartbeat-agent-status-recovery.test.ts

Queued run dispatch order

When an agent is at its concurrency limit, each freed slot starts the next queued run chosen by server/src/services/heartbeat/queued-run-dispatch-order.ts. The "place in line" a waiting task shows uses the same facts and order (queued-run-line.ts).

Runs fall into four tiers, started in this order:

  1. Person. A run a person asked for (an assignment, comment, chat message or Retry by a board user, itself or through its source chain), and any other non-recovery run on a task tree a person created. A task belongs to such a tree when it, or any task above it, has createdByUserId. This covers an agent's subtasks of a person's task and the parent's resume once they complete (issue_children_completed), so a half-finished task is not left behind newer tasks.
  2. Work. Other agent work, such as a tree an agent or routine started.
  3. Recovery and 4. old recovery backlog. System recovery wakes, even on a person's tree.

Within a tier, runs are ordered by readiness (in progress first), then task priority, then age, oldest first. A newer run on a person's tree never passes another person's equally ranked task that has waited longer.

Fairness bounds:

  • Oldest-run share. Every OLDEST_DISPATCH_SHARE (4th) start goes to the longest-waiting ready run that has waited past QUEUED_RUN_AGING_MS (60 minutes), preferring runs people wait on. So even while a deep tree keeps adding in-progress or higher-priority work, another person's new task starts within 60 minutes plus 4 x (aged person-tier runs queued before it + 1) starts.
  • Recovery share. Recovery takes every RECOVERY_DISPATCH_SHARE (4th) start from lower-tier work. It is held only while a run a person asked for is ready, not for the rest of a person's tree, so a deep tree cannot hold recovery indefinitely.
  • Scope. Queues are per agent, and agents belong to one company, so tree priority never moves one agent's or company's runs ahead of another's.

Tree membership is read in one recursive query per dispatch pass, walking each queued task's parent chain by the (company_id, id) unique index and stopping at the first person-created task (at most 32 levels). It runs only when the agent has more than one queued run.

Run history authorization reads

Run history reuses the authorization service's batch read API for distinct task bindings. Existence checks remain company-scoped and use batches of at most 500 IDs. Missing or foreign tasks stay unreadable in every privacy mode. The batch snapshot lasts for one request only; the next request checks current grants, membership, and run restrictions again. Adapters that expose only decide retain the individual-read path with duplicate bindings shared within the request.

Config Freshness

Agent, project, environment, secret, skill, and workspace config edits are sampled at the next run boundary. A heartbeat that is already running finishes with the config it started with. Native runners project explicitly configured task environment variables into provider processes and tool commands, including custom PAPERCLIP_* names such as PAPERCLIP_PAGE_BUCKET. Adding, changing, or removing a projected variable replaces a retained provider process at the next run boundary. Unchanged variables permit process reuse. The projection contains names only; secret values stay in the child environment. Ambient host secrets are not projected. Native projections allow up to 128 variable names, 64 KiB per entry, and 256 KiB of values in total. These native launch limits do not apply to legacy adapter configuration. Runtime authority, provider login credentials, and process-loader settings use their existing restricted paths.

Native OpenCode applies that projection when starting opencode serve, so its shell tools receive both plain bindings and resolved company secrets from adapterConfig.env. Forwarded values participate in diagnostic and provider-trace redaction. The isolated HOME, provider credential proxy, and GitHub broker retain their existing launch rules. OpenCode owns the OPENCODE_* and XDG_* configuration variables; task bindings cannot override them.

When effective run config changes, Paperclip may intentionally skip a saved adapter session, refresh persisted workspace runtime config, replace a reused execution workspace, or avoid reusing a sandbox/environment lease. Fresh execution can lose adapter-specific session, workspace, or sandbox state; correctness of the next run's config takes priority over continuity. Plain environment values affect freshness through value hashes; run result JSON and workspace operation logs expose only the non-sensitive freshness decision categories, without storing secret values, full env maps, provider credentials, or private path details.

The saved-session fingerprint covers only inputs that can change what the provider thread saw or may do, or what the native session digest compares before a follow-up turn attaches to its warm Runner. It leaves out the agent config revision row itself (its id and time), agent identity and display fields that tools read live, turn timeouts, the legacy kill grace, permission keys of a provider that is not selected, wake scheduling (runtimeConfig.heartbeat), the environment row's update time and selection source, and the environment's release, log-streaming, and RPC-timeout settings. Lifecycle inputs stay in: the lifecycle mode, the agent and environment warm idle windows, and the environment's lease reuse, because they become session.lifecyclePolicy, and a kept session with a different policy cannot attach to its warm Runner. server/src/services/heartbeat/session-fingerprint-inputs.ts classifies every input with the reason. Its tests fail when an agent revision field, a runner config schema field, a provider permission key, or an environment config key appears without a classification, when a new field reaches a hashed category, or when an excluded runner key reaches the native execution input. An unknown adapter or runtime config key is hashed. A session saved with the previous fingerprint composition is accepted once for the same inputs, so an upgrade does not discard saved sessions.

An explicit reuse_existing Git workspace keeps its recorded source repository, including after its project-workspace row is deleted or the project primary changes. The server verifies the original company, project, repository origin, and worktree registration before reuse. It does not attach a deleted association to the new primary. A conflicting explicit project-workspace selection, missing source identity, or an unverified legacy local checkout blocks the run. Restore the original source. Before intentionally clearing the existing-workspace binding to choose a new repository, review and preserve its retained work. Missing worktrees may be reconstructed only from the verified original source; current project defaults are not a fallback.

Heartbeat Service Extractions

Routine and issue-monitor dispatch uses saved future deadlines through server/src/services/scheduled-work.ts and the existing delivery coordinator. Startup restores the next deadline. Routine edits and monitor changes notify their worker after the enclosing database transaction settles, including native set_task_monitor writes. Deadlines crossed during a scan receive an immediate follow-up scan. Work already overdue at scan start retains heartbeat recovery. Each worker runs independently and retains the existing atomic claims and execution checks. The heartbeat interval still requests a recovery scan; it does not determine the dispatch time of a known future deadline. Failed scans retry at that same interval. Future timers do not hold idle work open, and standby, idle drain and shutdown retain the coordinator's admission gates. During idle drain, expired deadlines retry at the recovery interval while dispatch is paused; future deadlines keep their original timer so a canceled drain does not delay them. External database writers rely on the periodic recovery scan. Sparse cron searches reuse one parsed expression and skip impossible UTC dates before localized formatting.

Keep relevant heartbeat extractions and their focused tests in server/src/services/heartbeat/. Follow Heartbeat execution boundaries for state lifetime, admission, recovery ownership, cleanup ordering, and verification.

Runtime-service coordination is in server/src/services/heartbeat/runtime-services.ts. It validates service intents, starts configured services, records adapter-managed services and primary URLs, and posts workspace-ready notices. Process registries and leases stay in workspace-runtime.ts; the executor keeps release ordering in its existing cleanup path. Public-path tests use real local service processes.

Instruction working-copy coordination is in server/src/services/heartbeat/instruction-working-copy.ts. Its per-run controller prepares copies, tracks save receipts, and coordinates warm-directory handoffs. Live getters preserve updated copies and expose acquired reservations even if preparation fails. The executor keeps release and environment-cleanup ordering; a separate release closure defers cleanup failures without replacing run results. See tests/e2e/instruction-working-copy/README.md for isolated browser acceptance.

Task assignment Markdown is rendered by server/src/services/heartbeat/task-markdown.ts. heartbeat.ts re-exports it for existing callers. Keep prompt formatting changes in the renderer and its tests, separate from run orchestration.

Task context assembly is in server/src/services/heartbeat/task-context.ts. prepareHeartbeatTaskContext rebuilds prompt fields on the current run context: continuation summaries, wake comments, task plans, communication guidance, and full/compact assignment Markdown. It preserves quarantine and secret redaction before computing turn-context revisions. The executor supplies the already-loaded issue, session, and trust policy, and retains admission, resource ownership, and cleanup. Public-path characterization tests are in server/src/__tests__/heartbeat-task-context.test.ts; the workspace-bootstrap browser fixture also checks the persisted prompt variants and description revision from a real process adapter before completing its task.

Execution configuration is assembled in server/src/services/heartbeat/execution-configuration.ts: workspace and issue overrides, scoped credentials, managed AI identity, and runtime skill delivery. The executor receives credential ownership and failure-redaction updates as soon as they are available, so later setup failures still clean up acquired resources. Busy-account deferral returns explicitly before session/workspace preparation.

On Cloud-managed sandbox connections, an ACPX platform check can defer runtime preparation to the normal Runner launch. After the selected account completes its provider hello probe or live API-key check, that specific deferred check is informational. Authentication failures, unsupported platforms, other warnings, and self-hosted or SSH runtime checks retain their status. A passed connection test does not replace Runner package verification before execution. Public-path tests are in server/src/__tests__/heartbeat-execution-configuration.test.ts.

Concurrent skill preparations join the company inventory refresh before taking the version-cache lock. Each preparation then reads its actor's current skill visibility under that lock. Snapshot materialization and the durable run version pins stay in the same locked callback. Only the in-flight inventory work is shared; completed refreshes and permission results are not cached.

Browser Use skill assignment first checks for an enabled, active Browser Use connection in the run's company. With none, it skips full tool discovery. With one, the existing task, tool-policy and credential-grant checks still apply.

Session preparation is in server/src/services/heartbeat/session-preparation.ts. It selects fresh or resumable sessions from effective config and managed-account identity, then resolves adapter session identity and compaction after workspace preparation. Both phases keep their original ordering in heartbeat.ts; they acquire no runtime resources. Public-path characterization tests live in server/src/__tests__/heartbeat-session-preparation.test.ts.

Session compatibility uses effective workspace policy and settings. Generic issue and project row timestamps do not reset a session. This includes the first project-owned directory binding. Changes to effective execution configuration still require a fresh session.

Run-log formatting is in server/src/services/heartbeat/run-log.ts. It bounds stored event payloads, redacts and shortens log chunks, and caps stdout/stderr excerpts. run-output.ts owns per-attempt log handles, current-user redaction, output progress writes, live log delivery, and adapter event callbacks. The log store remains service-scoped. Completion reads live excerpt getters so output that arrives during finalization is retained. Existing public helpers and flush interval constants remain available from heartbeat.ts.

Workspace preparation is in server/src/services/heartbeat/workspaces.ts. It owns managed checkout materialization, workspace validation and reuse, referenced project resolution, and session/workspace configuration freshness. Its createHeartbeatWorkspaceResolver(db) factory binds the run workspace resolvers to a service's database. The checkout single-flight map stays at module scope so all service instances share in-flight materialization. Existing public helpers and the workspace validation error class remain available from heartbeat.ts. Keep workspace policy changes separate from scheduling and run execution changes.

Run preparation is in server/src/services/heartbeat/run-preparation.ts. It owns issue and wake context, responsible-user resolution, routine environment snapshots, skill mentions, adapter environment configuration, and MCP/tool access setup. createHeartbeatRunPreparation(db) binds the context loaders to a service's database without doing database work during construction. heartbeat.ts composes queue dispatch with cancellation and execution, and re-exports the existing public helpers and configuration-incomplete error class. Keep preparation policy changes in this module and its tests.

Queued-comment display reads select only wake identity, queue state, typed comment and interaction bindings, the displayed wait reason, and fresh-session flags. They read only the queued run's runtime mode. Mutation locks retain complete wake and run rows, so edits, ordering, interruption, and saved execution inputs keep their existing write contract.

The task composer offers Steer when the native session has an active provider turn that accepts steering. If steering is unavailable or unsupported, it offers an explicit Interrupt action instead. Interrupt records the board user's intent, requests cancellation, and delivers the saved queue through normal admission after the previous execution stops. Preparation and cleanup can keep messages queued; the action does not bypass Stop confirmation or execution gates.

Run retrieval and session state are in server/src/services/heartbeat/run-state.ts. It owns bounded run projections, database encoding checks, task session reads and writes, explicit resumes, session compaction, and usage/billing helpers. createHeartbeatRunState(db) binds these operations without doing database work during construction. The encoding-check cache belongs to each factory instance. Queued-comment polling reads current issue control fields, the typed issue binding, and the selected adapter type. Mutation and execution paths retain their full run reads. Workspace-operation polling reads the current run access scope and typed workspace binding, then applies the existing run and operation permission checks. heartbeat.ts keeps run execution and re-exports the existing public helpers. Keep session policy changes separate from run orchestration changes.

Explicit continuation admission reads only task/comment identity and dispatch evidence from the saved context. Its preflight reads the historical adapter type and workspace sync receipt. The coordinator and run locks retain their order. The locked reread retains the complete Runner profile because suspended-session and quarantined-cleanup proofs parse the exact saved execution input. Other run and dispatch paths retain their existing reads.

Measure these two admission queries with pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-continuation-run-reads.ts --baseline-ref=<pre-fix-commit>. The synthetic PostgreSQL fixture compares the admission result and reports exact text-format DataRow frame bytes, SQL samples, and execution plans. It excludes TLS and protocol metadata. To use an isolated external database, set PAPERCLIP_CONTINUATION_BENCHMARK_DATABASE_URL to a disposable database ending in _test. The harness applies migrations there and adds synthetic fixtures.

Continuation history reads select completed tool receipts and the typed conversation/workspace recovery fields used to choose the prior terminal run. Every completed receipt keeps its full result. Receipt order, company/agent/task scope, and the complete retained history stay unchanged. Saved results remain available to detail, recovery, and execution paths. Use pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-continuation-history.ts --baseline-ref=<pre-fix-commit> to compare generated SQL, complete continuation envelopes, and DataRow bytes. The benchmark includes a large required-receipt control. Its external database option is PAPERCLIP_CONTINUATION_HISTORY_BENCHMARK_DATABASE_URL, with the same disposable _test database restriction.

Retry scheduling is in server/src/services/heartbeat/retries.ts. It owns bounded retry schedules, connection and workspace contention deferrals, shared-workspace holder checks, due retry promotion, and retry-now requests. createHeartbeatRetries binds these operations to the service database and explicit lifecycle callbacks without doing work during construction. heartbeat.ts supplies status writes, run events, issue-lock release, plan-resume reporting, and worktree cutoffs. It re-exports the existing retry helpers and workspace-busy error class. Keep retry policy changes separate from this extraction.

Restart recovery and lease cleanup are in server/src/services/heartbeat/recovery.ts. It owns hot-restart snapshots and adoption, native restart recovery, shutdown draining, orphaned-run reaping, and active/pending-cleanup lease sweeps. createHeartbeatRecovery binds the service database and explicit lifecycle callbacks without starting work. The service supplies its shutdown flag callback and shared execution sets so separate service instances keep the same ownership and shutdown barriers. Cleanup single-flight state stays at module scope. Keep recovery policy changes separate from retry scheduling and execution changes.

Queue admission and wakeup dispatch are in server/src/services/heartbeat/queue.ts. It owns wake coalescing and batching, queued-run claims, daily heartbeat caps, concurrency and priority checks, timer admission, and native status wake intents. createHeartbeatQueue binds these operations without querying or starting work. The service supplies lifecycle effects, execution callbacks, worktree gates, and its process-wide execution and wakeup promise sets. Forwarding callbacks preserve construction order for retry and recovery services. Keep queue policy changes separate from this extraction and from adapter execution changes.

Run lifecycle handling is in server/src/services/heartbeat/run-lifecycle.ts. It owns status transitions, run events and progress, liveness classification, completion handoffs, issue-comment finalization, and runtime/cost settlement. createHeartbeatLifecycle binds the database and explicit reporting, recovery, and wakeup callbacks without starting work. The service supplies terminal reporting and forwarding callbacks preserve construction order for the other heartbeat modules. Existing public helpers remain re-exported by heartbeat.ts. Keep lifecycle policy changes separate from this extraction and adapter execution.

Run cancellation and cleanup are in server/src/services/heartbeat/run-control.ts. It owns Stop, agent and budget cancellation, environment lease release, and resumption of saved comments after execution stops. createHeartbeatRunControl binds the database, lifecycle and admission callbacks, and shared executor barriers without starting work. The service retains the process-wide execution and cancellation maps so scheduler and route instances observe the same owners. Forwarding callbacks preserve construction order for budget, retry, recovery, and queue services. Existing public helpers remain re-exported by heartbeat.ts. Keep cancellation policy changes separate from this extraction and execution.

An operator Stop during preparation can settle before any adapter starts. The preparation owner records executionCancellation.state = acknowledged only after it unwinds and the retained dispatch, process, session, event and lease evidence confirms that provider work never started. A terminal run status alone is not this proof. The locked run must have no budgetDispatch.handedOffAt marker. The dispatch gate writes that marker before adapter entry, so a missing invocation event or an old owner's local flag cannot prove that dispatch never occurred. The task page shows a confirmed operator Stop as Stopped and accepts a new message through normal admission. Stop does not automatically restart the task. Recovery holds, approvals, budget limits and uncertain termination keep their own gates. The receipt's preparation_settled proof identifies that evidence path. Both page projections read the same bounded evidence and require that proof; an acknowledged receipt from a different path does not clear the preparation failure notice. Only a user's operator Stop gets this acknowledgement and clears an obsolete automatic-recovery marker. A system cancellation keeps its existing recovery delivery metadata, so saved comments and continuation intents still reach admission.

Scheduling is in server/src/services/heartbeat/scheduling.ts. It owns timer ticks, issue-monitor claims and dispatch, exhausted-monitor recovery, and session-goal recovery. createHeartbeatScheduling binds explicit dependencies without querying or starting timers. Queue admission, heartbeat policy parsing, and atomic timer claims stay in queue.ts; the service supplies those operations and the current scheduling-suppression and worktree-cutoff callbacks. Monitor claims remain database-scoped so separate scheduler instances cannot dispatch the same claim. The public heartbeat methods keep their existing signatures. Keep scheduling policy changes separate from this extraction and execution.

Run completion is in server/src/services/heartbeat/run-completion.ts. createHeartbeatRunCompletion binds terminal result persistence, accounting, response presentation, task-session updates, and execution/setup failure handling. The executor calls these handlers from the original try/catch boundaries. It retains native recovery exceptions, dispatch ownership, and the outer cleanup. Completion reads the shared Stop barriers, uses live output getters, and reports trace finalization immediately so a later error cannot lose cleanup state. Existing public failure helpers remain re-exported by heartbeat.ts. Keep completion policy changes separate from this extraction and dispatch.

Runtime selection is in server/src/services/heartbeat/runtime-selection.ts. selectHeartbeatRuntime constructs or restores native execution inputs, selects completion contracts and sessions, and persists native or legacy runtime choices. Native selection retains the row lock that serializes it with cancellation and controller lease renewal. The executor receives an explicit selection result; lifecycle and ownership callbacks update its teardown state immediately, including when preparation later throws. Public sandbox lifecycle and model fallback helpers remain re-exported by heartbeat.ts.

Environment selection is in server/src/services/heartbeat/environment-selection.ts. selectHeartbeatEnvironment resolves environment policy and workspace reuse, including immutable native recovery bindings, managed sandbox enforcement, and Kubernetes provisioning. It validates native chat isolation and checks shared workspace contention before acquisition. It can defer a sandbox run or add a concurrency note to the task context. The executor retains lease ownership, dispatch gates, and terminal cleanup.

Workspace preparation is in server/src/services/heartbeat/workspace-preparation.ts. prepareHeartbeatWorkspace resolves and provisions the workspace, persists its metadata and issue binding, then acquires and realizes the execution environment. It preserves the immutable workspace binding used by native recovery and records preparation spans even when acquisition or realization fails. The executor retains dispatch gates and terminal lease cleanup. Persistence failures still clean newly created workspace artifacts before propagating.

Runtime execution is in server/src/services/heartbeat/runtime-execution.ts. executeHeartbeatRuntime dispatches a prepared native session or legacy adapter, captures usage, finalizes workspace operations, and releases gateway tokens and instruction copies. The executor supplies the atomic dispatch gate and retains admission, terminal run status, retry handling, and outer lease cleanup. An explicit dispatch result preserves early cancellation returns; ownership callbacks update teardown state before errors propagate. Instruction getters retain updates made during provider execution. Keep policy changes separate from this extraction.

Investigating polling load

Deferred-message recovery reads run admission fields before calling the release policy. The release coordinator reads authentication evidence and then refreshes completion identity after post-commit effects. These checks do not load stored prompts, Runner profiles, or output excerpts. Full execution and detail reads retain those fields. Queue cadence, delivery admission, and cleanup ownership remain unchanged.

Runtime progress updates recheck the persisted run status before publication. This check selects only the run ID, company ID, agent ID, and status through createHeartbeatRunState().getRunStatus. It does not fetch prompts, runner configuration, or provider results. Keep this read fresh so terminal or deleted runs cannot publish late progress from an old executor snapshot.

Native event and retry-exhaustion ingestion lock the run row to verify its company and agent bindings. That lock reads only those two fields. Sequence allocation, replay checks, and event insertion remain in the same transaction.

Sandbox-provider driver lookup filters plugin manifests by driver key and kind in Postgres before returning rows. Install order, status and worker checks still select the provider. Each lookup reads current registry state without a cache.

Run-to-issue links authorize through getRunReadAccess, which reads only the run ID, company ID, scope kind, and issue ID. The existing visibility policy then filters linked issues without loading the run's prompt, output, or result.

Run the deterministic native event replay with the normal server test runner:

pnpm --filter @paperclipai/server exec vitest run src/__tests__/native-observer-replay.test.ts

The test uses disposable PostgreSQL and the production event persistence and authenticated WebSocket paths. Its OBSERVER_REPLAY JSON line follows the existing benchmark stdout convention. Each case records the measured interval, active viewers and subscribers, polling cadence and actual read offsets, timing, query counts, and event/log/delivery/duplicate-replay assertions. Push-only cases mark polling as not applicable. Subscriber counts include one diagnostic listener that checks duplicate publication synchronously and does no database work. The polling fixture reads the production replay port; it does not model the full browser query graph. Query counts include Drizzle statement attempts on the measured connections, including authorization, duplicate replay, and separately counted delivery-time visibility proof reads. Seed, handshake, post-interval proof reads, and driver SQL are outside that count.

interval.workMs stops after event ingestion, delivery, duplicate replay, and visibility proofs. Performance comparisons use this elapsed time. The separate interval.wallMs includes the minimum observation window and polling drain, so that deliberate wait cannot hide work-time differences. Polling cases wait for two completed read batches with a bounded deadline before stopping observation. The slow-first-poll regression delays its first batch until 100 ms after work completion and still requires two completed real reads per viewer. That injected delay is explicit in the polling evidence and comparison matching rules.

Every viewer-sensitive case names its zero-viewer control. The report shows a matched push pair, unmatched viewer-count and polling pairs, and explicit observation-load experiments. Unmatched pairs have no timing ratio. A different fixture or observer churn stays unmatched even in an observation experiment. These local diagnostics do not establish hosted latency or provider performance. The output is test evidence; there is no product API or dashboard consumer to migrate. Keep future fields additive and check the schema and comparison status before interpreting timings.

The company heartbeat-run and live-run lists load secret registries in one company-scoped query per response. Registry reads project only paperclipSecretRedactions from the run context. They do not load the full prompt/context JSON. Decrypted values live only for that request and each run uses its own registry.

Hidden browser tabs suspend the company live-events connection and transcript log reads. Returning to a visible tab refreshes active queries once and resumes transcript reads from their retained offsets. A queued live-event invalidation that flushes after the tab hides marks data stale without starting a refetch.

Live-event delivery and workspace-operation run checks select only the stored company, run scope, and task binding. Event-list access also selects the run ID. These checks do not load run context, runner configuration, results, or output. Every delivery and request still checks current permissions. Grant revocation, deleted tasks, missing runs, and cross-company run IDs remain fail-closed. The developer-server health poll also stops in hidden tabs.

Workspace detail responses return stored metadata and runtime service information without inspecting Git, resolving delivery PRs, or reconciling remote storage. This also keeps operation-history reads, login handoffs, runtime controls and metadata updates from inspecting storage through their common workspace lookup. Delivery state stays unknown on these reads. Explicit close-readiness and final cleanup validation still inspect current Git state; neither uses a display result to authorize removal.

The terminal-workspace reaper uses the database work coordinator. Startup scans recover existing workspace rows in bounded pages. The first readiness sweep defers the new remote delivery-proof checks while preserving candidate deadlines; later scheduled work performs them. GitHub latency does not block readiness through this path. Issue status and tree changes, workspace lifecycle writes, finalization receipts, and pull-request work-product writes signal work on their transaction before writing. The coordinator wakes after the outer transaction settles and holds idle admission while writes or cleanup are active. Empty and nonterminal-only lanes have no recurring timer or query. Terminal trees schedule their actual cooldown deadline; unresolved delivery or cleanup eligibility is checked again after five minutes. Providers that explicitly declare supportsMergeAncestry: false use the latest successful finalization receipt as a candidate HEAD. Core checks GitHub before inspecting storage: the exact HEAD must be an ancestor of the intended destination tip, or an associated merged PR must have that exact final HEAD and destination, with its integration commit still reachable from the destination. Recorded PR work products are optional lookup hints. This supports squash merges without equating the squash commit with the original workspace commit.

Preparation captures the resolved destination branch when the provider supports it. Cleanup never silently adopts a changed project source or today's default branch for an ambiguous old workspace. Missing receipts or destinations park the workspace; finalization, workspace, issue, provider and work-product changes and startup reconsider it. A known commit awaiting delivery, or an unavailable remote lookup, keeps a five-minute remote-only retry. GitHub merges have no guaranteed local ingress for this lane, so this concrete unresolved delivery obligation can keep the worker scheduled until delivery is proved. There is no new polling loop. Background GitHub delivery checks use the source project's explicit GitHub authorization, with the same consent checks as provider inspection and removal. The stored source must still match the workspace repository. Each proof, including the final removal check, revalidates the selected grant, project access and trust. Public sources use anonymous access; missing or revoked private-source consent retains storage without falling back to another identity. No task identity is borrowed.

Only positive delivery evidence triggers a fresh storage inspection. The actual HEAD and branch must match the receipt. Cleanup revalidates delivery after its last inspection before recording removal intent. A provider can opt into recoveryPullRequestVersion: 1 to receive one specific PR number and original HEAD for recovery checks. A merged PR alone does not prove that the original commit remains recoverable after source-branch deletion. The provider must independently prove recovery of that exact commit and reject any local data. Missing capability declarations preserve existing provider protocols. Nonterminal trees, cooldowns, and active runs are checked before Git. The worker shares readiness across compatible rows within a bounded page. Each row retains its own task and delivery assessment. Scoped task/workspace hints invalidate backoff; changed tree or workspace revisions invalidate it too. Deadline selection reuses task trees read by those pages without a separate full-inventory query. Commits during a paged rotation request another bounded rotation so changes behind its cursor are recovered. The backoff retains at most 256 candidates and only applies to the background worker. Final cleanup still checks the current task tree, delivery evidence, ownership, lifecycle generation, and Git state under the existing cleanup locks.

These wakes cover the owning createDb scope in this server process. Direct SQL and other server processes require startup recovery. Local deadlines do not establish external wake-alarm coverage or whole-instance sleep safety. The local sleep inventory blocks host suspension until bounded recovery succeeds. Cooldown eligibility is an earliest allowed cleanup time; passive retained directories can wait for the next wake, when startup restores their rows. Existing idle-sleep blockers and cleanup admission holds remain authoritative.

The connection-health sweep selects only due IDs in SQL before applying its limit. Legacy paperclip_plugin placeholder connections are excluded: their tools run in plugin workers and do not have remote MCP endpoints. These rows remain available; the sweep does not disable or delete plugin connections.

When investigating an overloaded instance, distinguish request amplification from stored configuration problems. Verify connection transport and endpoint fields before disabling a connection. Verify workspace ownership, active runs, Git state, and runtime-service readiness before closing a workspace. A missing URL or old workspace timestamp alone does not prove that a row is disposable.

Budget accounting performance

Budget admission and settlement share the company accounting lock. They evaluate only the company, responsible agent, and applicable project. Running usage is an internal exposure projection, not a provisional ledger charge. Cancellation and lifecycle effects run through the delivery coordinator after accounting commits.

The final provider gate commits a dispatch claim before entering provider code. The reservation and native recovery lease generation fence duplicate handoffs; neither provider startup nor termination holds the company accounting lock. After accounting commits, a short ownership transaction rechecks the current task/run fences and covers the synchronous adapter handoff. It commits without awaiting provider bootstrap, results, or finalization. Hosted browser operations retain their own hold and admitted project in existing run metadata. Unknown provider outcomes retain that capacity until accounting establishes the charge. Browser holds are separate from their parent run's usage. Unfinished uncapped browsers and legacy rows without a reliable exposure record remain unknown usage when a new budget is introduced. The existing unpriced-usage policy controls admission until their terminal charge is known.

Use a disposable real PostgreSQL database to compare cost evaluation and complete admission:

pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-budget-enforcement.ts --baseline-ref=6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837 > /tmp/budget-forward.json
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-budget-enforcement.ts --baseline-ref=6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837 --reverse > /tmp/budget-reverse.json

The cost comparison keeps one company policy and no active runs while adding 0, 17, and 1,000 unrelated idle agents. It records 20 warm samples, SQL statement counts, company accounting locks, agent reads, and an EXPLAIN (ANALYZE, BUFFERS) plan. It asserts unchanged agent state and no agent reads in the new evaluation path. The harness also measures reservation, the final ownership gate, and an immediate offline provider result with one responsible agent and one active operation. It records transaction counts and asserts no reads of unrelated agents. Baseline service, lifecycle, reservation, and dispatch modules come from the supplied Git revision. Provider inference, network latency, stop acknowledgement, and hosted database latency need separate measurements. Do not run this benchmark beside test suites or builds when reporting timing results.

Public MCP and Dot event delivery

Public MCP task events and Dot mailbox events use separate topics in the shared transaction-aware delivery coordinator. They no longer scan SQL every two seconds. An enabled subscription without new events or unfinished deliveries is quiet, including when delivery history remains in the database.

Obligation Source and wake path
Task status, comments, documents, prototype work Existing activity rows; issue activity writes in persistActivity, the activity service, and wake recovery notify before their transaction commits
Dot task follow-up discovery Existing comments and assignments; insertIssueComment centralizes committed comment notifications, and assignment acceptance wakes the scanner
Dot mailbox delivery Existing mailbox rows; native broker events, explicit challenges, fencing, follow-up publication/consumption, and subscription reconnection notify through their owning transactions
Subscription creation, renewal, removal The subscription transaction wakes its worker; renewal can recover unreceived Dot assignment references
Feature disable/reenable The experimental settings transaction wakes both workers; disabled workers remain quiet until another change
Delivery retries and abandoned claims Existing mcp_event_deliveries.next_attempt_at drives the next deadline, reconstructed on startup
Admission and old subscription cleanup Opportunistic on startup and subsequent enabled sweeps; expiry remains enforced on use

The scanner preserves durable delivery receipts instead of advancing a timestamp cursor that could skip a late commit. Subscription scans use pages of 20; full source/delivery batches schedule another sweep until caught up. Subscription creation, refresh, and removal target their subscription ID (or the shared Dot binding for reconnect mailbox entries). Issue activity targets its task; comment discovery targets its company; mailbox and assignment changes target their Dot binding. Delivery retries do not rescan unchanged subscriptions. Only startup and instance-wide settings changes request a full scan. Settings changes retain an explicit wildcard when a transaction also contains scoped hints. A fresh request or failed sweep survives warm standby and idle drain; skipped dispatch never marks it complete. Shutdown stops admission of new batches and waits for the current bounded webhook attempt to settle.

Automatic Dot readiness challenges are issued once for a connected subscription. A pending or expired challenge does not imply periodic renewal; an explicit retest creates and signals another challenge. The worker keeps the existing challenge and authorization checks. No schema, OAuth, or webhook contract changes are required.

Notifications require supported writers in the same live server/database scope. Independent SQL writers and separate live server replicas do not wake this process; startup reconstructs their durable records, but continuous multi-writer discovery would require an ingress or notification contract before removing its backstop. The host sleep-safety checks remain conservative for MCP/Dot state. These retry deadlines are not yet exported to Paperclip Cloud's scheduled-wake contract.

events-work.test.ts exercises real PostgreSQL commits, rollback, startup/drain, retry restoration, producer reactivation, large batches, and SQL/network quiet. When embedded PostgreSQL is unavailable, use separate disposable databases ending in _test for PAPERCLIP_MCP_SLEEP_TEST_DATABASE_URL, PAPERCLIP_PUBLIC_MCP_TEST_DATABASE_URL, and PAPERCLIP_DOT_RUNNER_TEST_DATABASE_URL. The interaction-service regression suite also accepts PAPERCLIP_INTERACTIONS_TEST_DATABASE_URL. The suites migrate and write to those URLs; the sleep suite truncates company data between tests.