Commit Graph
4 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 f71aa135a0 fix(server): release sandbox leases for runs stopped before runner attach (#21788)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents in a `sandbox` environment run each task in a paid remote
sandbox that an environment lease tracks
> - When a run stops, the control plane must release that lease, and
terminating an agent waits until every lease of the agent is released
> - In a load test, a run stopped after its sandbox was ready but before
its runner started never released its lease, so the sandbox kept running
and the agent stayed in "terminating" forever
> - In the same test, terminating an agent whose failed run had a
scheduled same-run retry waited about 18 minutes, until the stale retry
time
> - This pull request makes lease release finish for every stop window,
records any release that must wait with a retry time, adds a hint-driven
safety net, and makes termination cancel a pending native retry at once
> - The benefit is that a stopped run never leaves a paid sandbox
running, and agent termination finishes in seconds without manual repair

## Linked Issues or Issue Description

**What happened?**
A load test with ten OpenCode agents in a `sandbox` environment stopped
its agents mid-run (pause, then terminate). Two defects appeared.
1. A run cancelled while it selected its provider pack (after the
sandbox was acquired and the workspace was prepared, before the runner
started) kept its environment lease `active` for more than 40 minutes.
The agent stayed in lifecycle `terminating`. The lifecycle worker
retried every pass and wrote `updatedAt` each time with no recorded
reason. Delete returned 409 "Complete termination before deleting the
agent".
2. A failed native run with a scheduled bootstrap retry kept its lease
until the stale retry time. Agent termination took about 18 minutes. The
run still showed "retry_scheduled" after the agent was terminated.

**Expected behavior**
A stopped run releases its sandbox lease within seconds. If release must
wait, the wait has a recorded reason and retry time, and a background
pass finishes it. Terminating an agent cancels a pending retry, releases
its lease, and reaches `terminated` promptly.

**Steps to reproduce**
1. Start a native run in a `sandbox` environment.
2. Stop it after the sandbox is acquired and the workspace is prepared,
before the runner starts.
3. Pause and then terminate the agent.
4. See that the lease stays `active` and the agent stays `terminating`.

**Paperclip version or commit**
Reproduced on `master` at 62033b9994 (load test build 2026.1106.0+2564).

**Relevant logs or output**
Root causes found in the code:
- `releaseEnvironmentLeasesForRun` returned with no record when
workspace retention refused a cancelled run. The refusal was permanent
when the run's native finalization owner had expired but was not
cleared. The owner is cleared only after `agentExecutionsHaveStopped`
sees every lease released, so the two waited on each other.
- `releaseRunLeases` silently skipped a lease whose prepared native
workspace was still "pending export", even for a run that never started
a runner and so never changed those files.
- The destroy path deferred on any busy warm session in the same
environment, also with no record.
- `stopInvocationsForAgents` returned on the first run that was still
live, so it did not release the leases of the other runs of the agent.
- The agent stop cancelled only `queued`, `running` and
`scheduled_retry` runs. A failed native run with a pending same-run
retry kept its lease until that retry fired.
- The lifecycle worker deferred a pending host step by 2 seconds every
pass, with no reason and no backoff.

## What Changed

- `releaseEnvironmentLeasesForRun` (`run-control.ts`):
- A native run with no runner launch evidence (no
`native.process_start_requested`, no runner event, no process id) skips
workspace retention and warm-session closing. No runner process exists
to fence, and no work ran against the prepared files.
- When retention is refused only because the run's native finalization
owner lease has expired, the release clears that expired owner
(compare-and-set on the exact expired owner of a cancelled run with no
result) and retries retention once. A recorded recovery then stops and
retains the sandbox, so the files stay recoverable.
- Every release that cannot finish now records
`metadata.releaseDeferral` (`reason`, `attempts`, `retryAt`,
disposition) on the active lease and publishes an `orphan-active-leases`
hint. The retry time backs off from 30 seconds to 15 minutes, then
retries hourly without end. The record keeps `failingSince` and the last
reason. A new Stop or terminate operation, or a server restart, triggers
an immediate attempt.
- `nativeWorkspaceExportPendingForLease` treats a cancelled run with no
runner launch evidence as owing no export.
- Safety net: `sweepOrphanedActiveLeases` (the existing hint- and
deadline-driven orphan-lease work) now retries the run's own release for
a due deferral or for a cancelled native run, after the native owner
lease has expired. The orphan-lease loader uses the stored `retryAt` as
the due time. No new timer or poller.
- Agent stop:
- It cancels a failed native run that has a pending same-run retry (or
an interrupted cancellation of one), through the existing
retry-cancellation path. A pause stamps `pauseStop` on those runs too,
so a budget resume can continue them.
- It releases the leases of every terminal run, and it skips a lease
whose deferral is not yet due.
- `agentInvocationStopBlocker` and `agentExecutionStopBlocker` return
why the stop must still wait.
- Lifecycle worker: a pending host or plugin step records
`pendingReason` and `pendingPasses` on the operation, and its retry
backs off from 2 seconds to a 60 second cap. When a release leaves a run
with no unsettled lease, the stopping agent's saved retry time is
cleared and the lifecycle queue is notified, so Pause or Terminate
finishes on the next pass.
- Cancelled-workspace preservation: a stopped run that never started a
runner has its prepared reference moved to
`nativeWorkspaceSyncReleasedUnchanged` when its lease is released, so
the preservation worker no longer selects it. The worker records the
same settlement itself for older runs, and waits on a live owner's exact
lease expiry.

## Verification

- Red on `master`, green on this branch (embedded PostgreSQL, fake
plugin sandbox provider):
-
`server/src/__tests__/heartbeat-lease-release-before-runner-attach.test.ts`:
agent pause then terminate, and board Stop, for each window between
sandbox acquire and runner attach (artifact discovery, pack select, pack
upload, runner launch, transport connect, session open). Each test
checks that the provider gets a stop or destroy call, the lease is
released, the run is cancelled, and the agent reaches `terminated`
within one extra worker pass. It also checks the orphan-lease safety net
for a leaked lease and a recorded deferral that the background pass
finishes when due.
-
`server/src/__tests__/heartbeat-terminate-pending-native-retry.test.ts`:
terminating an agent whose failed run has a bootstrap retry 17 minutes
away cancels the retry, releases the lease and reaches `terminated` at
once. The test covers an open task and a cancelled task.
-
`server/src/modules/agent-lifecycle/__tests__/pending-backoff.test.ts`:
a host step that stays pending records its reason, and its retry delay
grows to a 60 second cap.
- Related suites pass: run-control, recovery, orphaned active lease
sweep, environment runtime, agent lifecycle, native workspace
preservation and recovery, held comment wake.
- `server` typecheck passes.

## Risks

- A cancelled native run with no launch evidence now gets its prepared
sandbox destroyed instead of kept. The proof is the same launch evidence
that restart recovery uses.
- A launched cancelled run whose retention owes nothing is stopped and
retained, as before. A retention that throws (lock or write error, or a
live owner) now keeps the lease active with a recorded reason and retry
time, instead of a silent skip.
- Release now clears an expired native finalization owner of a cancelled
run. An old controller whose lease already expired then fails its next
owner check and stops, which is the normal lease rule.
- On the first start after deploy, the orphan-lease work releases leases
already leaked by cancelled native runs.
- Task state after pause then terminate: the run-scoped stop keeps the
task status (an open task stays open, a cancelled task stays cancelled),
and a run that held prepared files gets the existing workspace recovery
record on its stopped sandbox. No new state is added.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (code search, shell, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 12:54:23 -06:00
DottaandClaude Opus 5.5 763adbea09 fix(execution): a pause's stop never blocks the task behind an incident with no control (#20143)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A budget hard stop pauses the agent, which stops its running work
> - In the restart-chaos H1 scenario, the stopped task went `blocked`
behind a board action that said "inspect and reconcile" and offered no
button, and it stayed there after the board resumed the budget
> - A pause is an intentional stop, not a failure, so it must not leave
the task with no way forward
> - This pull request treats a pause's stop as intentional: the task
keeps its status and no incident opens, so the normal Retry works
> - The benefit is that a budget or agent pause never strands a task
behind a control-less incident

## Linked Issues or Issue Description

Refs #20113, #20101

**What happened?**
Restart-chaos wave w-20261108T210118-s9802 (H1). At the budget pause,
task CHA-1 went `blocked` (21:02:27.870) with an active board
`active_run_watchdog` action, cause
`native_continuation_requires_reconciliation`, next action "Inspect the
original failure and reconcile the previous execution before
continuing." After raise_budget_and_resume it stayed blocked,
`canRetry=false`, with no Retry. The run was cancelled by the agent
lifecycle pause (`cancellation: { reason: "Agent lifecycle stop
requested", initiator: system, expected: true }`). The wake-queue
release (`recordNativeTerminalRecoveryIfNeeded`) blocks every cancelled
native run that is not an acknowledged user Stop.

**Expected behavior**
A pause's stop keeps the task's status and opens no incident; the task
can be continued.

**Steps to reproduce**
Pause an agent (budget hold or company budget pause) while its native
run is active. Before this change, the task is blocked behind a board
action with no control.

## What Changed

- New `server/src/services/governance-pause-stop.ts`:
`isGovernancePauseStop(run)` matches an expected system cancellation
whose reason is the budget pause stop, or the agent lifecycle stop with
a `pauseStop` stamp.
- `wake-queue/adapters/postgres.ts`
`recordNativeTerminalRecoveryIfNeeded`: returns early for such a run, so
the release keeps the task's status and opens no reconciliation
incident.
- `agent-lifecycle-driver.ts`: the `pausing` host step stamps
`resultJson.pauseStop = { holds, companyPauseReason }` on the agent's
active runs before stopping them (one update, company pause reason by
subquery). Termination and cleanup share the lifecycle stop reason but
write no stamp, so they still open the incident.

## Verification

- `wake-queue/adapters/postgres.test.ts`: new cases. A run stopped by
the lifecycle pause or by the budget pause keeps the task `in_progress`
with no incident (both red on master). Agent termination, and another
system stop (company archive), still open the incident.
- `agent-lifecycle/__tests__/lifecycle.test.ts`: "records what a pause
stopped each active run for". `driver-scope.test.ts` updated for the
stamp's single update.
- `wake-queue` (107), `lifecycle` (68+), `driver-scope`,
`companies-service`: 199 passed. Server typecheck (`typecheck-layers`,
low-memory): no errors.
- Not run: the H1 chaos rerun (after the follow-up continuation PR).

## Risks

- A native run that a pause stopped no longer opens a reconciliation
incident. Resuming it is the normal Retry, which continues the
conversation. Termination, archive and failures still open one.
- One extra update per pause attempt on the agent's active runs. It is
idempotent (skips runs that already have the stamp) and not on any turn
path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 20:55:36 -06:00
DottaandClaude Opus 5.5 d33b3ac263 fix(wake-queue): save temporarily refused work intents and promote them through one path (#19956)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Users send work to agents as assignments, comments and Retries. The
wake queue delivers each one as a run
> - When an admission check refused a wake for a temporary reason, the
wake often ended as a terminal skip, so the work was lost
> - Restart-chaos waves found dropped comments and Retries (RC-37,
s7532) and double delivery (RC-53). Each fix covered one hold type
> - The durable ownership plan (section E) asks for one outbox rule: a
closed set of terminal outcomes, a saved deferral for every temporary
refusal, and one promotion path
> - This pull request adds that rule. It saves messages refused by a
pause, promotes deferred intents from every hold-release event through
one function, and adds a reconciler backstop
> - The benefit is that an accepted message is delivered exactly once
after a pause lifts, and the reconciler reports any intent that is stuck
or dropped

## Linked Issues or Issue Description

Refs #19831 (plan, section E), #19838 (inventory, wake-queue rows and
section 1b), #19927 (Phase 2 data: the I4 false positive), #19702,
#19741, #19829, #19826. Builds on #19944 (section D, merged).

**What happened?**
- A user commented while the agent was paused, or while the company was
paused. Admission refused the wake. Nothing saved the message as
pending, so when the pause lifted, nothing delivered it.
- Hold-release events woke different subsets of workers. A saved intent
then waited out a 30-second refusal backoff after its hold cleared.
- The I4 reconciler check matched free text with a regular expression.
Phase 2 found a false positive: a delivered retry whose error said
"retrying once".

**Expected behavior**
An accepted work intent ends only as delivered, consumed by a later run,
task cancelled, or hard stop. Every temporary refusal saves the intent
as `deferred` with its reason and its release trigger. Every
hold-release event promotes deferred intents through one function
(invariant 4).

**Steps to reproduce**
Pause an agent. Post a user comment on its task. Resume the agent. On
`master` the comment is never delivered as a wake.

## What Changed

- New `server/src/services/work-intent-outbox.ts`:
- `WORK_INTENT_TERMINAL_OUTCOMES`: the closed set of terminal outcomes.
- `TEMPORARY_REFUSAL_TRIGGERS`: maps each temporary refusal code to its
release trigger.
- `promoteDeferredWorkIntents`: the one promotion function. It removes
the 30-second refusal backoff from the matching deferred intents. It
then wakes run dispatch, the saved-receipt worker and stranded-work
recovery, and hints each saved intent of the issue.
- `heartbeat/queue.ts`: the #19829 save-as-deferred block is now a
shared helper. A user's message refused by a paused agent, a paused
company, or a budget pause that is not a reached hard stop is saved with
`executionWait: { reason, trigger, message }`. The caller still gets a
409. The message is saved only when the task's latest run is terminal,
because the release drain promotes a saved message behind that run. A
task that never ran keeps the message in the task, and stranded-work
recovery starts the task once the pause lifts. A `pending_approval`
agent and a task-tree hold are not changed here: the release drain still
fails a promoted wake for a pending approval, and tree holds have their
own resume path.
- Hold-release events call `promoteDeferredWorkIntents`:
  - accounting settled (`run-cost-accounting.ts`);
  - environment lease released (`environments.ts`, two sites);
  - native run projected terminal (`native-run-projection.ts`);
  - budget pause lifted (`budgets.ts`);
  - company reactivated (`companies.ts`);
  - task-tree hold released (`issue-tree-control.ts`);
  - agent lifecycle ready (`agent-lifecycle-driver.ts`).
- Reconciler (`recovery/ownership-registry.ts`,
`ownership-invariant-report.ts`, `native-finalization-reconciler.ts`):
- registers three new kinds: `wake_request_deferred`, `queued_run` and
`scheduled_retry_run`;
- I4 now reads the closed refusal code (the wake `reason`) of accepted
intents, not free text;
- a new I4 check reports a deferred pause refusal whose pause has lifted
for more than 5 minutes. The worker then promotes it as a backstop;
- a lifted pause inside its 5-minute grace window returns its deadline
through `nextDueAt`, and the worker also wakes on agent-lifecycle
signals (company and agent pause changes). So a missed release event is
still checked.
- `doc/SPEC-implementation.md`: adds the rule "Accepted work intents are
delivered exactly once".

## Verification

Each focused file ran alone in one vitest process (`--maxWorkers=1`) on
embedded Postgres. Results at `2875c85fd2`:

| Question | Loop | Elapsed | Result |
|---|---|---|---|
| Is a comment refused by a pause saved, then delivered once after the
pause lifts? | `heartbeat-held-comment-wake.test.ts -t "pause lifts"` |
8 s | 2 pass. With `queue.ts` from `master`, both fail: no intent is
saved |
| Does I4 use the closed code, and does it catch a stuck deferral? |
`ownership-reconciliation-report.test.ts` | 45 s | 20 pass. This
includes the Phase 2 "retrying once" false positive and the case
"paused, then lifted" |
| No regression on the paths that changed | held-comment (15),
archived-company guard (8), undispatched execution wait (32), deferred
receipt work (45), recovery pause durability (16), native run projection
work (55), tree-control routes and services (19, 1, 7) | 8–334 s each |
All pass |

- Exactly-once: each pause test runs two promotions (a release event,
then the reconciler backstop) and many dispatch passes. It asserts one
delivery. The existing consumption tests (RC-53) still pass.
- Server package typecheck passes.
- Speed: nothing new runs before `turn.start`. The new admission work
happens only on a refusal. Promotion runs in the release transaction and
in the background reconciler.
- Not run here: chaos waves. The coordinating session schedules them on
the shared harness.

## Risks

- `promoteDeferredWorkIntents` moves `updated_at` back for deferred
intents that were refused in the last 30 seconds. A busy company with
many release events can retry refused intents more often. Each retry
re-runs normal admission, and the refusal resets the backoff.
- More release sites now wake run dispatch and stranded-work recovery.
These notifications are coalesced per company.
- Lifting a pause is now the operator's explicit authorization to send
what the pause refused, as Send is after a Stop. When the pause stopped
the task's run, the lifted pause issues the task's next explicit wake,
which adopts the saved message once. Other messages queued behind an
acknowledged Stop keep the existing rule. If Stop recovery still holds
the task, the message waits for that recovery.
- An agent pause's stop sweep keeps the messages that the pause deferred
(`preservePauseDeferred`). Terminate and cleanup stops still end them.
- The backstop check reads only pause refusals (`company.inactive`,
`agent.not_invokable`), because those holds can be checked exactly.
Other deferred kinds keep their existing promotion paths.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 23:43:17 -06:00
DottaandPaperclip 30b753f614 perf(budgets): scope accounting and deliver enforcement asynchronously (#17046)
## Thinking Path

> - Paperclip manages AI agents and their paid work.
> - Budgets must prevent new work and stop reported overruns.
> - Cost settlement also ran serial lifecycle reconciliation for every
eligible company agent.
> - That made request latency grow with unrelated idle agents and
delayed results on stop delivery.
> - This change separates atomic admission and accounting from durable
asynchronous enforcement.
> - Scoped live usage detects overruns without writing provisional
ledger charges.
> - The result is constant query scope for ordinary settlement and
recoverable stop delivery.

## Linked Issues or Issue Description

**What happened?**

Settlement and budget checks entered company-wide lifecycle
reconciliation. A company with 1,000 unrelated idle agents performed
1,001 accounting transactions for one ordinary cost evaluation. User
requests could also wait for provider termination. Running usage
checkpoints did not enforce reported monetary exposure.

**Expected behavior**

Keep the final reservation gate authoritative. Commit charges, scope
blocks, and enforcement intent before returning. Deliver stop requests
after commit. Read only applicable policies and execution scopes for
ordinary charges. Stop long-running work when supported reported or
priced usage establishes an overrun.

**Steps to reproduce**

1. Create a disposable company with one company budget and idle agents.
2. Evaluate a charge below the budget with 0, 17, and 1,000 unrelated
idle agents.
3. Compare query counts, company accounting transactions, and duration.
4. Use the controlled partial-usage and delayed-stop fixtures to check
enforcement and request responsiveness.

**Paperclip version or commit**

The measured baseline is `6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837` from
`master`.

Related public work: Refs #16976, #2006, #3539, #13763, #14296, #16853,
#13075. These cover the affected service charge, cancellation, and
accounting paths. This change removes their shared reconciliation
overhead.

## What Changed

- Batch applicable policy exposure under the existing company accounting
lock. Preserve exact fractional money and zero-reservation dispatch
fences.
- Commit a durable final budget dispatch claim before provider I/O.
Retain a separate final issue/run ownership transaction through
synchronous handoff, without holding the company accounting lock or
awaiting provider work. Preserve native recovery lease checks and
duplicate handoff fencing. Browser handoff rechecks current policy; it
reuses a consumed rate slot only for the identical policy generation.
- Separate ledger spend, live observed spend, and effective held
capacity. Use the maximum of reservation and live observation for each
unfinished operation.
- Deliver budget cancellation through the existing coordinator.
Revalidate generations and persist queued/running stop fences before
external termination. Bound stop concurrency to four.
- Apply agent holds only to the affected agent. Project budgets leave
other projects running. Project real company transitions in bounded
lifecycle batches.
- Replace periodic broad reconciliation with durable pending work,
startup repair, receipt replay, and concrete retry/window deadlines.
Retain drain and idle-safety checks.
- Enforce accepted cumulative usage checkpoints without provisional
charges. Final accounting atomically replaces live exposure and settles
reservations.
- Add operator-gated budget spans, regression coverage, map updates, and
a real-PostgreSQL comparison harness. No schema or first-party telemetry
change.

## Verification

- Full workspace `pnpm -r typecheck` passes on the final ownership fix,
rebased onto `bbcf5dfe98`. Production `pnpm build` passes on final head
`4c5f4de8b1`. Map, module-boundary, token, and Storybook catalog checks
pass.
- Focused real-PostgreSQL coverage includes the 292-case rebased
accounting/dispatch run, 63 browser cases, 52 run-control/cancellation
cases, 42 policy-generation cases, 77 lifecycle/commentary cases, and
the exact Muse confirmation regression. Native executor unit coverage
passes all 768 cases. Final ownership coverage passes: 74
adapter/use-case cases, nine public regression cases, and the additional
deleted-project race. These exercise competing ownership, budget changes
and charges between phases, zero/nonzero duplicate claims, rollback,
recovery contention, read-only capacity rechecks, and controlled HTTP
startup after commit. Another 74 pure/startup/tracing cases pass. The
complete CI matrix and both 5/5 reviews passed on `1ac9b58`; its
external security scan identified a stale-owner handoff race. The final
fix restores synchronous issue/run ownership fencing after the
accounting transaction commits. Final head `8e84f55ce8` has fresh
Greptile 5/5 with zero new comments, Paperclip review 5/5, and a passing
security scan. The complete latest-head [CI
matrix](https://github.com/paperclipai/paperclip/actions/runs/39535189808)
passes, including budget, browser, Runner, build, and canary jobs. The
unchanged Runner close fixture passed on its sole same-head job retry.
- Broad local `pnpm test:run` invocations were stopped before required
source updates (rebase and review fixes). They are not claimed as full
passes. Final full-suite coverage comes from the complete latest-head CI
matrix; local focused coverage and the incomplete broad invocations are
reported separately.
- Reproduce performance with `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-budget-enforcement.ts
--baseline-ref=6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837`; repeat with
`--reverse`. The harness uses disposable real PostgreSQL, 20 warm
samples per variant/count, and records raw timings, SQL/lock counts,
agent reads, unchanged agent state, and query plans. Final
production-source measurement revision:
`eff713a4f72c7f4b7a4002a701cb481d4bfccd95`; the extended harness is
committed in `4c5f4de8b1`.

| Unrelated idle agents | Baseline median / p95 | New median / p95 |
Baseline SQL / locks | New SQL / locks | New agent reads |
| --- | ---: | ---: | ---: | ---: | ---: |
| 0 | 1.80 / 2.33 ms | 2.99 / 3.88 ms | 5 / 1 | 7 / 1 | 0 |
| 17 | 15.53 / 19.55 ms | 2.24 / 2.60 ms | 90 / 18 | 7 / 1 | 0 |
| 1,000 | 978.35 / 1,790.43 ms | 2.16 / 2.52 ms | 5,005 / 1,001 | 7 / 1
| 0 |

- These measurements use local PostgreSQL 18.1 and Node 26.4.0 arm64.
They isolate the cost-evaluation core and complete admission path,
without hosted-region or provider latency. Prior-revision reversed-order
controls also preserved the constant query counts; the final
measurements above use the final production source.
- Complete reserve → final ownership gate → immediate offline provider
result is measured separately, with one responsible agent and one active
operation at each unrelated-agent count. Admission p95 baseline/new: 0
agents **11.54 / 29.77 ms**; 17 agents **9.31 / 27.62 ms**; 1,000 agents
**24.51 / 74.05 ms**. Baseline admission was already scoped; the
stronger final fence adds bounded overhead. New counts stay constant at
53 Drizzle statements, three transactions, two company-accounting locks,
and five agent-reading statements bound to the responsible agent. Zero
unrelated-agent reads, exactly one provider callback per sample, and no
leftover runs or holds. All admission scenarios stay below 250 ms. No
external provider calls are included.
- The partial-usage/delayed-termination fixture passed 20/20: receipt
persistence p95 13 ms, receipt return to observed durable Stop fence p95
28 ms, complete receipt-start-to-fence p95 50 ms, unrelated database
request p95 1 ms while termination was blocked. After releasing the
controlled termination fixture, acknowledgement p95 was 14 ms. These are
separately measured phases; fixture acknowledgement is not live-provider
termination latency.
- Models and provider deadlines are unchanged. No deployment, live
inference, or staging acceptance is claimed. A quiet 30-minute follow-up
waits for the merged revision on staging, then repeats classifier and
controlled budget-stop acceptance in staging, recording hosted
bookkeeping, queue delay, and actual provider acknowledgement
separately.
- Development failures retained: finalizer notification regressions
(fixed; 27/27 affected cases passed), invalid browser admission fixture
assumptions (fixed; browser coverage now passes 63/63), and first-head
lifecycle/dispatch CI regressions (reproduced and repaired). A
prior-head Cases E2E test hit its unchanged announcement-dismiss handler
race while opening a filter; the same journey passed on `1ac9b58`. The
external security review then found a stale-owner race between
accounting commit and provider entry; final verification covers a
competing run in that gap while retaining provider I/O outside the
accounting lock. Early PostgreSQL startup attempts hit local semaphore
pressure; serial reruns succeeded. The final-head Runner lane had one
failure among 1,823 tests: the unchanged `timeout-attachment` fixture
failed at its first process close after 6.28 seconds. The sole same-head
job retry passed all 1,819 active tests (four skipped), including that
fixture in 1.85 seconds. Three isolated local probes could not run
correctly because the Rust fake-provider fixture binary was absent;
cleanup masked the original startup failure. CI builds that fixture
through its normal lane. These probes are inconclusive and are not
counted as passes.

## Risks

- This changes accounting, lifecycle, and cancellation ordering. Durable
generations, source/sequence checks, ownership leases, and company
locking remain required for correctness.
- Provider stop acknowledgement can be slow or fail. Stop intent remains
pending until existing termination proof succeeds; cancelled row status
alone does not acknowledge delivery.
- Mid-run limits depend on provider usage reports or supported price
calculations. Providers that report only at completion cannot provide
exact mid-call detection.
- Signals are process-local wake hints for the supported
single-active-writer model. Durable state and startup recovery remain
authoritative.
- There are no migrations, new queue tables, or cached balances. Repeat
the classifier and controlled budget-stop acceptance after normal
rollout.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, code execution,
and parallel coding agents. The host does not expose the exact backend
model revision or context-window size, so those values are not asserted.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 07:23:00 -05:00