Commit Graph
88 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 148d2b629c fix(heartbeat): renew legacy controller leases on their own connection, timed from the statement (#21042)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A legacy run keeps a controller lease on its `heartbeat_runs` row.
The server renews every lease in the process with one batched statement
every 10 seconds, and stops a run when its local lease deadline passes.
> - The batched renewal ran on the 2-connection liveness pool. Native
runner lease renewals use the same pool. Each native renewal holds a
connection for a row lock, about seven notifications, and an update.
> - With 18 native runner runs renewing every 20 seconds on a slow
database, the legacy batch waited 30 to 50 seconds for a connection. The
local deadline also counted that wait, so it fired 20 to 40 seconds
before the database lease could expire.
> - Runs then stopped with `controller_lease_lost` while no other
controller owned them, and their issues looped through retries.
> - This pull request gives legacy renewal its own connection and
measures the deadline from the time the statement starts on that
connection.
> - The benefit is that a slow server no longer stops runs that still
own their lease. A real takeover still stops the run.

## Linked Issues or Issue Description

**What happened?**

On staging, after the 05:37Z deploy, 63 of about 175 new runs failed
with `controller_lease_lost` ("Legacy controller lease lost") at about 3
per minute. All were legacy runs in preparation
(`executionRecovery.kind: bootstrap`, `providerWorkStarted: false`). One
example had `controllerLeaseLoss: {cause: "renewal_deadline",
renewalCount: 7, sinceRenewalMs: 42537, renewalPendingMs: 38798,
deadlineLateMs: 134}`. Several runs that failed at the same moment had
identical timing values, so they were members of the same renewal batch.
The previous successful renewal took 17 to 44 seconds, so the deadline
that it set was only 16 to 43 seconds away. At that time 27 legacy runs
were preparing and 18 native runner runs were renewing on the same
liveness pool. The issues looped: `issue_assigned`, lease lost,
`issue_continuation_needed`, lease lost, `transient_failure_retry`,
lease lost.

**Expected behavior**

A slow server or database must not cause a lease loss while the
controller still owns the lease. Only a real expiry or a takeover by
another controller must stop the run.

**Steps to reproduce**

1. Start legacy runs and native runner runs on one server.
2. Make the database slow, so that each native lease renewal holds a
liveness connection for several seconds.
3. The legacy renewal batch waits for a liveness connection, and its
next local deadline is shortened by that wait.
4. The next renewal also waits, and the local deadline fires first. The
run fails with `controller_lease_lost` (`renewal_deadline`).

**Paperclip version or commit**

Staging at 8cefbe9977. The fix is based on master at aa94939988.

**Deployment mode**

Hosted cloud (staging).

**Agent adapter(s) involved**

Legacy controller runs (Codex and Claude local adapters). Native runner
runs share the liveness pool.

## What Changed

- `packages/db`: add `controllerLeaseDb(db)`, a private 1-connection
pool (`paperclip-legacy-controller-lease`) for legacy controller
renewal. `closeLivenessDb` also ends it, so shutdown does not change.
- `legacy-controller-lease.ts`: the batched renewal runs on
`controllerLeaseDb` instead of the liveness pool that native lease
renewals use.
- `legacy-controller-lease.ts`: each batch records the time when its
transaction starts on the connection. The watcher measures the next
deadline from that time, not from the time the request was queued. The
database sets the new expiry at UPDATE time, which is later, so the
local deadline never passes the database expiry.
- The renewal statement, its boot-id and expiry conditions, and
revocation do not change. A takeover or an expired lease still rejects
the renewal and stops the run.

## Verification

- Red/green regression tests in
`server/src/services/legacy-controller-lease.test.ts`:
- `renews without queueing behind native lease transactions that hold
every liveness connection` (embedded Postgres). Before the fix, the
renewal waited 9.8 s behind four 5-second liveness transactions. After
the fix, it completes in less than 2 s.
- `keeps a controller owned when every renewal waits thirty-five seconds
for a connection before its statement runs`. Before the fix, the lease
was lost with `{sinceRenewalMs: 25000, renewalPendingMs: 20000,
renewalCount: 1}`, the same pattern as on staging. After the fix, the
controller stays owned for four lease periods.
- Updated the private-pool tests to check the new pool and its
application name.
- `npx vitest run src/services/legacy-controller-lease.test.ts`: 40
passed.
- `pnpm verify` (affected tests), plus `typecheck` for `packages/db` and
`server`.

## Risks

- Each server process opens one more database connection, only when it
has a legacy run.
- The local deadline can now be later than before, by the time that the
batch waited for a connection. It stays at or before the database
expiry, because the database sets the expiry after the recorded start
time. Readers that act on an expired lease are not affected.
- Low risk otherwise. There is no schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #20924, #20936

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 00:52:06 -06:00
DottaandClaude Opus 5.5 5a7378af56 fix(heartbeat): re-drive a queued run whose dispatch signal was lost after a restart (#20986)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts queued runs. The run-dispatch worker
re-runs the queue only when a write sends a "queued run dispatch"
signal.
> - A follow-up run can queue while its predecessor still holds its
sandbox lease. The claim then declines the run without a log line, and
the run stays `queued`.
> - After a controller restart, nothing sent a new signal once that
lease was settled. The run stayed `queued` forever. The issue had no
Retry, and the reconciler counted the queued run as a valid next action.
> - This pull request sends a dispatch signal from the native
finalization writes that can clear a predecessor's cleanup gate, logs
why a claim was declined, and bounds the wait. The dispatch worker
re-drives an undispatched queued run after 60 s as a backstop, and the
ownership reconciler reports and re-drives one still queued after 5
minutes.
> - The benefit is that a follow-up starts within seconds of its
predecessor settling, a lost signal delays it by about a minute instead
of forever, and a run that still cannot start is visible.

## Linked Issues or Issue Description

Refs #20909, #20935, #20953, #20970

**What happened?**

Restart-chaos wave `w-20261111T223714-s9512` (master `6154b9485e` +
#20907 + #20909, Daytona warm, Codex, W2, CR1 with a 360 s hold,
`--ephemeral-server`, seed 9512), task t02. Run `0b95b1cd` crossed the
outage with an accepted result. The next controller finalized it at
22:47:59Z. Its continuation run `1ad79418` was queued at 22:48:00Z
(`retryOfRunId` = `0b95b1cd`, wake `d967700c`, source
`native_status_decision`). The run stayed `queued` with `claimed_at =
null` until the wave ended 19 minutes later. The agent stayed `running`
with no running run. The task had no Retry. The grade was
`R1:no_rest_within_budget` and U1 `no_owner_no_action ... queued 485s`.

**Root cause**

`claimQueuedRun` (`server/src/services/heartbeat/queue.ts`,
`lockIssueExecutionClaim`) declines a queued run while
`hasPendingRunCleanup` (`server/src/services/execution-blocker.ts`) is
true for its predecessor. After the restart, the predecessor's warm
`reuse_by_environment` lease was still `active` when the continuation
queued, so the first claim was declined. The decline is correct, but
nothing owned the retry. `registerRunDispatchWork`
(`server/src/services/heartbeat/dispatch-work.ts`) has `next: async ()
=> null`: the queue is only re-run by a later dispatch signal. At wave
end the lease was `retained` with `cleanupStatus: success`, which does
not block a claim. A dispatch pass in that state starts the run
(verified locally). So no pass ran after the gate cleared. The ownership
reconciler listed the run as `queued_run` with owner `run_dispatch`
(always "alive"), and I7 treats any queued run as a next action, so
nothing reported it.

This is not the #20935 throttle. The wave base predates #20935, and a
throttled intent keeps its wake intent `queued` with a `nextAllowedAt`
deadline. Here the run row existed and its claim was never retried. On
current master #20935 parks this specific `response_wake` self-wake, but
any continuation, comment wake or Retry that queues behind a settling
predecessor after a restart has the same gap. The red test uses a
`same_agent` continuation, which master still enqueues.

**Steps to reproduce**

`server/src/__tests__/heartbeat-native-restart-suspension.test.ts`,
"starts a continuation that queued behind its predecessor's sandbox
release after a graceful restart".

## What Changed

- Event-driven dispatch. A lease release already signals queued-run
dispatch (`environmentsSvc.releaseLease`, `writeEnvironmentCleanup`,
stop-and-retain settlement). I enumerated every `environment_leases`
status writer: each one that moves a lease out of a blocking state
already signals. The writers that do not signal only acquire a lease or
set `pendingCleanupInFlight`, so they add a block and do not clear one.
The native coordinator side had no signal, so:
- `notifyNativeFinalizationWork` (the shared notifier of coordinator
commits and terminal transitions) now also signals queued-run dispatch
for the company.
- The coordinator writes that clear `leaseOwner` or end the phase
outside that notifier now signal in the same transaction:
`releaseLapsedCoordinatorOwner`, native startup cleanup,
observed-workspace preservation, `record_finalization_error` in the
interaction bridge, the wake-queue terminal recovery, and the
reconciler's ambiguous-state terminal failure.
- Not changed: three non-transactional pre-dispatch closes in
`run-completion.ts` and `agent-execution-stop.ts`. Their callers queue
or settle the successor after the write, so the successor's own enqueue
starts the queue.
- `claimQueuedRun` now logs the reason when it declines a queued run at
the issue execution claim (`predecessor_cleanup_pending`,
`predecessor_still_executing`, `issue_execution_owned_by_another_run`,
case handoff, task chat cleanup), when a predecessor is still settling
in this process, and when a temporary budget hold keeps it queued.
- New `server/src/services/queued-run-dispatch-liveness.ts`:
`listUndispatchedQueuedRuns` reads queued runs in active companies whose
ready, non-paused agent has no running run. It reads every page with a
keyset cursor until the set is exhausted, so older held runs cannot hide
a newer stranded run. `createQueuedRunRedriveSchedule` re-drives each
such run 60 s after it was created, then every 2, 4, 8 minutes, capped
at 15 minutes. Run ids that leave the set are dropped.
- `server/src/services/heartbeat/dispatch-work.ts`: the queued-run
dispatch worker's `next` now plans those re-drives. It re-reads only the
companies the pass just dispatched, keeps the deadlines of every other
company, and runs no query when the pass dispatched nothing. Only an
instance-wide pass (startup or explicit wake) reads every company. It
adds each due company to the dispatch scopes and returns the next due
time.
- `packages/db`: new partial index `heartbeat_runs_queued_created_idx
(company_id, created_at, id) WHERE status = 'queued'` (migration
`0470_heartbeat_runs_queued_created_index.sql`, `CREATE INDEX
CONCURRENTLY IF NOT EXISTS`, registered as a top-level migration in
`client.ts`). It returns `null` (no deadline, no SQL until a signal)
when no run is undispatched or scheduling is suppressed. Every re-drive
goes through the normal `startNextQueuedRunForAgent` and
`claimQueuedRun` gates, so it never bypasses a hold, budget, pause or
cleanup gate.
- `server/src/services/heartbeat.ts`: wires `listUndispatchedQueuedRuns`
into the worker (two lines).
- `server/src/services/recovery/ownership-invariant-report.ts`: new
check. A queued run still undispatched 5 minutes after creation is an I7
violation (`kind: queued_run`, `reason: queued_run_not_dispatched`),
written to the activity log once. A younger one sets the report's next
due time.
-
`server/src/services/native-runtime/native-finalization-reconciler.ts`:
`applyOwnershipReportBackstops` sends a queued-run dispatch signal once,
for each company with a newly found violation. Repeat passes do not
signal again, so a held run does not wake the report every 30 s.
- `tests/restart-chaos/README.md`: records the wave and the fix in the
CR results table. The existing CR1 cell (#20909) and the enforced U1
check already cover this class in the harness.

## Verification

- RED on master (`23909077ad`):
- `heartbeat-native-restart-suspension.test.ts` "starts a continuation
that queued behind its predecessor's sandbox release after a graceful
restart": the continuation is `queued` behind the predecessor's `active`
warm lease. The run-dispatch worker is registered, the lease then
settles to `retained`/`success` with no dispatch signal, and the run
stays queued: `AssertionError: expected null not to be null` after 75 s.
- `ownership-reconciliation-report.test.ts` "reports and re-drives a
queued run that no dispatch started while its agent had nothing
running": `expected [] to deeply equal [ ObjectContaining{…} ]` (no
violation for a run queued 19 minutes).
- `ownership-reconciliation-report.test.ts` "signals queued-run dispatch
when finalization work settles or a finished coordinator's lapsed owner
is released": `expected [] to include 'settled'`.
- GREEN on this branch: all pass. The restart test has two cases. When
the lease is released through `releaseLease`, the continuation starts
within 5 s. When the lease is settled by a write with no signal, the
backstop starts it within 75 s. A new report test puts three older held
runs ahead of a stranded run with a page size of 2, finds the stranded
run, and checks that the backstop signals once.
- `queued-run-dispatch-liveness.test.ts`: schedule bound, doubling, cap,
and going quiet.
- Focused suites pass (22 files, 495 tests):
recovery-deadline-acceptance, live-recovery-work,
heartbeat-run-dispatch-lifecycle, heartbeat-run-dispatch-work,
deferred-execution-receipt-work, heartbeat-native-cleanup-admission, the
ownership kinds suites, and others.
`heartbeat-native-restart-suspension` and
`ownership-reconciliation-report` pass in full. `tsc --noEmit` for the
server passes.

## Risks

- A queued run that is held on purpose (for example a temporary budget
hold or a case handoff waiting for release) is now re-checked at most
every 15 minutes after the first minute. Each re-check is one indexed
query and the normal claim gates. Such a run is also reported once as an
I7 violation.
- An instance with an undispatched queued run keeps a dispatch deadline,
so it does not idle while that run waits. An instance with no such run
is unchanged.
- Schema: one additive partial index, built concurrently. No data
change. State saved by an older version is read as is. Nothing runs
before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:04:58 -06:00
8818402e7a feat(inbox): route requests to people and record personal outcomes (#20959)
## Thinking Path

> - Paperclip helps people manage AI agents and their work.
> - The inbox collects requests that need a human decision.
> - Teams need to know who owns each request and who may act on it.
> - A shared close operation must differ from hiding a card for one
person.
> - This pull request adds personal routing and durable outcomes to the
existing attention service.
> - People can claim or transfer work and inspect what another person
did for them.
> - Domain permissions still decide who may resolve each request.

## Linked Issues or Issue Description

**Subsystem affected**

Server, shared contracts, and database. This is the routing foundation
for a stacked inbox UI change.

**Problem or motivation**

A company-wide attention feed does not give teams a clear personal
queue. Archive, withdrawal, transfer, and decisions by another person
need distinct scopes and records.

**Proposed solution**

Resolve a request audience from its addressee, task participants, and
configured owners. Provide personal, unassigned, company, all-assignee,
and archived views. Add claim, transfer, personal archive and snooze,
shared withdrawal, receipts, and personal history.

**Alternatives considered**

A single shared inbox leaves ownership and personal hiding ambiguous.
The service keeps one canonical request and projects its audience
instead of copying requests for each person.

**Roadmap alignment**

This extends the existing Multiple Human Users and Agent Reviews and
Approvals capabilities in ROADMAP.md.

**Additional context**

Refs #19363 and #19520 for the current inbox navigation and confirmation
replacement rules. Refs #20268 for related actionable-request work. That
PR filters actionability; this stack adds audience routing and personal
outcomes.

This is part 1 of a two-PR stack. The inbox UI PR #20961 targets this
branch.

## What Changed

- Add nullable company and project decision owners and a company budget
owner.
- Add audience views and personal badge counts to the canonical
attention projections.
- Add claim, transfer, withdrawal, personal archive, snooze, and restore
operations with permission checks and activity records. Lock addressed
transfers to the current addressee or routing administrators, and
validate the recipient against canonical resolution policies.
- Add receipts and scoped history for work handled by another person.
- Keep closed-task requests and replaced confirmations out of current
cards.
- Add an idempotent migration and a concurrent receipt-history index.
- Update typed fixtures so the existing UI remains compatible with the
new contracts.

## Verification

- Server layered typechecking passes, including all three test-tree
partitions.
- Seven routing and interaction-resolution suites pass all 249 tests
against a dedicated external PostgreSQL test database on `4caecc21d8`.
Six new regressions failed before the repair; they cover takeover
followed by approval, legitimate delegation, excluded creators/review
requesters, and unauthorized claims.
- Prior focused verification covers private approval history and
receipts, transfer visibility, audience escalation, withdrawal
governance, hidden-card counts, per-recipient confirmation replacement,
and SDK source contracts.
- A synthetic three-person company passed 37 routing and lifecycle
acceptance checks. The UI stack adds browser acceptance for the shared
controls.
- The complete [GitHub Actions CI
matrix](https://github.com/paperclipai/paperclip/actions/runs/43134997331)
passes on `4caecc21d891677323a7335f23b1a015cce24c13` (Linux/Node 24:
typecheck, build, general and serialized tests, Runner support, browser
shards, and aggregate gates). Greptile is 5/5 on that exact head with no
open findings. The fresh security scan passes after explicitly
rechecking the addressee-transfer repair.
- `pnpm verify` reports exit 2 because a CI-global script changed; full
CI supplies that broader coverage. Embedded PostgreSQL could not start
on this Mac because shared-memory slots were exhausted, so local
database checks use a separate external test database. Linux CI covers
the embedded setup.

## Risks

- Personal badge counts change to the viewer's assigned requests.
Accessible feed totals and task membership keep their existing scope.
- The migration adds nullable owner fields and a dismissal fingerprint.
Its index runs outside a transaction. The migration is classified and
rerunnable.
- Routing changes the addressee used by resolution authorization. The
current addressee or a routing administrator must authorize an addressed
transfer; the recipient must satisfy writer access, canonical resolver
exclusions, and task review policy under row locks.
- The shared validator export barrel crosses 1,000 lines because it
exports the new attention schemas. Schema implementation stays in the
dedicated attention validator module.

## Model Used

OpenAI GPT-6 through Codex. Used repository tools, code execution, and
browser checks. This session does not expose a more specific runtime
model ID or its context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
(OpenAPI contracts and docs/api/attention.md)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-11 17:49:39 -08:00
DottaandPaperclip cda23f08ac fix(db): preserve legacy project reads during rolling upgrades (#20860)
## Thinking Path

> - Paperclip controls agent work while web servers change versions.
> - The old and new servers can share a database during that change.
> - Older servers still read and write the project environment column.
> - Migration 0466 removes that column before those servers stop
serving.
> - A later repair alone leaves a committed missing-column window and
loses values.
> - This pull request applies the exact drop and repair in one
transaction.
> - Old reads and saved values survive while current placement behavior
stays unchanged.

## Linked Issues or Issue Description

Refs #20835 and #20852.

**What happened?**

Migration 0466 drops `projects.environment_id`. The previous server
still selects that column and reads it during admission. Applying a
separate repair afterward leaves a window in which those requests fail.
The drop also removes saved values.

**Expected behavior**

A database that has not yet applied 0466 must keep the column and its
values visible to concurrent old servers throughout the upgrade. Both
published migration entries must be recorded truthfully. Current project
responses must keep the new workspace placement contract.

**Steps to reproduce**

1. Use a database before migration 0466 with nonempty project
environment values.
2. Keep an old project writer transaction open and start the pending
migrator.
3. Start an old project read, then commit the writer.
4. Without this repair, the saved values disappear. With this repair,
the reader sees the preserved values and both migrations commit
together.

## What Changed

- Add migration 0467 to restore the legacy nullable UUID column and its
original foreign key. Leave the published 0466 SQL unchanged.
- Apply only the exact SHA-bound 0466/0467 pair in one transaction. Lock
the project table, retain nonempty values in a temporary table, execute
both real SQL files, restore values, and record both real journal
entries before commit.
- Apply the additive 0467 repair normally when 0466 was already
committed.
- Keep a deprecated database-only field and omit it from current project
API responses. Current placement logic stays unchanged.
- Add PostgreSQL regressions for concurrent old reads and writes,
idempotence, rollback, and an already-applied drop. Document the overlap
contract and retain the five newest generated snapshots.

## Verification

- Node 24 cached agent preflight passed.
- `pnpm exec vitest run
packages/db/src/project-environment-overlap-migration.test.ts
server/src/__tests__/project-environments.test.ts -t 'project
environment rolling migration|saves storage, not'`: 4 passed in 32.57
seconds against real disposable PostgreSQL.
- Before the owner fix, the regression had two failures: saved values
disappeared and the restore-failure case committed. The fixed case
proves that both DDL and journal entries roll back together.
- `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter
@paperclipai/db check:migrations`: passed.
- `git diff --check`: passed. Migration 0466 bytes are unchanged. The
generated snapshot set is 0463 through 0467.
- Review found a raw DELETE response that still exposed the
database-only field. The delete projection now omits it; a real
authorized HTTP DELETE passed with successful cleanup. The concurrency
test now observes the old reader waiting inside PostgreSQL before
releasing the old writer. All three migration cases passed with that
barrier. Earlier HTTP fixture setup/cleanup failures remain recorded as
failed attempts; they were not product-regression passes.
- Candidate `047178538c0df3a9e382259a51635a6e46da5646` has fresh
Greptile 5/5 with no open findings. [Required final-head
CI](https://github.com/paperclipai/paperclip/actions/runs/43062354612)
passed, including build, typecheck, migration checks, all test shards
and the aggregate verification gate. The separate Superagent security
scan ended inconclusive/neutral; it did not produce a passed scan.
Deployed acceptance remains pending. No hosted database or app
deployment was changed by this branch.

## Risks

The pair takes a table lock for its transaction. Operators must retain
their normal lock and statement timeouts. Both SQL files and their exact
hashes must match before the pair runs. A failed restore rolls back the
drop and both journal entries.

If 0466 was already committed, 0467 restores the column and foreign key.
It cannot recover values that were already deleted by the earlier
release. This repair does not claim to recover those values.

## Model Used

OpenAI Codex (GPT-6), with tool-assisted implementation and real
PostgreSQL qualification. The exact serving model ID and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-11 11:08:30 -06:00
DottaandClaude Opus 5.5 8e4c5e81b9 perf(issues): read created-from tasks by index and inbox review waits by page (#20573)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The task page shows "Created from this task". The inbox shows
blocked-inbox attention for a page of tasks.
> - On a hosted database, the created-from list ran about 4,000 times in
one afternoon at about 499 ms each. The task page sends it each time a
task opens.
> - Its filter was an `OR` of three provenance paths over the list's own
rows. PostgreSQL could not use an index for it. It read every task of
the company and tested each one, with the list's other per-row
conditions (such as the chat conversation state subquery).
> - The blocked-inbox attention map also read external review waits for
every in-review task of the company, not only for the tasks of the page.
> - This pull request reads created-from tasks from three index probes,
and reads review waits only for the in-review tasks that can change a
finding of the page.
> - The benefit is less database work on each task open and each inbox
page. The responses stay the same.

## Linked Issues or Issue Description

Refs #20400 (shares review-wait reads between the attention maps of one
request).
Refs #20551 (authorizes the review waits of a page with one read; this
change narrows the waits that the page asks for, and adds a test that
pins the batched read).
Refs #13241 (adds the created-from list).

**What happened**

- `GET /api/companies/:id/issues?createdFromIssueId=…` used
`createdFromIssueCondition`. Its filter was `origin_issue_id = source OR
(origin_issue_id IS NULL AND origin_run_id IN source runs) OR (… AND
EXISTS creation activity of a source run)`. The `OR` stops index use, so
the statement read every task of the company. The list's other
conditions, which include the chat conversation state subquery, ran for
each task before the task was excluded.
- `listIssueBlockedInboxAttentionMap` read review waits for every
in-review task of the company graph. The caller uses the findings of the
page rows only.

**Expected behavior**

The created-from list reads only the created tasks. The blocked-inbox
map reads review waits only for in-review tasks that the liveness
classifier can read for a page row. The rows, the order, and the
attention values do not change.

**Steps to reproduce**

1. Use a company with tens of thousands of tasks, runs, and activity
rows.
2. Open a task page. Look at the created-from list statement: it reads
the full task set of the company.
3. Open an inbox page with blocked-inbox attention. Look at the
review-wait read: it gets the ID of every in-review task of the company.

## What Changed

- `server/src/services/issue-creation-origin.ts`:
`createdFromIssueCondition` is now `issues.id <> source AND issues.id IN
(three branches)`. Each branch starts from an index:
  - recorded origin task: `issues_company_origin_issue_idx`;
- legacy origin run: the source task's runs
(`heartbeat_runs_company_origin_issue_idx`), then the new partial index
on `issues (company_id, origin_run_id)`;
- creation activity: `activity_log_run_id_idx` for the source runs, then
the task by primary key.
The precedence does not change: a recorded origin task wins, the run
path applies only without one, and the activity path applies only
without both. The activity path matches only an `entity_id` that is the
canonical text of the task ID, as before (it also compares `id::text =
entity_id`).
- `packages/db`: add `issues_company_legacy_origin_run_idx` on `issues
(company_id, origin_run_id) WHERE origin_issue_id IS NULL AND
origin_run_id IS NOT NULL`. Migration `0461` creates it with `CREATE
INDEX CONCURRENTLY IF NOT EXISTS` and is registered with the other
concurrent migrations. `doc/DATABASE.md` documents the index and the
benchmark command.
- `server/src/services/issues.ts`: add `blockedInboxReviewWaitIssueIds`.
It returns the in-review tasks in the blocker closure of the page rows.
The blocked-inbox map reads review waits only for these tasks. The
liveness classifier reads a wait only for a row or for a blocker that it
reaches from a row, and the caller reads only the findings of the rows.
The classifier does not change.
- Tests:
- `issue-created-from-routes.test.ts`: new cases for precedence in both
directions, a source-run comment, an activity row whose task has another
origin run, a non-canonical and a non-UUID activity entity ID, the
source's own creation activity, and other-company rows.
- `blocked-inbox-review-wait-scope.test.ts`: the narrowed set for
chains, cycles, shared blockers and off-page tasks. The rows get the
same findings with the narrowed waits as with the waits of every
in-review task, and each narrowed wait is needed.
- `issue-list-review-wait-authorization.integration.test.ts`: on real
PostgreSQL with two authorized waits, `listExternalReviewWaits` and one
issue-list request with blocked-inbox attention each send one bound-run
message-link statement. With the per-wait authorization from before
#20551, both tests fail (2 statements).
- `scripts/benchmark-created-from-issues.ts`: local benchmark. It loads
the baseline `issues.ts` and `issue-creation-origin.ts` from a pinned
commit, asserts identical complete row payloads, and reports alternating
timings and `EXPLAIN (ANALYZE, BUFFERS)`.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/issue-created-from-routes.test.ts
src/__tests__/blocked-inbox-review-wait-scope.test.ts
src/__tests__/issue-list-review-wait-authorization.integration.test.ts
src/__tests__/agent-task-hierarchy-depth.test.ts
src/__tests__/issue-blocker-attention.test.ts
src/__tests__/issue-list-assignee-filter-routes.test.ts
src/__tests__/issues-parent-id-alias.test.ts
src/__tests__/batched-chat-authorization.integration.test.ts
src/__tests__/issue-liveness.test.ts` passes (126 tests). One timing
test in `issue-list-assignee-filter-routes` failed one time under
machine load and passed on rerun.
- `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter
@paperclipai/db check:migrations` pass.
- `pnpm verify` asks for broader verification because the change adds a
file under `scripts/`. Required CI supplies that coverage.
- Benchmark: `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-created-from-issues.ts --baseline-ref=bd30ef8510`

Fixture (synthetic, local PostgreSQL 18.1): one company with 30,000
tasks (10,628 with a recorded origin task, 10,445 with only a legacy
origin run), 60,030 runs with wide context snapshots (145 MB), and
261,079 activity rows; a second company with 10,000 tasks and 20,000
runs. The busy source task has 63 created tasks through all three paths,
plus decoys. The quiet source task has 3. The UI's page: `sortField=id`,
`asc`, `limit=500`, routine executions included. Both versions return
identical complete payloads. 9 alternating pairs after 2 warm-up pairs:

| Case | List statement before | List statement after | List call
before, median (min to max) | List call after, median (min to max) |
| --- | --- | --- | --- | --- |
| Busy source (63 tasks) | 234.9 ms, 47,437 buffers | 0.90 ms, 437
buffers | 156.0 ms (114.5 to 480.7) | 31.6 ms (19.9 to 62.7) |
| Quiet source (3 tasks) | 90.7 ms, 47,413 buffers | 0.25 ms, 46 buffers
| 126.1 ms (90.6 to 300.4) | 10.5 ms (7.2 to 26.1) |

The list call time includes the enrichment reads (labels, runs,
attention), which do not change.

Inbox page (50 open tasks, blocked-inbox attention): the review-wait
read goes from 2 statements with 1,215 parameters (every in-review task
of the company) to 1 statement with 31 parameters. The page time does
not change in this fixture (median 801 ms before, 873 ms after, both
with a wide spread), because the fixture has no review waits and the
other graph reads of the map are most of the time. With real waits, the
server does an inherited GitHub review read for each wait it finds, so
the gain grows with the number of off-page waits.

Migration timing (local PostgreSQL 18.1): `0461` (measured as the same
statement under its earlier number) on 500,000 tasks (846 MB, 20
companies, one third with a legacy origin run) takes 1.6 s. The index is
12 MB.

## Risks

- The new index adds a small write cost to tasks with only an origin
run. New tasks with a source task record `origin_issue_id`, so the index
holds mostly older tasks. The migration does not lock writes.
- The created-from rewrite must keep the same rows. The new tests and
the benchmark payload check cover precedence, company scope, and ID text
forms. Visibility and read filters are the list's other conditions and
do not change.
- The review-wait narrowing depends on the liveness classifier reading
waits only for rows and their blocker chains. A unit test compares the
findings of the rows with full and narrowed waits. A future classifier
change that reads waits of other tasks must update
`blockedInboxReviewWaitIssueIds`. The function comment says this.
- The local timings are synthetic. Hosted acceptance needs a fresh
query-statistics window after deployment.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edits, local PostgreSQL) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 12:38:51 -06:00
DottaandClaude Opus 5.5 1805a1d247 perf(workspaces): probe workspace run bindings through indexes (#20568)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The terminal workspace sweep checks each Git workspace for recent
activity and for live consumers before it expires or moves the workspace
> - These checks find the runs of a workspace with one SQL scope,
`expirationScope`. The scope used one `OR` over all run bindings, and
one branch decoded the context JSON of each run
> - PostgreSQL could not use an index for that `OR`. Each check read
every run of the company and its large context from TOAST. On a hosted
database, each call took 2 to 5 seconds, about 5,200 s in total in one
window
> - This pull request splits the scope into one index probe per binding
and adds one concurrent expression index for the legacy
`executionWorkspaceId` context key
> - The benefit is that each check reads a few index entries instead of
all company runs. The cleanup decisions do not change

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The workspace activity and cleanup-safety checks:
`gitWorkspaceLastActivityAt`, `isGitWorkspaceExpirationEligible`,
`workspaceMoveEligible` and the orphan-folder consumer check. All of
them use `expirationScope` in
`server/src/services/git-workspace-expiration.ts`.

**Subsystem affected**
Execution workspace cleanup in the server
(`git-workspace-expiration.ts`, `isolated-repository-rehome.ts`,
`isolated-task-directory-retention.ts` through the shared scope) and the
`heartbeat_runs` schema in `packages/db`.

**Current behavior**
The `relevant_runs` CTE selects company runs where any of these is true:
`issue_id` or `native_issue_id` is in the issue tree or the bound
issues, the run is a bound issue's checkout or execution run, or the
context JSON has a matching `issueId`, `taskId` or
`executionWorkspaceId`. The planner does one sequential scan of the
company runs and reads each context from TOAST. The cost grows with the
run history of the company, not with the size of the workspace.

**Proposed behavior**
Each binding is a separate index probe. The `UNION` of run IDs goes to a
primary-key probe. The SQL gives the same rows.

**Reason and benefit**
Local SQL measurements show a 34x to 169x decrease in time. The decrease
grows with the run history (see Verification).

## What Changed

- `expirationScope` builds `relevant_runs` from a `UNION` of index
probes, one for each binding. The key sets go in as arrays (`=
ANY(ARRAY(...))`). With `IN (subquery)`, the planner chose a hash join
over a full scan, because it does not count the cost of reading TOASTed
contexts.
- The legacy context keys use `context_snapshot ->> 'key'`. This
expression matches the existing indexes
`heartbeat_runs_company_ctx_issue_created_idx` and
`heartbeat_runs_company_ctx_task_created_idx`. For a non-object context,
`->>` gives `NULL`, as the strict JSONPath decode did. Such a run can
still match through a native binding.
- New partial index `heartbeat_runs_company_ctx_execution_workspace_idx`
on `(company_id, context_snapshot ->> 'executionWorkspaceId') WHERE
context_snapshot ? 'executionWorkspaceId'`. The query repeats the
predicate. For an array context, `?` can be true, but `->>` gives
`NULL`, so the result does not change. The index is partial so that it
serves only this probe. A full index that starts with `company_id` made
the planner choose it for an unrelated company-scoped query. Migration
0460 builds it with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. The
migration is in the list of migrations that run outside a transaction.
- New tests: a legacy `issueId` or `executionWorkspaceId` binding on a
run that is bound to a different issue still blocks cleanup. A plan test
with `enable_seqscan = off` checks that no binding scans
`heartbeat_runs` and that the three context indexes are used.
- The benchmark script `scripts/benchmark-workspace-activity.ts` gets a
`--runs-per-workspace` option. Unrelated fixture runs now have distinct
context IDs, as real history does.
- `doc/DATABASE.md` describes the probes and the new index.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/git-workspace-activity.test.ts
src/__tests__/native-harness-backup-expiration.test.ts`: 132 of 132
tests pass. The second file checks that the native session index stays
the planner's choice for its query. These include the existing route
matrix (root, child, bound, native, context issue, context task, context
workspace, checkout, execution run), exactly-once counts, malformed and
uppercase contexts, company isolation, controller leases and lease
vetoes.
- `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter
@paperclipai/db check:migrations` pass.
- Benchmark (local SQL measurement, PostgreSQL 18.1, synthetic fixture,
8 workspaces per sample, before and after alternate in each iteration).
The script compares the complete ordered caller results and the complete
`relevant_runs` multisets of the baseline and treatment. They are equal
in every run.

`pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-workspace-activity.ts --baseline-ref=origin/master
--runs-per-workspace=<N>`

| Fixture | Check | Before median (min–max) | After median (min–max) |
Speedup |
  | --- | --- | --- | --- | --- |
| 4,800 runs, 330 MB | last activity | 523 ms (460–598) | 12 ms (8–15) |
44x |
| 4,800 runs, 330 MB | eligibility | 452 ms (432–473) | 13 ms (11–15) |
34x |
| 4,800 runs, 330 MB | idle eligibility | 465 ms (442–573) | 12 ms
(10–13) | 40x |
| 24,000 runs, 1.7 GB | last activity | 2,240 ms (2,214–2,454) | 13 ms
(11–14) | 169x |
| 24,000 runs, 1.7 GB | eligibility | 2,245 ms (2,097–2,377) | 17 ms
(14–19) | 133x |
| 24,000 runs, 1.7 GB | idle eligibility | 2,196 ms (2,179–2,381) | 15
ms (13–18) | 143x |

Before: one `Seq Scan` on `heartbeat_runs`. After:
`heartbeat_runs_company_issue_created_idx`,
`heartbeat_runs_company_native_issue_contract_id_uq`, the three context
indexes and `heartbeat_runs_pkey`.
- Index build time: on a synthetic table with 200,000 runs and 8 KiB
contexts (1.8 GB), the full form of this index took 3.7 s to build
concurrently and used 15 MB. The partial index reads the same rows and
holds only runs that have the key.

## Risks

- Cleanup safety: a wrong "allowed" result could delete an active
workspace. The scope keeps every binding of the old query. The benchmark
checks exact equality of the run sets, and the existing tests check each
binding route and each veto.
- Migration: the index build reads every run context once. It runs
concurrently and does not block writes. Until migration 0460 builds the
index, the new query is still correct but the `executionWorkspaceId`
probe scans.
- Plan choice: the array form makes index probes the cheapest plan for a
small tree. A very large issue tree gives a large array, but each
element is still one index probe.
- The sweep frequency does not change. This pull request reduces only
the cost of each check.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 12:17:46 -06:00
DottaandClaude Opus 5.5 d5c1f48180 perf(native): index the backup lease lookup by scope digest (#20571)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native harness backup expiration removes old session backups. Before
it removes a backup, it finds the environment leases that reference the
backup root
> - This lookup reads leases by run ID, by the
`metadata.nativeHarnessBackup.sessionScopeSha256` digest, or by the
identity's own lease ID, in one `OR` query
> - The metadata arm has no index. So PostgreSQL reads every lease of
the company, and the metadata of each lease, for each lookup
> - On a hosted test database, this statement ran 21,736 times at a mean
of 158 ms in one window. After other fixes, it still ran 2,364 times at
a mean of 121 ms
> - This pull request adds a small partial expression index on the
digest
> - The benefit is that each lookup reads a few index entries instead of
all company leases. The query text and the result do not change

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The lease lookup in `backupDependencies` in
`server/src/services/native-runtime/native-harness-backup-expiration.ts`.
`reconcileBackup` calls it for each backup root when the expiration
worker loads or checks a root.

**Subsystem affected**
Native harness backup expiration and the `environment_leases` table
schema.

**Current behavior**
The query is `company_id = $1 and (heartbeat_run_id in (...) or
metadata->'nativeHarnessBackup'->>'sessionScopeSha256' = $n or id =
$m)`. The run arm and the ID arm have indexes. The digest arm has no
index. A `BitmapOr` plan needs an index for every arm, so the planner
reads all company leases and evaluates the JSON expression on each row.
Lease metadata holds sandbox details, so these reads are large.

**Proposed behavior**
The partial index `environment_leases_native_backup_scope_idx` keys
leases by the digest. It holds only leases that have a backup stamp.
Every arm of the `OR` now has an index, and the planner combines three
index probes. The SQL and the returned lease IDs do not change.

**Reason and benefit**
This statement was in the top ten by total database time on a loaded
hosted database. A local reproduction shows a query that is more than
6,000x faster and reads 9 buffers instead of about 436,000.

**Breaking changes**
None. The API, the lease-matching rules and the results do not change.

## What Changed

- Add the partial expression index
`environment_leases_native_backup_scope_idx` on
`(metadata->'nativeHarnessBackup'->>'sessionScopeSha256')`, where that
value is not null.
- Add migration `0459`. It uses `CREATE INDEX CONCURRENTLY IF NOT
EXISTS`. The migration runner runs it as a top-level statement, the same
as migrations 0454–0458.
- Add a regression test. It checks the lease-matching rules: run leases,
a lease with only the backup stamp, and the identity's own lease match.
Another digest, a non-string stamp, a top-level key with the same name,
and the same digest in another company do not match. The test also
checks that the custom plan and a forced generic plan of the generated
SQL use `BitmapOr` with the new index and no sequential scan.
- Add `scripts/benchmark-environment-lease-backup-lookup.ts`. It
captures the SQL that the real service generates and measures it with
and without the index.
- Document the index in `doc/DATABASE.md`.

## Verification

Cause and thesis. The cost comes from the one `OR` arm without an index,
which forces a scan of all company leases and their metadata. Thesis: an
index on the digest lets the planner use a `BitmapOr` of three index
probes, so the cost no longer depends on the company's lease count. The
results stay identical. The benchmark confirms the thesis.

I did not change the SQL. With the index, the existing `OR` query
already gets the index plan. A `UNION` rewrite would not help without
the index, because the digest arm would still scan the company.

Local SQL benchmark:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-environment-lease-backup-lookup.ts
```

Fixture (synthetic): one company with 100,000 leases with 4 KiB of
incompressible metadata each. 30% of the leases have a backup stamp for
another session. A second company has 20,000 leases. The table is 678
MB. The backup root has 3 run leases, 1 lease with only the stamp, and 1
identity lease. The same digest is also in the other company. Embedded
PostgreSQL 18 on an Apple laptop, which other work loaded heavily. The
hosted database is Neon with different compute, so absolute times are
local SQL measurements only.

The script measures in the order: no index, index, no index, index. Each
measurement has 2 warmups and 11 samples of `EXPLAIN ANALYZE` execution
time and of the full `reconcileBackup` call.

| Measurement | Without index | With index |
| --- | --- | --- |
| Lease query execution, run 1 | median 196 ms (138–562 ms) | median
0.010 ms (0.007–0.017 ms) |
| Lease query execution, run 2 | median 185 ms (144–295 ms) | median
0.030 ms (0.020–0.045 ms) |
| `reconcileBackup` wall time (includes file reads), runs 1 and 2 |
median 188 ms and 216 ms | median 1.4 ms and 4.2 ms |
| Plan | `Gather > Seq Scan` | `Bitmap Heap Scan > BitmapOr` of
`heartbeat_run_idx`, the new index and `pkey` |
| Buffers | 365,880 hit + 70,050 read | 9 hit |

These numbers are from head `65bd9d7ac6`. The final head only renumbers
the migration to 0459 after rebases; the SQL, the index and the service
code are the same. Two earlier runs on the same fixture, SQL and index
gave 236–425 ms without the index and 0.023–0.033 ms with the index. The
machine had heavy load from other work, so the spread without the index
is wide. Output check: in all four measurements, the service returned
the same 5 lease IDs, and the complete query rows were equal.

Index build (`CREATE INDEX CONCURRENTLY` from the migration file):
0.8–2.4 s for the 678 MB table over all runs. Index size: 3.4 MB with
30% of leases stamped.

Tests:

- `cd server && npx vitest run
src/__tests__/native-harness-backup-expiration.test.ts`: 109 passed. The
new test failed when I dropped the index before the plan check
(confirmed red), and it passes with the index.
- `pnpm --filter @paperclipai/db check:migrations`: passed.
- `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter
@paperclipai/server typecheck`: passed.
- `pnpm verify`: it asks for broader coverage (exit 2) because of the
new script. See the CI results for this head.

## Risks

- Migration time. A concurrent index build reads the full
`environment_leases` table and does not block writes. Locally, the build
took 0.8–2.4 s for 678 MB. The time grows linearly with the table size.
I did not time it on a production-size clone. I expect a few seconds to
tens of seconds on a large instance, which is in the 1–2 minute budget.
This estimate is not proven.
- If the build stops, PostgreSQL can leave an invalid index. The
migration runner repairs invalid concurrent indexes when it runs again,
and `IF NOT EXISTS` makes a rerun safe.
- The index expression must stay textually equal to the query
expression. If someone changes the JSON path, the planner cannot use the
index. The new test catches this for custom and generic plans.
- The index is partial, so ANALYZE does not collect statistics for the
expression. The planner uses the default equality estimate. On the large
fixture and in the small test, the planner still selects the index plan.
A company with very few leases can still get a sequential scan, which is
cheap.
- Each lease write with a backup stamp writes one more small index
entry. The write cost is small.
- This change does not reduce how often the worker calls the lookup. It
makes each call cheap. The other digest lookups in backup retention
(`native-harness-backup-retention.ts`) and session retention checks can
also use the index.
- Previous-release story: no persisted state changes. The index is
derived from existing lease metadata.
- The hosted improvement is not verified until deployment. After
deployment, compare a new query-statistics window for this statement.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 11:30:05 -06:00
DottaandClaude Opus 5.5 765ccd8691 perf(chat): index chat-admission checks by company and payload issue id (#20562)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Chat connections save each external message as a `chat_actions` row
before the server admits it as an agent wakeup.
> - The server checks if an issue holds chat input that it did not
admit. Recovery skips such an issue. Manual task start refuses it.
> - On a hosted instance, the recovery read with this check used the
most total database time: 488 calls at about 1 s each. The manual-start
check ran 155 times at about 0.9 s.
> - The check matches `chat_actions.payload->>'issueId'` against the
issue. No index covers this value, so each call reads the full
`chat_actions` table and its payloads.
> - This pull request adds one concurrent partial index for the check.
The check SQL does not change.
> - The benefit is less background database load, so request reads wait
less. The results stay the same.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The chat-admission check `unadmittedChatWakeupCondition`. Recovery uses
it in the per-issue candidate read. Manual task start uses it in the
held-input check.

**Subsystem affected**
Server chat admission, issue recovery, manual task start, and the
database schema.

**Current behavior**
The check finds `inbound_wakeup` actions by `company_id` and
`payload->>'issueId'`. Two nested checks find `failed_run_retry` actions
and later `inbound_wakeup` actions on the same company and payload issue
ID. `chat_actions` has no index on `company_id` or on the payload issue
ID. Each call reads the full table and reads each payload. On a hosted
instance, the two statements used about 630 s of database time in one
statistics window.

**Proposed behavior**
The check and its nested checks read a small set of rows from one index.
The admission result does not change.

**Reason and benefit**
The hosted database had many concurrent active queries. Request reads
waited behind background work such as this check.

**Breaking changes**
None. The migration adds an index only.

## What Changed

- Migration `0458_chat_admission_issue_index` builds
`chat_actions_company_issue_kind_idx` with `CREATE INDEX CONCURRENTLY IF
NOT EXISTS`. It runs through the existing top-level statement path,
which repairs an invalid index on retry.
  - Keys: `(company_id, payload->>'issueId', kind, status)`.
  - Partial: `kind IN ('inbound_wakeup', 'failed_run_retry')`.
- One index serves the outer input lookup, the failed-run retry lookup,
and the later-input lookup. All three match on the same company and
payload issue ID.
- The Drizzle schema declares the index. The snapshot chain moves to
`0458` and keeps five snapshots.
- New regression test
`server/src/__tests__/chat-admission-condition.test.ts` pins the
admission semantics on real PostgreSQL:
- Pending and failed input without a receipt holds the issue. Received,
processed and cancelled input does not. Other action kinds do not.
- A skipped receipt, a receipt for another agent, or a receipt in
another company does not admit input. A failed or cancelled run still
counts as admission.
- The issue ID is compared as company-scoped text. Uppercase, array,
null, other-issue and other-company values do not match.
- Only an admitted retry in the same conversation and issue that lists
the action releases skipped input.
- Only later admitted input in the same conversation, issue and session
generation releases a failed input. Pending input stays held.
  - The check reads `chat_actions` only through the new index.
- `scripts/benchmark-chat-admission-index.ts` measures both caller
statements with and without the index and fails if the rows differ.
- `scripts/benchmark-chat-admission-index-migration.ts` times the
migration through the normal migration runner.
- `doc/DATABASE.md` documents the index and the benchmark commands.

## Verification

Local SQL measurements on synthetic data. PostgreSQL 18.1 (embedded),
default settings. The fixture has 100,995 `chat_actions` rows (662 MB):
60,000 GitHub actions with 6 KiB incompressible payloads, 2,000 issues
with 20 admitted inbound messages each (3 KiB payloads), and 1,000
failed-run retries. Each case has 2 warmups and 9 samples of `EXPLAIN
ANALYZE` execution time. Runs alternate: no index, index, no index,
index.

| Caller statement | Case | No index, median (range) | Index, median
(range) |
| --- | --- | --- | --- |
| Manual-start held check | quiet issue | 3,093 ms (2,150–3,519) / 3,586
ms (2,375–11,235) | 0.036 ms (0.034–0.052) / 0.040 ms (0.033–0.047) |
| Manual-start held check | admitted input | 3,555 ms (3,310–8,080) /
5,205 ms (3,353–9,525) | 0.049 ms / 0.052 ms |
| Manual-start held check | unadmitted input | 3,002 ms (2,864–3,221) /
4,179 ms (3,200–6,585) | 0.051 ms / 0.044 ms |
| Manual-start held check | failed, later admitted | 3,586 ms / 3,722 ms
| 0.066 ms / 0.054 ms |
| Manual-start held check | admitted retry | 2,733 ms / 4,953 ms | 0.049
ms / 0.053 ms |
| Recovery candidate read | quiet issue | 43 ms (41–48) / 96 ms (44–219)
| 0.047 ms / 0.055 ms |
| Recovery candidate read | admitted input | 203 ms (48–575) / 174 ms
(44–314) | 0.055 ms / 0.058 ms |
| Recovery candidate read | unadmitted input | 66 ms / 144 ms | 0.053 ms
/ 0.054 ms |

- Buffers per call fall from about 40,000 shared hits to 5–23.
- Locally, the recovery read without the index uses a hashed full scan
of `chat_actions`. The held check uses a full scan per call. On the
hosted instance both took about 1 s per call. The local fixture is
smaller than the hosted table.
- Correctness: complete positional rows match across all four runs for
all six cases (quiet, admitted, unadmitted, failed then admitted,
retried, other-company decoy). The script asserts the expected held
result for each case.
- Migration timing: 600,001 `chat_actions` rows (1,759 MB), 300,000 of
them in the index scope. First run through `applyPendingMigrations`: 4.2
s. Rerun after the history row is removed: 0.5 s. Concurrent build
alone: 3.4 s. Index size: 30 MB.

Commands:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-admission-index.ts
OTHER_PAYLOAD_KIB=2 pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-admission-index-migration.ts
cd server && npx vitest run src/__tests__/chat-admission-condition.test.ts   # 6 passed
pnpm --filter @paperclipai/db check:migrations                               # passed
pnpm --filter @paperclipai/server typecheck                                  # passed
```

## Risks

- The migration builds one index concurrently. It does not block writes.
On the 1.76 GB local fixture it took about 4 s. A larger hosted table
takes longer, but the build does not lock out reads or writes.
- Writes to `inbound_wakeup` and `failed_run_retry` actions maintain one
more index. Status updates on these rows were already not HOT updates,
because other partial indexes use `status`.
- The planner has no expression statistics for a partial index. The new
test pins that the check uses the index on an analyzed table with
unrelated rows.
- The check SQL does not change, so admission behavior does not change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 10:44:51 -06:00
DottaandClaude Opus 5.5 56aa160a4e perf(native): index heartbeat runs by company and native session (#20560)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs share a native session. Retention, recovery, resume and
archive code reads all runs of one session from `heartbeat_runs`.
> - On a hosted instance, the native retention lookup ran about 67,000
times in one day at about 22 ms, and about 10,000 times at about 11 ms
in a later window.
> - No index has `native_session_id`. Each read walks the agent's or the
company's run history and filters the rows.
> - This pull request adds one concurrent partial index on `(company_id,
native_session_id)`. The SQL does not change.
> - The benefit is less background database load. Request reads wait
less, and the results stay the same.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The native session-run lookups. The largest caller is
`sessionRetentionBlocker` in native harness backup and checkpoint
retention. It selects `id, status, finished_at`, an incomplete-recovery
flag and the issue ID for one company, agent and native session.

**Subsystem affected**
Server native runtime (retention, recovery, resume and archive reads)
and the database schema.

**Current behavior**
`heartbeat_runs` has no index on `native_session_id`. PostgreSQL reads
the whole run history of the company (or of the agent through
`heartbeat_runs_company_agent_started_idx`) and filters by session. The
cost grows with the run history, not with the size of the session.

**Proposed behavior**
The index finds the few runs of one session directly. Every caller
already filters by `company_id` and `native_session_id`, so one index
serves the agent-scoped and the issue-scoped reads.

**Reason and benefit**
This statement family used about 1,500 s of database time in one day on
a busy hosted database. The local proof below shows 12.3 ms to 0.04 ms
per call.

## What Changed

- `packages/db`: add `heartbeat_runs_company_native_session_idx` on
`(company_id, native_session_id) WHERE native_session_id IS NOT NULL`.
Migration `0457_native_session_run_index` builds it with `CREATE INDEX
CONCURRENTLY IF NOT EXISTS`. The migration is registered as a
top-level-statement migration, the same way as 0454–0456.
- Add a regression test in `native-harness-backup-expiration.test.ts`.
It captures the real retention lookup and proves that its plan can use
the index. It also pins the company and session scope: a live run of
another company with the same session ID does not protect a checkpoint.
A live run of the same agent in another session does not protect it. A
live run of the same session does protect it.
- Add `scripts/benchmark-native-session-run-index.ts`. It runs the real
checkpoint retention pass (dry run), captures the exact SQL, and replays
it with and without the index. It also times the migration through the
normal migration runner.
- `doc/DATABASE.md`: describe the index and the benchmark command.

## Verification

Local SQL measurements, synthetic data. PostgreSQL 18.1 (embedded test
server). Fixture: 100,000 runs (50,000 of the measured agent, 50,000 of
a second agent), 4 runs per session, 4 KiB incompressible
`runner_profile_json` per run, 638 MB table. Order: no index, index, no
index, index. 15 `EXPLAIN (ANALYZE, BUFFERS)` samples after 3 warmups.

| Pass | Plan | Median execution | Min / max | Shared buffers |
| --- | --- | --- | --- | --- |
| No index (1) | Seq Scan | 12.30 ms | 10.33 / 35.27 ms | 3,607 |
| Index (2) | Index Scan using
`heartbeat_runs_company_native_session_idx` | 0.036 ms | 0.021 / 0.115
ms | 38 |
| No index (3) | Seq Scan | 12.38 ms | 10.24 / 26.14 ms | 3,607 |
| Index (4) | Index Scan using
`heartbeat_runs_company_native_session_idx` | 0.043 ms | 0.022 / 0.098
ms | 38 |

- Correctness: the service SQL is the same in every pass. The returned
rows (all columns) and the retention result are equal in every pass. The
script fails on any difference.
- Migration: the first run through the migration runner took 525 ms on
the 638 MB fixture. The rerun took 273 ms and skipped the existing
index. A direct concurrent build took 145–221 ms. Index size: 1.9 MB.
- `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-native-session-run-index.ts`
- `npx vitest run
src/__tests__/native-harness-backup-expiration.test.ts` (server): 108
passed.
- `pnpm --filter @paperclipai/db check:migrations`: passed. Five
snapshots remain.
- `pnpm verify` exits 2 (a new script under `scripts/` is a global CI
input). Required CI supplies the broad coverage.

## Risks

- Low risk. The migration adds one index concurrently and does not lock
writes. The SQL of every caller is unchanged.
- The index adds a small write cost to `heartbeat_runs` inserts and to
updates of `native_session_id`. Rows with a null session are not
indexed.
- Persisted run state: no saved run, session or checkpoint shape
changes. State saved by the previous release carries forward unchanged.
- Hosted latency is not measured yet. The numbers above are local SQL
measurements.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, local PostgreSQL).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 10:04:22 -06:00
DottaandClaude Opus 5.5 347d85fef3 perf(chat): index GitHub review lookups and share review-wait reads (#20400)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - GitHub chat endpoints let agents review pull requests. The server
reads review state when it lists issues and when it updates GitHub
checks.
> - On a hosted instance, two GitHub review queries used the most total
database time. One query ran about 660,000 times at about 33 ms. The
other query ran about 10,000 times at about 1 s.
> - The first query finds a source comment's chat link. No index covers
`chat_message_links.comment_id`, so each call reads the full table.
Issue lists also read the same review waits two times per request.
> - The second query joins deliveries, links and runs on a JSON key. The
planner walks the agent's run history in creation order and reads each
run's context.
> - This pull request adds four concurrent indexes, splits the run
lookup into two small reads, and shares review-wait reads in one
issue-list request.
> - The benefit is less background database load, so request reads wait
less. The results stay the same.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
GitHub review state reads. These are the inherited review-wait lookup by
source comment, and the review-check lookup of the run that a delivery's
comment woke.

**Subsystem affected**
Server chat channels (GitHub review waits and checks), issue list
attention maps, and the database schema.

**Current behavior**
- `readInheritedGitHubReviewWait` filters `chat_message_links` by
`company_id`, `comment_id` and `direction`. No index has `comment_id`,
so PostgreSQL reads the full table on each call. The run-origin check
does the same lookup first.
- An issue list calls the external review-wait read from the review
attention map and again from the blocked-inbox attention map. The two
maps read the same in-review issues.
- `githubReviewCheckState` joins deliveries, links and runs with
`context_snapshot->>'wakeCommentId' = comment_id::text`, sorts by
`created_at desc` and takes one row. The planner reads the agent's runs
in creation order and reads each run context. If no run matches yet, it
reads all of them. The delivery filter on `normalized_event` and the
link join on `delivery_id` also have no index.

**Proposed behavior**
The same rows come back from small index reads. Each issue's review wait
is read one time per issue-list request.

**Reason and benefit**
These two statement families used about 33,000 s of database time in one
cumulative statistics window. The database had about 50 concurrent
active queries, and request reads waited behind this background work.

**Breaking changes**
None. The migration adds indexes only.

## What Changed

- Migration `0456_chat_review_lookup_indexes` builds four indexes with
`CREATE INDEX CONCURRENTLY IF NOT EXISTS`. It runs through the existing
top-level statement path, which repairs an invalid index on retry:
- `chat_message_links_company_comment_idx` on `(company_id, comment_id)`
where `comment_id is not null`.
- `chat_message_links_delivery_idx` on `(delivery_id)` where
`delivery_id is not null`.
- `chat_deliveries_github_automatic_delivery_idx` on `(endpoint_id,
normalized_event->'githubAutomatic'->'context'->>'deliveryId')` where
that value is not null.
- `heartbeat_runs_company_wake_comment_idx` on `(company_id,
context_snapshot->>'wakeCommentId')`. This index is not partial. The
planner does not use expression statistics from a partial index. Without
these statistics, it estimates many matches and keeps the creation-order
walk.
- `githubReviewCheckState` reads the delivery's comment IDs first. If
there is a comment, it reads the newest same-agent run with one of these
`wakeCommentId` values. If there is no comment, it skips the run read.
The comparison stays text equality on `comment_id::text`.
- `createExternalReviewWaitReader` shares one read per issue between the
review and blocked-inbox attention maps of one issue-list request. The
reader exists only for that request.
- `doc/DATABASE.md` describes the indexes and the new read order.
- `scripts/benchmark-review-lookup-indexes.ts` compares the pre-fix and
new code with and without the indexes.
`scripts/benchmark-review-lookup-index-migration.ts` times the migration
on a large synthetic database.

## Verification

Local SQL measurements on a synthetic fixture. They are not production
timings. Environment: embedded PostgreSQL 18.1, baseline revision
`02bfe71a6b`. Fixture: 50,000 links, 5,000 deliveries with about 2 KiB
events, and 4,000 runs with about 12 KiB incompressible contexts. The
wake run is in the middle of the run history. Each value is the median
server execution time from `EXPLAIN ANALYZE` (2 warmups, 9 samples). The
order was: no indexes, indexes, then no indexes again.

| Caller and statement | Before (no indexes, two passes) | After |
Shared buffers before → after |
| --- | --- | --- | --- |
| Inherited review wait, link by source comment (same SQL) | 2.72 / 2.36
ms | 0.076 ms | 929 → 12 |
| Review check, wake run found | 31.9 / 21.9 ms | 0.29 ms (two
statements) | 25,776 → 62 |
| Review check, no wake run yet | 71.0 / 47.5 ms | 0.06 ms (two
statements) | 34,861 → 12 |

- Indexes alone do not fix the review check. The old SQL with the new
indexes took 8.6 ms (run found) and 29.7 ms (no run), because it still
walked the run history. The new SQL without the new indexes was neutral
(23.5 and 35.1 ms).
- Service wall times on this shared machine were noisy, so the table
reports server execution time.
- Correctness: the benchmark fails unless the inherited statement
returns the same positional rows in each index state. The check service
result and the final wake-run rows of the new code must also match the
pre-fix module for both cases.
- Migration timing, normal migration runner, synthetic data: 200,000
runs (1.8 GB with 8 KiB incompressible contexts), 500,000 links and
100,000 deliveries. First run 4.2 s; rerun 0.38 s. Separate builds:
links by comment 1.5 s, run wake comment 1.1 s, deliveries 0.7 s, links
by delivery 0.3 s. All indexes were valid. The run index was 8.3 MB.
- New tests:
- `server/src/__tests__/github-review-check-state.test.ts` checks that
the newest same-agent run is selected. It also checks that other agents,
other companies, uppercase IDs, non-string values, list-only
`wakeCommentIds`, comment-less links and unknown deliveries do not
match. It checks that the generated statements can use the new indexes.
- `server/src/services/native-runtime/external-review-wait.test.ts`
checks that the reader reads each issue one time, keeps concurrent
callers correct, and rejects all waiting callers when a read fails.
- Red/green: with the pre-fix `review-check-state.ts`, the behavior test
passes and the index test fails. With this change, both pass.
- Focused runs pass: `issue-blocker-attention`,
`heartbeat-issue-liveness-escalation`, review-wait, review-check-state,
external-review-wait, review-repair (106 tests), and
`external-chat-wait.integration` (285 tests). `pnpm --filter
@paperclipai/server typecheck` and `pnpm --filter @paperclipai/db
check:migrations` pass.

Commands:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-review-lookup-indexes.ts --baseline-ref=02bfe71a6b
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-review-lookup-index-migration.ts
```

## Risks

- The migration builds four indexes concurrently. It does not block
writes. On a large database, the run index reads every run context one
time. The local run took 1.1 s for 1.8 GB from a warm cache. Cold
storage on smaller compute can be slower.
- The run index adds one entry for each run insert and for each run
update that is not a heap-only update. Many runs have no
`wakeCommentId`, so these entries hold a null key.
- The review check now sends two statements instead of one when a
delivery has a comment. Each statement reads only a few index entries.
- When two matching runs have the same `created_at`, either can be
selected. The previous query had the same tie behavior.
- This change does not change the number of in-review issues that one
issue list reads. The blocked-inbox map still reads all in-review issues
of the company. A smaller scope needs a separate change to the liveness
classifier.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code with
tool use (shell, file edits, local PostgreSQL benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 18:47:34 -06:00
DottaandClaude Opus 5.5 0116e45958 perf(db): index pending budget Stops instead of scanning all run results (#20391)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Budget enforcement stops agent runs when a budget hard limit is
reached. The stop intent is saved in each run's `result_json` until run
control delivers it.
> - The budget worker, run control, and the idle-sleep check find these
pending stops with a predicate on `result_json`. No index supports that
predicate.
> - Each check therefore reads every heartbeat run and its large result
JSON from TOAST. On a hosted database, this statement family was among
the most expensive: about 8,000 calls at about 1 s each, about 9,800 s
in total.
> - Pending stops are rare. The predicate is constant, so a partial
index can contain only those rows.
> - This pull request adds that partial index. The SQL that the server
sends does not change.
> - The benefit is that each check reads a few index entries instead of
the full table. The database has more capacity for request work.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The check for committed budget Stops that are not delivered yet. The
budget worker runs it each time it looks for work
(`hasPendingBudgetEnforcement`), and again for each delivery pass
(`listPendingBudgetCancellationScopes`). Run control and the idle-sleep
check use the same predicate.

**Subsystem affected**
Budget enforcement in the server
(`server/src/services/heartbeat/budget-cancellation.ts`,
`server/src/services/budgets.ts`,
`server/src/services/heartbeat/run-control.ts`,
`server/src/services/idle-sleep-safety.ts`) and the `heartbeat_runs`
schema in `packages/db`.

**Current behavior**
The query groups all rows of `heartbeat_runs` and reads
`result_json->'startupCancellation'->'budgetEnforcement'` for each row.
A limit of 1 does not help, because the group and sort occur before the
limit. Cost grows with the run history, not with the number of pending
stops.

**Proposed behavior**
A partial index, `heartbeat_runs_pending_budget_cancellation_idx`,
contains only runs with a pending budget Stop. The planner uses it for
all four callers. The results are the same.

**Reason and benefit**
Local SQL measurements show a 300x to 1,000x decrease in time and buffer
reads (see Verification).

**Breaking changes**
None.

## What Changed

- Added the partial index
`heartbeat_runs_pending_budget_cancellation_idx` on `heartbeat_runs
(company_id)`. Its predicate is identical to
`pendingBudgetRunCancellationPredicate`.
- Added migration `0455_pending_budget_cancellation_index.sql`. It uses
`CREATE INDEX CONCURRENTLY IF NOT EXISTS`, and it is in the list of
migrations that run outside a transaction, as `0436` is.
- Moved the scoped run-control read and the idle-sleep check into
exported builders in `budget-cancellation.ts`, next to the scope list
query. The SQL does not change. The callers and the test now share the
same statements.
- Added a real PostgreSQL regression test. It checks that delivered,
malformed, and missing intents are not pending. It also EXPLAINs the
statements that the callers send (scope list with limit 1 and 100,
scoped read, `EXISTS` check) and requires the new index in each plan. If
a person changes a query shape, the server predicate, or the index
predicate, the test fails.
- Added `scripts/benchmark-budget-cancellation-scan.ts`. It captures the
SQL that the real service sends, then compares the plans, timings, and
complete rows with and without the index.
- Updated `doc/DATABASE.md`.

## Verification

Red/green regression test:

- Before the migration, the new test failed: the plan was `Seq Scan`
with the predicate as a filter.
- After the migration: `npx vitest run
src/services/heartbeat/budget-cancellation.test.ts` passes (17 tests).
- Also passed: `src/__tests__/budgets-service.test.ts`,
`src/__tests__/idle-sleep-diagnostics-postgres.test.ts`,
`src/services/budget-resume-continuation.test.ts`.
- `pnpm --filter @paperclipai/db check:migrations`, and typecheck of
`@paperclipai/db` and `@paperclipai/server` pass.

Performance and result equality (local SQL measurements only):

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-budget-cancellation-scan.ts --runs=20000 --payload-kib=16 [--reverse]
```

- Fixture: embedded PostgreSQL 18.1. 20,241 runs in 2 companies, with
incompressible result JSON of about 16 KiB (table 379 MB). 40 pending
stops in 6 scopes, 200 delivered or malformed decoys, and 1
cross-company intent.
- The SQL is the same in both treatments. The script toggles only the
new index. 2 warmups and 7 samples per query. Rows are compared outside
the timed part. Ordered queries must match in order. The scoped
run-control read has no `ORDER BY`, so it is compared as a multiset.

| Query (caller) | Baseline median | With index median | Shared buffers
|
| --- | --- | --- | --- |
| `listPendingBudgetCancellationScopes(db, 1)` (worker `hasPending`) |
688 ms | 2.1 ms | 243,500 → 1,158 |
| `listPendingBudgetCancellationScopes(db)` (delivery) | 799 ms | 1.4 ms
| 243,500 → 1,158 |
| Scoped pending runs (run control) | 357 ms | 1.1 ms | 121,835 → 322 |
| `EXISTS` pending (idle-sleep check) | 440 ms | 0.37 ms | 240,560 → 2 |

- Reverse order (index first) gave the same result: 524 ms → 1.6 ms, 464
ms → 1.4 ms, 306 ms → 1.0 ms, 306 ms → 0.42 ms.
- Plans change from `Seq Scan` (or a broad bitmap scan) to `Index Scan`
or `Index Only Scan` on the new index.
- All result rows are equal in every run.

Migration duration:

- 20,241 runs (379 MB): index build 2.0 s to 3.9 s.
- 200,241 runs with 8 KiB result JSON (2.1 GB): index build 5.6 s. With
this table size, the four queries change from 1,148 / 879 / 514 / 4,392
ms to 1.1 / 1.2 / 0.85 / 8.3 ms.
- The build is concurrent, so it does not block run writes.

## Risks

- Migration: the build reads every run's result JSON one time. It is
concurrent and resumable (`IF NOT EXISTS`). The executor repairs an
invalid concurrent index on retry.
- Write cost: run writes evaluate the index predicate. The index
contains very few rows, so the extra cost is small.
- Predicate drift: if a later change edits only one of the two
predicates, the planner cannot use the index. The new regression test
detects this.
- The hosted timings are not measured yet. The local numbers come from a
synthetic fixture.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 18:14:52 -06:00
DottaandClaude Opus 5.5 c985541a57 perf(db): index retry-exhaustion receipts for the attention scan (#20386)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The attention feed shows runs that failed after automatic retries
stopped
> - To find these runs, the feed reads every `heartbeat_run_events` row
of the company and filters the rows by a message prefix
> - Run output makes this table very large, so each feed request does a
large scan
> - On a hosted test database, this one statement used the most total
database time: 923 calls at a mean of 5 s (4,709 s in total)
> - Request authentication waits behind this background work, so all
users feel the load
> - This pull request adds a small partial index that holds only the
retry-exhaustion receipts
> - The benefit is that the same query reads a few index entries instead
of all company events

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The attention feed query that finds retry-exhausted runs
(`listAttentionExhaustedRuns`).

**Subsystem affected**
Server attention feed and the `heartbeat_run_events` table schema.

**Current behavior**
The query reads all company rows in `heartbeat_run_events`, filters them
by `event_type = 'lifecycle'` and the `Bounded retry exhausted` message
prefix, and then joins the few matches to failed or timed-out runs. The
cost grows with all run output. It does not grow with the number of
exhausted runs.

**Proposed behavior**
A partial index holds only the matching receipts. The query plan reads
the index and joins a few rows. The query text and the result do not
change.

**Reason and benefit**
The scan was the most expensive statement by total time on a loaded
hosted database. A local reproduction shows a 54x faster query and about
116x fewer buffer reads.

**Breaking changes**
None. The API, the feed rows, and their order do not change.

## What Changed

- Add the partial index `heartbeat_run_events_retry_exhausted_idx` on
`(company_id, run_id, id DESC)`. The index covers only rows where
`event_type = 'lifecycle'` and `message like 'Bounded retry
exhausted%'`.
- Add migration `0454`. It uses `CREATE INDEX CONCURRENTLY IF NOT
EXISTS`. The migration runner runs it as a top-level statement, the same
as migrations 0432, 0436 and 0450.
- Add a regression test. It pins the row order (newest run first, then
the newest receipt on a tie) and the latest receipt per run. It also
checks that the query plan can use the new index. If the query predicate
changes so that the index no longer applies, the test fails.
- Add `scripts/benchmark-exhausted-run-attention.ts`. It measures the
same query with and without the index, and the time to build the index.
- Document the index in `doc/DATABASE.md`.

## Verification

Cause and thesis. The scan cost comes from reading all company events,
not from the few exhausted runs. Thesis: an index that holds only the
receipts makes the query cost depend on the number of receipts, and
keeps the result identical. The benchmark below confirms the thesis.

Rejected alternative without a migration. I first tried a LATERAL lookup
that starts from the failed and timed-out runs. On the same fixture it
was about 2x slower (median 379 ms against 194 ms). It reads the full
event history of each failed run with random I/O, and that costs more
than one sequential scan.

Local SQL benchmark (the SQL that the service generates; same SQL with
and without the index):

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-exhausted-run-attention.ts --runs=30000 --events-per-run=100 --samples=9
```

Fixture (synthetic): one company with 30,000 runs (3,819 failed or timed
out) and 3.0 M events, plus a decoy company with 0.5 M events. The table
is 1.4 GB. About 5% of events are `lifecycle` events. Events of 20
concurrent runs are interleaved. 1 in 23 runs has a receipt, some runs
have two receipts, and output decoys contain the prefix. Two agents are
terminated. Embedded PostgreSQL 18.1 on an Apple laptop. The hosted
database is Neon with different compute, so the absolute times are local
SQL measurements only.

The script runs two cycles (without index, then with index; then with
index, then without index) to expose cache bias. Each measurement has 2
warmups and 9 samples.

| Query | Without index | With index | Samples and spread | Output check
|
| --- | --- | --- | --- | --- |
| `listAttentionExhaustedRuns` | median 157 ms (96–1,271 ms) | median
3.3 ms (2.2–5.7 ms) | 18 each | 133 rows, positional rows equal in all 4
measurements |
| Buffers (EXPLAIN ANALYZE) | 127,743 blocks | 5,580 blocks | | plan
uses the new index (asserted) |

Speedup: 48x (median), on head `ae6037b2a0` (the same SQL and index as
the final head). The machine was under heavy load from other work, so
the spread without the index is wide. An earlier run on the first head
gave 248 ms against 4.6 ms (54x). Index build (`CREATE INDEX
CONCURRENTLY` from the migration file): 1.7 s and 2.5 s for the 1.4 GB
table. Index size: 152 KiB.

Review fix: the query now writes `event_type = 'lifecycle'` as a SQL
literal, not a bound parameter. With prepared statements, PostgreSQL can
reuse a generic plan. A generic plan cannot prove that a bound parameter
matches the index predicate, so it would not use the index.

Tests:

- `cd server && npx vitest run src/__tests__/attention-service.test.ts`:
44 passed. The new test checks a fresh plan and a forced generic plan
(`plan_cache_mode = force_generic_plan` with `PREPARE`). It failed with
the bound `event_type` parameter and with a changed message prefix
(confirmed red), and it passes with the fix. Locally, the file's
`afterAll` database cleanup hook timed out on a heavily loaded machine
(load average about 28). The timeout also occurs when the new test is
excluded.
- `pnpm --filter @paperclipai/db check:migrations`: passed.
- `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter
@paperclipai/server typecheck`: passed.
- `pnpm verify`: see the CI results for this head.

## Risks

- Migration time. A concurrent index build reads the full
`heartbeat_run_events` table and does not block writes. Locally, the
build took 1.2–2.5 s for 1.4 GB. The time grows linearly with the table
size. I did not time the build on a production-size clone. On a database
with a 5 s full company scan, I estimate the build at tens of seconds,
which is in the 1–2 minute budget. This estimate is not proven.
- If the build stops, PostgreSQL can leave an invalid index. The
migration runner repairs invalid concurrent indexes when it runs again,
and `IF NOT EXISTS` makes a rerun safe.
- The index predicate must stay textually equal to the query predicate.
If someone changes the message prefix or binds a predicate value as a
parameter, the planner cannot use the index. The new test catches both
cases, for fresh and generic plans.
- Each new receipt writes one more small index entry. Receipts are rare,
so the write cost is very small.
- The hosted improvement is not verified until deployment. After
deployment, compare a new query-statistics window for this statement.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL). Standard
context window and reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 17:37:20 -06:00
DottaandPaperclip facd409073 perf(workspaces): index legacy workspace companion lookups (#20372)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Online workspace migration must retain prepared sources and recovery
holds.
> - Companion discovery reads saved workspace references from run
profiles.
> - The selector can scan large saved profiles even when it returns few
rows.
> - This pull request adds one company and workspace expression index.
> - Companion discovery uses the same expression as existing migration
lookups.
> - The result preserves each source check and reduces history scan
work.

## Linked Issues or Issue Description

**What happened?**

The companion selector used a JSON path expression with no matching
index. A read-only plan on about 22,000 run rows showed a sequential
scan over a table with about 1.3 GB of table and TOAST storage. A
repeated inspection reached its statement timeout. Existing migration
lookups use a different JSON operator form, so one expression index must
match both callers.

**Expected behavior**

Use a company-scoped indexed lookup. Keep prepared and finalized
references, the exact text comparison, native issue references, and run
order. Keep all source and recovery validation.

**Steps to reproduce**

1. Retain many run profiles with saved workspace references and wide
JSON fields.
2. Inspect a workspace with few matching companion runs.
3. Explain the company and saved-workspace selector. Without the new
index, it can scan run history and read unrelated JSON values.

**Paperclip version or commit**

Baseline: `0e3fdebea8257bdff11573081c76b306f12db230`.

**Deployment mode**

Server with PostgreSQL. The local proof uses disposable PostgreSQL 18.1.
The observed server plan used PostgreSQL 17.11.

Related: [batched companion discovery in
#19735](https://github.com/paperclipai/paperclip/pull/19735). The
existing batching remains in place. The duplicate search found no open
workspace index change. This branch also carries the isolated API
route-summary correction shared with
[#20370](https://github.com/paperclipai/paperclip/pull/20370).

## What Changed

- Add one concurrent index on company ID and the saved native workspace
ID expression. The index has no state, runtime, or status predicate.
- Use nested JSON operators in companion discovery. Existing migration
lookups already use this expression. Keep both accepted states and the
same ordering.
- Register the migration on the existing top-level migration path.
Retain its advisory lock, interrupted index repair, and completion
journal behavior.
- Add a real PostgreSQL migration and ordered-result test. Update the
existing companion query counter for the equivalent SQL form.
- Generate the schema snapshot and retain the five newest snapshots.
- Describe the task case-list route as “Find the cases for a task.” This
keeps the route on the first discovery search page. Search logic and
test assertions are unchanged.

## Verification

- `pnpm --filter @paperclipai/db exec vitest run
src/heartbeat-workspace-companion-index.test.ts --maxWorkers=1`: **1
passed**. It builds the concurrent index through the canonical migrator,
checks index validity, preserves prepared/finalized and
cross-issue/null-issue results, excludes foreign and malformed
references, and checks one journal entry after a second migration call.
- `pnpm --filter @paperclipai/server exec vitest run
src/services/remote-workspace-online-source.test.ts -t 'discovers all
Git companions' --maxWorkers=1`: **1 passed; 86 not selected**. This
uses the real source owner and retains finalized recovery holds, company
isolation, and a fresh obligation between inspections.
- `pnpm --filter @paperclipai/db typecheck`, migration numbering/safety
checks, and `git diff --check`: **passed**.
- One small local SQL comparison used the captured baseline selector,
the changed selector, 3,012 synthetic rows, about 37.8 MB of table
storage, and the new expression index. Ordered results were identical.
One client timing changed from **8.70 ms to 0.86 ms**. This is one local
warm-cache comparison, not a server latency claim. Hosted improvement is
**unverified**.
- Initial setup attempts failed before the affected test ran: registry
access, local tsx IPC, and a missing SDK build. The canonical Node
24.22.2/pnpm 9.15.4 setup and SDK prerequisite resolved these setup
failures. The local SQL timing client used Node 26.4.0; the focused
application tests used Node 24.22.2.
- [The first CI
run](https://github.com/paperclipai/paperclip/actions/runs/42705129724/job/129460702073)
failed the inherited case-history API first-page search assertion. The
same failure was reproduced on the unchanged base: the route ranked
sixth for “case for task.” The isolated route-summary correction from
#20370 passed the same three assertions. The search algorithm and
assertions were not changed.
- No public command changes require a documentation update. The schema
and migration record the added index. No hosted DDL or deployment was
performed for this PR.

- Final head `c50101dabf46ade42b545d2051d724d93f413ccd`: [full
CI](https://github.com/paperclipai/paperclip/actions/runs/42707623920)
passed with 60 successful checks and three configured skips. [Fresh
Greptile
review](https://github.com/paperclipai/paperclip/pull/20372#issuecomment-6752305627)
is 5/5 on this head with no open threads.

## Risks

- The index adds storage and write cost when a run profile changes.
- A concurrent build reads the table and can wait for old transactions.
The existing migration path runs it outside a transaction and can repair
an interrupted invalid index.
- Existing nested-operator lookups can use the index. The companion
selector needs this code change to use it. Other JSON path queries are
unchanged.
- The small local result does not establish end-to-end import speed or
PostgreSQL 17 hosted latency. Fresh deployed measurement remains
necessary.

## Model Used

OpenAI Codex, GPT-6 family. The exact runtime model ID and context
window are not exposed in this session. Used reasoning, code editing,
shell tools, and local PostgreSQL tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 16:17:13 -06:00
DottaandClaude Opus 5.5 c49ab6b9ee feat(issues): wake the parent once when a child or blocker stalls on the board (#20098)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent task delegates to children and waits on blockers. Its agent
learns of progress only through wakes
> - Today only completion (`issue_children_completed`) and resolution
(`issue_blockers_resolved`) wake a waiting issue
> - In restart chaos `w-20261108T181313-s9701` t05, a blocker stopped on
a recovery action that needs the board, and its parent sat silently
blocked for 50+ minutes
> - This pull request wakes the waiting agent once per such stall, tells
it what it can do, and shows the stall on the parent's page with the
child's Retry
> - The benefit is that a stuck child or blocker no longer silently
strands its parent

## Linked Issues or Issue Description

Refs #19831 (durable ownership plan; restart chaos campaign). Restart
chaos wave `w-20261108T181313-s9701`, task t05 (M1, claude).

**Problem**

From the kept database:
- CHA-8 `blocks` CHA-5.
- CHA-8's board-owned recovery action (`native_session_retry_exhausted`)
was created at 18:44:02.909 and resolved with outcome `blocked` at
18:44:08, when CHA-8 moved to `blocked` for the board.
- CHA-5's only wake in the whole wave was its `issue_assigned` at
18:20:48. Its agent never heard about CHA-8.

**Expected behavior**
- When a child or blocker stalls on the board, the waiting agent is
woken once. It can then re-delegate, escalate, or block with a reason.
- The board sees the stall on the waiting task, with the child's Retry.

## What Changed

- **Shared:**
- `ISSUE_BLOCKER_STALLED_WAKE_REASON` (`issue_blocker_stalled`),
`ISSUE_BLOCKER_STALL_RELATIONS`, and the types
`IssueBlockerStallRelation` and `IssueBlockerStalledWakeContext`;
  - `Issue.stalledDependencies`.
- **Server, `services/recovery/blocker-stall-wakeups.ts`:**
- A stall is an issue that is not done or cancelled, whose latest
recovery action needs the board. That means either open
(`active`/`escalated`) and board-owned or escalated, or board-owned and
resolved as `blocked` while the issue is still `blocked`. The second
case is what t05 needed: recovery resolves its action that way when it
hands the issue to the board.
- Waiters are the parent, and every issue the stalled one `blocks`, when
they are open, visible and agent-assigned.
- Each waiter gets one wake per stall. The idempotency key
`issue_blocker_stalled:{waiter}:{stalled}:{recoveryActionId}` is checked
through the indexed `(company_id, payload->>'issueId')` path. A later
stall has a new action id, so it wakes again.
  - Each wake writes activity `issue.blocker_stalled_wake`.
- The pass runs from the recovery-action sweep
(`reconcileActiveRecoveryActions`) and from the per-issue blocked sweep
(`reconcileResolvedDependencyWakeBackstop`, which visits each blocked
issue). No new worker.
- A waiter's agent that cannot read the stalled issue (`issue:read`,
which honors private visibility) is woken without its title, identifier
or recovery details; the activity entry withholds them too.
- **Database:** migration `0450` adds the partial unique index
`agent_wakeup_requests_issue_blocker_stalled_uq` on `(company_id,
idempotency_key)` for `issue_blocker_stalled:` keys in delivered or
pending statuses, so two concurrent sweeps cannot both deliver a stall.
It is built `CONCURRENTLY IF NOT EXISTS` outside a transaction (listed
in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`).
- **Queue:** `allowsDependencyBlockedWake` (in
`run-dispatch/domain/wake-context.ts`) lets a stall wake run while the
waiter's blockers are unresolved, at both dependency gates. Without it
the wake was parked until the stalled blocker resolved, which is never.
Pause holds are unchanged.
- **Agent guidance:** the wake payload carries `blockerStall`, and
`renderPaperclipWakePrompt` adds a "Stalled dependency" section. It
names the task and its recovery action and the board's next action, then
the three choices: re-delegate, escalate with a comment, or block with a
reason naming the stalled task.
- **Other reason lists:** `issue_blocker_stalled` joins the
responsible-user wake reasons, the conversation execution-wake reasons
(it never drives a chat turn) and the wake diagnostic's known reasons.
- **Board surface:**
- `GET /issues/:id` returns `stalledDependencies`, readable issues only
(`readableIssueEdgeIds`).
- `StalledDependencyNotice` shows on the issue's chat notices, next to
`IssueBlockedNotice`: each stalled child or blocker, its board next
action, a link, and the child's own Retry (`useTaskRetry`), offered only
when the child's interruption allows it.

## Verification

Red/green:
- `server/src/__tests__/issue-blocker-stall-wakeups.test.ts` (14 tests).
  - On master: no `issue_blocker_stalled` wake is raised.
- With this PR: the parent is woken once per stall, and re-woken after a
new stall. A blocker's dependent is woken.
- The t05 shape (an action resolved as `blocked`) wakes once, and
nothing more after a Retry.
- No wake for an agent-owned action, a done, unassigned or hidden
waiter, or an action resolved another way.
  - `listStalledDependencies` lists children and blockers.
- `heartbeat-dependency-scheduling.test.ts` "admits an
issue_blocker_stalled wake while a blocker is unresolved". On master the
wake was parked (`null`); once admitted, the run-start gate cancelled
it. With this PR it runs with `dependencyBlockedInteraction`. The file:
14 passed.
- `packages/adapter-utils` `server-utils.test.ts`: the new prompt test
failed on master (no `blockerStall`). The file: 141 passed.
- `ui/src/components/StalledDependencyNotice.test.tsx`: 3 passed.
`IssueChatThread.test.tsx`: 109 passed.

Regression and static checks:
- Nearby server suites (recovery durable wait, dependency wake routes,
recovery actions, run-preparation): 149 passed.
- `tsc --noEmit` passes for `packages/shared`, `server` and `ui/src`.
- `pnpm check:token-gates` is clean.

Review round 1 (Greptile, Superagent), each with a regression:
- A backlog waiter is never woken. The waiter's status and assignee are
guarded at enqueue.
- One stall per run: no coalescing, so a merged run cannot keep only the
last stall's context.
- Only a delivered or pending wake ends a stall. A refused one
(wake-on-demand off) is retried after 10 minutes, and an agent that
takes no wakes is skipped without writing a row.
- Company-wide passes resume after the last stall a capped pass reached.
The per-issue sweep also checks the issue's own children and blockers,
which covers an unassigned stalled blocker.
- The dependency gates no longer trust the reason string.
`verifyBlockerStallWake` requires a system request, the stalled issue's
current stall action, and a real child or blocker edge. An
agent-requested wake with the same context is parked (tested).
- The notice shows a refused Retry's reason, and a lost response as
"Taking longer than usual…", with Retry still available (tested).

Review round 2 (Greptile), each with a regression:
- Race: two passes claiming one stall leave one delivered row (unique
index; the loser's insert is skipped).
- The company recovery sweep selects companies whose only stall is a
resolved-`blocked` board action, so an unassigned stalled blocker with a
todo waiter is reached.
- Private stalled issue: the waiter's wake and activity carry no title,
identifier or recovery details.
- `GET /issues/:id` always returns `stalledDependencies`, so a cleared
stall replaces the cached list.
- The notice shows a failed child read with a Reload button (tested).
- `pnpm --filter @paperclipai/db check:migrations` passes.

Review round 3 (Greptile), each with a regression:
- A stall wake that fails to enqueue is retried: the recovery-action
pass reports `blockerStallWakeFailures` and the dependency pass counts
it in `enqueueFailed`, so the worker throws and retries at its backoff
(`issue-recovery-work-lifecycle.test.ts`).
- The notice drops a stall once the child's refreshed status has moved
on (`StalledDependencyNotice.test.tsx`, 7 tests).

## Risks

- A deploy wakes the waiters of every issue that is stalled today, once
each. That is intended (they are stranded now), and it's bounded at 200
stalls per pass.
- A stall wake may run while blockers are unresolved, like a comment
wake. It is bounded to one per stall, and pause holds still apply.
- The agent skill (`skills/paperclip`, CODEOWNER-owned) does not yet
list the new reason. The wake prompt carries the guidance.
- One additive index, built concurrently; no persisted run state change.
Existing rows cannot violate it before deploy, since no
`issue_blocker_stalled:` keys exist yet.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 18:26:07 -06:00
DottaandClaude Opus 5.5 576ba29fff feat(recovery): settle the legacy controller lease on the shared ownership path (#19953)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Legacy (non-native) runs hold a controller lease on `heartbeat_runs`
while the server prepares and dispatches the adapter. The shared
ownership reconciler now settles proven-gone owners (Phase 2 of the
durable ownership plan)
> - The lease stamped only a boot id. The reconciler could never prove
its owner gone (`missing_identity` in the Phase 2 data), and recovery
released it by age: lease expiry (60 s), then a 5-minute age guard on
dispatched runs
> - A hard-killed server's legacy run therefore stayed `running` for up
to 6 minutes after the next boot
> - This pull request records the controller process with the claim,
registers a compare-and-swap settle, and makes live-run recovery use the
shared prover instead of the age guard when the controller is proven
gone
> - The benefit is that a dead controller's run is settled at once,
while a live, unprovable, or previous-release controller keeps today's
behavior

## Linked Issues or Issue Description

Refs #19933 (the settle hook, merged). Stacked on #19950 (migration
0446, registry) and targets its branch until it merges; then it
retargets to master.

**What happened?**
After a hard kill, a dispatched legacy run stayed `running` until its
controller lease expired and its last update was 5 minutes old.

**What did you expect?**
The next boot proves the old controller gone and fails the run as a lost
process at once.

**Steps to reproduce**
1. Start a legacy (non-native) run and let it dispatch the adapter.
2. Kill the server (SIGKILL).
3. Start the server. The run stays `running` for up to 6 minutes.

## What Changed

- `heartbeat_runs.controller_process` (jsonb, migration 0447): `{bootId,
pid, hostHash, processStartedAt}` of the claiming server process. It
names the controller only while its `bootId` equals
`controller_boot_id`. Revocation swaps the boot id, so a revoked lease
never keeps it.
- `legacyControllerClaim` writes it in the same UPDATE (from
`currentControllerProcessStamp()`, no I/O).
- `legacy-controller-lease.ts`: `legacyControllerOwnerStamp`,
`legacyControllerProvenGone` (calls `proveOwnerGone`), and
`settleLegacyControllerLease` (compare-and-swap expiry of the proven
controller's outstanding lease after it wakes live-run recovery).
- `ownership-registry.ts`: `legacy_controller_lease` registers `settle`
and `listForRuns`. The owner is the bound process; other rows are
boot-id fences. The listing covers outstanding leases only.
- `heartbeat/recovery.ts` (3 small hunks): a controller of another boot
that the shared prover proves gone has its live lease settled through
`settleOwnableKind` and skips the age guard. The local child-process
check after it is unchanged, so a live child still keeps the run
running.
- Migration 0447 builds the partial index with `CREATE INDEX
CONCURRENTLY` (heartbeat_runs is a known-large table) and is added to
`MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`.

Bespoke code deleted: none. The kind had no liveness proof; the age
guard stays for controllers that are not proven gone.

## Verification

- `cd server && npx vitest run
src/services/legacy-controller-lease.test.ts`: 33 passed.
- `cd server && npx vitest run
src/__tests__/ownership-reconciliation-report.test.ts`: 17 passed.
- Red with `recovery.ts` and `ownership-registry.ts` from the base
commit (tests, schema and lease module from this branch): 4 failed, 29
passed. Red: the crash-point reaper test, the expired-lease age-guard
test, the reconciler settle test, the exact-controller CAS test. The
tests were written after the code.
- New tests:
  - the claim stamps this process, and a changed boot id unbinds it;
- crash point: a controller dispatched the adapter and was killed with
its lease live; the startup reaper (default 5-minute age guard) fails
the run as `process_lost` at once;
  - a dead controller's expired lease skips the age guard;
- counter-cases keep the run and lease through the reaper and a
reconciler pass: a live earlier-boot PID, a dead PID in another PID
space, and a previous-release row with no `controller_process`; an
expired lease of a live controller stays behind the age guard;
- a read-only pass (`record: false`) never settles; a recording pass
expires the lease and leaves the run to live-run recovery;
  - settle refuses a controller that changed after the proof.
- Not run locally: typecheck (CI runs it) and the wide suites.

## Risks

- Previous-release story (rules/upgrade-safe-run-state.mdc): rows saved
by the previous release have no `controller_process`. They stay boot-id
fences with today's release (expiry, revocation, age guard). A rollback
ignores the column.
- Unprovable controllers (another host or PID space, unknown space, a
live PID with an unreadable start time) never release early. Their
handoff stays the lease expiry plus revocation, and the age guard. A
different host is never proof.
- Behavior change: a dispatched run whose controller is proven gone is
failed as `process_lost` without the 5-minute wait. The retry decision
is unchanged.
- The settle expires the lease only. Live-run recovery still settles the
run under its own checks. Accounting and workspace obligations are not
touched.
- The index build is concurrent. A cancelled build is retried by the
migrator's invalid-index handling.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 23:27:54 -06:00
839d41aabb fix(server): retire owned plugin listeners before idle sleep (#19358)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Plugins receive durable agent lifecycle events through a registered
host.
> - The host uses a PostgreSQL LISTEN connection to wake delivery.
> - An idle report could return `none` while that host-owned connection
stayed open.
> - This change gives that listener an exclusive owner and closes it
during the current idle hold.
> - Release or expiry restores listening before journal replay.
> - Failed close and pending work continue to block sleep.

## Linked Issues or Issue Description

Refs #18988, which is now merged. This focused ten-file change now
targets master. Current integrated candidate:
`0f83005122ef1bdb1f1e08f8f58424c9a840ebb1`, with actual master
`644ee66f5d4ae8365fe8b9eca57df2be7405b5ab` preserved through normal
merges. No history was rewritten.

**What happened?**

A normal local App with an installed lifecycle plugin could return an
idle `none` report while its lifecycle LISTEN backend and socket stayed
open.

**Expected behavior**

The host must close its own listener before it reports idle safety. It
must retain other subscriptions, durable events, company scope, retry
deadlines and useful reactivation.

**Steps to reproduce**

Start the normal server with a lifecycle plugin. Create an agent and
wait for its durable acknowledgment. Stop the server, create another
agent while it is stopped, and restart with the same files. Request an
owned idle hold after ordinary replay. Check the exact listener backend
and matched Node socket before shutdown. Let the valid hold expire
naturally, then make an ordinary authorized agent update and check its
durable plugin acknowledgment.

Live executing or persistent preview services remain active. The
separate hosted preview dependency
[paperclip-cloud#478](https://github.com/paperclipai/paperclip-cloud/pull/478)
retains its own App and operator prerequisites. This PR does not claim
deployed preview or suspension acceptance.

## What Changed

- Add a private, exclusive database listener owner that inherits the
normal client policy.
- Prepare plugin lifecycle retirement after native retirement under the
same current idle hold.
- Use one delivery-eligibility helper for dispatch and idle journal
inspection. It preserves the same manifest, capability, company-disable
query, and skip rules.
- Close only the owned listener. Keep failed close ownership and
inspection failures visible.
- Resume listening on normal release or expiry, before journal replay.
- Preserve disabled-feature retry history and the original deadline on
re-enable.
- Await lifecycle disposal during normal shutdown and consumer teardown.
- Add owner regressions and document the lifecycle contract.

## Verification

**Current integrated source:**
`0f83005122ef1bdb1f1e08f8f58424c9a840ebb1`. Fresh normal Runner
TypeScript passed in 0.785 seconds. The complete normal App layered
semantic check passed in 28.284 seconds. All tracked source, the
lockfile and the 18 separate private witness inputs stayed unchanged.
Known compiler owners closed and the isolated temporary home was
removed.

**Current product qualification:** at measured integration
`896cc9b62993e6ed10418d7a7417e1ddbf8dc89c`, all three whole files passed
with no failures or skips: lifecycle delivery 40, runtime hooks 20, and
runtime assignments 24. The actual physical LISTEN and natural-expiry
controls passed. All owned clients and captured processes closed.
Independent final session checks were zero before normal deletion of
each successful test database.

**Normal full-App physical retirement:** the original case passed in
47.574 seconds. It exercised real startup, readiness, SDK callbacks,
durable ACK, foreign-company exclusion, physical stop, offline input and
cold replay. The initial unknown hold and its ordinary release remain in
the evidence. After the original 30.5-second cadence, a new valid hold
returned quiescent. The exact current cold-App LISTEN backend and its
matched Node socket closed before server stop.

**Normal full-App natural expiry:** the separate original case passed in
77.391 seconds under its original 90-second body cap. The final valid
30-second hold expired naturally. A new listener backend and matched
current App socket appeared before fresh input. An ordinary authorized
agent-name PATCH produced journal event 20, the actual SDK callback and
durable ACK. Foreign exclusion and readiness assertions passed. The
final measured hold had no manual release, replacement hold or shortened
TTL. The original preparatory unknown hold, ordinary release and
30.5-second cadence remain explicit.

Both App cases used the normal current server helpers and actual SDK
bundling/worker entry. A private compatibility change supplied the same
owned HOME at the exact existing child-fork boundary. It did not change
the product helpers, business assertions or timing limits. These
separate acceptance fixtures remain outside the focused product diff.

**Source carry to the current candidate:** all nine other owned files,
listener mechanics, actual producer paths, SDK paths and measured hold
behavior are unchanged. The sole owned intersection from newer master is
an independent environment-delete cleanup callback in `app.ts`. The
plugin witnesses do not call that route. Normal merges preserve every
unrelated master blob and file mode. The fresh current semantic checks
above qualify the integrated source. New master Native crash-recovery
and saved-copyback Retry capabilities are not claimed by these plugin
cases.

**Actual master merge target:** newer master contains retained plugin
Runner release/resume and other Native recovery fixes. The projected
combined target passed fresh normal Runner TypeScript in 0.799 seconds
and the full App semantic check in 29.715 seconds. All ten plugin-owned
blobs and modes remain exact to the published head; every unrelated
master change is preserved. The measured plugin cases create no Native
provider turn or warm Native session. The unchanged
native-first/plugin-second ordering still checks the native result
before listener retirement. Later sparse-checkpoint and Native
response-publication changes do not enter those paths. This is a scoped
source carry, not new hosted Runner acceptance.

**Failures retained:** an earlier reactivation fixture's real timer
returned one millisecond before its original wall-clock expiry. Its
private fixture now waits any positive remainder and retains the same
expiry assertion and limits. The earlier partial-composition `unknown`
result remains distinct from normal-startup proof. Current qualification
also exposed missing normal workspace dependencies and a private guard
that assumed loose Git refs. Normal owned dependency setup and an exact
loose-or-packed reader fixed those setup failures. No business
assertion, timeout, permission boundary or product sleep gate was
relaxed.

Repeat focused product coverage with a fresh external fixture database
and the normal server Vitest command, one worker:

```sh
pnpm --filter @paperclipai/server exec vitest run src/__tests__/plugin-lifecycle-delivery.test.ts --maxWorkers=1 --no-file-parallelism
pnpm --filter @paperclipai/server exec vitest run src/__tests__/agent-runtime-hooks.test.ts --maxWorkers=1 --no-file-parallelism
pnpm --filter @paperclipai/server exec vitest run src/__tests__/agent-runtime-assignments.test.ts --maxWorkers=1 --no-file-parallelism
```

Exact current head `0f83005122ef1bdb1f1e08f8f58424c9a840ebb1` has 60
successful checks, three skips and one neutral result, with no pending
or failing contexts. Greptile completed its full review at 5/5 with no
actionable issue. All root and nested review pages were read; the
earlier eligibility finding is resolved. Ponytail's latest turn stopped
before a verdict. A normal retry was requested; no current Ponytail
verdict or score is claimed. Previous `3ca86ec` CI and scores are
historical. A full local repository typecheck/test/build or browser run
is not claimed. Controlled local App/SDK/PostgreSQL proof does not
establish live provider, deployed Railway/Neon suspension, signed Cloud
alarms or global exactly-once delivery.

## Risks

- A listener close failure or uncertain inspection must continue to
block sleep.
- Release, expiry, configuration changes, and journal arrival can race
with retirement. Epoch and current-hold checks retain conservative
results.
- Delivery remains at least once across uncertain ACK boundaries. This
change does not add global exactly-once delivery.
- The database helper closes only its exclusive listener client. Other
owners and subscriptions keep their resources.

## Model Used

OpenAI `gpt-6.1-sol`, reasoning and code/tool use. The managing Codex
agent implemented shared database/composition seams and reviewed the
change; its exact model ID was not exposed. The runtime does not expose
a verified context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-11-06 22:56:47 -06:00
DottaandClaude Opus 5.5 7e3caa7bac fix(tree-control): resume work a pause gate refused after release (#19421)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The board can pause an issue tree. A pause stops work. Resume or
release must continue that work once.
> - PR #19394 made release re-dispatch a run that a pause Stop
cancelled.
> - That rule covers only a run that an operator Stop ended and that the
hold snapshot named.
> - A scheduled retry that comes due during a pause is not stopped. The
dispatch gate cancels it with `issue_paused`. The release did not see it
as owed work, so the task stalled with no run.
> - This pull request makes every gate that refuses work for a pause
record the covering hold. Release then re-dispatches that work exactly
once.
> - The benefit is that a paused task with a parked retry or a queued
run continues after release, with no manual wake.

## Linked Issues or Issue Description

Refs #19394

**What happened?**
A child task had a `workspace_busy` scheduled retry. The board paused
the parent tree. The retry came due during the pause. The
scheduled-retry gate cancelled it with `issue_paused` ("Scheduled retry
suppressed because the issue is held by an active subtree pause hold").
The board released the hold. Four minutes later the child was still
`in_progress` with no run, no execution blocker, and no new activity.

**Expected behavior**
Release (or `mode: "resume"`) re-dispatches the refused work exactly
once. No reconciliation and no user message are necessary.
Stranded-issue recovery does not add a second run.

**Steps to reproduce**
1. Give an assigned child task a parked `workspace_busy` scheduled
retry.
2. Pause the parent tree.
3. Let the retry come due while the pause is active. The gate cancels it
with `issue_paused`.
4. Release the pause. No run starts for the child.

**Paperclip version or commit**
master at 4d49a31286 (includes #19394).

**Agent adapter(s) involved**
Not adapter-specific (core bug).

## What Changed

- `issue-tree-control.ts`: add `ISSUE_PAUSED_ERROR_CODE` and
`pauseSuppressionResult()`. A gate records `treeHoldSuppression: {
holdId, rootIssueId }` on a run that it refused for a pause.
- `findOwedPauseResume` now treats the latest run as owed when it is an
`issue_paused` cancellation under a pause hold. It uses the recorded
hold. For an unmarked run (a scheduled retry cancelled before this
change), it uses a snapshot pause that was active at cancellation. The
operator Stop rule does not change.
- Scheduled-retry gate (`run-dispatch/adapters/postgres.ts`,
`cancelSuppressedRetryInTx`): locks the pause row `FOR SHARE` and
records the covering hold. A concurrent release either sees the refusal,
or the retry sees no pause and runs normally.
- Queued-run claim gate (`heartbeat/queue.ts`): cancels with
`issue_paused` and the hold marker, after the agent start lock is
released. Before, the cancel re-entered that lock from inside it, and
the agent's queue stalled until the 30-second stale-lock timeout. After
the refusal is saved, the gate re-checks the hold and resumes the task
itself if release already finished (`resumeIfPauseAlreadyReleased`).
- Release (`issue-tree-resume.ts`, route): revisits the snapshot members
and every task with a run refused for that hold, also when `wakeAgents`
is set. This covers tasks created under the tree after the snapshot and
tasks moved out before release.
- `packages/db`: partial index
`heartbeat_runs_tree_hold_suppression_idx` (migration 0436, `CREATE
INDEX CONCURRENTLY`, registered as a top-level migration) for that
lookup.
- Docs: `doc/execution-semantics.md` (gate-refused work, race handling,
non-assignee and pre-fix scope) and `doc/DATABASE.md` (index).

## Verification

- Red before the fix, green after: `pnpm --filter @paperclipai/server
exec vitest run src/__tests__/issue-tree-pause-resume-run.test.ts` (21
tests).
- "resumes a parked retry the pause suppressed exactly once after
release / resume / legacy": before, `expected [] to have a length of 1`
(no resumed run). After, one `issue_tree_resumed` run with
`interruptedRunId` = the suppressed retry, no execution blocker, and the
task finished. `reconcileStrandedAssignedIssues` added no run during the
hold or after release.
- "resumes a queued run the claim gate refused during the hold exactly
once": before, the test timed out at 15 s (start-lock re-entry). After,
the run is cancelled with `issue_paused` and the hold marker, and
release re-dispatches it once.
- "resumes a refused retry on a task moved out of the paused tree before
release" and "... on a task outside the snapshot when release also wakes
members": review findings, one resume each.
- "leaves a due retry parked when the pause is released while the gate
refuses it": the test waits until the gate blocks on an uncommitted
release. The retry is not cancelled and runs once. Without the `FOR
SHARE` lock the gate does not wait and the test fails.
- "resumes work a gate refused after the release that should have
resumed it had finished": the late-refusal re-check resumes once, and a
repeated check does not dispatch again.
- "does not resume a paused-code cancellation that no pause hold
covered": negative case.
- Related suites pass locally: `src/__tests__/issue-tree*`,
`src/modules/run-dispatch`, `src/services/heartbeat/queue.test.ts`,
`heartbeat-dependency-scheduling`, `heartbeat-workspace-busy`, and
`packages/db` `client.test.ts`.
- I did not run a live staging check here. Staging APIs are not
reachable from this environment.

## Risks

- Migration 0436 adds a partial index on `heartbeat_runs`. It is built
concurrently and covers only cancelled runs with the marker, so it is
small.
- The scheduled-retry gate now takes a `FOR SHARE` row lock on the pause
hold before it cancels. Release updates that row, so the two serialize
briefly. Release does not lock issues or runs, so the lock order cannot
deadlock with the gate's issue-then-run locks.
- The queued-run claim gate now records `issue_paused` in place of the
generic `cancelled`. Stranded recovery already classifies `issue_paused`
as non-retryable. The pause resume owns that work, so recovery does not
add a second run.
- Scope limits (documented): a non-assignee run refused by a pause is
not replayed, the same as non-assignee wakes that admission drops during
a hold. Among pre-fix rows, only scheduled retries carry `issue_paused`,
so only those fall back to the snapshot. A pre-fix row whose hold is
already released does not self-recover; the staging task that showed
this bug needs one manual wake, or a new pause and release.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 03:03:15 -06:00
DottaandClaude Opus 5.5 5c185db77c fix(tree-control): resume work a subtree pause interrupted (#19394)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Task tree control lets the board pause a task tree and then resume
it.
> - A pause stops the runs that are active in the tree. The release must
then continue the work that was pending.
> - A child task whose run was active at pause time did not start again
after release. The child became blocked and asked for a user message.
> - A pause is an operator hold, not a failure. It must not strand work,
and it must not require reconciliation for a run that never started.
> - This pull request re-dispatches interrupted work one time on
release, and it settles a legacy Stop that never entered the adapter.
> - The benefit is that pause and resume keep a task tree moving without
manual recovery, and real safety gates stay in force.

## Linked Issues or Issue Description

**What happened?**

A parent task delegated two children. One child's run was active (in
host workspace preparation, before the adapter started) when the board
paused the parent's tree. The pause cancelled that run. After the board
released the hold, no new run started for that child. The child became
`blocked` with the execution blocker cause
`legacy_execution_requires_reconciliation`. Later the next action became
"Send a message to continue." A sibling child that had no active run at
pause time resumed normally.

**Expected behavior**

A run that a subtree pause cancels is an operator hold. On release, the
pending work starts again one time, automatically. It does not go
through execution reconciliation when the run never entered its adapter.
It does not wait for a user message.

**Steps to reproduce**

1. Create a parent task and a child task that is assigned to an agent.
2. Wake the child. While its run is in host preparation (or queued),
call `POST /api/issues/:parentId/tree-holds` with `{ "mode": "pause" }`.
3. Call `POST /api/issues/:parentId/tree-holds/:holdId/release` without
`wakeAgents`.
4. Before this change, no new run starts, and `getExecutionBlocker`
reports `legacy_execution_requires_reconciliation` for the child.

**Root cause**

- (a) The pause records each interrupted run in the hold member snapshot
(`activeRunId`). Nothing used that record on release.
- (b) Release woke tasks only when `metadata.wakeAgents === true`.
Without it, release relied on stranded-issue recovery. That recovery
stands down when the latest run was cancelled by an operator
(`isOperatorCancelledRun`). So the interrupted child had no owner.
- (c) The pause Stop of a legacy run in preparation is written directly
as `cancelled`. Legacy terminalization classifies a cancelled run with
no bootstrap evidence as an unknown outcome. It records
`legacy_execution_requires_reconciliation`, although the gated dispatch
never entered the adapter.
- Native runs: the verified native Stop does not create the
reconciliation hold, but gap (b) applies to native runs too. The release
change is runtime-independent.

## What Changed

- `issueTreeControlService.findOwedPauseResume` decides if a task is
owed one resume. The task's latest run must be an operator Stop that a
pause hold member snapshot names, recorded while that hold was active.
The task must still be executable by the same agent, and no pause may
still cover it. The obligation belongs to the run, so overlapping pauses
keep it until the last covering release.
- New `services/issue-tree-resume.ts` re-dispatches owed work with wake
reason `issue_tree_resumed` (`holdId` and `interruptedRunId` in the
context). A compare-and-set claim (`treeHoldResume`) on the interrupted
run makes the resume happen at most once. A failed wake returns the
claim.
- Pause release and `mode: "resume"` call it for every member of the
released hold, also without `wakeAgents`. When `wakeAgents` is true, the
existing loop wakes every executable member.
- A task that still has an execution blocker is not replayed. The
release writes `issue.tree_resume_wake_skipped`, the blocker stays
visible, and the obligation stays unclaimed.
- `releaseHold` updates only an `active` hold. A repeated release gets
`409`.
- `settleCancelledLegacyPreparation`: when the executor that owned a
cancelled legacy run proves that dispatch never entered the adapter, it
records bootstrap evidence and resolves that run's
`legacy-execution:<runId>` hold as `false_positive`. It also corrects an
automatic no-replay disposition and restores a `blocked` status only
when that disposition made the change. Then it re-dispatches owed work,
for a release that happened while the Stop was unwinding. It requires a
requested Stop with no existing recovery receipt, and no process,
session, native finalization, unreleased lease, or execution-start
event.
- `settleUnrecoverableExecutions` waits while a legacy executor still
holds its controller lease, and records a `legacyCancellationBlock`
receipt when it blocks a task for a cancelled legacy run.
- Docs: `doc/execution-semantics.md` and `doc/composer-stop.md` describe
the resume rule.

## Verification

- New regression
`server/src/__tests__/issue-tree-pause-resume-run.test.ts` uses embedded
Postgres, the real heartbeat, and the real tree-control routes:
- A run paused during preparation: nothing runs during the hold, there
is no blocker, and release creates exactly one `issue_tree_resumed` run
that succeeds. A second release returns `409` and creates no run.
- Release before the stopped preparation finishes unwinding (with the
periodic disposition run in between): exactly one resumed run after the
executor settles.
- An automatic no-replay disposition that ran first: the blocker clears,
the status returns to `in_progress`, and release resumes once.
- Overlapping child and parent pauses: no resume while the parent pause
remains, then exactly one resume after it is released.
- A queued run cancelled by the pause: release creates exactly one
resumed run.
  - A task reassigned to a user during the hold: no resume.
- A mid-turn legacy Stop with unknown outcome: no replay, the
reconciliation blocker remains, and `issue.tree_resume_wake_skipped` is
logged.
- Red before each fix: the first 4 cases failed on master; the 3 review
cases failed on the first PR commit. All 7 pass now.
- Also passed locally: `issue-tree-control-routes`,
`issue-tree-control-service`, `heartbeat-workspace-busy`,
`heartbeat-undispatched-execution-wait`, `execution-control-work`,
`legacy-workspace-restore-recovery`, `legacy-continuation-authority`,
`task-retry`, `legacy-execution-recovery`, `agent-conversations`,
`heartbeat-retry-scheduling`, `satisfied-connection-intents`.
- Not verified here: a live staging pause/resume. Re-check on staging
after deploy.

## Risks

- Behavior change: release now starts the work that the pause
interrupted, also without `wakeAgents`. Before, that work stayed idle.
Other idle tasks still wake only with `wakeAgents`.
- The settlement trusts the executor's adapter-entry flag. The flag is
set only inside the atomic dispatch gate, which refuses a cancelled run.
The settlement also requires no process, session, lease, or execution
event.
- A native run whose Stop has not finalized at release time is not a
target. Stranded recovery and explicit wake still apply.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 22:08:01 -06:00
DottaandClaude Opus 5.5 bf10bc695a feat(task): show the agent's context window in task properties (#19273)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents run through a harness (Codex app-server, or Claude through
ACPX), and the runner turns harness events into Paperclip events
> - Both harnesses report how full the model's context window is. Codex
sends `modelContextWindow` with its token usage. Claude sends `used` and
`size` in ACP `usage_update`
> - Paperclip dropped both values. The task view could not show how
close an agent is to compaction, and a compaction left no visible mark
in the task chat
> - This pull request carries the context-window occupancy through the
runner, stores it with the existing usage events, and shows it as a
"Context" row in the task Properties panel
> - It also shows a "Context compacted · 230k → 38k tokens" divider in
the task chat when the agent compacts its context
> - The benefit is that operators can see when a long task is near its
context limit and when the agent lost detail to a compaction

## Linked Issues or Issue Description

Refs #19252 (same pattern: a canonical harness event turned into
task-chat UI). Refs #19267 (status notices; it turns Claude's compaction
banner into a notice, and this PR reads that notice too).

**What existing behavior does this improve?**
The task view shows no context-window information, and a compaction is
invisible in the task chat.

**Subsystem affected**
Runner usage normalization (Rust and TypeScript transport), the PRP
usage schema, a read-only issue API, the task Properties panel and the
task-chat transcript.

**Current behavior**
Codex `modelContextWindow` and ACP `used`/`size` are dropped in the
runner. `context.compacted` always has null sizes, Claude compactions
emit no `context.compacted` event at all, and the live runner turn hides
every marker except "interrupted".

**Proposed behavior**
`usage.reported` carries an optional `context {usedTokens,
windowTokens}`. The task Properties panel shows a Context row ("10% of
258k") with a hover breakdown (tokens used, input and cached input,
output, cost, compactions). The transcript shows a compaction divider,
with sizes from the usage before and after the compaction.

**Reason and benefit**
Operators can see when a long task is close to its context limit, and
they can see where the agent's earlier context was summarized.

**Breaking changes**
None. The new schema field is optional, and the new endpoint is
read-only.

## What Changed

- PRP `usage.schema.json`: add an optional `context {usedTokens,
windowTokens}` object. Regenerate the schema bundle and the protocol
manifest.
- Rust `provider_events.rs`: add `codex_context_usage` (last request
`totalTokens` and `modelContextWindow`) and `acpx_context_usage` (ACP
`used`/`size`). ACPX `usage_update` now sets `context`.
- Rust `provider_backend.rs`: Codex `thread/tokenUsage/updated` sets
`context` on live (non-historical) usage.
- Rust `provider_events.rs`: emit `context.compacted` for the completed
`context_compaction` provider notice that #19267 produces from Claude's
compaction banner. Only this path marks a Claude compaction, so the
server counts each compaction once.
- `runnerd-codex-transport.ts`: `rehydrateRunnerdUsageNotification`
keeps a valid `context` on `tokenUsage`, so it reaches the stored
`item.completed` usage item.
- Server: new `GET /api/issues/:id/context-usage` (issue read access is
required) and `issueContextUsage` service. It reads only the assignee's
current session (the session of its newest usage report), and returns
that session's latest context reading, non-zero totals, cost, and the
count of compactions across all of the session's runs. Added to OpenAPI.
Shared type `IssueContextUsage`.
- DB: migration 0432 adds two partial indexes on `heartbeat_run_events`,
built concurrently. One indexes usage items by run and time. The other
indexes `context.compacted` events by company and session. `EXPLAIN`
shows that both lookups use them.
- UI: new `ContextUsageMeter` (bar, percentage, hover card) and
`IssueContextUsageRow` in the Execution section of Properties. The row
refreshes every 8 s while a run is active, and once more when the run
ends.
- UI transcript: `native-run-events.ts` maps `context.compacted` to a
`context` activity entry with pre/post sizes from neighbouring usage
items. `transcript-adapter.ts` turns it into a new `context_compacted`
marker variant. The live runner turn keeps that marker as its own
timeline row.
- Storybook: `task-context-window.stories.tsx` (Properties panel in a
task, all states, hover breakdowns, divider).

## Verification

- `cargo test -p paperclip-runner-core --lib provider_events` (new
tests: ACPX context, Codex context, banner compaction, notice
compaction)
- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/live/runnerd-codex-transport.test.ts`
- `cd ui && npx vitest run src/components/ContextUsageMeter.test.tsx
src/components/transcript src/components/task-chat
src/components/issue-properties`
- `cd server && npx vitest run
src/__tests__/issue-context-usage.test.ts` (embedded Postgres)
- `node scripts/typecheck-low-memory.mjs`, `pnpm check:token-gates`,
`pnpm --filter @paperclipai/paperclip-runner run check:protocol-types`
- Live check on a local worktree instance through the runnerd path:
- A Codex `gpt-5.5` task showed "10% of 258k" (27k of 258k, input 77k
with 49k cached, output 1.1k).
  - A Claude Sonnet 4.6 task showed "17% of 200k" (34k of 200k).
- Storybook: Tasks and conversations / … / Task chat / Context window.
- Not verified live: a real compaction. The divider is covered by unit
tests through the same projector path the task chat uses.

## Risks

- The schema field is optional, and older runners omit it. The UI
renders nothing until a harness reports a window.
- Migration 0432 builds two partial indexes concurrently on
`heartbeat_run_events`, which is a large table. The indexes cover only
usage items and compaction events, so they stay small. They add a little
write overhead to those events.
- Claude in-progress usage rows still record zero tokens with `requests:
1`. The final turn usage is correct. This PR does not change accounting.
- A "Compact now" action is out of scope. It needs a compaction-only run
that does not produce a run result. It will come in a follow-up.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, in-app browser for Storybook and
the live instance).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:54:27 -06:00
sentry[bot] 18abf7c905 fix(db): tolerate unparseable connection URLs during teardown (#18684)
closeRegisteredClients now uses hostPortKeyOrNull and returns early for URLs that new URL() rejects, matching createDb, which never registers clients under such URLs.

Fixes PAPERCLIP-BACKEND-5C
Fixes #18685
2026-11-02 15:33:36 -08:00
DottaandPaperclip 006d5691a6 feat(server): support volume-free web server handoff (#18518)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its web server also owns controller admission and user commands.
> - A rolling deployment starts a replacement before the old server
exits.
> - Shared database state alone does not preserve local agent files,
browser chunks, or in-process commands.
> - This pull request adds an opt-in ephemeral server contract and a
local overlap test.
> - The benefit is a web server that can relinquish traffic and local
files without losing acknowledged work.

## Linked Issues or Issue Description

Refs #18403, #18327, #18335, #18341.

**Current behavior**

The control-plane web server still has local agent-file authority and
local command targets. A new container can miss saved bytes or send a
command to the wrong process. An existing browser tab also needs chunks
from its original build.

**Proposed behavior**

Eligible servers share only PostgreSQL, object storage and stable
secrets. A replacement waits for readiness. The old server drains
admitted requests, keeps owner commands available through handoff, and
releases controller leases. Explicit maintenance adopts existing files
before the volume is removed.

**Reason and benefit**

This is the application prerequisite for rolling app-only deployments.
Schema migrations and permanent multi-replica operation are outside this
change.

## What Changed

- Store agent-file manifests in PostgreSQL and immutable bytes in S3.
Treat local roots as disposable projections. Preserve existing
permission checks and edit conflicts.
- Add an explicit migration, eligibility and object-retention command to
the production runtime bundle.
- Add `/api/ready`, admitted-request drain, browser restart signaling
and retained immutable UI chunks.
- Carry transaction-aware work hints across controllers during handoff
and active execution. Keep listeners closed when idle.
- Deliver durable Stop and runtime responses to the execution owner.
Recheck request lifecycle and access before response dispatch.
- Reject local-only configuration in ephemeral mode. Keep the feature
disabled for existing installations by default.
- Add an isolated PostgreSQL/S3/proxy/browser deployment harness with
independent server homes and retained failure evidence.

## Verification

- All CI checks pass on `ce17b95019d3cf1cabd357d19b6d37f96782cc5c`:
workspace typecheck, build, Runner checks, server/general tests and
browser shards. Greptile reviewed the runtime changes at 5/5 with all
findings resolved; the final commit updates maps and the plan.
- Local targeted checks passed: coordinator 27; Stop/recovery 33;
PostgreSQL command delivery 6; route/workspace regressions 8; bootstrap
Stop 1; restart recovery 75. PostgreSQL command tests simulate the
provider edge.
- Local browser proof passed 91 reads and 19 saves; reconnect took
1,013ms. The tab opened Audit from S3 after deletion of A's whole
home/build. This uses compiled workspace code and local-trusted auth,
not released containers or live agents.
- Local workspace typecheck/build passed; current server and runtime
bundle passed. The earlier broad local suite was stopped after three
hours and is not a passing receipt. Its retained failures and isolated
reruns are documented in the plan.
- Stacked #18526 adds stricter graceful-exit proof, an unready
replacement control, live Runner qualification and a Stop handoff
correction. Hosted enablement remains gated on those results and Cloud
#1096. No hosted stack has been enabled.

## Risks

- Apply migration 0422 before enabling the feature. Existing data must
pass each owner's migration and eligibility checks. Do not discard a
volume based on a single migration page.
- Durable manifests remain authoritative after the opt-in is disabled.
Rolling back to a binary that ignores them is unsupported.
- Cold directory restoration and complete shutdown timing still need
qualification before enabling a hosted stack.
- Retained UI assets must cover the supported browser and rollback
window. Cleanup is explicit; no new idle maintenance polling is
introduced.
- This change does not claim production zero downtime or authorize a
fleet-wide enablement.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and
context-window limit are not exposed in this session. Used reasoning,
code editing, shell execution and browser automation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-01 01:58:54 -06:00
DottaandPaperclip 492d537340 fix(native): recover controllers across webserver replacement (#18343)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native execution runs in a remote Runner while the webserver owns
its controller.
> - Encrypted PRP checkpoints and session snapshots already survive a
webserver restart in Postgres.
> - Ordinary warm and per-turn admission still needs local markers and
supervisor metadata to find that state.
> - Replacing the webserver with an empty native controller directory
can therefore lose a usable retained session.
> - This change adds bounded recovery metadata to the existing snapshot
transaction and reconstructs local projections from the selected durable
source.
> - The benefit is native execution continuity across webserver
replacement, with the existing ownership and agent lifecycle policies.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Native remote controller recovery and retained-session continuation
during a webserver upgrade.

**Subsystem affected**

server/ — native runtime, checkpoint persistence, recovery scheduling
and internal upgrade reporting. Product E2E qualification also changes.

**Current behavior**

Without both the local PRP marker and supervisor file, normal
continuation can return before discovering the saved Postgres
checkpoint. The snapshot already exists in Postgres, but its local reuse
compatibility envelope cannot be reconstructed.

**Proposed behavior**

Select the existing durable source, validate its identity and ownership,
restore its PRP state and supervisor projection, then use normal
authenticated admission. Missing or inconsistent evidence holds
recovery. It does not authorize fresh provider work.

**Reason and benefit**

A replacement webserver must recover active execution and later continue
its retained session without the old server's native files. This is one
component of deployment continuity. HTTP traffic switching, workspace
storage migration and Railway rollout remain separate.

**Breaking changes**

No new public endpoint, schema migration or database pool. The private
run-profile envelope is additive. Disk-independent rollback also
requires a reader for the staged controller format and compact
checkpoint part; the first production release containing these readers
is the floor. Older file-backed sessions need an owner checkpoint or the
bounded internal importer.

**Additional context**

Related foundations: #17401, #17916, #17254, #14621, #14622. The
implementation contract and open acceptance gates are in [the dated
plan](doc/plans/2026-10-30-native-controller-checkpoints.md).

## What Changed

- Persist a versioned recovery envelope, at most 4 KiB, beside the
redacted session snapshot in its existing transaction. Keep it out of
public run projections.
- Compare frozen admission separately from prepared attachment
instructions during staged recovery, preserving exact
template/identity/authority checks.
- Bound the existing lazy checkpoint pool at three connections after
measured two-slot burst backlog; keep request/renewal capacities and SQL
operation counts unchanged. The tuning is a separate commit.
- Discover and restore the exact selected retained source without local
native markers. Preserve the 32-candidate window, progress barriers and
current authority checks.
- Fence snapshot and result writes. Acquire coordinator, issue and run
locks in order and check expiry after lock waits. Combine snapshot
ownership/read work into one statement.
- Reuse existing recovery deadlines and drain/resume scheduling.
Retained-owner waits preserve the original unstarted run and actual
lease deadline. Stop, ownership and provider-progress fences precede
deferral. Add bounded native evidence reporting, retained-source import
and retention protection.
- Admit a settled warm source with an authenticated idle goal-read
suffix while retaining active-work and mixed-authority holds.
- Permit lifecycle `turn.stop` checkpoints after accepted results under
existing ownership fences. Claim the existing successor coordinator
before retiring the old warm session, and stage shared attachments only
after verified retirement. Preserve omitted isolated-home defaults and
existing fresh skill-selection order during recovery without hiding real
configuration changes.
- Carry the exact durable six-field PRP binding during active recovery
after a healthy warm handoff. Keep the Runner mismatch guard and
preserve opaque attachment IDs.
- Preserve an already verified active or idle warm process during remote
preparation; revalidate its proof/liveness and retain later
authenticated PRP attachment. Ordinary resumed leases retain suspension
checks.
- Drain retained source events under their original authority, reconcile
immutable source accounting and block unknown source tool dispatch
before target work.
- Persist compact staged handoff intent through authenticated adoption
and recover direct/listener command-loss windows without another Runner
or duplicate attachment.
- Rebuild suspended lifecycle receipts for resumed prepared tasks, and
retry missing lifecycle reads after partial PRP materialization without
overwriting contradictory evidence.
- Add normal warm/per-turn diskless continuation coverage, real Postgres
races and SQL-cost checks. Add a reproducible paired load workload.
- Give only the explicit outage fixture a 15-minute provider idle
window, with a red/green catalog regression preserving all production
timing and lifecycle policies.
- Publish the goal-less task-session pointer in the existing
post-acceptance terminal checkpoint transaction, with current ownership,
accepted-result, reset and newer-progress fences. Keep a private bound
receipt so late completion cannot duplicate or resurrect publication.
- Use a retained warm sandbox in the dedicated continuity fixture, and
accept an exact output token followed by an explanation. Keep the strict
identity oracle.
- Extend the explicit Daytona cell to a five-minute active outage and
another replacement after completion. Require retained provider
identity, a new tool receipt and a useful follow-up result. Add an owned
loopback Postgres E2E fixture for hosts that cannot start embedded
Postgres.

## Verification

**Merge recommendation — `fc68584ce6231a02e47ecd743f55b036ccea034f`:
ready for the qualified native-controller component.** Full CI, fresh
review, local application paths, controlled performance qualification
and the strict live Linux/Daytona case pass. Independent resource
cleanup is complete. Broader rollout, browser reconnection and the
unexecuted transport/platform combinations remain integration gates.
Integrated 484 passed full local recursive typecheck/build and ordinary
active→completion→cache-loss→C component qualification. Fresh review
then found a valid attached-wake failure: preparation appends attachment
instructions before the verifier compares execution to the frozen
admission. 9f4 fixes that by separately checking the pre-preparation
admission and exact prepared execution. Both hashes are captured
privately per admission; the full template, identity, configuration and
authority checks remain. Its regression fails before the fix; 36 focused
cases, 42 staged source/target cases and server typecheck pass
afterward. The review thread is resolved. Exact-head full CI passes
([run
41168228822](https://github.com/paperclipai/paperclip/actions/runs/41168228822):
48 successful jobs, one intended skip); the complete `pnpm test:run`
partitions are forced, with the five native suites in the Runner lane.
This is not a claim that an unsharded local command passed. The fresh
exact-head Greptile check succeeds with 5/5 and all six threads
resolved.

The original two-connection database comparison closes **steady PASS and
burst FAIL**. Every relative CPU/WAL/request-p99 gate passes and all
operations/ownership remain correct, but candidate arm 3 drains in
1,015.99 ms and baseline arm 4 in 1,059.67 ms against the unchanged
1,000 ms absolute gate. Those results remain failed. A separate 2→3→4→2
diagnostic reproduces drain times 1,051.57 / 956.79 / 882.02 / 1,065.72
ms: three is the smallest tested passing cap, and the repeated two-cap
failure survives run-order/cache effects.

Separately reviewable `fc68584ce6` changes the existing lazy checkpoint
pool default to three connections. Request capacity stays ten and
liveness two; idle release, shutdown fencing, transaction counts and
pool count are unchanged. Thirteen DB lifecycle/cap tests and DB
typecheck pass. Full recursive typecheck/build pass at exact fc68584,
with the matching server build stamp. The final paired native
feature-cost comparison explicitly uses the **same cap-three policy and
common current DB factory in both variants**; its baseline is
controlled, not the unmodified historical cap-two configuration.
Original gates and workload counts remain unchanged. The one strict
Linux/Daytona case started at 04:40:56 UTC only after both new profiles
passed; it passed on the first attempt with all 49 workflow checks.
Independent cleanup confirms all three owned sandboxes absent, owned
Postgres/controllers/observer gone, VM terminated, root volume absent
and ephemeral SSH key/agent removed.

**Final controlled cap-three qualification: PASS.** Baseline native
implementation `59da15066a` and candidate `fc68584ce6` use the same
current DB factory and checkpoint policy (request/liveness/checkpoint =
10/2/3). Eight 65-second ABBA arms cover 300 controllers and 10,000
historical tasks. All correctness, absolute and 10% relative gates pass;
original cap-two failures remain unchanged.

| Mode | DB CPU µs/op, baseline → candidate | WAL B/op, baseline →
candidate | DB-request p99 ms, baseline → candidate | Maximum drain |
| --- | --- | --- | --- | --- |
| Steady | 542.40 → 554.83 (+2.29%) | 1509.01 → 1558.27 (+3.26%) |
107.11 → 111.90 (+4.48%) | 26.35 ms |
| Burst | 1036.36 → 1063.11 (+2.58%) | 4153.12 → 4317.64 (+3.96%) |
103.736 → 103.841 (+0.10%) | 930.14 ms |

Every matched arm preserves operation/statement/transaction counts,
including 3,900 PRP and 3,900 snapshot transactions. All three actual
pools are included; there are zero lost owners, missed lease deadlines,
dropped operations or errors. Peak Postgres CPU is 18.31%/9.47% of two
cores and memory 614.29/684.42 MiB (steady/burst). The benchmark uses
one Node process simulating 300 controllers; it does not qualify
two-webserver overlap, HTTP latency or hosted capacity. Its observer SQL
contributes to measured database CPU. The original unmodified cap-two
baseline is separately retained as failed; this cap-three comparison is
explicitly controlled.

The exact final-source 8/512/8,192-chunk test also passes: zero recovery
SQL, transactions or changed rows in each text-only interval, zero idle
SQL, seven lifecycle transactions and twelve changed rows per sample.
Existing lease traffic remains separate. The final ordinary managed
application replay passes active recovery, completion, native-cache
removal and C continuation; its UNKNOWN local debug-daemon provenance
and partial instruction-observer limits remain recorded. The live case
uses the attested Linux producer instead.


**Live qualification: PASS, one attempt, no retries.** Campaign
`native-controller-retained-fix-fc68584ce6-ec2-20261031T044055Z` started
04:40:56 UTC on the exact final Linux webserver source; suite definition
hash `12f9bb97823c1537e43f0398e7cddf39890f77bce7b413ca4dfd944f4a30e5d5`.
The existing immutable image and attested Bookworm Runner artifact are
unchanged.

- A was SIGKILLed and its native cache discarded. B started after
300,149 ms, naturally acquired generation 2, authenticated at 316,624 ms
and resumed at 317,682 ms within the original 420,000 ms grace. The same
active run, Runner, provider PID/start fingerprint and one
submitted/started/accepted turn were preserved; one final
result/terminal and workspace finalization completed, with no
retry/recovery action.
- After active and queued completion, B was SIGKILLed and its native
cache discarded again. C received exactly one ordinary task-comment API
follow-up, completed a new tool and produced a useful reply while
preserving native/provider/Runner identities. This closes the earlier
failed post-C gate. A separate C OS-process PID field was not exported;
the physical-PID proof is specifically A→B.
- Four distinct results were accepted and published; all four
finalizations committed at attempt 1, all four accounting-v2 projections
settled, and the task-session pointer advanced to C. The coordinator
independently checked these facts and all 49 canonical checks against
the sanitized receipts.
- Canonical cleanup passed and independent checks confirmed three exact
sandbox 404s plus complete inventory/zero owned remnants, no owned PG or
controller/observer processes, EC2 terminated, root volume absent, and
the ephemeral SSH key/agent removed by 04:54:25 UTC.

Live scope is classic remote Codex warm continuity with non-native
storage dependencies available. Prepared-task completed B→C, full
per-turn application disk loss, browser reconnection, two-server load
and overall Railway/HTTP continuity were not live-qualified here.
Earlier failed campaigns and the original cap-two burst failure remain
failed historical evidence. A report-renderer cache setup failure was
preserved; the normal sanitizer subsequently succeeded with an explicit
cache path and a new output directory, without changing validators.


The repair keeps retained events under their original six-field
authority until persistence and accounting accept them. Unknown semantic
effects stay unresolved. A compact staged target survives source
bootstrap, cached-command delivery and authenticated handoff, including
an uncertain commit response. Direct and same-guest-listener recovery
require the verified original Runner and the exact endpoint. Target
snapshot publication marks readiness in memory, with retirement
coalesced into an existing command write. Headers contain only the
staged descriptor/digest; a prequeue template is written once as a
changed checkpoint part and then recovered from the existing
command/receipt. No second session snapshot transaction, streaming SQL
path, new pool or recurring idle scan is added.

| Current repair gate | Evidence and scope |
| --- | --- |
| Source-tail regression | Real Runner protocol RED reproduced
attachment rejection. Frozen core/transport pass five source-tail cases:
source usage attribution, held persistence/no premature ACK, lost commit
response plus cache loss, unknown semantic hold and accounting rejection
hold. |
| Handoff loss windows | Direct and listener modes pass
queued/prepared/activated/completed boundaries (eight cells), preserving
the actual Runner/provider process. Missing adoption proof and changed
endpoints reject recovery. |
| Compact storage | A 31 KiB attachment grant writes its separate part
once, leaves it unchanged across source snapshots and removes it at
queueing. Headers do not duplicate the grant. |
| Accounting | 21 real-Postgres usage tests and 19 source-binding tests
pass. Recovery hydrates immutable source receipts without resetting
their producer or billing classification; conflicting/incomplete usage
holds. |
| Admission and malformed evidence | Driver 16, staged evidence 42,
restart classifier 27 and source usage 19 focused cases pass. Four
malformed-file regressions pass at d6a7, whose executor is
byte-identical at 484, and server typecheck passes. Full executor file:
990 pass; 16 embedded-Postgres `initdb` setup failures before
assertions, retained separately. |
| Required checks | Exact fc68584 recursive typecheck and build pass
locally, with matching server stamp. Forced full CI succeeds (48 jobs
pass, one intended skip); all PR checks are clean. Maps/token gates pass
for their unchanged source surfaces. Fresh exact-head Greptile is 5/5,
all six threads resolved. |
| Ordinary application path | Exact fc68584 managed component passes
active recovery after natural expiry, completion, native-cache loss and
ordinary C continuation. Production bytes match before/after; temporary
fixtures restored and owned DB/services cleaned. UNKNOWN local
debug-daemon/partial instruction-observer limits remain; the separate
live case uses the attested Linux producer. |
| Producer | Locked Bookworm build from exact 5b7 Rust tree
d03e494217, equal to final fc68584 Rust
source. SHA-256
65b8f8cfdd6099e9ad24f1ca154e8a260777269cc82f6e23c6e6a180c531eaaf;
38,515,328 bytes. Actual Debian 12 execution and live admission pass.
Controller/admission/DB changes do not relabel the producer source. |
| Historical cap-two comparison | Exact 59da baseline vs 5b7 candidate,
300 controllers/10,000 history, ABBA 4×65 s per mode, all three actual
pools and real snapshots. Steady CPU/op 541.36→554.64 µs (+2.45%);
WAL/op 1512.13→1560.99 B (+3.23%); DB-request p99 102.78→105.99 ms
(+3.12%): PASS. Burst CPU/op +4.71%, WAL/op +5.29%, p99 −0.10% pass
relative gates, but absolute drains above fail. Identical operation/SQL
counts, zero lost leases/drops/errors. One Node process simulates 300
controllers; this does not qualify two-webserver overlap or HTTP
capacity. |
| Live acceptance | Strict final-source Linux classic Codex case passes
1/1 on attempt 1, all 49 checks. Active recovery within grace and useful
retained C continuation succeed; four
accepted/published/committed/settled outcomes. Independent provider,
PG/process, VM/root and SSH cleanup complete. Prepared/task and broader
rollout limits are explicit above. |

**Historical qualification snapshot through `9eb9ac6`.** The head at
this snapshot was `9eb9ac6f9e312f3ebe8040f5a7565e88ae8824f3`. Server and
Runner package sources are byte-identical to `283e05ffaa`; the two later
commits change the explicitly scoped qualification and its documentation
only. It includes master through the storage-owner fix
`45a3a92e6dc52d20923b9aa34a22fee2ec5443f9` (#18357). At that snapshot,
CI/review passed but the strict post-C native recovery gate failed; the
PR stayed draft and required another native repair. Prior `39a4343` CI
and review passed, but its full live workflow failed; that grade remains
unchanged. The first repaired `435ccd2` live attempt failed before any
injected outage. It exposed a legitimate terminal snapshot before result
acceptance; no accepted result ID or publication receipt was observed.
Its canonical report remains 0/1 `candidate_failure`, with cleanup
passed and independent provider absence confirmed. `0862062` defers
publication until result acceptance and fixes the fresh review's
coordinator/issue lock-order finding. Both failures have deterministic
real-Postgres regressions. The `0862062` live attempt then proved
accepted-result publication but failed active recovery: a resumed-lease
preparation guard demanded suspension after the exact live Runner was
already verified for adoption. `64c0b746` preserves that verified
active/idle-warm process, revalidates its proof/liveness, and keeps
ordinary suspension checks and later PRP authentication. Its live run
passed preparation but exposed a second active-warm boundary: all
durable binding fields matched except opaque turn/item attachment IDs,
which the server had regenerated. `7a203274` carries the complete
verified six-field binding and keeps the strict Runner mismatch guard.
Its cloud case then passed active adoption, completion, publication and
accounting. The queued continuation exposed accepted-result retirement
writes and an implicit-home fingerprint drift; both are fixed with
deterministic regressions in `39a4343` before this fresh full
qualification.

The prior live case exposed a real publication gap. A goal-less native
turn could persist its accepted result and terminal snapshot before
provider cleanup, while the task-session pointer was still empty. Normal
later admission then selected a fresh session. The fix publishes that
pointer and a private bound receipt in the **existing post-acceptance
terminal checkpoint transaction**, before cleanup. Earlier terminal
observations persist without a handled receipt or continuation pointer.
No result-acceptance or snapshot transaction was added. The late
completion path recognizes the receipt. This also prevents a late
callback from restoring a pointer after a user reset. Goal/control
publication keeps its existing owner.

The dedicated Daytona fixture now uses the existing warm/reusable
environment. Its sandbox, Runner and provider storage must survive. The
former generic per-turn fixture disposed the sandbox after completion
and kept provider-history backup bytes under the discarded directory.
Whole per-turn disk-loss acceptance therefore has a separate
provider-backup storage dependency. It remains distinct from the passing
native per-turn component checks with backup storage available. The
60-second lease, 300-second outage and 420-second grace are unchanged.

| Gate | Current repair evidence |
| --- | --- |
| Assembled regression | Baseline fails with an accepted result and
terminal snapshot but a NULL task pointer while real provider cleanup is
held. The fix passes normal continuation and reset cases, with no
pointer seeding. Exact merged `92b7447` passed 2/2 twice, including
after normal package TypeScript output refresh. Production files are
byte-identical at frozen `435ccd2`. |
| Persistence and concurrency | 19 helper/transaction cases and all 59
existing control-port cases pass on `0862062`. They cover bound result
IDs, reset/newer epochs, Stop, lock waits and expiry, receipt replay,
missing stubs, foreign bindings and atomic 4 KiB rollback. Existing
snapshot transaction count stays one. The new tests reproduce the exact
live missing-result-ID error and the inverse replay/status lock order
against `435ccd2`, then pass after the correction. Replayed results
still reject stale generations. |
| Completion | All 32 completion-boundary tests pass on merged
`92b7447`; the source is unchanged at `435ccd2`. Native result-shaped
session metadata is preserved. |
| Qualification fixture | A valid review report reproduced three
rejected correct replies. Exact token-boundary matching fixes them. 140
focused E2E/catalog cases and Product E2E typecheck pass. Wrong tokens,
marker prefixes/suffixes and wrong authors remain rejected;
retained-identity checks are unchanged. |
| Required checks | Full recursive typecheck and standard build passed
at `435ccd2`. Both exact commands also passed on `0862062`, with its
exact server build stamp; both also pass on `7a203274`, with its exact
server build stamp. Fresh `0862062` Greptile passed 5/5, with zero
unresolved threads. The `64c0b746` normal CI then passed all 49 jobs (48
success, one expected skip) and Greptile passed 5/5 with zero unresolved
findings. The `7a203274` normal CI also passed 48 jobs with one expected
skip, with fresh Greptile 5/5 and zero unresolved findings. Current
`39a4343` recursive typecheck and standard build both pass with
persisted exit codes and its exact build stamp; fresh Greptile is 5/5
with zero unresolved findings. Its normal CI completed successfully: 48
jobs passed and one expected job skipped. The fresh live result remains
separately failed. The `0862062` normal CI had 46 passing jobs, one
expected skip, one worker communication failure and its aggregate
failure; GitHub states the self-hosted runner lost communication. No
failed assertion or specific OOM/spot cause was established, and no
blind retry was used. |
| PostgreSQL supplement | One corrected all-pool pair at the preceding
repair passed 300-controller/10,000-history correctness and absolute
gates. SQL 76,590→75,990; transactions 9,013→8,713; UPDATE rows 11,613
each; 300 correct pointer writes each. PG CPU 31.99→30.46 s, WAL
70.36→67.87 MB, request p99 100.98→103.12 ms. This is one pair, not a
repeated improvement claim or an exact `0862062` CPU measurement. Its
two later changes defer premature publication and reorder existing
locks; they add no transaction, stream/idle writer, pool or timer. The
first collapsed-pool attempt remains invalid and retained. This is not
an exact final-head CPU/WAL measurement; later fixes change ordering,
identity selection and in-memory skill replacement without adding
snapshot transactions, pools, stream writes or idle SQL. |
| Live publication proof (`0862062`) | Baseline run
`06fcc806-f186-4190-9b57-087b61d75533` accepted result
`58f912dc-94d9-4b4d-bd3d-d5f689c83fd3`; the publication receipt names
the same result, the task pointer has parameters and `lastRunId`
matches, and the normalized native identity matches the following active
run. Accounting v2 settled. |
| Active recovery (`0862062`) | **FAIL.** SIGKILL/cache discard at
23:07:28.666 UTC; hold 300,000.859 ms; B ready at 308,896.486 ms;
natural generation-2 ownership at 312,906 ms. At about 319 s,
preparation failed `runner_harness_state_mismatch:
resumed_not_suspended`. No authenticated after-receipt passed within 420
s. Exact original Runner PID 668/start/fingerprint was independently
still alive; provider auto-stop was 15 minutes. No retained C follow-up
qualified. Canonical 0/1 `cleanup_failure`/cleanup-failed report remains
failed; subsequent independent cleanup deleted all three owned resources
with exact 404s and confirmed full intended-account inventory with zero
owned resources. |
| Verified adoption preparation (`64c0b746`) | 91 focused cases pass,
including actual preparation for four active providers and a distinct
prior/current warm proof. Unproved resumed state, dead process and stale
warm owner remain rejected. The earlier options-only fixture missed this
gate. No spawn, backup restore or PRP-authentication bypass was added.
Server typecheck passes. |
| Active warm identity (`64c0b746`) | **FAIL.** B claimed generation 2
at 312,995 ms. Preparation passed, then strict PRP recovery rejected the
generated attachment IDs at about 319 s. The saved and desired Runner,
lease, normalized session and run IDs matched; only
`turn_attachment_…`/`item_attachment_…` differed from generated
`turn-<run>`/`item-<run>`. Canonical 0/1 cleanup-failed report remains
failed; independent cleanup confirmed deletion/404s and zero owned
provider resources. |
| Six-field identity repair (`7a203274`) | Actual server-factory
regression RED4→GREEN38. A real Rust Runner test passes healthy first
run → random-ID warm handoff → held active second run → restored
controller. It verifies a new live authenticated connection, exact
identity, same Runner/provider OS identities, no duplicate launch/turn,
and no redispatch or invented receipt for an uncertain tool. The
synthetic wrong pair still fails the unchanged guard. This protocol case
does not claim business completion. Server and Runner typechecks pass. |
| Active recovery/completion (`7a203274`) | **PASS for the active
component.** Full five-minute SIGKILL/cache-loss hold. B ready at
308,936.651 ms; natural generation 2 observed 312,979 ms; authenticated
at 317,720 ms and resumed processing at 318,527 ms. Same run
`1e1ccb00-0728-4935-97da-57562d81cb42`, Runner
`246f73f7-a689-4177-a9c2-7ce711d63aed`, remote PID 669/start
23:48:27.470, provider lease/session and one submitted turn. Accepted
result `49097529-5310-4ac3-af95-5bf9b91d19f4`, publication, finalization
and accounting v2 all settled. |
| Full workflow (`7a203274`) | **FAIL, preserved.** Queued
late-requirement run `5e61f38d-c7d1-4fc1-bcbf-bce9d46ce4eb` failed
`runner did not durably suspend before checkpoint` before outage 2.
Canonical 0/1 candidate failure, cleanup passed, evidence valid. C
retention was not exercised. Independent provider absence,
controllers/PG cleanup, EC2 termination and root-volume absence all
confirmed; closed receipt SHA-256
`676a3b15eb9937b6bf995c36fb8666c88964d509e5ceb9109fb02da0aa2bc909`. |
| Retirement and stable configuration (`39a4343`) | Real Runner +
Postgres RED reproduced the exact suspension failure and nine owner
rejections. `turn.stop` is now lifecycle checkpoint maintenance under
unchanged owner/result/generation fences. GREEN25/25 proves actual
persisted suspension, exact identity and completed stop/drain/suspend;
unaccepted, foreign, stale and new-work negatives remain rejected. Two
default-home fingerprint REDs become GREEN9/9; explicit mode changes
still invalidate. No global matching exemption or historical fingerprint
rewrite. |
| Fresh full Daytona warm cell (`39a4343`) | **FAIL, preserved.**
Launched 00:15:29 UTC with automatic retries disabled. Active adoption
passed: actual offline hold 300,000.848 ms; B ready 309,922.155 ms;
natural generation 2 at 311,821 ms; authenticated 317,989 ms; resumed
330,440 ms. Same run `3e126819-e786-4b3b-b923-c975cf4f5b79`, Runner
`4079138a-8d7c-44a4-8558-07a5411d1f55`, PID 668/start and
provider/session identities. Accepted result
`08e504af-4dec-48a8-bf05-bf0a32280559` and publication passed. Queued
run `31d168c9-7ab9-4700-abb6-fafb90912237` failed the same strict
suspension error before outage 2. Its row selected a new native
identity, but no new third provider process or invocation was observed.
Recovery-time configuration still reset only `adapterConfig`; native
provider, context and workspace subtrees were unchanged. Canonical 0/1
candidate failure, evidence valid, normal cleanup passed; all campaign
provider/PG/process resources independently absent. Closed campaign
receipt SHA-256
`5e7d6281c13c812f3a946667f97f3c7d339b5ad86a3a819bd185b2fa73bc4a19`;
EC2/SSH remain temporarily held for the concrete follow-up repair.
Original 60/300/420-second limits and strict queued/second-replacement
checks remain unchanged. Direct-process path; no new live prepared-task
claim. |
| Native retirement authority (`283e05ffaa`) | Exact `39a4343` facade
regression RED3→GREEN9. The successor now uses its existing production
coordinator claim and prompt renewal before strict retirement. A real
Runner/Postgres case starts in the actual native-observed/unclaimed
phase, claims through production code, then closes and durably suspends
the predecessor in 1.657 s. All 25 real-Postgres cases pass;
unclaimed/foreign writers remain rejected. Shared attachment staging
stays after retirement; Stop and denied claims neither close nor stage.
No new claim or SQL path was added. |
| Configuration continuity (`283e05ffaa`) | Real shared/project-primary
environment selection → configuration with four skill versions
reproduces `reset: true`, changed category exactly `adapterConfig`,
after a Postgres JSONB read. Pinned restoration had reordered the
generated preference while actual runtime skills stayed identical.
Replacing existing pinned rows in place preserves the original fresh
name/key order and fingerprint. The assembled regression passes, as do
three existing pinned-version/rename/alias cases and server typecheck.
Genuine model/skill-selection changes still invalidate. No
fingerprint-format change or broad compatibility exemption. |
| Final production repair (`283e05ffaa`) | Full recursive
typecheck/build and [normal
CI](https://github.com/paperclipai/paperclip/actions/runs/41151145106)
PASS: 48 jobs plus one expected skip. Fresh review 5/5, zero unresolved
threads. Live baseline, recovered run and queued continuation all
succeeded, each accepted/published/committed once with accounting v2
settled. Their native/provider identities and configuration fingerprint
match; both recovery and continuation report no reset, no changed
categories and task-session reuse. Active outage/authentication remains
within original timing. |
| Browser-dependent continuation (`283e05ffaa`, definition 6) | **Whole
cell FAIL, preserved.** After B was killed at 00:46:38.928 and its cache
discarded, C became ready in 9,151.410 ms. `openParent()` then hit a
30-second `page.goto`/DOMContentLoaded timeout before posting the final
request. No fourth native run or C continuation was exercised. Cause of
navigation timeout is unknown. C task had no execution lock, queued
comments, handoff or recovery blocker; all three prior runs were
committed/accounted. Normal cleanup passed; independent provider
absence, controller/PG cleanup, EC2 termination and exact root-volume
absence confirmed. Sanitized report bundle SHA-256
`2a3b841d62efb2ae30ef14f96a431ac4fa7fe3a782f1144831512f78edc65022`. |
| Native API continuation qualification (`9eb9ac6f9e`, definition 7) |
**FAIL, preserved.** The real post-C API request dispatched run
`c71fbb18-f979-4fe4-93ce-9c4be46dbf21`, which failed
`native_session_interrupted` / `native_session_recovery_failed` at
01:10:00.640 UTC. It selected the same normalized native identity,
Runner and checkpoint-provider identity as the three succeeded turns. No
C result was accepted; pending accounting remains explicit. Active
recovery passed: actual offline hold 300,001.098 ms, B ready 308,335.019
ms, natural generation 2 observed 311,818 ms, authenticated 315,540 ms,
resumed 316,612 ms. Queued continuation succeeded/accepted/published. B
was then killed/cache discarded at 01:09:34.500 and C ready in 8,783.803
ms. Original classifier `transient_infrastructure`, whole-case failure
and cleanup passed are unchanged; that classifier does not establish the
root cause. All owned provider/PG/process resources, EC2, root volume
and SSH agent independently absent. Closed receipt SHA-256
`7f21ac4a22efc07b849fa7e17866083b6127b3321af12671b4fc054ce57039a9`;
sanitized bundle
`71ffef14a30a553e008320a31840a6d6c0de85c1bfdb60f783fdb53a6aae812a`.
Exact backend detail is unavailable in approved structured evidence. |
| Current required checks (`9eb9ac6f9e`) | Recursive typecheck, standard
build and exact build stamp PASS. [Normal
CI](https://github.com/paperclipai/paperclip/actions/runs/41152908417)
PASS: 48 jobs plus one expected skip. Fresh Greptile 5/5, latest edited
summary checked, zero unresolved threads. Exact8/512/8,192 stream probes
again report zero recovery SQL/transactions/rows and settled-idle SQL;
seven control transactions/twelve changed rows each, with 36 separate
legitimate renewals in the longest case. CI success does not supersede
the failed live C gate. |

The completion supplement observed secondary request/liveness/checkpoint
transaction p99 increases of 23.0%/45.7%/15.6% in its single pair.
Renewal p99 stayed below 0.865 s, with 900 successful renewals per arm
and no lost leases. The named pool caps stayed 10+2+2. An aggregate
activity sample peaked at 15 non-observer entries without backend-type
classification, so no claim is made that all were client connections. No
sampled lock episodes does not establish zero lock waits.

Twelve pure reader/selector cases also pass with the new adjacent
receipt. This is not another executed process rollback. Development
reader `5cf7aead` can parse the recovery state, but its old run-detail
redactor can expose receipt IDs/outcome to an authorized run reader. The
first production release containing recovery readers, complete
attachment-identity handling and both private-field redactions is the
rollback floor. The implicit-home fix prevents new recovery-time
fingerprint drift; it does not rewrite previously stored fingerprints
from failed candidates. The development probe is not a published
supported rollback target.

Latest actual `0862062` Runner tests independently passed 8/512/8,192
text chunks with zero recovery SQL, transactions and changed rows, plus
zero settled-idle SQL. All three observed seven setup/control
transactions and twelve changed rows. The 8,192-chunk case separately
recorded 31 legitimate lease-renewal statements, with no lost lease.
These are component SQL observations, not a production CPU/WAL claim.
The new adoption and identity branches affect recovery preparation only.
The `64c0b746`, `7a203274` and exact `39a4343` CI independently passed
all three stream sizes with the same zero-SQL/idle result; each latest
8,192-chunk case recorded 36 separate renewal statements. Exact
`39a4343` Runner 2/2 passed 133 files / 2,317 tests plus two skips; its
separate native group passed 150 tests plus eight skips.

The first API fixture revision omitted the mandatory final screenshot;
this was caught before paid launch and allocated no provider/model work.
The correction captures the actual retained Chromium page with an
explicit unqualified-browser label, without changing the standard report
validator.

Automatic approval review rejected exporting raw private API/workflow
snapshots because of sensitive payloads and insufficient
destination-specific authorization. Sanitized reports and
bound/hash-only receipts are retained; raw snapshots were not copied
through another channel. This does not change the failed grade.

Repair characterization also preserves its failed attempts. An
unconditional staging relocation broke three existing same-scope
scripted cases; the final implementation defers staging only for an
actual superseded warm scope, and all nine order/race cases pass. One
baseline transform overlapped an edit and is explicitly invalid; the
actual immutable `39a4343` loader then produced the three genuine REDs.
The independent task-directory hypothesis produced a RED but was
excluded by the live shared-workspace evidence; no directory policy
change was landed. A shared-workspace fixture authorization omission was
corrected separately from the four-skill semantic RED.

Setup failures are retained separately from semantic outcomes. These
include an old host port after the owned Postgres fixture restarted and
a sandbox-denied TSX IPC pipe. The supported external-Postgres run
passed, and the final exact recursive typecheck passes with normal
process permissions. Seven stale declaration errors after the master
merge cleared after compiling the actual merged SDK/Runner TypeScript
sources. No declarations were edited by hand.

<details>
<summary>Earlier qualification measurements and failed attempts
(historical, before this repair)</summary>

**Historical blocked gate (before the repair below).** Final native
implementation is `abed921e12d543b62f57a56435419c0dafb0bc33`; current
branch head is `af9e2d157dcc349e55374ad186c9be2d6cb96bf9`. The two later
commits change only qualification fixtures, their matcher and
documentation; `server/` and `packages/` are unchanged. It includes
master through `fd008979f5794fda3d5f54151d36acd31d0de140`.

The live Daytona measurement proves five-minute active recovery,
once-only completion and accounting, but **fails retained-session
identity after the second replacement**. The saved checkpoint and
recovery metadata exist. The task-session pointer remains empty, and
ordinary admission creates new native/provider sessions. The exact
skipped publication boundary remains unproven. A useful saved-file
result alone does not satisfy retained-session continuation. This
application-path gap blocks merge independently of CI.

A separate new-master CI failure is owned by the storage workstream (fix
in #18357). No storage guard, session-identity oracle, lease/grace limit
or provider result was relaxed.

| Gate | Observed result and exact scope |
| --- | --- |
| Distinct source recovery and compatible rollback | PASS at A
`5cf7aead09a2d9c7e2f2929d761f535ef3c1d798` → B
`5ea45f8d17e33c9a62bd79982dcf0375b07733d0` → fresh B → A. Four
controller processes, natural 60-second lease expiry, absent native
projections before replacements, three succeeded/committed unique
results, one provider thread, unchanged Runner identity, and strict idle
cleanup. 177.782 seconds. |
| Immutable installed native component | PASS at installed A
`5cf7aead09a2d9c7e2f2929d761f535ef3c1d798` → installed B
`eaef0d56552c087aaa4fa68a8655282ffbd12667` → fresh B → A. Each
controller imports its contained compiled server and DB. Three unique
committed results, stable Runner/thread, no extra provider restart on
same-contract rollback, strict cleanup. 106.672 seconds. This is
compiled component proof, separate from full CLI-native application
acceptance. |
| Normal public CLI startup | PASS for B
`eaef0d56552c087aaa4fa68a8655282ffbd12667`, Node 24.22.1, darwin-arm64:
fresh external DB, 415 migrations, health ready, root HTML 200, company
POST 201 plus GET readback. The installed tree remained unchanged after
execution. |
| Five-minute component outage | Actual controller absence 300,168 ms.
Natural generation-2 claim at 407.196 seconds after crash, same provider
attempt and Runner, original run succeeded/committed. The larger
540-second case timed out before its retained follow-ups, so it is an
active-adoption pass, not a complete four-controller pass. The complete
short source/installed cases above independently cover follow-up and
rollback. |
| Drain/resume review regression | Fixed in
`226f03e736d1ee99fea1762ab32ea3a8a9241c20`. Five facade-level cases
pass: overdue normal/restart waits, future timer cancellation, another
facade resuming the original owner, and transient retry. Against exact
prior scheduler `5ea45f8d17e33c9a62bd79982dcf0375b07733d0`, four fail
and one passes. The later head changes documentation only. |
| Final corrected-fixture Daytona | **PARTIAL: active recovery PASS;
retained identity FAIL**, exact live source `0b6afc499f`. A was
SIGKILLed and native cache discarded; actual offline hold 300,000.375
ms. B was ready at 308,326.961 ms, naturally claimed generation 2 at
312,059 ms, authenticated at 315,996 ms and resumed processing at
316,519 ms, within the original 420-second boundary. The same run,
Runner PID/start, session, lease and event prefix survived; one turn
submitted. Original execution succeeded and settled accounting. After
completion, B was killed and its native cache discarded; C was ready in
9,077.779 ms. C executed the saved project and returned correct
`queue_r`, with all four runs succeeded/accounted. However, native and
true provider-session identities changed: the unchanged retained oracle
fails. No whole-workflow pass is claimed. |
| Prior matched-producer Daytona | FAILED at exact `abed921e12`. The
matching current-source Runner passed normal admission, completed the
first task and settled its accounting. During the active revision, A was
SIGKILLed and its native cache discarded. Actual offline hold was
300,000.727 ms; B was ready at 308,304 ms and naturally claimed
generation 2 at 309,651 ms. At about 318 seconds, the remote process
check reported `process_not_running` for the original admitted Runner.
No authenticated after-receipt or retained follow-up passed. The current
failure is preserved; no automatic retry, lease/grace relaxation, debt
edit or provider replay is used. Bounded forensic read found preserved
exact-bound state with no recoverable failure, a transport-reset
diagnostic and no terminal delivery; this supports external process loss
without proving the exact cause. The sandbox had a five-minute auto-stop
policy, conflicting with the required surviving-sandbox assumption.
Owned cleanup precedes the separate corrected-fixture case. |

**Live retained-session boundary.** Prior completed run `3806…` and C
follow-up `8adf1df3-6744-40bf-b35d-100451f29d9f` both have database
session snapshots and `nativeControllerRecovery`. Their tool
fingerprint, provider configuration, workspace descriptor,
runtime-context digest, driver and work mode match. There is no recorded
force-fresh/workspace-selection change, and this is an ordinary task
rather than a conversation-generation reset. The same issue/agent
task-session row has `session_params_json: null` and `last_run_id: null`
from baseline time. Both native normalized UUID and actual provider
session/thread hashes differ after C. Full application admission
therefore did not select the retained source. No fallback to file
execution or relaxed identity comparison is accepted as proof of
continuity.

**Independent cleanup completed.** Full current-account provider
inventory confirms no resources owned by either final fixture, with
exact-ID absence checks. Owned controllers and campaign Postgres are
stopped/removed. EC2 `i-01551c09af098dde5` is terminated, its
delete-on-termination root volume returned `InvalidVolume.NotFound` at
22:14:08 UTC, and the temporary SSH key/agent were removed. The
canonical failed report and immutable oracle input are retained. Closed
qualification receipt SHA-256:
`b4b5fefce287ebfdbf2cc2238efaad62d8ad485aed0e9c0628ae95dbc18a8575`.

**Original measurement and grader repair remain separate.** The C reply
contains the requested marker followed by a newline and the correct
output. The old workflow waited for one literal space, so its normal
receipt collection stalled; that original attempt was closed and its
canonical `cleanup_failure`/interrupted result is preserved. Commit
`af9e2d157d` repairs only this reply predicate, with 144 focused cases,
E2E typecheck, discovery and diff checks passing. Ordinary string
matching stays literal. The corrected predicate accepts the saved reply,
but the unchanged retained-identity oracle still fails. Pure
active-recovery grading of the captured measurement passes 7/7; this is
not a rewritten passing campaign.

The sandbox idle-window conflict was corrected only for this explicit
fixture (15 minutes). Actual baseline and later sandbox metadata confirm
that policy; the original active box was disposed before a post-gate
policy read, so its 15-minute setting is established by the normal
create/environment configuration rather than an after-read of that box.
Production native lifecycle and the 60/300/420-second constants remain
unchanged.

The development compatibility reader A is a genuine backport commit
based on `59ff89a18fe1312b5f0bb908439bbdb1237b0252`, with the required
idle-goal reader fixes. It is not a previously published release. Its
older executor differs from B. The rollback floor must include this
reader capability. The source component cases admitted the Linux ARM64
Runner packaged at `59ff89a18fe1312b5f0bb908439bbdb1237b0252`; the
installed component case admitted the `5cf7aead` package. Their Runner
bytes are identical: SHA-256
`b1f04caa1ce04e720dec7eb62ede170eb3621948c0849b8f305ea5c5116bced8`. Each
admitted artifact remained unchanged throughout its case.

[Final public package
build](https://github.com/paperclipai/paperclip/actions/runs/41121460635)
and [reader-floor
build](https://github.com/paperclipai/paperclip/actions/runs/41122170506)
both passed all four daemon builds, package assembly and both Darwin
clean consumers. Both Linux ARM consumers failed before provider work
with the same existing local Codex permission-profile error: bubblewrap
attempts `activity.fifo/.git` and reports “Not a directory.” This
matched baseline failure remains open; it is not a native remote
continuity pass. Full post-run tree and build-stamp revalidation also
passed after the compiled native case. The installed component trees are
A `c29114c1bf71c5af00f761aeb9e3d84ed3d3075b9f5600dee3ef2543eb2b6ec5` and
B `eda575505373a3bdc630fa4919862b359791fe3a7ef637961a25b73b38efe4b0`.

**Controlled PostgreSQL qualification passed.**
Baseline/candidate/candidate/baseline workloads used 300 controllers,
10,000 historical tasks, real snapshot/encrypted PRP writes and
unchanged pools. PostgreSQL and client were each limited to two CPUs and
1536 MiB on an isolated eight-vCPU Linux host.

| Workload | PG CPU per operation | WAL per operation | Database request
p99 |
| --- | ---: | ---: | ---: |
| Steady | +1.18% | +1.81% | +1.91% |
| Aligned burst | +3.40% | +2.56% | +3.81% |
| Two logical servers | +3.11% | +2.56% | +9.68% |

All arms passed absolute/correctness gates with unchanged control
transaction counts, no dropped operations and no owner loss. Overlap p99
has little margin below the relative threshold. It uses two clients in
one Node process, not two HTTP servers. [Closed measurements and
limitations](doc/plans/2026-10-30-native-controller-qualification-evidence.json)
and exact-source reproduction patches are retained. [Earlier
contended-host
failures](doc/plans/2026-10-30-native-controller-load-evidence.json)
remain unchanged. The production DB-claim probe claimed 300 owners in
2.84 seconds and restored/committed controls and snapshots in 8.74
seconds, with zero owner loss or settled-idle SQL; its provider
references were synthetic. The measured checkpoint/snapshot/renewal
paths retain the same persistence behavior after integration; later
accounting and business writes have separate owners.

Actual Runner stream tests at `da8c7f6e1837ddd896dfe83a7524812fee813487`
passed 8/512/8,192 chunks: zero recovery SQL, transactions and changed
rows during text intervals; zero settled-idle SQL. Each retained seven
control transactions and twelve changed records. Existing renewal SQL
was measured separately. The guard/scheduler fixes add no stream path,
pool or recurring idle scan.

The live fixture used current-source producer SHA-256
`f0000c90cb83880191479da639201e659aa6ce0fc7d67c859faa172ec977849e`, a
normal locked exact-head build with pinned Rust 1.97.1 in Bookworm.
Actual network-isolated, read-only ABI execution passed (GLIBC
requirement 2.34 against runtime 2.36). The immutable base image
remained
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:36069788f53c51f228764fa36180575443dfb5d1f84a342d7c0daceb28dc61cc`,
source `a0ef3ce7`. This was an unpublished development producer,
explicitly distinct from the base image and admitted before the fresh
fixture began. The earlier host build failed ABI preflight before
admission. The preceding old-producer fixture completed its initial task
but lacked the usage-basis marker required by current master; it
remained pending/unpriced and correctly paused. It was independently
cleaned without debt or budget changes. Two preceding local Linux setup
attempts OOM-killed before provider allocation, and their cleanup
passed. These attempts remain separate from the final recovery failure.

**Failures retained and resolved separately.** The earlier `98f05fc`
probe rejected a valid settled PRP v2 goal suffix; `eaef0d5` fixed that
reader and the fresh follow-up passed. Later component cleanup failures
were traced by a scoped exception observer to the fixture adapter
rejecting directory downloads. A bounded owned-directory transfer with
exact-byte/mode preflight corrected the fixture; source and installed
strict-cleanup cases then passed. No production guard was relaxed. The
initial local integrated control-port run hit its 30-second embedded
Postgres startup hook before executing 59 cases; 18 other focused cases
passed. A bounded rerun used a fresh owned external Postgres fixture,
applied normal migrations before Vitest, and passed all 59 cases with
the original timeouts unchanged. Its container and data were removed;
the failed attempt remains retained.

The additional opt-in full Product E2E Vitest run is not green: 5,313
passed, seven failed and three skipped. An actual `abed921e12` archive
with the same dependencies, Node and IPC permissions reproduces all
seven failures with the same reasons. Five are unchanged
workflow-policy/profile-pin assertions, including Pi profile version 37
versus the test pin 35. Two database suites failed during embedded
Postgres startup before their assertions; semaphore exhaustion is
confirmed in the observer log. The observer passes its supported
external-Postgres rerun with original timeouts and source unchanged. The
controller-database client-kill/postmaster assertions remain unexecuted
after `initdb` bootstrap failure; no external override is supported. The
unit launcher stopped before its Node-test phase, which was then run
independently and passed 181/181. The original failed full run is
preserved. These results remain distinct from the passing focused checks
and normal repository CI. The first local Product E2E typecheck also
encountered stale compiled Runner declarations; a genuine package-local
TypeScript compile refreshed them and the unchanged check passed,
without a dependency install or producer rebuild.

Earlier [Daytona campaign
41108162900](https://github.com/paperclipai/paperclip/actions/runs/41108162900)
failed at sandbox creation before any agent, with the underlying
provider error unavailable. Its [sanitized
report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-41108162900-2/index.html)
remains a `cleanup_failure`. Both old creation names are absent in the
current authenticated account, but the old CI account identity was not
retained, so account equivalence is unproven. The subsequent exact
`eaef0d5` source Product E2E attempt passed image/agent setup and the
initial project, then held the controller offline for 300,003.7 ms. Its
Mac supervisor failed owned-process inspection before B readiness; no
adoption passed. Full current-company lease/label checks, provider
deletion, exact-ID 404s and complete inventory confirmed cleanup of that
attempt. Its local server and owned Postgres stopped. The prior AWS
campaigns and database qualification VM were terminated with root
volumes removed.

The old admitted test Runner lacks the newer usage-basis marker
introduced on master. The integrated component B turns retained
unpriced, incomplete accounting obligations with `costAccountingPending:
true` and no settled charge. The compatibility-reader A turn used its
legacy receipt interpretation but likewise retained pending accounting
and no settlement timestamp. This proves obligation preservation, not a
whole-run accounting migration. Runner/receiver release ordering remains
governed by the usage owner's contract.

At native implementation `abed921e12d543b62f57a56435419c0dafb0bc33`,
Greptile was 5/5 with zero unresolved threads and CI completed with 54
successful checks and five expected skips. At `0b6afc499f`, Greptile
again passed 5/5 with zero unresolved threads, but CI ended with 51
successful checks, five skips and three failures. Server 9/12 fails a
deterministic managed-skills maintenance assertion newly inherited from
master `198218bf` in merge `c80e00ad`; that test is absent from the PR
head, and the storage owner has the explicit-StorageService fix in
#18357. Runner 2/2 passed 2,283 unit cases but its runner received a
shutdown signal during the long stream probe; its cancelled step and
aggregate failure are preserved. No blind retry was used. Current
`af9e2d157d` checks are pending; the matcher-only change passes 144
focused cases and Product E2E typecheck. Earlier scheduling-sensitive CI
failures and the interrupted unsharded local test run remain recorded in
the dated plan. CI runs the repository's normal test partitions. Final
verification does not claim Railway rollout, uninterrupted HTTP service
or whole-webserver filesystem loss.

</details>

## Risks

- The task-pointer repair passes its assembled regression. The first
live repair exposed pre-acceptance ordering and the fresh review exposed
a replay lock-order issue; both are fixed and pass real Postgres tests
at `0862062`. The following live attempt exposed the
verified-running-versus-suspended preparation gate. Its narrow fix
passes 91 focused cases at `64c0b746`. The next live case exposed
synthetic attachment IDs; `7a203274` fixes that field selection and
passes real Runner authentication. The subsequent queued continuation
exposed strict retirement maintenance and a false default-home change;
`39a4343` fixes both with real Runner/Postgres and fingerprint
regressions. The `39a4343` live continuation failed again despite green
CI. `283e05ffaa` repairs the actual native-observed admission ordering
and reproduces/fixes the JSONB skill-order fingerprint drift. Both pass
targeted real-Postgres regressions and the live active plus queued
paths. The definition 6 post-C browser navigation timed out before
admission. Explicit definition 7 now verifies native continuation
through the ordinary comment API; its strict post-C native gate failed
after a real dispatch and remains historical failed evidence. The final
fc68584 repair now passes its fresh strict 49-check Linux/Daytona
campaign; the prior failed grades remain unchanged. Browser reconnection
is handed to its own workstream with the original failed evidence. No
prior failed campaign is reclassified as passing.
- Incorrect checkpoint/source binding or stale ownership could duplicate
external work. Missing evidence remains a hold; existing receipt and
finalization owners keep uncertain effects unresolved.
- Recovery must retain maintenance writes after accepted results without
allowing a stale business turn to write. Real lock-wait and generation
tests cover that boundary.
- The current two-connection paired burst profile fails its absolute
drain gate in both baseline and candidate, despite passing relative load
gates and all correctness checks. A causal capacity diagnostic selected
three slots; the final controlled cap-three profiles pass unchanged
thresholds. Full application replacement and two-server overlap capacity
remain separate. There is no new streaming writer, pool or recurring
idle scan.
- The remote sandbox and Runner must survive the recovery window.
Provider auto-stop is a separate lifetime bound from reconnect grace;
the explicit qualification fixture uses a longer idle window.
- Workspace and provider backup availability remain external
dependencies. This PR does not certify whole-webserver filesystem loss,
uninterrupted HTTP service or a Railway rollout.

## Model Used

Implementation and initial verification used OpenAI GPT-6 through Codex;
the initial hosted snapshot was not exposed. The follow-up qualification
used three OpenAI `gpt-6.1-sol` workers coordinated through Codex, with
repository tools, code execution, CI and scoped test infrastructure.
Context-window size is not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused and exact-source
application/cost checks pass; full forced CI passes; earlier local
embedded-PG setup failures remain recorded)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green (exact fc68584 full CI and
auxiliary checks pass; forced complete test partitions)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(exact fc68584 check and edited summary pass; all six threads resolved)
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-31 07:33:52 -05:00
DottaandPaperclip 8ff812cdde feat(chat): deliver proactive task messages to agent chats (#18374)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agent Chat gives each person a persistent conversation with each
agent.
> - Tasks and routines produce results outside that conversation.
> - Those runs need a safe way to bring a requested result back to the
person.
> - The server must select the agent's own chat and check the source
task's access rules.
> - This pull request adds proactive chat messages and unread
indicators.
> - People can see the result, open its source task, and reply in the
same chat.

## Linked Issues or Issue Description

**Subsystem affected**

Server, native runner tools, shared contracts, database, and UI.

**Problem or motivation**

An agent can report progress on a task or comment on an authorized task.
It cannot send a routine result to its own standing chat with the
responsible person. A morning review routine therefore cannot leave a
visible digest in that person's chat.

**Proposed solution**

Add `send_chat_message({body, idempotencyKey})` and `POST
/api/agents/me/chat/messages`. The server selects the company, agent,
and responsible person from the active run. It saves an attributed
comment and audit record in one transaction. The chat shows a
source-task link and per-person unread state.

**Alternatives considered**

A destination parameter on a generic comment tool would expose
unnecessary routing authority. A new delivery worker would add another
durable delivery path. This change uses a dedicated tool and the
existing comment and live-event paths.

**Roadmap alignment**

This extends the experimental Agent Chat capability in `ROADMAP.md`. The
existing `enableAgentChat` setting controls it. Related public work:
Refs #13282, Refs #13283, and Refs #14408. Searches for proactive chat
and unread chat found no duplicate PR.

## What Changed

- Add a focused messaging service shared by the native tool and HTTP
route. It checks current run ownership, execution mode, recipient
membership, task access, and task privacy.
- Reuse the canonical company/agent/person conversation. Save the
comment and audit record together. Keep task completion and chat
execution separate from delivery.
- Add source-task provenance and source-scoped retry keys. Identical
retries return the saved receipt. Changed content returns a conflict.
- Enforce the conversation's fixed agent at the shared comment-write
boundary. Keep normal reply cancellation and session checks.
- Add chat unread state, accessible agent dots, aggregate Chat
indicators, and visible-message read cutoffs. Refresh through existing
live events.
- Add the request-key constraint, native tool contracts, documentation,
capability maps, stories, and regression coverage.

## Verification

- Local delivery/security proof: all 38 proactive-message cases pass.
Eight credential-scope regressions and four quota regressions fail
against the prior source. The repaired delivery and canonical
cross-issue cap suites pass all 49 cases; all 31 OpenAPI tests, server
typecheck, and capability-map checks pass. The shared per-run cap covers
native and HTTP sends, mixed ordinary mutations, concurrent last-slot
contention, and retry/recovery without double charging.
- Credential tests preserve the authenticated person and authorizer;
reject bridge/test credentials and persisted skill-test runs after
task-mode changes; and recheck API-key revocation, scope, and
responsible person after destination locking. The earlier
delivery/native-authority suites passed all 74 cases before the four
quota additions.
- Four PostgreSQL contention cases cover native and HTTP sends plus
retries during a native chat mutation. All four deadlock against the
earlier source and pass with destination-before-source/run locking.
Recovery and human-reply delivery enqueue only the human comment; a
proactive digest never wakes the chat agent.
- Passed 108 companion conversation, native-authority, and OpenAPI
tests; 112 runner catalog/workflow-report tests; 47 completion-delivery
tests; and 25 project-handoff tests. Focused runner tests cover native
tool availability without generic API tools.
- All 327 affected UI tests, UI typecheck, token gates, and catalog
checks pass. The read hook retries at most three times, respects server
retry delay, coalesces only seen cutoffs, recovers on reconnect, and
cancels on scope changes.
- Real browser journeys with a deterministic process agent verify
routine-to-chat delivery, first-chat creation, unread state across
reload, the source link, secondary chat navigation, and a follow-up
reply referring to the digest. This does not qualify live GitHub
retrieval or model output quality.
- The Storybook showcase uses production components with deterministic
network fixtures. Rendered interactions pass for the landing, ordinary
read success, transient read recovery, and persistent failure after
bounded retries. Both Chat navigation variants update from two unread
chats to one while only Kodi's dot clears. Mobile and light variants
were reviewed. The existing Coordinator Recents label contrast finding
is unchanged; no Linux visual baseline archive exists.
- Current master's independent sandbox migration retains 0418.
Regenerated the feature migration as 0419 from the combined schema and
retained exactly the newest five snapshots; numbering/safety, replay,
and schema-drift checks pass.
- Workspace typecheck and build passed before PR preparation; canonical
server/UI typechecks passed again on the combined tree. Migration
`0419_slippery_impossible_man.sql` passes safety and schema drift checks
on disposable PostgreSQL, including interrupted index repair, concurrent
migrators, one-time owner read-state backfill, preserved receipts, and
two SQL replays. A default five-second local snapshot comparison timed
out; a diagnostic 30-second run passed without drift.
- Earlier failures are retained in the record: full local `pnpm
test:run` and broad shards encountered embedded-PostgreSQL startup,
timing, and provider-recovery failures. Thirteen selected
external-channel cases pass in isolation on the original and changed
sources; an unordered comment-row assertion now explicitly orders both
reads. The earlier route registry, native tool count, runner catalog,
and accessible Chat badge-name failures were fixed and passed full
remote CI on `920b3bb7db`.
- Repeated managed-skill server8 failures came from CI's newer synthetic
merge: its Cloud guard correctly rejected the old local-disk fixture.
Master independently repaired the fixture in #18385; this branch keeps
that repair unchanged. All 57 managed-skill and Cloud prerequisite cases
pass on the combined tree, and that fixture has no remaining PR diff.
- Current reviewed head `dc16f314739a6d977bed21cb1446d0e80a541578`:
Greptile 5/5 with no findings, Superagent security assessment passed,
advisory Paperclip Review 5/5 with no findings, and zero open review
threads. All 55 current-head check runs pass, with five expected skips
and the advisory review neutral at 5/5; Snyk also passes. No checks are
pending or failing. Protected squash auto-merge is enabled and is
waiting for required code-owner approval. A normal exact-head merge was
refused by the base-branch policy; no admin bypass was used. The
previous head's Superagent quota finding is fixed and covered above.

## Risks

- Chat remains company-visible. Delivery rejects a source task with
private-task, project, or ancestor restrictions.
- Read state uses a comment timestamp cutoff. The server retains
database precision and never moves a read receipt backward.
- The new unique constraint changes comment query planning. An existing
regression now uses an explicit row order. The migration builds the
unique index concurrently, then takes a brief table lock to attach it.
It uses the existing top-level migration and interrupted-index recovery
path.
- The upgrade backfills missing chat-owner read-state rows at the latest
existing agent comment. Existing read receipts remain intact. New chats
can still be unread before the human participates.
- Quota observation commits atomically with delivery. A rejected send
rolls back its attempted denial audit row and any first-chat creation;
the canonical warning remains in server logs, and previously spent quota
remains enforced. Identical/conflicting retries and receipt recovery
spend no extra slot.
- This version delivers in-app only. It adds no external-channel, push,
or email delivery.
- Local full-suite acceptance remains limited by the failures recorded
above. The feature stays behind `enableAgentChat`.

## Model Used

OpenAI Codex. PR preparation and CI work used `gpt-6.1-sol` with tool
use and code execution. The earlier implementation also used Codex, but
its exact model ID was not recorded in the visible session. The
context-window size was not provided.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run focused tests locally and they pass; the broad local
failures are recorded above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-30 21:39:07 -05:00
DottaandPaperclip 2a6d4d1694 perf(db): index canonical task activity lookups (#18043)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/177/changes/paperclip/177
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip helps people manage AI agents and their tasks.
> - Task lists rank tasks by their latest canonical activity.
> - The existing lookup reads every event for each task before
pagination.
> - Query-only alternatives failed a narrow visibility control or the
performance goal.
> - This PR adds an index that lets the unchanged lookup stop at the
newest qualifying event.
> - The result is less history work with the same permissions, ordering,
and response.

## Linked Issues or Issue Description

Refs: #13803. That broader proposal stores a new activity field with
triggers and a backfill. This index was authored independently and uses
the existing query.

**Subsystem affected**

Task-list activity ranking and the database schema.

**Current behavior**

The list computes `MAX(activity_log.created_at)` for every eligible
task. The existing entity index reads 50 events per task across 1,199
candidates in the representative fixture.

**Proposed behavior**

Add `(company_id, entity_type, entity_id, created_at)`. Keep the
production task service and permission helpers unchanged. PostgreSQL
reads backward until an event passes the existing exclusions.

**Reason and benefit**

Fresh paired measurements on the current base improve the
full-description scoped SQL from **44.60 to 20.33 ms (2.19×)**. Activity
buffer accesses fall from **63,547 to 4,796**, and history rows per
probe fall from 50 to one. The complete service median falls from 50.63
to 31.31 ms. Complete positional SQL results and service responses
match.

## What Changed

- Add generated migration `0414_closed_edwin_jarvis.sql` and the
matching Drizzle index.
- Use `CREATE INDEX CONCURRENTLY IF NOT EXISTS` through the existing
top-level runner and invalid-index retry cleanup. The client change is
one whitelist entry.
- Preserve all upstream SQL and journal entries; retain snapshots
0410–0414.
- Cover full and index projections, canonical ordering, invalid-index
recovery, concurrent migrators, and incomplete fixture rejection.
- Repair benchmark seeding, apply requested work memory to both
connections, and record both settings. Hosted defaults now include six
Slack conversations, 642 GitHub conversations, and 53 removed
conversations.

## Verification

- **All final-head CI gates passed**, including full workspace
typechecks, tests, build, Runner checks, and all eight E2E shards.
Greptile is **5/5** on `67ca7ae1cf15fc054ef515ffd8205b1fe30a0d55`; no
unresolved review threads remain.
- Local database/server typechecks and migration numbering/safety checks
pass. Three server regression tests and three migration regression
suites pass on real PostgreSQL.
- The original pre-index work guard fails at eight rows per probe; the
indexed guard now checks both the full-description query and index
projection. The service oracle covers cleanup/inbox exclusions, raw text
IDs, company scope, comments, ties, sort directions, visibility before
pagination, empty lists, and one child per parent.
- The documented pre-index seed command reproduced the review failure
before the fix and now succeeds. A paired run with non-default memory
confirms both independent connections use `16MB`.
- Fresh proof pins pre-index base
`929e840595c8933ca485ac7a2e99413e1e11ca64`, with identical cloned data,
unchanged generated SQL and parameters, two warmups, and five
alternating samples per arm. All **18 comparison cells** preserve every
positional row and complete service response. The 15-case default-memory
matrix passes its physical-work, speed, and control bounds.
- The one-visible-task control is 6.64 → 7.25 ms SQL; the 20-task
control is 3.71 → 3.72 ms. Both pass the declared 25% or 2 ms budget.
Full-query memory controls use matching `4MB` or `16MB` settings.
- Committed 1,000-row write batches measured 40.09 → 43.17 ms median
(+7.7%). Ranges overlap (34.67–71.85 and 34.27–48.05 ms); this is an
observed local cost, not a precise throughput guarantee.
- The final-numbered migration builds a valid index through the normal
runner on the fresh 300,000-row fixture in 0.55 s. Earlier large-fixture
proof of the same index SQL used 5,000 tasks and 1.25M activity rows:
2.58 s first apply, concurrent write/read probes 5.16/2.43 ms,
interrupted invalid-index cleanup, successful retry, and one receipt.
Current migration recovery tests verify the renumbered path.
- [The proof
document](https://github.com/paperclipai/paperclip/blob/67ca7ae1cf15fc054ef515ffd8205b1fe30a0d55/docs/performance/task-list-activity-index.md)
retains the historical baseline, rejected rewrites, and the earlier
matrix invalidated by severe host contention. Those historical timings
are not substituted for the fresh measurements above.

## Risks

The migration is an exception after measured query-only alternatives
failed. The index adds storage and work to every activity write: about
30 MiB for 300,000 rows and 123 MiB for 1.25M rows. Concurrent creation
avoids the writer blocking observed with an ordinary index build, but
long transactions can delay it. Local timing does not establish Neon
build duration.

The query still ranks all eligible tasks, and excluded newer events
still need to be skipped. These are local SQL/service measurements;
deployed HTTP latency remains unverified. No hosted DDL or deployment
was performed.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, code execution, and tool
use. The environment does not expose the exact model identifier or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue using
the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal issue
id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-29 09:06:41 -05:00
79ce86630a fix(server): schedule governed review response recovery (#17702)
## Thinking Path

> - Paperclip manages agents, tasks and their external chat responses.
> - A completed chat turn can owe a response while an earlier review
gate remains pending.
> - Periodic recovery revisits that response history even when no work
can advance.
> - This change uses the existing delivery coordinator and saved retry
deadlines.
> - Approval, interaction and authority changes wake the affected issue
or run after commit.
> - The existing presentation owner verifies the durable result and
current authority.
> - Settled and dormant history becomes quiet without losing future
eligible responses.

## Linked Issues or Issue Description

Refs #17696 (merged) and #15625. This is slice 49, now based on master.
It owns governed-review response presentation. Accepted responses and
pending decision effects keep their existing owners.

**What happened?**

Native finalization recovery repeatedly inspects committed review
responses. Removing the scan without covering approval, interaction,
task-policy and authority changes would strand responses that become
eligible later.

**Expected behavior**

Recover an owed response once from existing durable facts. Enforce
authority at publication, wake after relevant outer transactions commit,
retain real retry deadlines and restart/drain recovery, and stop
recurring SQL when no response can advance.

**Steps to reproduce**

Complete an external chat turn while an earlier completion review
remains pending. Block its response behind an approval or interaction,
then resolve that blocker. The response must appear once without a
periodic scan or resolving the original review gate.

## What Changed

- Add a scoped governed-review worker using the existing coordinator,
finalizations, decisions and comment receipts. No queue table or
transaction wrapper is added.
- Keep recovery and publication policy in the native finalizer. Preserve
binding, source/session authority, ownership, deadline and
deleted-comment fences; match coordinator-before-issue lock ordering.
- Notify through actual approval, interaction, OAuth, task-policy and
lifecycle producers before their owned writes. Serialize approval
resolution with task linking so a concurrent link retains its wake.
Nested transactions still settle at outer commit.
- Scope ordinary milestones, finalization and coalesced input to their
issue/run. Use the explicit company authority boundary for genuine
global restoration, including membership changes. Both response workers
share this boundary.
- Register under maintenance and drain gates, disable the two periodic
repair entrances, and retain fresh finalization and explicit recovery.
Add the concurrent 0413 comment-receipt index and migration
registration; retain five snapshots.
- Preserve the producer inventory, source qualifications and earlier
failures in the owner audit. Retain a real Photon regression proving
routine ownership assertions stay quiet and concurrent authority
restoration wakes each worker once.

## Verification

- Scoped repair source `2341714720b5c273f0ce011b48345af0ef39227f` passes
**2053 distinct cases**: 1703 in 15 complete producer/authority/recovery
files, including all 273 governed-response cases and 95 wake-queue
adapter cases, plus 350 selected chat cases. Another 1463 chat cases are
outside that selection. Thirteen fresh PostgreSQL 17 fixtures migrate
through unchanged 0413; the one temporary queue-fixture import restores
exactly. All sixteen captured hashes match.
- Two actual repeated coarse-hint SQL regressions fail before repair and
pass afterward. Actual membership restoration passes before and after.
The same assertions and time limits prove dormant work stays quiet and
real authority restoration still publishes once. Real coalesced input
covers both queues after commit.
- Combined source `586f5674cd9798b33cd22437255e0480b5726d20`, including
independent master #18129, passes **1297 distinct cases**: 1093 in five
complete server files, 185 in five runner files, 14 Node protocol cases
and five UI registry cases. This includes the complete 22-case Photon
file for both response workers. The external Photon transport is mocked;
Paperclip callbacks, transaction fences and observed SQL are real. The
earlier and combined case counts are separate, with no cumulative
double-counting.
- A normal merge retains both complete owned and independent master
binary patches. All twelve new-context hashes and sixteen earlier repair
hashes match. Three dedicated fresh 0413 fixtures plus the live-event
helper's own fresh database use PostgreSQL 17 without source
adaptations. Database build, server types/build, boundaries, full-SHA
migration order and generated TS/Rust profiles pass. The new inherited
Runner vendor inputs rebuild successfully.
- Database bytes remain identical to the previously qualified 0413
source. The recorded populated 0412-to0413 upgrade retains all 411 prior
receipts and 38 companies, appends one receipt and none on replay; the
index is valid and ready. The earlier 2828-case acceptance, CLI journey,
setup failures and their repairs retain their recorded source scope in
`doc/plans/2026-10-26-governed-review-response-sleep-audit.md`.
- Parent #17696 merged with 58 clean checks, Greptile 5/5 and Ponytail
5/5. Its Photon warning was disproved by real callback/SQL evidence and
the security check became successful. This PR still requires its own
fresh exact-head CI and review gates. No deployed sleep, external alarm
or fleet savings claim is made.

## Risks

- Notifications use the existing process-local writer contract.
Independent SQL writers and uncoordinated replicas are outside its
coverage.
- Genuine startup and company authority reconstruction remain
proportional to the owed backlog. Queries page 100 rows, but discovery
accumulates the candidates; this is not a constant-cost global
operation.
- Current publication authorization, durable binding, ownership and lock
ordering remain mandatory. The worker does not resolve the original
review gate.
- The concurrent index uses existing migration replay handling. Other
recovery duties and Cloud sleep eligibility remain separate.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell
execution and PostgreSQL tests. The runtime does not expose a more
specific serving snapshot or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 02:54:46 -05:00
ec3b558277 fix(server): schedule accepted native response recovery (#17696)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs store accepted results and their finalization decisions.
> - A completed run can still owe its public answer after an
interruption.
> - Periodic recovery scans for that answer even when no new work
exists.
> - This pull request wakes recovery from committed changes and saved
retry deadlines.
> - The existing owner still verifies the result, source, and current
publication authority.
> - Settled or dormant response work no longer needs its own periodic
scan.

## Linked Issues or Issue Description

Refs #17690 (merged). This is slice 48 of the sleep migration stack, now
based on master. Related: #17661 handles GitHub terminal publication;
this change handles native accepted-response presentation.
Governed-review presentation remains a separate duty.

**What happened?**

Native finalization recovery repeatedly checks already accepted results
for a missing answer. Removing that check without complete notifications
would lose responses when evidence arrives late, an agent resumes, or
admission settles after execution.

**Expected behavior**

Recover the answer from its saved result when a supported change makes
it eligible. Retain real retry deadlines. Stop querying when the answer
is selected or cannot currently be published. Reconstruct work after
restart.

**Steps to reproduce**

1. Commit a native accepted result with no selected response comment.
2. Run finalization recovery while its source is unavailable or locked.
3. Restore that source or release the lock. The answer must appear once
without repeating provider execution.

## What Changed

- Honor recorded non-create decisions and skip-comment policy in both
discovery and recovery. Apply durable conversation/result/continuation
guards before recovery locks. Preserve genuine unanswered and legacy
recovery.
- Scope routine response wakes to the affected issue/run. Coalesced
input names its active run; pending-effect settlement names its issue.
Sixteen existing global authority/configuration hooks use one explicit
company invalidation boundary. No new transaction wrappers or tables are
added.
- Add an accepted-response worker using the existing delivery
coordinator and scoped deadline helper. Preserve saved retry deadlines
and pending-effect fences under the owner's lock.
- Share the finalization notification boundary. Wake on finalization
settlement, exact accepted-result/final-message evidence, agent
availability, retry admission, and existing chat authority changes.
- Keep lifecycle policy in services. A narrow pre-write status hook lets
lifecycle retain ownership of its transaction and status transitions.
- Register the worker during enabled maintenance startup. Remove its
authority subscription on shutdown. Periodic replay skips this duty on
both committed and fresh-commit paths; ordinary finalization and
explicit recovery retain their behavior.
- Add a partial index and expression statistics for unpresented native
runs. Migration 0412 supports concurrent replay and adds no queue table.
- Add real PostgreSQL and lifecycle coverage. Preserve the
implementation and producer audit in the plan document.

## Verification

- Additional real Photon callback acceptance on the published production
code disproves the security warning: ten active assertions cause zero
response-recovery hints/SELECTs, while three concurrent assertions
during an actual attention-to-active recovery cause exactly one
wake/SELECT. The existing locked status guard already enforces this
boundary. No production change was needed; the resolved finding now has
a successful current-head security check. The regression will be
retained with the next slice.
- Eligibility repair source `892f36e5f66ba1af123034936a7160c64c3ab883`
passes **1489 distinct server cases**: 1139 in six complete
recovery/finalizer/monitor/summary/workspace suites (81.02 seconds),
plus 350 selected chat cases (297.68s). Another 1463 chat cases are
outside the pattern; the complete 1813-case file is not claimed. All
eleven new regressions are included in that count. Five dedicated fresh
PostgreSQL 17 databases migrate through unchanged 0412, with no
fixture-source adaptations. Database build, server types/build,
boundaries and full-SHA migration ordering pass. All seven test source
hashes match.
- The exact published baseline first fails eight behavioral assertions
while the three genuine-answer cases pass. The same corrected test
source passes all eleven after repair: recorded none/reuse remain
suppressed; five ineligible history families take zero recovery row
locks across repeated issue hints; missing/legacy/create-null answers
still publish once; fresh periodic finalization defers presentation to
its worker. Earlier fixture setup and variable/compiler failures remain
explained in the audit, rather than erased by later success.
- Repair source `2fc8b0614aaf9ac7337045de4ced7325f95edc45` passes **1821
server cases**: 1496 in 13 complete producer/lifecycle/authority suites
(110.63 seconds), plus 325 selected chat cases (278.99 seconds).
Coverage includes all native-session cases, queue admission, status
decisions, membership/board access, the complete GitHub workflow,
accepted-response lifecycle and real authority restoration. The other
1477 chat cases are outside this selection; the complete 1802-case file
is not claimed for this repair run.
- Actual PostgreSQL regressions first reproduce a coalesced-input hint
missing its run scope and two routine hint paths producing 12
response-history SELECTs each after three notifications. The same
assertions pass with run-scoped coalescing and zero response-history
SELECTs for unscoped routine hints. Actual membership restoration still
publishes the waiting response once. Assertion time limits are
unchanged; this measures real driver SQL and persisted producer work.
- Eleven dedicated fresh PostgreSQL 17 databases migrate through 0412.
One embedded-only wake-queue fixture import temporarily uses its own
external database and restores byte for byte. All 14 selected source
files match pre-run hashes. This qualifies external-database behavior
rather than reproducing embedded CI locally.
- Master advanced independently with #18126's Pi archive retry fix.
Normal merge `2a535bc61259ef99387369e8b32498f774e5ba2c` preserves both
complete binary patches. All accepted server/database/UI/test/CLI trees
stay identical; the three independent runner installer/manifest files
are retained. All 13 installer tests pass, including stalled header/body
cancellation. Combined server types/build, boundaries and full-SHA
migration order pass. A subsequent normal merge of #18128 at
`e8b78cce5bb557b086bee9ca930eeaab1d09b244` retains its three test-only
file changes with both complete patches preserved. The changed HTTPS
backfill case passes on a fresh PostgreSQL fixture (1 selected case; 178
outside the pattern), and combined types/build, boundaries and migration
order pass again. All 14 repair source hashes remain unchanged. The
independent browser fixture changes are preserved; no local browser run
is claimed.
- The earlier source `9feec170ef8174011fc3c5d3d6d76f0a0f39fea0` passed
the 4171-case full baseline before this scope repair. That scope is
preserved in the audit rather than represented as a current full rerun.
All 0412 database bytes remain unchanged: original concurrent
index/statistics/ANALYZE, prior executable history, journal entries and
retained snapshots. Five snapshots 0408 through 0412 remain. The
populated 0411-to0412 upgrade preserved 410 prior receipts and 38
companies, appended one receipt and none on replay; index/statistic
checks passed.
- The earlier P2 security failure, Greptile's former conditional
dependency on #17702, and the subsequent actual Ponytail eligibility
findings are recorded in
`doc/plans/2026-10-25-native-response-presentation-sleep-audit.md`. This
PR now includes the issue-scoped pending-effect notification fix itself,
so it is independently correct before #17702. Fresh published-head CI,
Greptile 5/5 and a clean security scan remain required. No
deployed-sleep, external-wake or fleet-savings claim is made.


## Risks

- The writer contract is process-local. Independent SQL writers and
other replicas are not covered by these notifications.
- Unpresented backlog reconstruction still has memory and time cost
proportional to candidate count. Pages contain at most 100 rows;
execution uses the shared fair batch. Selected history is excluded.
Health-only signals do not trigger reconstruction.
- Publication policy remains with its current owner. Digest, company/run
binding, source authorization, selected-answer suppression, and
ownership fences remain required.
- The rest of native recovery and governed-review presentation still run
separately. This PR does not relax Cloud sleep eligibility or add
external wake coverage.
- The stack parent has merged and its actual ancestry is reconciled.
This publication still needs fresh current-head CI and review.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell
execution, and PostgreSQL tests. The runtime does not expose a more
specific model identifier or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 02:31:20 -05:00
a8a038e659 fix(server): schedule native pending effect recovery (#17690)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native status decisions store effect targets and a delivery ledger.
> - Historical pending or failed effects still need recovery.
> - The periodic finalizer queries that ledger even when work is
complete.
> - This pull request gives pending effects their own commit-driven
worker and saved deadlines.
> - Completed history becomes quiet while incomplete effects remain
recoverable.

## Linked Issues or Issue Description

Refs #17687 (merged). This is slice 47 of the sleep migration stack, now
based on master.

**What problem does this solve?**

Periodic native finalization repeatedly checks effect delivery. A tenant
with completed history should not perform recurring SQL for this duty.
Historical unresolved effects must still recover without duplicating
their targets.

**Proposed solution**

Reconstruct current pending effects at startup. Wake from committed
coordinator changes. Use the saved retry and ownership deadlines. Keep
the existing dispatcher as the only recovery mutation owner.

**Alternatives considered**

A slower unconditional scan still creates idle traffic. A new queue
table duplicates existing durable state. Replaying materialization can
duplicate effects. This change uses the existing ledger, transaction
signals, and shared scheduler.

## What Changed

- Add a scoped worker with 100-row discovery pages and fair 25-entry
dispatch batches. Delivered, superseded, and terminal-failure history
has no recurring scan.
- Preserve exact company/issue/run/decision binding and validate
materialized targets before settling effects. Recheck phase, saved
result, retry, and lease under the coordinator lock. Missing targets
retain recovery.
- Notify from decision replay, accepted-result recovery, coordinator
claim/retry/release, attention replay, and execution-control recovery.
Only two DB-only producer writes need a new transaction wrapper. Nested
transactions use the existing outer-commit boundary.
- Exclude pending-effect candidates from periodic finalization so it
cannot skip ahead of their owner. Keep explicit and startup recovery
behavior compatible. Respect maintenance disablement, drain, standby,
and shutdown.
- Add concurrent, idempotent migration 0411 with partial indexes for
pending effects and their current coordinator binding. Retain the five
newest schema snapshots.
- Document producers, query evidence, failure history, and scope in
`doc/plans/2026-10-25-native-pending-effects-sleep-audit.md`.

## Verification

- Source `ba38ee465df02b6026286b5d5e15ae49fe07a367` passes all 2284
distinct cases across 25 complete files, without skips. The main
selection passes 2225 in 203.94 seconds; safe-replacement passes 59 in
26.03 seconds. Coverage includes all 968 native-session cases, the
94-case status-authority/pending-effect corpus, actual decision/lease
producers, Stop, queue, interaction, reconstruction and Slack consumers.
The audit gives per-file counts.
- The complete 94-case corpus passes again on the same database in 16.03
seconds. Retained PSC/OUT rows and an unrelated sentinel remain
unchanged. The repeat is not counted again as distinct coverage.
- Twenty-one separate fresh PostgreSQL 17 databases migrate through 0411
before setup. Four embedded-only fixture imports temporarily use their
own external database on this IPC-exhausted host and are restored byte
for byte. Assertions and timing limits stay unchanged. This qualifies
external-database behavior; it does not claim an unmodified embedded CI
run.
- Database build, server typecheck/build, module boundaries and
exact-head migration order pass. Original runtime/schema edits and
unrelated upstream blobs are preserved. Both original concurrent index
statements, all prior executable history, journal entries and retained
snapshot bytes are unchanged; five snapshots 0407 through 0411 remain.
- Populated 0410-to0411 upgrade preserves all 409 prior migration
receipts and 38 companies, appends one receipt and adds none on replay.
Both indexes are valid and ready on fresh and upgraded databases.
- Adopting the latest prepared prerequisite changes ancestry and parent
audits only. All accepted core trees remain identical. The audit retains
earlier failures and qualifications. Actual landed-parent ancestry is
now reconciled; fresh published-head CI and exact-head Greptile 5/5
remain required. No deployed sleep, Cloud-wake or fleet-cost claim is
made.

- Current parent context: integration
`5f7cd7bc589ad0169dfb5f33df611ead56510ca3` changes only the parent's
GitHub observation bound and independent cleanup assertion. All
production code, packages and 25 accepted test files remain unchanged.
Both focused GitHub idle cases pass in 15.26 seconds and the complete
six-case cleanup suite passes in 8.47 seconds on two fresh 0411
PostgreSQL fixtures. This does not claim the full parent workflow was
rerun here.

## Risks

- A missing producer can delay work until restart. The audit covers
current supported writers. Independent SQL and other processes remain
outside the local notification contract.
- Recovery validates existing targets; it cannot recreate a missing
target or silently mark it delivered. Such work retains an error and
retry.
- The indexes add write/storage cost and use the existing restart-safe
concurrent migration runner.
- Other finalization duties still poll until their separate migrations.
This PR does not prove host/database suspension, extend Cloud alarms, or
claim fleet savings.
- The actual prerequisite squash is adopted through a normal merge. Its
complete tree equals the reviewed parent, and the complete accepted
child tree is preserved. Fresh published-head CI and review remain
required.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning and code-execution tools.
The exact serving model ID and context-window size are not exposed in
this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 00:28:00 -05:00
DottaandPaperclip 4e715eb7ea fix(server): schedule historical unsafe export recovery (#17687)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs can retain accepted results after an unsafe workspace
export fails.
> - Historical recovery can retire an old notice or resume the accepted
result without another provider turn.
> - The periodic reaper repeatedly scans failures that cannot currently
recover.
> - Existing rows and committed changes can identify the actual recovery
obligation.
> - This pull request wakes recovery from those changes and retries only
owed work.
> - Dormant history becomes quiet while ownership and explicit operator
decisions remain protected.

## Linked Issues or Issue Description

**What happened?**

The native finalization reaper repeatedly visits historical unsafe
workspace-export failures. Cancelled tasks, newer runs, changed
contracts, and explicit operator decisions can make those rows
permanently ineligible. They still produce periodic database reads.

**What did you expect?**

Retire stale unsafe-export notices and recover eligible accepted
results. Stop reading dormant history until a relevant committed change
occurs. Keep a retry while a real obligation is blocked by ownership or
a transient failure.

**Steps to reproduce**

Retain an unsafe-export terminal failure whose task is cancelled or
superseded. Leave the instance idle after startup. The periodic reaper
continues reading that history.

Prerequisite #17680 is merged into master. Related: #14402 established
safe export finalization; #14329 was an earlier historical recovery
proposal. #17679 owns separate policy-review cleanup. This PR changes
only historical unsafe-export scheduling and its producers.

## What Changed

- Register a scoped worker through the existing coordinator and
scheduler. Remove this duty from periodic reaping. Preserve the
manual/startup recovery entry point.
- Distinguish stale notices, eligible accepted results, dormant history,
and concrete ownership contention. Reuse the existing mutation owner and
lock order. Preserve explicit settlement vetoes and exact
result/contract identity.
- Signal relevant task, run, action, and deletion writes before they
commit. Existing transaction boundaries handle nested commits and
uncertain outcomes. Notifications contain only scope IDs.
- Retain a retry only for owed work. Reconstruct work at startup,
preserve drain/resume, and stop through the shared lifecycle.
- Add concurrent, idempotent partial index migration 0410 for unsafe
terminal failures. Keep the five newest generated schema snapshots.
- Document producer coverage, query plans, supported writer boundaries,
and test failures in the unsafe-export audit.

## Verification

- Source `2a13dd08a6cc52f0adab09e3df9ce31e97b746b9` passes all 2188
cases across 24 complete files, without skips. The main selection passes
2129 cases in 192.58 seconds; safe-replacement passes 59 in 26.04
seconds. Coverage includes all 968 native-session cases, real
export/action producers, explicit continuation, workspace
reconstruction, interaction routing, Stop, queue and Slack consumers.
The audit gives per-file counts.
- Twenty separate fresh PostgreSQL 17 databases migrate through 0410
before setup. Four embedded-only fixture imports temporarily use their
own external database on this IPC-exhausted host, then are restored byte
for byte. Assertions and timing limits stay unchanged. This is qualified
external-database acceptance, not an unmodified embedded CI run.
- Database build, server typecheck/build, module boundaries and
exact-head migration order pass. Normal integration preserves the
complete original runtime/schema patch and unrelated upstream blobs.
Original concurrent index SQL, all prior executable history, journal
entries and retained snapshot bytes are unchanged; five snapshots 0406
through 0410 remain. Added comment removal preserves the full
non-comment TypeScript token streams.
- A populated 0409-to0410 upgrade preserves all 408 prior migration
receipts and 38 companies. It appends exactly one receipt; replay adds
none. The index is valid and ready on fresh and upgraded databases.
- The owning audit retains earlier fixture failures, skipped checks and
notice-retirement regression history. Older combined stack checks retain
their original scope in their owning audits. Actual parent
`73d24aa5acfcf17b3f676b4da7cd05eb70b60b99` is reconciled through a
normal merge. Its tree equals the reviewed ancestor and the complete
accepted child tree is unchanged. Fresh published-head CI and Greptile
5/5 remain required. No deployed sleep, fleet-cost or live provider
qualification claim is made.

- Inherited prefix correction: current integration
`8f43c9f81fac23b50997e7f0e1d0d9cf383a8109` changes only two values in a
previously unselected corpus fixture. All production code, the complete
package tree and all 24 accepted test files remain byte-identical to the
2188-case source above. The additional complete 72-case corpus passes in
14.00 seconds and again on the same database in 13.08 seconds. Retained
PSC/OUT and sentinel rows remain unchanged. The repeat is not
double-counted. The parent audit retains the original collision and
rejected cleanup attempt.

- Latest parent context: integration
`4d51231abe8b68c21ce896d0d2ac16865978c8ab` carries only the parent's
one-argument GitHub observation correction and an independent master
cleanup assertion. All production code, packages, 24 accepted suite
files and the additional corpus are unchanged. The two focused
native-presentation cases pass in 15.14 seconds and the complete
cleanup-removal suite passes all six cases in 8.31 seconds on two fresh
0410 PostgreSQL fixtures. This does not claim the full parent workflow
was rerun here.

Current head `b28399babac72ba9db1c4a4ef882c5e87aded38c` has passing CI
and Greptile 5/5 with no actionable findings. The first CI attempt lost
runners: fourteen jobs explicitly received simultaneous shutdown
signals, two reported lost runner communication, and two aggregate gates
failed downstream. One failed-job retry on unchanged source passes. The
underlying host shutdown cause remains unconfirmed; no code fix or extra
retry is claimed. [CI run and
attempts](https://github.com/paperclipai/paperclip/actions/runs/40819408528).
No Ponytail verdict is present.

## Risks

- A missing future producer can delay dormant-history discovery until
restart. The audit names current producers. Independent SQL writers and
other server processes remain outside the process-local signal contract.
- Physical ownership and uncertain operations retain retries. Time alone
does not authorize taking workspace ownership. No provider turn or
unsafe archive retry is added.
- The recovery owner remains authoritative if state changes after
discovery. Terminal UI status does not authorize resource destruction or
accepted-result replay.
- The partial index adds matching-row write/storage cost. It builds
concurrently through the existing restart-safe migration runner.
- This proves the changed worker becomes quiet. It does not prove host
suspension, external wake coverage, or fleet cost savings. Other sleep
migrations remain. This slice still requires fresh current-head CI and
reviews.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning and code-execution tools.
The exact serving model ID and context-window size are not exposed in
this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-29 00:13:22 -05:00
DottaandPaperclip 51c3f8924c fix(server): schedule cancelled workspace preservation (#17677)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A cancelled native run may still owe preservation of prepared
workspace files.
> - The existing recovery receipt and source lease record that
obligation.
> - The orphan reaper still searches for these cancellations on every
pass.
> - This PR registers committed cancellations with the shared deadline
worker.
> - The existing retention owner keeps file pinning and ownership
checks.
> - Idle and completed history no longer causes this preservation scan.

## Linked Issues or Issue Description

Refs #17674 and #17145. Stopped instruction collection is merged into
master. It owns file-preservation discovery only. The existing
environment cleanup worker continues to own stop-only teardown.

**What needs to be improved?**

`preserveCancelledNativeWorkspaces` runs from every orphan-reaper pass.
It queries even when no cancelled native workspace needs a file-recovery
hold. Current foreground Stop can also defer retention after a lock or
database failure, so startup-only recovery would be insufficient.

**Why is this improvement important?**

Empty queries keep idle tenants active. Removing them must not lose user
files, weaken Stop ownership, replay cancelled work, or turn a retained
allocation into destructive cleanup.

**Proposed solution**

Register the existing durable obligation from committed run-status,
cancellation-acknowledgement and workspace-reference writes. Reconstruct
it at startup. Keep retries while actual preservation remains
unresolved. Disarm after the existing owner records the hold. Related-PR
search and ROADMAP.md review found no duplicate of this scheduling
migration.

## What Changed

- Add a company/run-scoped preservation worker with 100-row keyset
reconstruction and fair batches of five.
- Separate pending file debt from immediate coordinator/lease readiness.
Retain a 30-second retry for known unfinished preservation and shared
failure backoff for errors.
- Signal the existing status, acknowledgement, workspace-reference,
deletion and settings transactions. Add no table or independent timer.
- Keep exact source checks, no-workspace contracts, task/coordinator/run
lock order, retention receipt and no-replay recovery card with their
owners.
- Recheck cancellation acknowledgement, run status and coordinator phase
under the retention locks. Preserve the existing foreground preterminal
API.
- Keep startup/manual recovery; skip only this duty in periodic reaping.
Preserve drain/standby and shutdown behavior.
- Add concurrent, replay-safe partial index migration 0409 and retain
the five newest snapshots.
- Add the producer and lifecycle audit in
`doc/plans/2026-10-25-cancelled-workspace-preservation-sleep-audit.md`.

## Verification

- Current source `21b46a8e4293ca670235117458b984493a181759`: all 1472
cases in 16 complete files pass without skips. The owner passes 84 cases
in 14.93 seconds. Fifteen consumer files pass 1388 cases in 133.46
seconds, including all 968 native-session cases and Stop, dispatch,
restart, workspace-sync and upgrade-readiness coverage.
- Twelve separate fresh PostgreSQL 17 databases migrated through 0409
before setup. The embedded-only lifecycle fixture temporarily used its
own external database on this IPC-exhausted host and was restored byte
for byte. Assertions and timing limits stayed unchanged. This qualifies
external-database behavior; it does not claim an unmodified embedded CI
run.
- Database build, server typecheck/build, module boundaries and
exact-head migration order pass. The complete original runtime/schema
patch and unrelated current-parent blobs are preserved. Original
concurrent index SQL, all earlier executable history and retained
snapshot bytes are unchanged; five snapshots 0405 through 0409 remain.
- A populated 0408-to0409 upgrade preserves all 407 prior receipts and
38 companies. Exactly one receipt is appended; replay adds none. The
preservation index is valid and ready on fresh and upgraded databases.
- Actual parent `82529741418e138d9d623bf13d2fa22c1ff79d36` is merged
through a normal ancestry reconciliation. Its complete tree equals the
reviewed parent. The entire accepted child tree and all source hashes
are preserved. The audit retains earlier failures, qualifications and
per-file counts. Published head
`b34f0589847f33754ecde9413fd36bf007353d2c` has Greptile 5/5 and all 57
checks passing or skipped. No Ponytail review is present. These checks
do not establish deployed host/database suspension or Cloud alarm
delivery.

## Risks

- Signals cover instrumented single-process writers. Independent SQL
writers and additional replicas need integration.
- Status signals may inspect a run without relevant file debt once. They
do not create recurring work. Discovery and memory still scale with
actual candidates.
- Missing ownership or stop evidence remains pending. Active retries are
intentional while a concrete file obligation is unresolved.
- Retry timing is process-local; startup reconstructs durable debt. The
worker does not overwrite another native-finalizer deadline.
- Admitted database operations may finish after drain begins. Shutdown
joins them. Provider teardown remains with the existing cleanup owner.
- The partial index adds storage and write cost. Its concurrent
migration uses the existing interrupted-index repair path.
- Cloud alarm coverage, sleep eligibility and host suspension are
unchanged. This proves worker behavior, not fleet savings.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL
tests. The runtime does not expose a more precise model ID or
context-window size for this turn.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 22:27:11 -05:00
DottaandPaperclip 8252974141 fix(server): schedule stopped instruction collection (#17674)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Runs can stop while their private instruction edits still need
collection.
> - Existing copy rows and stop evidence describe that unfinished work.
> - The orphan reaper still queries for it on every maintenance pass.
> - This PR activates collection from committed copy and run
transitions.
> - It uses saved retry deadlines when stop proof or collection remains
pending.
> - Active copies and completed history no longer cause collection
polling.

## Linked Issues or Issue Description

Refs #17671 and #17670. Both prerequisites are merged into master.
Captured saves and post-save release remain separate reasons with their
own workers.

**What needs to be improved?**

`recoverStopped` queries instruction copies on every orphan-reaper pass,
even when no terminal run owes collection. A run can become terminal
without changing its copy row, so copy-only notifications would miss
real work.

**Why is this improvement important?**

Repeated empty queries prevent idle tenant instances from becoming
quiet. Removing them must preserve edits, stop proof, current ownership,
and recovery after restart.

**Proposed solution**

Use the existing transaction signals, scoped deadline worker, and
scheduler. Reconstruct eligible copies at startup, signal supported copy
and terminal-run writers, and retain saved retry deadlines for
unresolved work. Preserve the existing collection and directory owners.

Related PR search found the warm agent-file ownership work (#14695),
verified stop boundary (#14311), and task-workspace cleanup (#16464).
This PR does not replace those contracts. ROADMAP.md has no duplicate
migration.

## What Changed

- Add scoped stopped-instruction collection with keyset reconstruction
and fair batches of 20.
- Signal before copy preparation, mutation and adoption writes, and
before terminal run writes in their existing transactions.
- Cover heartbeat, native finalization/restart/session recovery,
execution-control, queue and dispatch producers.
- Signal settings changes and company/agent deletion. Preserve drain,
standby, restart and shutdown ownership.
- Recheck terminal-run eligibility in the due-time claim. Keep terminal
status separate from process-stop proof.
- Retain existing stop checks, attempt limits, saved candidate bytes and
remote-resource fencing.
- Skip only this lane in periodic orphan recovery. Preserve
manual/startup recovery.
- Add concurrent, replay-safe partial index 0408 and retain the five
newest snapshots. Add no table.
- Record the owner and producer audit in
`doc/plans/2026-10-25-stopped-instruction-collection-sleep-audit.md`.

## Verification

- Source `cb2818a8f46ef961f744977a258f0e9042e31a42`: all 1469 cases
across 14 complete files pass without skips in 141.69 seconds. This
includes all 968 native-session cases, filesystem saves, startup,
terminal-run producers, dispatch, queue, execution-control, finalization
and lifecycle.
- Eleven separate fresh PostgreSQL 17 databases migrated through 0408.
The embedded-only lifecycle fixture temporarily used its own external
database on this IPC-exhausted host and was restored byte for byte.
Assertions and timing limits stayed unchanged. This qualifies
external-database behavior; it does not claim an unmodified embedded CI
run.
- Database build, server typecheck/build, module boundaries and
exact-current migration order pass. The original concurrent index SQL is
unchanged. All prior executable migration history, journal entries and
retained snapshot bytes are preserved; the five newest snapshots are
0404 through 0408.
- A populated 0407-to0408 upgrade preserves all 406 prior migration
receipts and 38 companies. It appends exactly one receipt. Replay adds
none. The collection index is valid and ready on fresh and upgraded
databases.
- The normal merge of actual parent
`267842fc3c6e6542deed8a1bcc2c78bec8df7862` preserves both complete
patches, all independent sidebar changes from #18117, and the full
accepted server/package/test trees. The complete sidebar test file
passes all 36 cases on the combined tree. React act warnings remain.
- Prior failures and fixture qualifications remain in the audit.
Published head `d176d731f083c7edd2ede688cb901d1e66d8f1d8` has Greptile
5/5, all 57 checks passing or skipped, and no unresolved review threads.
No Ponytail review is present. These local tests do not establish
deployed host/database suspension or Cloud alarm delivery.

## Risks

- Process-local signals cover instrumented writers. Independent SQL
writers and additional replicas still need integration.
- Terminal-run signals can cause one scoped inspection even if no
instruction copy exists. They do not create a recurring scan.
- Reconstruction and memory scale with actual pending candidates; active
candidates are filtered by the run join.
- Admitted file/provider operations can outlive a later drain request.
Shutdown joins them. Uncertain ownership remains pending rather than
being discarded.
- The partial index adds storage and write cost. Its concurrent
migration uses the existing interrupted-index repair path.
- Cloud alarms, sleep eligibility and provider suspension are unchanged.
Worker quiet is not a claim of deployed host/database sleep or measured
savings.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, shell tools, and
PostgreSQL/filesystem tests. The runtime does not expose a more precise
model ID or context-window size for this turn.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 22:13:57 -05:00
DottaandPaperclip 267842fc3c fix(server): schedule owed agent directory cleanup (#17671)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agent runs can save their directory before temporary-file cleanup
finishes.
> - The existing receipt records stop proof and unfinished cleanup.
> - The orphan reaper still searches for this debt on every maintenance
pass.
> - This PR activates cleanup from committed changes and saved retry
deadlines.
> - The directory owner keeps its resource checks and file lock.
> - Settled directory history no longer causes recurring cleanup
queries.

## Linked Issues or Issue Description

Refs #17670, the merged captured-instruction commit slice. This PR
targets master. It does not include stopped instruction collection or
change the saved-file contract from #14695.

**What needs to be improved?**

The post-save part of `recoverCaptured` queries directory copies on each
orphan-reaper pass, even after all temporary directories have been
released.

**Why is this improvement important?**

Empty database work prevents idle tenant instances from becoming quiet.
Cleanup must still survive a crash after a save and preserve files when
stop or resource ownership is uncertain.

**Proposed solution**

Use existing copy receipts as the source of truth. Notify the shared
coordinator inside relevant copy-write transactions. Reconstruct pending
work at startup and register each saved retry deadline. Keep physical
cleanup in the directory owner. Add a partial index over unfinished
cleanup rather than a new queue table.

Related-PR search found workspace archive and SSH task-workspace cleanup
changes (#17175 and #16464). They own different resources and do not
replace this stopped agent-directory receipt recovery. ROADMAP.md has no
duplicate migration.

## What Changed

- Add a scoped directory cleanup worker with keyset discovery and
batches of 20.
- Signal relevant copy transitions, company/agent deletion, and worktree
execution policy changes.
- Preserve the due-time claim, current receipt read, directory lock,
stop proof, and exact remote-resource checks.
- Read the receipt after release; a returned promise does not establish
cleanup completion.
- Keep unresolved cleanup on its saved retry or shared failure backoff.
Final receipts disarm the worker.
- Register the worker during startup and skip only this lane in periodic
recovery. Keep manual recovery compatible.
- Add concurrent, replay-safe partial index migration
`0407_greedy_star_brand.sql`. Retain the five newest schema snapshots.
- Add the producer audit and lifecycle evidence in
`doc/plans/2026-10-25-directory-cleanup-sleep-audit.md`.

## Verification

Published head: `210fcb1215a3b5e5ed84e420ddb73ab5952d9365`. Source
acceptance: `70a6800eea874bdb87685bf6eea66de79c48b577`. Parent #17670
landed as `b294de546bf9cd3a8578a7a66d7e8ec0eb939799`; normal
squash-ancestry reconciliation preserves the entire accepted descendant
tree. Final follow-ups change audits only.

- All 284 cases in six complete filesystem, startup and heartbeat owner
files pass, without skips, in 48.88 seconds. Four unused fresh
PostgreSQL 17 databases migrated through 0407 isolate destructive
fixtures.
- Three fixtures use supported external database paths. The
embedded-only lifecycle fixture temporarily uses its own external
database on this IPC-exhausted host, then is restored byte for byte. All
assertions and original timing limits remain unchanged. This qualifies
the local result; CI must still pass with the original embedded fixture.
- Actual database build, server typecheck/build, module boundaries and
exact-current migration order pass. The original concurrent idempotent
index SQL bytes are unchanged. All prior executable migration history,
journal entries and retained snapshots stay intact. The five snapshots
are 0403 through 0407.
- Populated 0406-to0407 upgrade preserves all 405 old receipts and all
38 companies, adds exactly one receipt and adds none on replay. The
cleanup index is valid and ready on fresh and upgraded databases. The
entire current database tree equals the source used for this proof.
- Both complete original and upstream patches are preserved. The actual
parent squash tree equals the already-adopted reviewed parent; resolving
eleven duplicate-ancestry text conflicts leaves the whole accepted tree
unchanged, not just selected files. Complete server, package, UI and
test trees still match the 284-case source acceptance.
- The audit retains earlier ownership, stopped-resource, receipt,
fixture and local-runner failures. The no-projects build filter was not
counted as a build pass; corrected actual build and complete reruns
passed before publication. Old draft review scores do not count for this
new head.
- Current-head Greptile gives 5/5 with no actionable findings, and all
57 checks pass. These checks establish worker behavior, not deployed
host/database suspension or measured savings.

## Risks

- Signals cover the existing single-process writer model. Independent
SQL writers and replicas need separate integration.
- The worker retains identifiers for actual pending work. Its index
excludes completed history but does not make large pending backlogs
free.
- File/provider cleanup can outlive an admission change once started.
Shutdown joins admitted operations. Unknown remote ownership remains
pending and no replacement environment is acquired for cleanup.
- The partial index adds storage and write cost. Its concurrent
migration uses the existing interrupted-index repair path.
- Cloud alarm coverage and host-sleep safety remain unchanged. This
establishes worker quiet, not deployed host/database suspension or
measured savings.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, shell tools, and
PostgreSQL/filesystem tests. The runtime does not expose a more precise
model ID or context-window size for this turn.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 21:51:47 -05:00
DottaandPaperclip 27270e9bf8 fix(server): wake interrupted retry cancellation from committed intent (#17667)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A parked scheduled retry has no executor to finish an interrupted
Stop request.
> - Its saved cancellation fence already records the work that remains.
> - The orphan reaper currently scans for that work on every heartbeat,
including idle periods.
> - This PR registers the existing records with the shared delivery
scheduler and wakes recovery after committed writes.
> - Recovery keeps the existing cancellation guards and settlement
behavior, then becomes quiet.

## Linked Issues or Issue Description

Refs #17666, the merged preceding PR in the recurring-server-work
migration stack. This PR targets master. Related cancellation work found
in the dedup search: #12301 and #7340. This PR changes the trigger for
interrupted parked-retry cancellation; it does not replace their
cancellation or retry policies.

**What happened?**

The periodic orphan reaper queries scheduled retries with cancellation
fences even when no cancellation needs recovery. Its first-20-row scan
can also stop behind malformed evidence.

**Expected behavior**

Committed cancellation intent wakes bounded recovery. Failed work
retains a retry. Startup reconstructs durable work. Empty or settled
work performs no recurring SQL for this duty.

**Steps to reproduce**

1. Start a server with heartbeat scheduling enabled and no interrupted
retry cancellations.
2. Observe the orphan reaper repeatedly query for cancellation fences.
3. Cancel a scheduled retry and fail the final status write after its
fence commits. Recovery must complete the existing Stop request without
running a provider or dropping its metadata.

**Paperclip version or commit**

Current parent squash `7afee8f0622e273d10506b0febfb68f143d168b1`. Source
acceptance `66107ae0a7ad85fd0476f91c4f8f98ae74a7a048`.

**Deployment mode**

Self-hosted server with the existing single-process transaction signal
model. Local evidence uses PostgreSQL 17.

## What Changed

- Add one domain-owned recovery worker over existing heartbeat run rows.
No new table. It delegates to run control and preserves original
cancellation, budget, and operator metadata.
- Signal cancellation fences, lifecycle transitions, dispatch
deferral/promotion/suppression, relevant settings changes, and deletion
within their owning transactions. Wrap only the ordinary database fence
in a short transaction; provider calls stay outside it.
- Keep startup/manual recovery compatible and remove this duty from the
periodic orphan sweep. Other reaper duties remain for separate PRs.
- Extend the shared scoped worker to retain pending work with a
memory-only admission retry while its owner is paused. Without pending
work, it schedules nothing. Existing GitHub consumers retain their
behavior.
- Add a concurrent idempotent partial index and lifecycle/producer
tests. Enable guarded disposable PostgreSQL URLs in affected test
harnesses.
- Reconstruct one hundred candidates per scope per scheduler pass.
Resume from the saved scope cursor and retain fair twenty-item execution
batches. Manual recovery starts cancellation before reading the full
backlog.
- Require the recovered browser reply to be saved before testing reload.
Keep all original assertions and time limits.
- Record the full producer audit, lifecycle limits, and failed attempts
in `doc/plans/2026-10-25-scheduled-retry-cancellation-sleep-audit.md`.

## Verification

Published head: `405e533394b26215e22d0452e9b0207d3f784a79`. Source
acceptance: `a35fa43b98e2bd378d4a920d2775ce243654838b`. The final commit
changes the audit only.

- All 363 backend cases in ten complete owner and consumer files pass
without skips. This includes actual Stop/retry producers, transaction
settlement, interrupted-write recovery, pause/resume, shutdown, fair
backlog processing and quiet after settlement. Seven separate fresh
PostgreSQL 17 databases prevent fixture interference.
- The two embedded-only backend fixtures temporarily use separate
external databases on this IPC-exhausted host. Their fixture imports and
native client teardown are restored byte for byte afterward. Assertions
and timing limits are unchanged. CI must still pass with the original
embedded fixtures.
- Two paging regressions fail on the prior worker and pass on the fix.
They cover cancellation before full reconstruction, a malformed first
page, fair batches, a newly committed intent behind the cursor, and no
later idle SQL. Per-candidate admission reads remain deliberate safety
checks.
- The repeated CI reload failure is reproduced with a temporary
twelve-second fixture settlement delay. The trace retains the same
five-second missing-reply assertion, even though a cleanup race masked
it in the top-level error. Requiring exactly one saved reply from the
resumed run passes under the identical delay. The fault is removed. All
six browser journeys pass on the final source in 40.5 seconds, with zero
retries and the original 120-second case and five-second reload limits.
- Browser acceptance uses the canonical CLI/server/static-UI path and a
fresh external database. Only fixture database-URI setup is temporarily
adapted and then restored. This is not an unmodified embedded
CI-environment reproduction. The six cases include the controlled case;
repeated runs are not added to the distinct count.
- Actual database build, server typecheck/build, module boundaries and
exact-current migration order pass. The database source tree is
unchanged from the verified 0405-to0406 upgrade: all 404 prior receipts
and 300 companies stay intact, exactly one receipt is added, replay adds
none, and the index is valid on upgraded and fresh databases. The five
snapshots remain 0402 through 0406.
- All earlier failures remain recorded in the audit. Prior head
`48b3fcb83f305616df02a4337a42ec0867bb4b80` had exact-head Greptile 5/5
but failed the repeated browser assertion. It was not retried blindly.
Its completed Ponytail review's backlog concern and old migration
paragraph are addressed here. Old scores do not satisfy new-head gates.
- Current-head CI passes. Exact-head Greptile and Ponytail both give
5/5, with no remaining actionable findings. These checks do not prove
deployed instance or database suspension.

## Risks

- Only supported in-process writers produce local transaction signals.
Independent SQL writers or other replicas need a separate ingress
contract. Startup reconstructs saved obligations.
- A real failure retains bounded retries. Unknown commit outcomes retain
the shared intent hold. Neither is discarded to report quiet.
- Per-owner drain uses a memory-only admission recheck. The shared
helper change affects existing consumers, which have focused regression
coverage.
- The partial index is concurrent and idempotent; it adds index storage.
Each scoped load reads at most one hundred rows. The shared helper
retains discovered pending IDs between passes, so a large unresolved
backlog still uses memory.
- Cloud alarms, idle-safety exemptions, and other reaper duties remain
separate. Merge requires passing current-head CI and review gates.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, shell tools, and local
PostgreSQL tests. The runtime does not expose a more precise model
identifier or context-window size for this turn.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 21:07:55 -05:00
DottaandPaperclip 8bc051245f fix(server): wake stale GitHub review repair from committed changes (#17664)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/215/changes/paperclip/215
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - GitHub checks and task admission can finish in either order.
> - A check can reject an old head before admission creates its review
row.
> - The chat maintenance loop repeatedly scans for that late review.
> - Existing cancellation and delivery receipts already identify the
repair.
> - This pull request wakes repair from committed changes and leaves
settled history quiet.

## Linked Issues or Issue Description

**What happened?**

GitHub chat maintenance repeatedly scans all review projections. Empty
instances and retained history incur database work even when no repair
is owed.

**What did you expect?**

A committed stale-head cancellation or admitted review should wake
repair. Only the exact delivery's unassessed queued/running review
should become superseded. When no repair remains, this duty should stop
making database calls.

**Steps to reproduce**

Configure a GitHub chat endpoint and leave it idle after all reviews
settle. The previous maintenance callback continues to call
repairSupersededReviews. A rapid push can also cancel a check before
admission inserts its review, which explains why recovery must remain.

Refs #17662, the merged review-check publication parent. Related #17661
and #17656 migrate terminal responses and acknowledgements. This slice
owns only late stale-head projection repair.

## What Changed

- Move the repair predicate and execution into the GitHub provider
owner. Match the original admitted delivery and cancellation receipt.
Recheck state and assessment in the update.
- Register a scoped worker with the existing coordinator. Use keyset
pages and fair bounded dispatch. Keep retries only for unresolved repair
or failed reconstruction.
- Signal before cancellation, automatic receipt insertion,
automatic-review binding, and check-driven review-state writes. Preserve
existing transaction ownership and outer commit behavior.
- Remove the periodic provider wrapper and publication-maintenance
callback. Preserve explicit manual recovery and independent
missed-webhook recovery.
- Add two concurrent, idempotent partial indexes. Use a guarded delivery
UUID lookup and a bounded exact-receipt lookup. Add no tables.
- Add ten real-database lifecycle and producer cases. Document the
producer audit, failure history, and remaining limits.

- Keep YAML browser reads scoped to the editor after focus changes, with
the original journey assertions and deadlines.

## Verification

Current source: `0886a0229b1c4f68d2f582da86aa1f248e9124f3`.

- At `dd6bca0485c3444d6e3d0ee10ec4aa651e801392`, **287 distinct cases
pass**: 274 complete GitHub workflow cases and 13 startup cases. The
GitHub command filters out 1,485 unrelated cases. The workflow suite
uses a new, pre-migrated PostgreSQL database.
- Coverage includes both orders of cancellation and admission, outer
commit and rollback, restart, drain, standby, failed-update retry,
concurrent assessment under an observed row lock, shutdown during that
wait, more than one candidate page, manual history, and settled-history
quiet.
- Database build, server typecheck/build, module boundaries, and target
migration order pass.
- Canonical migration `0403_natural_cerise.sql` preserves both original
index definitions and concurrent idempotent creation. All 401 parent
migration receipts remain unchanged during upgrade. One new receipt is
added. Replay adds none. Both indexes are ready and valid on fresh and
upgraded databases. The five newest snapshots remain.
- The normal parent merge preserved the prepared tree. The later browser
fixture repair changes only its central read helper and verification
note. Entire server, package, and UI trees remain identical to the
accepted source.
- [The owning
audit](https://github.com/paperclipai/paperclip/blob/0886a0229b1c4f68d2f582da86aa1f248e9124f3/doc/plans/2026-10-25-github-review-projection-sleep-audit.md)
records the exact evidence and earlier failures. Earlier query-plan
measurements retain their historical revision scope.
- Fresh current-head CI and Greptile review remain required. No deployed
sleep, full repository-suite, or fleet-savings claim is made.

Failure accounting: browser shard 7 failed twice on
`1453a4447c0d4d33f713ae3e00734c7950f95426` because the YAML helper read
whole-page clipboard text while main was focused. Three isolated
unchanged local passes did not resolve that failure. A controlled native
focus-loss probe reproduced the same error before the repair and passed
after it. The helper now clears stale clipboard data, focuses the
editor, and waits within the existing expectation budget for a fresh
YAML copy. All six complete YAML journeys pass with zero retries in 50.9
seconds. Journey assertions and deadlines are unchanged. The temporary
fault is removed. The exact CI focus-change trigger remains unknown.
[Both CI
attempts](https://github.com/paperclipai/paperclip/actions/runs/40777754306)
remain part of the verification record. Fresh new-head checks are
required.

Ponytail follow-up: the 4/5 review of `b3c81ba` was conditional on
fixing a stale startup-test slice marker, so the merge was held.
Boundary assertions reproduced the invalid -1 end index. The test now
uses the existing timer declaration and checks both bounds; all 13
startup/hostname cases pass. The optional no-op coordinator cleanup is
already in #17666. The six YAML journeys and all production code remain
unchanged. Fresh review and CI are required for this corrected head.

## Risks

- Missing a future producer signal could delay discovery until restart.
Supported writers share the current database owner's process-local
notification scope. Independent SQL writers and other server processes
need their own integration.
- The migration builds indexes concurrently. The existing migration
runner handles interruption and invalid index recovery.
- Direct GitHub missed-webhook discovery remains periodic. This change
does not make the whole chat service idle.
- Existing Cloud wake coverage and sleep blockers remain unchanged.
These tests do not prove deployed host or database suspension.
- Each remaining sleep slice needs its own review and CI checks before
merge.

## Model Used

OpenAI Codex (GPT-6), with reasoning and code execution. The exact
serving model ID and context-window size were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 18:42:54 -05:00
DottaandPaperclip 65fa1cd617 fix(server): schedule GitHub review checks from committed changes (#17662)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/213/changes/paperclip/213
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - GitHub review checks report the state of admitted agent work.
> - The chat maintenance loop searches for check work even when none
remains.
> - Existing action rows already record unfinished publication and retry
deadlines.
> - Committed changes can wake those rows through the shared
coordinator.
> - An unresolved check must still detect a remote head change when a
webhook is missed.
> - This pull request schedules only outstanding checks and leaves
settled history quiet.

## Linked Issues or Issue Description

**What happened?**

The GitHub chat maintenance callback repeatedly queries review-check
actions. Empty queues and completed history still incur database reads.
Local state changes wait for the maintenance callback.

**Expected behavior**

Committed check creation, review binding, run changes, and
publication-lease release wake the worker. Existing retry and
remote-head deadlines apply only to unfinished checks. A settled worker
makes no recurring database or provider calls.

**Steps to reproduce**

Configure a GitHub bot with no outstanding checks, or finish all of its
checks. Leave the instance idle. The old maintenance callback continues
to call the check service.

Refs #17661, the merged terminal-response parent. Related #17656
migrates response acknowledgements. Related #16385 changes branch-policy
admission; this PR preserves that separate policy boundary. Late
stale-head projection repair remains a separate follow-up slice.

## What Changed

- Move the existing check-state reader into the GitHub provider owner.
Preserve current authority, run selection, assessment ownership, and
check replacement behavior.
- Register a scoped worker with the existing coordinator and scheduler.
Existing received and retryable-failed actions remain durable truth. No
new table is added.
- Signal check enqueue, task review receipts, manual review
insertion/binding, run lifecycle changes, and publication-lease release
through existing transaction ownership.
- Preserve the saved failure backoff, eight-attempt policy, and
competing lease expiry. Keep five-second remote-head reconciliation only
while a received check remains unresolved.
- Check admission before each HTTP request. Shutdown leaves durable work
recoverable without spending a provider failure attempt.
- Remove check dispatch from the periodic GitHub callback. Keep the
separate late-projection repair and the manual combined recovery method
compatible.
- Add an idempotent concurrent partial index for unfinished check
selection. Retain the newest five generated snapshots.
- Add eleven real-database worker cases and startup wiring coverage.
Document producer coverage and the external reconciliation limit.

## Verification

Current reviewed source: `2bba08403b195bf646eda48992b46d6f5cebf756`.

- Normal merges preserve the complete review-check patch, the landed
terminal-response parent, and current master changes. No landed SQL
migration or journal entry changes.
- At `6995f84b240a14a7b8660e1cf095b3aa1e1e3168`, **331 distinct tests
pass**: 264 complete GitHub workflow cases, 30 completion cases, eight
issue-tree producer cases, 16 login-cleanup cases, and 13 startup cases.
The GitHub command filters out 1,485 unrelated cases. Login cleanup uses
its real embedded PostgreSQL fixture.
- Combined-master acceptance: **277 distinct tests pass** at
`9c538aed1251fd7f7cecb829a0ecbce6110fb1ed`: 264 complete GitHub workflow
cases and 13 startup cases. The GitHub command filters out 1,485
unrelated cases. A new PostgreSQL database receives all migrations
through 0402..
- Migration `0402_ancient_malice.sql` preserves the original partial
index definition and concurrent idempotent creation. All 400 prior
migration receipts stay unchanged during upgrade. One new receipt is
added. Replay adds no receipt. The index is ready and valid on fresh and
upgraded PostgreSQL databases. The five newest snapshots are retained.
- A rollback-only query probe adds 10,000 processed actions and one
pending action. The actual parameterized candidate predicate selects the
one pending row through the partial index, with two shared buffers and
0.007 ms execution time. This measures candidate discovery, not all
lease or provenance reads.
- Database build, server typecheck and build, module boundaries,
migration safety and migration order pass. Audit: [GitHub review-check
scheduling](https://github.com/paperclipai/paperclip/blob/2bba08403b195bf646eda48992b46d6f5cebf756/doc/plans/2026-10-25-github-review-check-sleep-audit.md).
- Fresh checks and Greptile review are required for this head. Prior
scores do not count. These checks do not prove deployed host suspension
or fleet savings.

## Risks

- Unresolved queued/running checks still make remote-head requests at
the existing five-second deadline. GitHub webhook receipt does not prove
future pushes cannot be missed. This PR does not claim active reviews
become network-idle.
- Run and configuration hints rebuild outstanding checks for the
affected company. Enqueue, review, and lease hints narrow to the
affected endpoint/repository/PR. Startup and scope reconstruction read
existing pending rows; they are not constant-cost operations.
- Delivery admission failures remain discoverable through the unresolved
check's external reconciliation deadline. Not every shared delivery
mutation gains a new immediate signal.
- Notifications use the existing process-local database ownership
contract. Independent SQL writers and other replicas need their own
supported integration.
- Provider markers, publication leases, and saved actions preserve
recovery after uncertainty. This change does not expand Cloud alarm
coverage or prove deployed host/database suspension.
- Each remaining sleep slice needs its own review and CI checks before
merge.

## Model Used

OpenAI Codex (GPT-6), with reasoning and code execution. The exact
serving model ID and context-window size were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 17:08:27 -05:00
DottaandPaperclip f8226b0515 fix(server): wake GitHub terminal responses from committed changes (#17661)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/210/changes/paperclip/210
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip coordinates agent work and presents results in chat.
> - GitHub acknowledgements already persist the response action and
provider receipt.
> - A terminal run can still owe a final response after acknowledgement.
> - Periodic maintenance searches those rows even when no relevant state
changed.
> - Committed run, issue, presentation, lease and publication changes
identify when recovery can advance.
> - The owner should recover from those events and real deadlines, then
become quiet.

## Linked Issues or Issue Description

Refs #17656, which is merged. This existing terminal-response slice
targets master.

**Problem:** A saved GitHub response can remain nonfinal while its run,
native presentation, publication or source is unresolved. Removing
maintenance without covering those producers loses recovery. Polling
settled history keeps the tenant active.

**Expected behavior:** Committed supported changes wake the terminal
owner. It reconstructs existing actions, preserves provider authority
and pending publication ownership, waits at saved deadlines where
needed, and becomes quiet after settlement.

## What Changed

- Register terminal GitHub response work with the shared scoped deadline
coordinator, using existing action/delivery/publication/run rows.
- Notify response availability from supported run milestones, native
identity, recovery resolution, issue updates and tree/workspace
restoration writes. Preserve outer transaction commit/rollback.
- Keep GitHub response interpretation in its platform owner. Use the
bounded run projection for both direct authorization and scheduling,
including native presentation guards and deliberate passive outcomes.
- Retain the existing publication lease, markers, source/principal
checks and final receipt precedence. A publication owner or uncertain
outcome keeps its existing authority; the worker does not replay it.
- Remove only recurring terminal-response recovery, preserving the
manual path and neighbouring duties. Add no table or scheduler service.
- Regenerate the original sparse terminal-work index as
`0401_yummy_ultimates.sql`, concurrently and idempotently through the
existing top-level migration runner. Preserve all prior SQL/journal
entries and the five newest snapshots.

## Verification

Published head: `88a209e28b969b5229e9a621aeddfd3cbae9b0cd`, integrated
with landed #17656 (`ba5bba86dcdde32b14d4d7dec1b51898e5f6f22d`) and
master `52b86c2032a189b0ea259c5b0cb7158b07ef788c`.

- **74 distinct current cases pass:** 17 terminal-response
lifecycle/producer cases (1,721 other chat cases skipped), all six
queued-comment consumption regressions for the new upstream query
optimization, all 30 heartbeat completion cases, all eight real/unit
issue-tree control cases and all 13 startup cases. Database build,
server typecheck/build, module boundaries and four fresh migrations
through 0401 pass.
- The earlier **304-case** prepared-parent matrix passes at
`aa72f32641b2e2c680342cd49477272349aca9db`: all 253 GitHub workflow
cases, 30 completion cases, eight issue-tree cases and 13 startup cases.
It retains that revision scope. Repeated cases are not added to the
current 74.
- The terminal matrix covers finalized-history quiet, deleted Board
sources, pending presentation/passive outcomes, outer rollback/commit,
paused/new runs, restart/drain/standby, committed native identity, lease
waits, full-page unrelated history, uncertain publication ownership and
tree/workspace/exhausted-recovery producers.
- A populated 0400 database upgrades to 0401 with all 399 prior
migration receipts identical, exactly one added, and none added on
replay. The terminal index is valid/ready on fresh and upgraded
databases. Its SQL matches the original draft after whitespace
normalization; generation keeps snapshots 0397–0401.
- Normal integration `3c8339a541bf68bc9957b9d5a8df80d3a7c3452d`
preserves the complete 24-path owned binary patch and all five new
master paths. The landed parent tree also equals the previously
published parent tree byte for byte. Both normal merge parents and the
exact composition tree are checked. No force push or rebase occurs.
- [The owner
audit](https://github.com/paperclipai/paperclip/blob/88a209e28b969b5229e9a621aeddfd3cbae9b0cd/doc/plans/2026-10-25-github-terminal-response-sleep-audit.md#acceptance-on-the-landed-github-parent-and-current-master)
preserves the original producer, quietness, failure-isolation,
skipped-suite and presentation-wait results with their scopes. Current
passes do not erase earlier failures or turn skips into passes.
- The audit-only follow-up leaves all owned source/test/migration hashes
unchanged. **Fresh exact-head CI and Greptile 5/5 with no actionable
findings remain required.** No full local repository suite, live GitHub
tenant or deployed sleep result is claimed.

## Risks

- Process-local notifications cover supported application writers and
shared pools. Independent SQL writers and other replicas gain no new
wake guarantee.
- Native terminal status alone does not authorize fallback publication.
Pending presentation and existing publication ownership remain separate
obligations.
- Stop can leave an in-flight provider request settling; admission
fences stop a subsequent request. Unknown outcomes retain their existing
recovery rules.
- Index creation can take time on large histories. Concurrent/idempotent
migration and existing replay recovery remain in charge.
- Worker quiet does not establish host/database suspension. Cloud alarm
coverage and sleep admission remain unchanged.

## Model Used

OpenAI Codex, using reasoning, repository inspection, code execution and
real PostgreSQL tests. The exact model identifier and context-window
size are not exposed in this session.

## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 16:51:57 -05:00
DottaandPaperclip ba5bba86dc fix(server): wake GitHub acknowledgements from committed changes (#17656)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/206/changes/paperclip/206
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip coordinates agent work and presents its progress in chat.
> - GitHub deliveries save acknowledgement actions and provider
receipts.
> - Periodic maintenance currently searches for unfinished
acknowledgements even when idle.
> - Committed delivery, lease, response and presentation changes
identify when work can advance.
> - The existing owner should wake on those changes and recover only at
saved deadlines.
> - Completed history should become quiet without losing unfinished
provider effects.

## Linked Issues or Issue Description

Refs #17653, which is merged. This existing GitHub acknowledgement slice
targets master.

**Problem:** A saved GitHub acknowledgement can wait for a publication
lease, fast-response result or native presentation decision. Polling
empty or unchanged work keeps the tenant active. Removing maintenance
without covering these producers can strand the acknowledgement.

**Expected behavior:** Committed changes wake the scoped owner. It
reconstructs existing actions on startup, retries at saved deadlines,
parks unchanged presentation waits and becomes quiet when settled.

## What Changed

- Register GitHub acknowledgement work with the shared scoped deadline
coordinator. Reuse existing action, delivery, lease and fast-response
rows.
- Add transaction-aware response-availability notifications at supported
writes. Keep GitHub protocol details in its platform owner and shared
notifications limited to IDs.
- Share the bounded latest-request/run projection between scheduling and
direct response authorization. An absent native presentation decision
parks the action until its real writer commits.
- Retain publication leases, source/principal authority, version
fencing, response markers and final receipt precedence. Check admission
before each new provider request; let in-flight work settle. Stop does
not consume the provider retry budget.
- Remove only recurring acknowledgement recovery and keep its manual
path. Add no work table or scheduler service.
- Regenerate the three original indexes as
`0400_absent_phantom_reporter.sql`: pending acknowledgement IDs,
fast-response delivery lookup and publication comment/destination
lookup. Each uses concurrent/idempotent creation and the existing
top-level migration runner. Preserve all landed SQL/journal entries and
the five newest snapshots.

## Verification

Published head: `09ca00b9f029838434ec042cda691345d67b493e`, on landed
continuation parent `063cfd43813cabf18417605fb9eefb9aacf95c8d`.

- **87 distinct current cases pass:** 18 real PostgreSQL acknowledgement
lifecycle/presentation cases, all 26 fast-response service cases, all 30
heartbeat completion cases and all 13 startup cases. Database build,
server typechecking/build, module boundaries and three fresh migrations
through 0400 pass. The current Runner bundle rebuild also passes after
upstream provider-event changes.
- The lifecycle matrix covers empty/history quiet, lease release,
restart, drain/standby, fast-response failure/expiry/publication/unknown
outcome, streaming quarantine, outer rollback/commit, saved provider
retry, stop between requests and receipt state. The real presentation
writer proves quiet while pending, rollback, committed reactivation and
final quiet.
- The earlier **305-case** matrix (236 full GitHub workflow cases, 26
fast-response service cases, 30 completion cases and 13 startup cases)
passes at `337f272853d4f8a51b5b9300900a46eb4e13f698`. It retains its
prepared-parent scope. Repeated cases are not added to current 87.
- A populated 0399 database upgrades to 0400 with all 398 prior
migration receipts unchanged, one new receipt and no additional receipt
on replay. All three indexes are valid/ready on fresh and upgraded
databases. Their SQL definitions match the original draft after
whitespace normalization; migration history and retained snapshot bytes
are preserved.
- Normal integration `9b3607e159153e00ccdfe538d52b591bd38d6499`
preserves the full 28-path owned binary patch and all 20 unrelated
master paths, including native retry, provider-event and evaluation
workflow changes. Both normal merge parents and the exact tree are
checked. No force push or rebase occurs.
- [The owner
audit](https://github.com/paperclipai/paperclip/blob/09ca00b9f029838434ec042cda691345d67b493e/doc/plans/2026-10-25-github-chat-maintenance-sleep-audit.md#acceptance-on-the-actual-landed-continuation-parent)
retains the original missing-notification, JSON precedence, receipt
lifecycle, external-database fixture and migration-registry failures.
Earlier 69/228/broad counts keep their original scope. Skips and failed
revisions are not relabelled as current passes.
- The final audit-only edit leaves all owned source/test/migration
hashes unchanged. **Fresh exact-head CI and Greptile 5/5 with no
actionable findings remain required.** No full local repository, live
GitHub tenant or deployed sleep result is claimed.

## Risks

- Process-local transaction hints cover supported application writers
and shared pools. Independent SQL writers and other replicas do not gain
a new wake guarantee.
- Provider effects remain guarded by the existing publication lease and
current source/principal authority. Unknown outcomes retain their
existing recovery semantics.
- Presentation waits depend on their supported milestone/transaction
producers. A missing producer can defer recovery until another supported
change or restart.
- Index creation can take time on large histories and runs concurrently
outside a transaction. Existing migration recovery handles replay; prior
SQL is unchanged.
- Worker quiet is distinct from host/database suspension. Cloud wake
coverage and sleep admission remain unchanged; later maintenance duties
are separate slices.

## Model Used

OpenAI Codex, using reasoning, repository inspection, code execution and
real PostgreSQL tests. The exact model identifier and context-window
size are not exposed in this session.

## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 16:06:42 -05:00
DottaandPaperclip 063cfd4381 fix(server): wake interaction continuations from committed responses (#17653)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/204/changes/paperclip/204
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip coordinates agent work and human approvals.
> - Human decisions already persist the continuation action that resumes
an agent.
> - A response can commit before its callback or publication finishes.
> - Periodic maintenance repeatedly discovers those rows, including when
idle.
> - Committed writes should wake the existing owner; saved retries and
claims supply real deadlines.
> - Startup reconstructs durable work, and settled history becomes
quiet.

## Linked Issues or Issue Description

Refs #17651, which is merged. This existing continuation slice targets
master and removes its duty from periodic chat maintenance.

**Problem:** A saved `interaction_wakeup` action can survive a committed
human response whose callback crashes. Without producer coverage,
removing maintenance strands the continuation. Repeated empty
maintenance also keeps the database active.

**Expected behavior:** The outer transaction's settlement wakes the
continuation owner. It handles ready conversation publications,
dispatches under existing authority, preserves interrupted/retry work,
and becomes quiet after settlement.

## What Changed

- Register continuation recovery with the shared scoped deadline
coordinator. Reconstruct existing issued/processing actions and retain
the existing retry and stale-claim deadlines.
- Notify from both enqueue producers and owner-local claim/settlement
writes. Preserve outer commit/rollback and avoid moving provider or
heartbeat calls into database transactions.
- Give the existing publication queue an optional
company/endpoint/conversation scope. The continuation owner gives ready
cards in that conversation a bounded delivery opportunity before
claiming the action; FIFO, capacity, ambiguity and provider authority
remain with the queue.
- Carry admission/cancellation through publication preparation and check
before every new send. Join in-flight requests and retain pending
actions/publications through drain, standby, shutdown and resume.
- Remove only recurring continuation recovery; keep the manual API. Add
no table. A concurrent partial index on existing pending continuation
actions and matching company/id keyset paging prevent retained history
from being scanned. Parent #17651's indexed confirmation discovery and
current master are retained.

## Verification

Published head: `a1d16f84b3fae361d5d648670a5964486370c3a3`, on actual
landed terminal-confirmation parent
`ee67d2ece9bda157dfad9bdf25175b1d004ed341`.

- **75 distinct current cases pass:** 32 selected real PostgreSQL chat
cases (14 continuation lifecycle/producer/stop cases, one new
continuation query-cost case, and 17 parent confirmation/query-cost
cases; 1,671 skipped), and all 43 cases in complete
interaction-publication and app/startup suites. Database build, server
typecheck/build, module boundaries and fresh migrations through 0399
pass. The original landed-parent matrix passed 74 cases before the
additional query-cost repair; those are not added again.
- The new real-SQL regression seeds 20,000 completed/unrelated actions
and one pending processing claim. Before repair it examines 20,345 rows
and hits 20,421 buffers. The final startup and company-scoped queries
must examine at most two action rows and fewer than 20 action buffers;
empty scoped recovery must also stay below 20 buffers. Planner settings
are unchanged. Migration `0399_ambitious_onslaught.sql` is
concurrent/idempotent and uses the existing top-level migration runner;
prior SQL/journal entries and the five newest snapshots are preserved.
- The matrix covers callback recovery, concurrent responses, malformed
prefixes, more than one page, saved retries/stale claims, outer
rollback/commit, deferred/future/paused publication states, restart,
drain and stop before each new provider request. Quiet assertions
inspect real selects/transactions across virtual idle time.
- Normal integration `db5a84c0f8b73a0ce863f4d2fbaee0a115cf044f`
preserves the complete 12-path owned binary patch and all 12 unrelated
current-master paths, including managed Cloud sandbox changes. A
verified semantic base resolves squash-ancestry conflicts; both normal
parents and the exact tree are checked. The subsequent index/keyset
repair is a deliberate additional change, tested separately; the final
patch is not claimed identical to the original 12-file implementation.
No force push or rebase occurs.
- The same 74 pre-index cases passed on the earlier prepared and actual
landed parents. The new query-cost regression failed before its repair,
then passes on the repaired source. Earlier 57-case and broad acceptance
retain their original revision scopes; repeated cases are not added to
the current count. The audit-only follow-up leaves all owned server
source/test hashes unchanged.
- [The owner
audit](https://github.com/paperclipai/paperclip/blob/a1d16f84b3fae361d5d648670a5964486370c3a3/doc/plans/2026-10-25-interaction-continuation-sleep-audit.md#acceptance-on-the-actual-landed-terminal-confirmation-parent)
retains the original missing-signal regression, the skipped standalone
publication suite, and the review's stop-before-send failures with their
corrections. Skips and earlier failing revisions are not relabelled as
current passes.
- **Fresh exact-head CI and Greptile 5/5 without unresolved actionable
findings remain required.** The former remote head's reviews are stale.
No full local repository suite, live provider or deployed sleep result
is claimed.

## Risks

- Process-local transaction hints cover supported application writers
and shared pools. Independent SQL writers and other replicas do not gain
a new wake guarantee.
- The publication queue's existing FIFO, capacity, retry/ambiguity and
authorization contracts still govern effects. In-flight provider
requests can settle after Stop; the fence prevents a subsequent request
from starting.
- Missing a supported producer can defer recovery until another change
or startup. Producer and rollback tests cover this slice's supported
paths.
- Worker quiet is separate from host/database suspension. External alarm
coverage and Cloud sleep gates are unchanged.

## Model Used

OpenAI Codex, using repository inspection, code execution and real
PostgreSQL tests. The exact model identifier and context-window size are
not exposed in this session.

## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 15:45:18 -05:00
DottaandPaperclip ee67d2ece9 fix(server): wake terminal confirmation recovery from committed changes (#17651)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/202/changes/paperclip/202
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip persists confirmation clicks and interaction decisions
before dispatch completes.
> - A crash after a decision commits can leave a processing confirmation
action.
> - Periodic publication maintenance repeatedly scans those actions even
when idle.
> - Saved decisions and the supported board/admin writers already
identify when settlement can change.
> - This owner reconstructs durable actions, reconciles real changes and
then becomes quiet.

## Linked Issues or Issue Description

This existing terminal-confirmation slice follows merged #17649. It
removes only terminal-confirmation recovery from periodic chat
maintenance while retaining the explicit manual API and existing durable
action/interaction/publication rows.

**Problem:** An accepted response can commit before its dispatch
settles. Removing periodic recovery without covering board decisions,
expiry and ownership changes would strand processing actions. Pending
human decisions must remain recoverable without polling indefinitely.

**Expected behavior:** Committed supported changes wake the scoped
owner. It reconciles terminal responses once, parks unresolved human
decisions, preserves concurrent commits/restart/drain and becomes quiet
when settled.

## What Changed

- Register terminal-confirmation work with the existing delivery
coordinator and scoped lifecycle. Reconstruct processing action IDs,
bound selection pages, preserve concurrent wakes and keep current
authorization in the existing interaction owner.
- Add the existing transaction-aware confirmation notification topic at
real resolution writes and callback settlement. Reuse milestone signals
instead of adding per-caller recovery wrappers.
- Cover ordinary and checkbox confirmations, suggestions and connection
intents through their owning service. Cover board accept/reject,
cancellation/withdrawal, opt-in comment supersession, expiry,
reassignment and already-satisfied connection resolution.
- Preserve task/actor authority, one-use callback tokens and immutable
provider publication semantics. Explicitly fail malformed missing
bindings instead of silently retaining unrecoverable processing actions.
- Remove only this duty from recurring publication maintenance. Add no
table or scheduler infrastructure. A concurrent partial index on
existing action rows prevents recovery discovery from scanning completed
history.

## Verification

Published repair head: `a35e78d076c1b618bc643391b7520fa6952fc996`,
targeting master after normal integration of
`644937c93150c423f12674992c7ea1e8b5dbc98c`. [Detailed acceptance and
preserved
failures](https://github.com/paperclipai/paperclip/blob/a35e78d076c1b618bc643391b7520fa6952fc996/doc/plans/2026-10-25-terminal-confirmation-sleep-audit.md#bound-retained-history-reads--october-28-review-repair).

- **New query-cost regression:** Capture the actual SQL emitted by
startup and company-scoped confirmation discovery. Seed 20,000
completed/unrelated action rows and one pending human decision. Before
the index, the regression fails because recovery examines 20,009 action
rows. After the partial index, startup and scoped discovery must examine
at most two action rows and fewer than 20 table/index buffers; the
no-pending scoped path must also stay below 20 buffers. Planner settings
are unchanged, and returned IDs remain correct.
- Match startup keyset ordering to `(company_id, id)`, the partial index
order. The index alone still let PostgreSQL choose the primary key and
scan 20,206 rows for `ORDER BY id`; the final query fixes this without
forcing a planner setting or adding a second index. Fixed kind/status
predicates stay literal for prepared-plan index eligibility.
- Add the generated `0398_yielding_random.sql` migration, using `CREATE
INDEX CONCURRENTLY IF NOT EXISTS` and the existing top-level migration
runner. Keep the five newest generated snapshots and preserve every
prior SQL/journal entry. No work table is introduced.
- **110 distinct cases pass:** 17 final confirmation
lifecycle/query-cost cases (1,671 skipped), seven real Runner descriptor
variants (295 skipped), and all 86
app/startup/bundled-plugin/plugin-database cases. Server/database
typechecking and builds, migration safety/numbering, module boundaries
and a fresh PostgreSQL migration through 0398 pass. The final test-only
vacuum precondition is followed by another 17-case confirmation pass and
server typecheck. Counts are not added across repeated runs. Normal
integration of master `644937c93150c423f12674992c7ea1e8b5dbc98c`
preserves the complete owned patch and all 18 unrelated master paths.
- Previous landed-parent acceptance: 29 distinct cases (16 confirmation
lifecycle/producer cases with 1,671 skipped, plus all 13 startup cases)
passed on `32e64a7e9ea5b457e54ca7b268be90e1e155b451`; the earlier
80-case matrix remains scoped to
`c1e8542b588a4fa59b62f9829541d3daea4f1ddc`. Counts are not added across
repeated runs.
- Actual producer cases prove parking/quiet, caller-owned outer
transaction rollback, commit-driven automatic settlement, restart/drain
and concurrent commits. Checkbox, suggestion, connection-intent and
Telegram/Slack callback paths retain existing authority and immutable
publication semantics.
- **Fresh exact-head CI and Greptile 5/5 with no unresolved actionable
findings are required.** The preceding head's Runner CI failed its
timeout-attachment durable-suspension case (1 failed/1,955 passed/4
skipped); that failure is retained and not counted as a pass. The new
head must pass this job. No full local repository, live provider or
physical suspension claim is made.

The owner audit retains the earlier missing-board-producer regressions
and their corrections, including the invalid supersession fixture that
assumed a new checkbox expires an older one. Existing behavior was
preserved by using the supported opt-in human-comment path. Prior
scheduler/backoff, route and plugin test failures remain at their
original revisions; no old pass is substituted for current remote gates.

## Risks

- Process-local notifications cover supported application writers and
their shared pools, not independent SQL writers or other replicas.
- Processing actions waiting on a human response are deliberately
parked. Supported interaction resolution/expiry writers must use the
notification boundary; a missing producer delays settlement until
restart.
- Invalid bindings have explicit failure evidence. Source, actor, task
and publication authority remain checked by their existing owners.
- Following continuation/GitHub duties remain separate. Cloud sleep
gates and external alarms are unchanged; local worker quiet does not
establish physical host/database suspension or fleet savings.

## Model Used

OpenAI GPT-6 / Codex, with reasoning, repository inspection, code
execution, and PostgreSQL tests. The exact runtime model identifier and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 15:00:31 -05:00
DottaandPaperclip 8049fbf51d fix(server): schedule voice call reports from saved deadlines (#17586)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Voice calls retain reports and project guest transcripts into intake
tasks.
> - Chat maintenance repeatedly scans recent calls even when no report
is due.
> - Empty scans keep the tenant database active.
> - Existing session and report rows already contain the work and its
deadlines.
> - This pull request moves report refresh to the shared coordinator and
removes the final voice maintenance lane.
> - Idle voice configuration and old call history no longer cause report
scans.

## Linked Issues or Issue Description

Refs #17583, #17580, #17578, #17576. This is slice 25 of the
recurring-server sleep stack. It now follows merged #17583 and targets
master.

**What existing behavior does this improve?**

Voice call-report refresh and guest transcript projection.

**Current behavior**

The chat interval scans terminal calls for reports. It repeats the scan
even when there are no eligible calls or all saved deadlines are in the
future.

**Proposed behavior**

Committed session transitions and signed callbacks wake a scoped worker.
Startup reconstructs existing rows. Later refreshes use saved deadlines.
The worker becomes quiet after all eligible windows expire.

**Reason and benefit**

Stop empty database scans without losing late transcript and billing
updates. Keep the existing 24-hour refresh policy, provider response
ordering, and guest-task trust restrictions.

## What Changed

- Add a voice-owned report worker using the existing scoped deadline
helper. No queue table or field is added; the existing session table
receives a report-window index.
- Save each attempt and its retry before provider I/O. Compare that
ownership on settlement so an older response cannot overwrite a later
callback or claim.
- Preserve available reports when a response is incomplete or stale.
Preserve their finite refresh window.
- Route voice-change and reply call-ended writes through
transaction-aware report signals.
- Preserve guest transcript locks and use the observed document revision
when updating an existing transcript.
- Register the worker in app startup. Remove voice reconciliation from
the chat interval. Keep explicit one-shot diagnostics.
- Document producer scope, lifecycle behavior, tests, and external wake
limits.

## Verification

Published head: `758cfe15c3e6283d38ba512c8416ccce4d200862`. The
follow-up after `c28de398` changes only two documentation files; the
verified production, migration and test source is unchanged. It normally
merges landed retention/master
`980775e0e956eecdc28c73c9b4aca0407ad45e1a`. The 20 owned paths remain
the complete slice, including officially regenerated migration 0397, its
snapshot and required pruning of snapshot 0392. The non-database
reviewed patch was preserved during ancestry integration; the only
subsequent source edit makes one existing report test’s provider
responses call-specific. All production report behavior and unrelated
master changes are preserved. No shared worker, workflow or lockfile
change belongs to reports. All 58 current-head checks are now clean,
including the previously failing approval shard; exact-head Greptile is
5/5 with no new actionable issues. Ponytail’s current turn stopped
without a verdict, an authorized nonblocking state. The prior
conditional documentation requirement was fixed; old CI failures remain
recorded below.

- **Prior-head evidence at `9bf20467df`: 303 distinct targeted cases
passed**, not rerun wholesale for this integration: all 132 voice
integration cases; 170 cases across 11 complete report, retention,
cleanup, recovery, company-removal, shared-worker and startup/hostname
suites; and one complete snapshot/schema drift case. This includes 15
report, 13 retention and 14 cleanup lifecycle cases. Voice integration
and the four other database owners use separate initially empty
databases and migrate themselves. Report cases use their dedicated
database after the setup repair.
- Five real connection saves over 205 sessions with saved future report
deadlines selected 1,025 session rows before the notification repair.
They now select zero and preserve the original deadline. The report
notification helper accepts only exact session scopes. Configuration,
authority and connection hints retain their existing live
recovery/reply/cleanup behavior but do not rebuild report history. Real
session terminal writes, callback hints, report claims and receipts
still reactivate reports.
- Database/server typechecks and builds, migration numbering/safety, and
module-boundary checks pass. Final server types/build also pass after
integrating the retention review repairs. The parent 205-admission,
five-configuration-save test still observes zero retained-history rows
and preserves its deadline.
- The old eager reader selected all 105 saved-report headers before
yielding and failed the 100-row budget. Registered reconstruction now
uses the existing paged contract and yields at 100 rows per scope
refresh. A partial scan can restart on the same owner and reconstruct
every saved deadline. At due time all 105 report IDs are fetched exactly
once. Future deadlines remain quiet between actual retries.
- The company-fairness case places a small company deterministically on
page two and requires it in the next 25-work batch after discovery, with
all 105 reports processed before the failed provider's retry. Guarded
mutations catch a missing cursor reset and bypassed company rotation.
Original source was restored before final acceptance.
- At prior head `9bf20467df`, migration 0396 was generated with `pnpm
db:generate` on the parent 0395 lineage. It creates the partial
`(ended_at, id)` report-window index concurrently and replayably through
the existing migration runner. All 394 historical SQL files and the
journal prefix are unchanged; snapshots 0392 through 0396 remain. Fresh
application migration and two replay invocations succeed with a valid
index. The real plan case retains 20,001 old sessions, 205 recent
sessions, precision-safe keysets, and fewer than 1,000 rows removed by
filtering on each page.
- Existing real hangup/callback/reply-error producers, callback
ownership fencing, monotonic report results, saved retry before provider
contact, guest transcript projection after a document lock, transaction
rollback/nesting, drain and shutdown remain covered. No table, field or
provider protocol is added. Available reports retain the existing finite
24-hour refresh policy.
- [Current-parent acceptance and failure
ledger](https://github.com/paperclipai/paperclip/blob/9bf20467dfd50b81848e1c8dd7ddc905d206418f/doc/plans/2026-10-25-voice-call-report-sleep-audit.md#current-parent-reconstruction-and-migration-acceptance-october-28)
records the failed fresh fixture, eager read, fairness-clock attempts,
guards, commands, counts and limits. Older 174/138-case and
workspace-wide results belong to their stated heads. No new complete
local repository suite, live provider, Cloud wake, host/database
suspension or savings claim is made.
- [Report configuration-scope
repair](https://github.com/paperclipai/paperclip/blob/9bf20467dfd50b81848e1c8dd7ddc905d206418f/doc/plans/2026-10-25-voice-call-report-sleep-audit.md#configuration-scope-review-repair-october-28)
records the current 303 cases and measured rescan repair. The prior
`80207ba6bb` head had Ponytail 5/5; that review is stale after this
change. Fresh exact-head Greptile 5/5 and all CI remain required. Actual
failing Ponytail findings must be fixed; a stopped or paused turn
without a verdict is an authorized nonblocker. Active no-verdict review
waits are capped at 30 minutes from the original request/start
(25-minute monitor cutoff). Older scores and parent checks are not
inferred as current passes.

- **Current-master verification: 185 distinct cases pass**: final
complete voice integration 132, complete report/cleanup/retention
15+24+13, and snapshot/schema drift 1. Each database owner used its own
empty fixture database. The first combined run had 183 passing cases and
one report-test failure; it is preserved in the audit. A deliberately
earlier-ended competing call reproduces the old one-shot mock consuming
the intended response. The corrected test selects responses by provider
session ID, verifies independent costs for both calls, then rejects
stale target data. The isolated repair and full 132-case suite pass;
production behavior and timeouts are unchanged. These 185 cases are not
added to the older 303 count.
- **Current migration lineage:** official `pnpm db:generate` after
actual master creates `0397_known_madrox.sql`; the reviewed
concurrent/idempotent report-index SQL is retained byte-for-byte. All
395 existing SQL files and journal entries are unchanged; snapshots
0393–0397 remain and the new snapshot links parent 0396. The application
migrator boots a fresh database, repairs a deliberately invalid index
during two concurrent replay invocations, records one history row, and
replays again successfully. Database build, server build/typecheck,
migration checks and module boundaries pass. [Full current-master
acceptance and failure
ledger](https://github.com/paperclipai/paperclip/blob/c28de398fef074f650eff532952ab6f5563739a8/doc/plans/2026-10-25-voice-call-report-sleep-audit.md#landed-master-migration-and-acceptance-october-28).

- **Review follow-up and preserved CI timeout:** Ponytail’s prior
`c28de398` review was 4/5 advisory, with no must-fix production defect
and a request to fix the stale design-doc migration reference before
shipping. That document now names migration 0397 and links current
185-case acceptance; older 302/303-case evidence is explicitly
historical. The optional suggestions about one-shot diagnostic inventory
cost and inherited test-clock helpers remain disclosed. CI run
`40703490340` failed the unchanged approval-queue backoff test on its
existing 15-second timeout and then an activity-log FK in teardown
(2,097 pass/one fail/one skip in that shard). The complete unchanged
28-case approval suite passes on a new isolated database; the exact CI
timing cause remains unproven. No timeout, workflow or production logic
was changed, and no old-head rerun or failure waiver is claimed. This
documentation head requires fresh CI and exact-head Greptile. [Review
and timeout
ledger](https://github.com/paperclipai/paperclip/blob/758cfe15c3e6283d38ba512c8416ccce4d200862/doc/plans/2026-10-25-voice-call-report-sleep-audit.md#review-documentation-and-preserved-remote-timeout-october-28).

## Risks

- Process-local commit signals do not observe independent SQL writers or
another server process.
- Available reports still refresh for 24 hours. The provider's complete
flag does not promise immutable final data.
- Transcript projection retries remain within the existing report
window. This change does not add indefinite projection repair.
- One-shot diagnostic `reconcile(limit)` inventories the eligible window
before slicing; it remains a manual diagnostic cost, not recurring idle
work. Some inherited test helpers advance fake time during assertion
retries, an optional reliability follow-up noted by review.
- A local deadline cannot wake a stopped host. Cloud alarms and sleep
eligibility remain unchanged. No deployed sleep or cost saving is
claimed.
- This PR depends on the preceding stack. Final master reconciliation
and remote review gates remain required.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, code editing,
and local test execution. The exact serving model ID and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 09:41:15 -05:00
DottaandPaperclip 71b448b9cb fix(server): schedule inbound voice cleanup from durable obligations (#17580)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Incoming voice calls need approval, expiry, and safe provider
cleanup.
> - The remaining voice maintenance tick scans admissions every second,
including idle endpoints.
> - Admission status does not prove that the provider call has ended.
> - Existing admission rows can retain the cleanup owner, confirmation,
and retry deadline.
> - This pull request replaces the cleanup scan with committed changes
and actual deadlines.
> - Idle cleanup stops querying while unfinished calls keep a
recoverable path to termination.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Inbound voice admission cleanup. Its reply-delivery parent #17578 has
landed; this PR targets master. Live-session recovery is in #17576. It
preserves separate ownership for those services.

**Current behavior**

The voice maintenance tick scans expired admissions even when no cleanup
is due. A session can commit before the separate approved-admission
write. Expiry can then terminate that authorized call. A rejected guest
call can be labeled ended before provider termination succeeds.
Seven-day retention can delete that unfinished cleanup.

**Proposed behavior**

Wake cleanup from committed admission and session changes. Schedule the
original expiry or saved retry. Claim current state under the admission
lock before provider I/O. Commit approval with session reservation. Keep
unconfirmed termination durable and transfer bound calls to live-session
recovery.

**Reason and benefit**

Configured idle endpoints and confirmed history cause no recurring
admission cleanup SQL or provider requests. Approval, retries, crash
recovery, and cleanup remain supported. History retention and call
reports stay separate migration items.

## What Changed

- Recover terminal and rejected legacy admissions whose session binding
was lost. Copy known termination or bind a proven matching live session
for cleanup without granting approval.
- Register inbound cleanup with the shared coordinator and scoped
deadline worker. Yield after each 100-row scope page and reset cursors
on registration. Remove only cleanup from periodic voice maintenance.
Keep complete manual reconciliation.
- Add commit-aware notifications at admission, approval, rejection,
guest binding, pending-tool expiry, lifecycle confirmation, and session
termination writes.
- Commit authorized session binding and admission approval together.
Recheck cleanup ownership under the same admission lock. Recover older
interrupted bindings only from matching session evidence.
- Add nullable cleanup confirmation, next-attempt, and provider-account
fields to the existing admission table. No new queue table is added.
Generated migration 0396 backfills only provable historical termination
in committed 1,000-row keyset batches and adds a concurrent partial
index.
- Save the retry before provider contact. An unconfirmed end receipt
retains cleanup. Bound calls remain owned by live-session recovery.
- Verify the signed provider account before using a rotated key for
unbound cleanup. A different account cannot inspect or end the saved
call. Legacy admissions require their original credential fingerprint.
- Prevent history pruning and company deletion from erasing unfinished
cleanup. Company deletion locks endpoints before the voice owner's
check, matching ingress serialization.
- Document producer coverage, ownership, deadlines, verification, and
remaining sleep limits.

## Verification

Published head: `a4e7ccf976fc5793a8914e4e4861aa080d43a4da`. The reply
parent #17578 has landed. A normal merge integrates actual master
`e7ea2e2961863171956e70fb05d3d1f3790f7909`, preserves its unrelated
changes, and regenerates this slice’s migration after landed migration
0395. Fresh exact-head reviews and CI are required.

- The previous Ponytail 3/5 finding is repaired: legacy terminal
admissions no longer enter approval-only repair, and rejected admissions
with valid matching live sessions retain their rejection and persist the
binding needed for session-scoped cleanup confirmation. Five regressions
failed with ownership conflicts before the repair. Five
conflicting-session regressions prevent an unrelated leg from being
claimed. No cleanup operation grants approval or starts a provider call.
- **24 complete cleanup lifecycle cases and 122 complete voice-store
integration cases pass** on the repair. After actual-master integration
and migration regeneration, the complete 24-case cleanup suite passes
again on a new empty PostgreSQL 17 database, and the database migration
integration test passes separately. The cleanup owner, voice-store test
and admission schema are byte-identical to that tested repair; the
cleanup fixture only changes the migration filename. Migration
numbering/safety, database build, production server typecheck and build
pass.
- Migration `0396_tan_paladin.sql` was generated with `pnpm
db:generate`. Every landed SQL migration and the complete parent journal
prefix are unchanged. Generated snapshots retain exactly 0392–0396,
pruning 0391. The reviewed cleanup SQL, concurrent idempotent index and
committed 1,000-row historical backfill are retained byte for byte under
the new generated filename, registered for top-level execution. The real
application migrator boots a fresh database, repairs an interrupted
concurrent index, serializes replays and verifies eligible-history
backfill across actual transactions.
- The preceding published head
`48d8356e73308579470c182e759eead6e83173d2` received Greptile 5/5 but
failed CI server shard 9/12: one existing connected MCP rate-limit test
expected two real-clock calls to share a fixed 60-second bucket. A
controlled window rollover reproduced its exact failure. That existing
fixture now freezes only `Date` around the same-window 429 assertion,
verifies acceptance after advancing one window, and restores the real
clock in `finally`. All 92 gateway acceptance cases pass. Production
rate limiting, timeouts and assertions are unchanged. This adds one
existing CI fixture to the slice; the original failed run remains
recorded.
- The first regression invocation skipped the suite because it used the
wrong database variable; that is not a pass. The corrected invocation
reproduced five ownership conflicts. The first fixed cleanup run passed
22 and failed two new low-level deletion test calls on a
retained-session foreign key; the regressions now check the cleanup
deletion guard at its owning boundary, with production deletion ordering
unchanged. The final full suite passes all 24.
- The first complete voice-store invocation passed 121 and timed out on
one transcript case at the unchanged 15-second limit. That case passed
in isolation on a fresh database. One subsequent complete invocation,
with builds finished, passed all 122 without source, assertion or
timeout changes. The underlying timeout cause is unproven; the original
failure remains recorded. A new PostgreSQL startup also failed from
exhausted host semaphore capacity; fresh databases on the already-owned
server were used without removing shared IPC resources.
- Prior source `b8b12199b9` passed 266 distinct cases and carried three
guarded mutation proofs; those older results are not added to this
repair’s case count. Its CI run 40641168596 failed E2E shard 3 when
legacy cancellation hit a one-second PostgreSQL lock timeout. GitHub
initially rejected a retry while the run was active, then accepted one
targeted retry after completion; attempt 2 passed on that same old head.
The lock holder and underlying cause remain unproven. The unchanged Node
policy checker’s older failure on ignored generated Copilot manifests
and passing tracked-input mirror are also retained in the audit; no
workflow, checker, vendor manifest or ignore rule was modified.
- [Current repair and failure
record](https://github.com/paperclipai/paperclip/blob/a4e7ccf976fc5793a8914e4e4861aa080d43a4da/doc/plans/2026-10-25-voice-inbound-cleanup-sleep-audit.md#legacy-terminal-admissions-missing-their-session-binding--october-28).
Current-head CI and Greptile 5/5 gate merge. A completed failing
Ponytail verdict must be fixed; missing or interrupted reviews do not
block, and active pending review waits are capped at 30 minutes under
the maintainer’s explicit instruction. No new full local repository
suite, real provider, Cloud alarm, host/database suspension or savings
claim is made.

## Risks

- The migration distinguishes admission status from physical
termination. Legacy ended guest rejections without proof remain cleanup
obligations. Missing original credentials can keep those old obligations
unresolved.
- Future writers must use the existing notification boundary.
Independent SQL writers and multiple processes need their own wake
contract.
- Pending inbound cleanup now blocks company deletion instead of
allowing cascade deletion to lose the provider obligation. Existing
voice-history deletion restrictions remain. This deliberately includes
an active anonymous inbound ring and an old call whose termination
cannot be verified. An operator must stop ingress and settle the call
before deletion; this slice adds no abandonment override and never
treats missing credentials as proof of termination. This is the retained
cleanup ownership policy, not a claim that every deletion attempt can
complete immediately.
- History retention and report processing still run in the remaining
maintenance lane. This PR does not extend Cloud wake coverage or claim
measured host/database suspension.

## Model Used

OpenAI Codex (GPT-6), with reasoning and code execution. The exact
serving model ID and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 08:30:15 -05:00
DottaandPaperclip 7d8bab773d perf: avoid full JIT and activity scans in Mine unread badges (#17973)
## Thinking Path

> - Paperclip helps people manage AI agents and their work.
> - Mine includes tasks the person created, was assigned, or
participated in.
> - Its unread badge ranks the visible personal top 100 before checking
unread state.
> - Correlated participation and privacy predicates can trigger
expensive PostgreSQL JIT work.
> - This change joins native participant IDs before ranking a narrow
candidate projection.
> - A partial actor index avoids scanning company activity history for
each badge.
> - Local representative scoped reads improve while dense unrestricted
overhead remains explicit.

## Linked Issues or Issue Description

**What existing behavior does this improve?** The Mine unread badge in
the company sidebar.

**Current behavior:** A fresh five-minute window recorded 402 ranking
calls and 31.7 seconds of SQL time. Correlated predicates can spend more
time compiling than executing.

**Proposed behavior:** Build the same company-scoped participant set,
use native issue keys, apply fresh authority before the top-100 limit,
then retain the original canonical rank and second unread query.

**Reason and benefit:** Reduce full JIT compilation and avoid an
unindexed company-history activity scan. Realistic activity-only
participation is included in the measurement.

**Breaking changes:** No API or policy change. This PR adds one
concurrent partial index migration. Related Mine rules in #16356 and
#17756 are preserved.

## What Changed

- Join distinct native participant IDs; guard exact legacy text before
UUID conversion.
- Skip redundant activity participation already covered by
creation/assignment. Sort a narrow eligible candidate projection before
canonical activity ranking.
- Share existing pending-request and user-activity predicates with
ordinary Mine reads.
- Add partial `activity_log(company_id, actor_id, entity_id)` for
user-authored issue activity. Register its idempotent concurrent
migration with the existing top-level/invalid-index-repair runner.
- Add real PostgreSQL coverage for visible top 100,
unread/archive/source semantics, exact IDs, company scope and fresh
grant/private-project access.
- Add a pinned service benchmark with separate pre-index/indexed
databases, identical data, full positional equality, ordinary-call
timings, plans and real migration/cancellation proof.

## Verification

- Fourteen focused PostgreSQL 17 tests pass (16.37 s), including
assembled badge routes and grant revocation. Full workspace typecheck
and build pass on the final production sources.
- The earlier full local suite is incomplete: native PostgreSQL startup
exhausted system semaphores, independently reproduced with empty initdb.
All sixteen failed cases passed with task-owned Docker PostgreSQL 17.
This is not counted as a full local-suite pass. Full Linux CI passes on
final head `f44e46ce580e8345209d7165f57bd8bf696c517c`. Greptile and
Paperclip Review are both 5/5 with zero unresolved comments.
- Baseline `70c864633744cd96c0dc7394d631fc7119cd17d2`; source and DDL
hashes retained. Two warmups and five alternating samples per arm/cell.
Capture and EXPLAIN occur outside timed ordinary service calls.
- All twelve 300k-history cells match complete positional rows in both
SQL phases and final returned IDs. Half the personal tasks qualify
through activity alone; the actor index contains matching and
other-human rows.

| Local service case | Before median | After median | Before / after |
| --- | ---: | ---: | ---: |
| dense / unrestricted / JIT on | 18.33 ms | 25.06 ms | 0.73x |
| dense / scoped / JIT on | 909.30 ms | 221.10 ms | 4.11x |
| sparse / unrestricted / JIT on | 13.64 ms | 5.84 ms | 2.34x |
| sparse / scoped / JIT on | 719.07 ms | 10.79 ms | 66.64x |

Small paths add at most 0.6 ms at the median. Dense unrestricted paths
regress by 6.7–7.7 ms; dense scoped JIT-off is approximately neutral
(218.34→223.24 ms). Sparse JIT-off paths improve too. This is not a
uniform speedup.

The actual baseline fails the compiler-work assertion. The final query
avoids full JIT optimization/inlining and exceeds 1.5x where the old
plan triggers those passes. Small/sparse controls meet the 25% or 2 ms
allowance. Dense no-JIT cases are explicit diagnostics. The optional 60
ms absolute goal still fails at 221.10 ms on dense scoped reads:
substantial privacy-predicate execution remains after compilation is
removed. The failure is retained rather than counted as green.

The actual normal migration runner upgraded 300k activity rows in 668.79
ms including inspection, index build/validation and journal work. Rerun
took 21.83 ms; a deliberately cancelled build left an invalid index, and
retry repaired it in 187.71 ms. Concurrent reads/writes completed; the
final index was 188416 bytes. No migration was run on the hosted
database.

The million-row mixed-participation sparse control also passes all four
visibility/JIT pairs with complete equality: unrestricted JIT-on
13.44→5.76 ms; scoped JIT-on 807.76→11.59 ms. The actor branch visits 36
indexed rows rather than scanning company history. Actual
migration+runner took 255.06 ms; rerun 20.32 ms; cancellation
repair/retry 241.73 ms. Concurrent reads and writes completed without
failures.

Reproduce:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-mine-unread-badge.ts --baseline-ref=70c864633744cd96c0dc7394d631fc7119cd17d2 --history-decoys=240000 --verify-migration --assert-performance --evidence-dir=<directory>
```

Add `--fixture=sparse --history-decoys=940000` for the million-row
control. The default harness starts disposable embedded PostgreSQL and
ignores DATABASE_URL. Recorded evidence used PostgreSQL 17.11 through a
scratch constructor provider because the host native runtime has
semaphore limits. All normal database allocation/migration/client paths
are retained.

Source-only EXISTS/lateral alternatives were measured and rejected after
company scans or sparse regressions. A time bound would lose historical
participation, and existing indexes cannot select a user's activity.
This justifies the exceptional partial index migration. The exact sample
ranges, plans, failures, source hashes, migration footprint and
feedback-loop timing are in
`doc/plans/2026-10-27-mine-unread-badge-query.md`.

## Risks

- Dense unrestricted reads are slightly slower, and dense scoped reads
retain substantial authority work. The PR removes a compilation/lookup
bottleneck rather than all badge latency.
- The concurrent build scans activity history and can wait for
long-lived transactions. Local build/cancellation timings do not
guarantee the duration on larger or slower hosted databases.
User-authored issue activity gains one narrow index to maintain.
- Participation, visibility, ranking and legacy ID semantics must remain
exact; the real database regressions exercise those boundaries.
- Hosted improvement is unverified. The affected ranking SQL averaged
78.8 ms while HTTP badge requests averaged 752 ms without correlated
traces. No merge or deployment is included.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use and local code
execution. The runtime does not expose the exact model ID or context
window to this agent.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 07:20:08 -05:00
DottaandPaperclip 09ad9adce2 feat(server): adopt running remote agents through durable controller checkpoints (#17439)
## Thinking Path

> - Paperclip must keep admitted remote work alive while its main server
is replaced.
> - Runner owns recovery mechanics and exposes opaque checkpoints
through #17097.
> - The server owns company scope, encryption, Stop, tool authority and
execution ownership.
> - A successor must prove current ownership before it adopts the
original execution.
> - Checkpoint bursts must also leave capacity for normal requests and
lease renewal.
> - This PR persists and adopts that authority through the existing
runtime boundaries.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Remote agents retain the original execution during a bounded main-server
outage.

**Current behavior**

A server restart can lose the controller's local state and connection.
An unsafe replacement could repeat a turn or an uncertain mutation.
Checkpoint writes in the request pool can also delay ordinary requests
during an aligned burst.

**Proposed behavior**

Restore encrypted control records from Postgres and adopt the original
runner under a fenced lease. Recover known tool receipts through current
tool authority. Give control checkpoints a shared pool capped at two
connections per database owner.

Depends on #17097. Successors are #17455 and #17456. Qualification and
presentation remain separate review slices.

## What Changed

- Add company/run-bound encrypted checkpoint records and migration 0394.
Keep the SQL idempotent after renumbering against current master.
- Recover startup and expired ownership through native runtime and
heartbeat. Preserve the sandbox lease, admitted runner artifact, harness
configuration, native session and active turn.
- Keep Stop, finalization, preparation ownership and stale-writer
fencing transactional. Preserve graceful detach and natural crash
takeover.
- Recover committed tool receipts through the existing authority.
Reauthorize reads and hold uncertain mutations.
- Bound startup retention with the existing coordinator and preserve
zero idle scans.
- Share at most two checkpoint connections per database client. Keep
application and liveness capacity separate. Close the checkpoint pool
during shutdown; late writes cannot reopen it.
- Count SQL across both application and checkpoint clients. Prove
request/liveness progress, command recovery and failure after pool
shutdown with real Postgres.

## Verification

Project creation recovery now uses the existing project activity receipt
owner. It retains authenticated input normalization, fingerprint checks
and current run authority after the receipt lock. An absent receipt
returns no result before mutation. The new lost-HTTP-response and
no-effect regression uses real API routes and Postgres. The seven
transport cases pass locally. This Mac skips all 20 database cases. On
current source `5de803c2cd`, Linux [job
122593415890](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415890)
executes all 20 project-tool cases successfully (9.480 seconds),
including the committed-project lost-response regression. Its shard
passes 1,627 cases with three unrelated skips. Server typecheck passes.
The transport regression initially has two failures; a subsequent
incorrect expectation about schema defaults was corrected before the
final pass. Ordinary execution adds no checkpoint writes or recurring
work.

Current candidate `5de803c2cd` includes the project-creation receipt
recovery fix and master `2cd747d1b6`. Migration 0394 retains the prior
idempotent checkpoint DDL. Every master SQL file and journal entry is
preserved. Snapshots 0390–0394, migration checks and server typecheck
pass.

The ordinary warm-turn regression fails on its second turn before
`a219ab51ad`. After correction, three completed warm turns select the
correct prior checkpoint and preserve the original session. All 288
provider-wiring cases pass. The control plane, external persistence
callback, identity rotation and filesystem are real; the backend and
database-load seam are scripted. The 13 warm admission/configuration
variants also pass after final master integration. This does not qualify
an underlying harness.

At current source `5de803c2cd`, Linux [job
122593415907](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415907)
passes all 940 native-session-executor cases (61.431 seconds), including
the warm external-checkpoint behavior; the shard passes 2,358 cases with
one skip. Current [Runner job
122593415475](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415475)
passes the native recovery lane: 115 passes and five container-only
skips, including the encrypted restore guard, fresh preparation (63.189
seconds), blocked claim lock (1.226 seconds) and recovery beyond 101
blocked owners (3.897 seconds). The first lane passes 2,215 cases with
one skip. The SQL cost test reports zero idle and streaming SQL for both
output volumes; this is component evidence, not a production capacity
claim. At prior candidate `1ff7bb44f4`, Linux job 122549307215 passes
all 938 native-executor cases, including the three-turn
external-checkpoint regression. Job 122549306469 passes the native
recovery lane with 115 passes and five container-only skips, including
the encrypted-restore session guard, fresh preparation (63.664 seconds),
claim-lock contention (3.961 seconds), and 101-owner recovery (3.824
seconds). This Mac skips the real-Postgres suite. The earlier full local
executor attempt has 921 passes and 16 failures during embedded-Postgres
initialization before assertions. The original fixture collisions and
their corrections remain recorded.

At earlier candidate `8fbe62b41d`, server shard job 122535673841 has
3,876 passes, one failure and one skip in an unchanged expiry-worker
retry/drain timing test. The same server/database tree on qualification
source `b64cac6951` passes all 20 expiry cases and all 3,877 shard
cases. The single diagnostic retry, job 122541891915, passes all 20
expiry cases and all 3,877 shard cases (one skip). No assertion was
weakened. The original failure remains recorded and its exact timing
race is not proven. All Actions checks and legacy status passed on
`8fbe62b41d`, with completed Greptile 5/5 and no open feedback. The next
candidate `1ff7bb44f4` also passed all Actions checks and Greptile 5/5.
Its Paperclip Review remained queued before the new master conflict
required integration. The new candidate needs fresh exact-head CI and
reviews; old checks do not qualify it. Migration generation is
checkpoint-only, all master SQL and journal entries remain unchanged,
and migration checks, server typecheck and all 13 warm-owner variants
pass after integration. The first migration check was blocked by TSX
sandbox IPC; its permitted repeat passes. The first warm-test command
used an incorrect path and ran no tests; the corrected command supplies
the 13-case receipt.

All 16 database pool-lifetime/options/teardown cases pass after granting
loopback permission; the first batch had two EPERM failures and 14
passes. Earlier semantic-tool/HTTP recovery, adoption, encryption-batch,
zero-streaming-write and idle-SQL receipts remain source-attributed in
the [qualification
record](https://github.com/paperclipai/paperclip/blob/2ff66f57da0da35bac1c19a2366ea4b84d87a986/doc/plans/2026-10-22-hot-upgrade-qualification.md).
That record retains earlier assertion, setup and cleanup failures and
their corrections. No failed gate is waived.

The historical 300-session component campaign passes four after phases:
steady read p99 is 107–108 ms at 500 reads/s and 100 writes/s;
aligned-burst read p99 is 103–105 ms after the bounded pool, versus
584–689 ms before. This comparison isolates the persistence corrections.
It does not measure the final stack's hosted active-agent capacity. The
hosted zero-agent API pair leaves SQL and CPU essentially unchanged, but
both arms fail absolute latency gates and candidate peak cgroup memory
fails its gate. The active-agent/viewer matrix, different-artifact
upgrade and rollback remain open.

Run `PAPERCLIP_TEST_DATABASE_URL=<owned-disposable-postgres> pnpm
--filter @paperclipai/server exec vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`.

Previous head `f2db8a6568` had one failed workspace setup gate: [job
122555672441](https://github.com/paperclipai/paperclip/actions/runs/40608984238/job/122555672441)
passes all 4,169 UI tests and 285 CLI cases (one skip), but the company
import/export suite reaches its 30-second health deadline during doctor
startup. Its output does not reach the server-start step. The CLI,
server, database, shared, adapter and plugin trees are identical on
qualification source `a866619d7b`, whose [job
122555886663](https://github.com/paperclipai/paperclip/actions/runs/40609057937/job/122555886663)
passes that fixture and all 286 CLI cases. The one diagnostic retry,
[job
122561057225](https://github.com/paperclipai/paperclip/actions/runs/40608984238/job/122561057225),
passes the formerly failing fixture and all 286 CLI cases. All Actions
checks and legacy status on `f2db8a6568` pass. Its exact-head Greptile
completed at 5/5 with no open feedback. The current integrated head
requires fresh checks and a completed Paperclip Review before merge. The
original failure remains recorded; its exact timing cause is unproven. A
repeated failure requires investigation and repair rather than another
retry.

All current-head CI, legacy status, exact-head completed Greptile 5/5,
resolved feedback and ordinary GitHub merge eligibility remain required
before merge.

The latest integration preserves master’s published Copilot v70 profiles
and every published migration. Checkpoint migration
`0394_conscious_whizzer.sql` was generated with `pnpm db:generate` and
retains the prior idempotent DDL. The Runner vendor rebuild and replay
parity checks pass. Server typecheck and all 13 warm-owner variants pass
after that rebuild. The imported report-lock patch is byte-identical to
master; its intentional patch-context whitespace is retained, and the
feature delta passes whitespace checks. No hosted experiment was run.

## Risks

The pool adds up to two active connections per instance, shared across
agents. Checkpoint delivery can wait behind that bound; it still
requires its commit. Steady component-test Postgres CPU rises by about
1.7 percentage points of a two-CPU allocation. These corrections do not
remove the stack's control and renewal writes. The migration and key
must survive replacement. Fresh-database tests and local load do not
establish different-artifact hosted upgrade or rollback continuity.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools and code
execution. The runtime does not expose a more specific model ID or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 21:39:23 -05:00
DottaandPaperclip 8406047b4e fix(server): schedule join request retention from durable deadlines (#17508)
## Thinking Path

> - Paperclip manages agent companies and their access requests.
> - Pending join requests appear in the attention feed and can age into
automatic archival.
> - The maintenance tick reads those requests even when no archive is
due.
> - Existing request rows already store the activity timestamp needed
for retention.
> - Committed changes can wake a worker that registers the next actual
deadline.
> - This PR gives join-request retention its own bounded worker and
removes its recurring feed scan.
> - Idle and resolved history stop producing retention traffic without
changing access approval.

## Linked Issues or Issue Description

**What happened?**

Every retention maintenance pass reads pending join requests through the
attention feed. Empty, resolved, kept, and future-due states still
participate in recurring discovery.

**Expected behavior**

Committed request and retention changes wake the owner. It waits until
the next actual archive deadline and stops querying when no eligible
request remains. Existing approval and visibility behavior is preserved.

**Steps to reproduce**

Start an idle company with no pending join requests, or with requests
younger than the 90-day archive threshold. Observe the maintenance feed
repeatedly select join_requests. No state change is needed to cause
those reads.

This PR targets master after #17505 merged. Approval retention is
#17497. Other attention source kinds remain separate migration duties.
No duplicate open join-retention PR was found.

## What Changed

- Add a join-request retention owner that preserves pending status,
active-company visibility, matching invite ownership, and keep/archive
controls. Invite expiry or revocation retains the existing feed policy.
- Reuse one `createDecisionRetentionSweep` for approval and join paging,
atomic retention and finite scan cycles that revisit failures despite
continuing arrivals. Eligibility and source-row locks stay with each
owner.
- Register real saved deadlines, rounded upward in SQL to millisecond
precision, with the existing coordinator. Preserve 100-row pages and
thirty-second notification coalescing without idle polling.
- Materialize unseen join state inside the existing authorized
keep/archive/revive transaction, preserving archive choices and source
locking.
- Notify before request inserts and updates in the existing acceptance
and Slack transactions. Use a small SQL-only helper for replay and
approval/rejection updates. Company status, budget pause/resume,
deletion, and retention controls notify this owner too.
- Preserve the approved-only API-key consumption path without a new
wrapper; it cannot produce pending retention work.
- Exclude join requests from maintenance feed collection while retaining
normal UI reads.
- Add an idempotent concurrent partial pending index and prune old
schema snapshots through the existing generator.
- Document producer coverage, boundaries, failure evidence, and
remaining work.

## Verification

Test revision: `df1815fbba`. Normal integration of landed workspace
precision `d23dd19750` has an identical complete source tree. This
remains only the join-retention duty and its shared approval-retention
mechanics, targeting master.

- **55 worker cases pass**, including two new PostgreSQL regressions
that failed before repair: continuing overdue arrivals no longer
postpone a recovered failure indefinitely, while an entire failed page
still allows later healthy rows to progress.
- The nine complete suites passed 341 cases but failed an attention
cleanup hook after its 16,400-run case. After vacuum/analyze of only
that disposable fixture, the full 31-case attention suite passes
unchanged. Together with the original eight passing suites (311 cases),
all 342 distinct cases pass across complete-suite executions. Recovery
is established; the original cleanup slowdown is not claimed explained.
Server typecheck and production build pass.
- Earlier CI at `df1d4d0` failed the unchanged decision-expiry restart
deadline assertion (a thirty-second retry appeared before the saved
expiry). All twenty unchanged expiry cases pass locally, but the CI
timing cause remains unproven. No retry, timeout relaxation or assertion
change is used to dismiss it; fresh final-head CI remains required.

- **Five regressions failed before repair:** three unseen join controls
returned 404, and both owners advertised a microsecond deadline one
millisecond early. All pass. Two old real-producer expectations failed
after the correct rounding change; they now derive exact expected
milliseconds independently from database integer microseconds instead of
truncating a JavaScript Date.
- The initial timing-only concurrency test still passed a missing-lock
mutation. It now observes the actual PostgreSQL wait before releasing
the writer. Production passes; removing the lock fails with zero waiters
instead of one. Source is restored before commit. The archive-first
committed-order case confirms that later activity stays archived until
explicit revival; no broader race guarantee is claimed.
- Generated concurrent/idempotent `0393_open_grandmaster.sql` adds only
the pending join index. All 391 parent journal entries and historical
SQL files remain unchanged. Five generated snapshots remain (0389–0393).
Migration safety/numbering, schema drift and actual apply/replay on six
isolated PostgreSQL fixtures pass.
- Existing lifecycle cases retain real HTTP invite
acceptance/approval/rejection, Slack identity linking, rollback/outer
commit, source visibility, failure rotation, drain/resume and shutdown
coverage. Earlier stack setup and CI failures remain in the owning audit
and stack ledger; they are historical evidence, not this head's
acceptance.

Commands: server Vitest for `join-request-retention-work`,
`approval-retention-work`, `approvals-service`, `attention-service`,
`decision-retention-service`, `budgets-service`, `tool-gateway-service`,
`server-startup-feedback-export`, `scoped-deadline-work`; server
typecheck/build; DB migration checks, snapshot drift and actual
apply/replay. Fresh exact-head CI, security and Greptile/Ponytail 5/5
remain required. No full local repository-suite, deployed suspension or
cost claim is made.

## Risks

- A future supported producer must signal its transaction. Independent
SQL writers and another active server process are outside the
process-local notification contract.
- A writer that wins the source lock first is rechecked before archival.
If archival commits first, later activity preserves the archived state
until explicit revival. Existing invite replay expires before the
retention cutoff and Slack replay deduplicates. Future producers needing
renewed visibility must request revival. Cross-table visibility retains
the existing contract.
- The worker registers only under the existing maintenance enablement
switch. The internal deadline does not add Cloud alarm coverage or relax
conservative sleep blockers.
- New eligibility can wait within the thirty-second aggregate
notification window; known deadlines preempt it.
- Archival preserves pending approval status. Other attention sources
and aggregate maintenance remain separate work.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 20:03:16 -05:00
DottaandPaperclip 553a004926 fix(server): schedule approval retention from durable deadlines (#17497)
## Thinking Path

> - Paperclip manages agent work and should become idle when it owes no
background work.
> - The attention feed archives old pending approvals under the existing
90-day policy.
> - The server rebuilds the feed on each maintenance tick, even when
approval state has not changed.
> - Some approvals have never been read, so retention rows alone cannot
define the work.
> - Approval records and committed eligibility changes define the real
deadline and wake triggers.
> - This PR gives approval retention its own deadline worker and reuses
the existing retention and archive-delivery contracts.
> - The approval lane stops idle scans while other attention sources
remain separate migration work.

## Linked Issues or Issue Description

Stacked on #17490. This PR owns approval retention only. Archive
notification delivery is already separated in #17475. Other
attention-source retention and remaining recurring server duties remain
in the stack inventory. The related-PR search found no overlapping
approval-retention worker migration.

**What existing behavior does this improve?**
Automatic archival of old pending approvals from attention.

**Current behavior**
The shared maintenance tick builds the attention feed for each active
company. That query discovers pending approvals and materializes
retention records before checking their age. An unchanged or empty
approval source is read repeatedly.

**Proposed behavior**
Reconstruct eligible approvals at startup. Schedule the earliest
approval activity timestamp plus 90 days. Wake after committed changes
to approvals, links, issue visibility, company activity, or retention
controls. Stop scheduling when no eligible approval remains. Archive
records through the existing version checks, audit writes, and
notification outbox.

**Reason and benefit**
Idle approval retention should generate no recurring SQL. Unseen
approvals still need eventual archival. A stale candidate must not
archive an approval that was resubmitted while the worker waited for its
source lock.

**Breaking changes**
None intended. The archive age, eligibility, keep controls, audit, and
notification policy remain. Normal attention reads still include
approvals. No Cloud sleep blocker is relaxed.

## What Changed

- Restrict tool-review signals to their existing terminal transition
owners (approve/reject/cancel). Creation/link/revision/resubmit signal
retention only.
- Coalesce retention notifications under one thirty-second aggregate
read budget. Known deadlines still preempt it; no notification or
durable deadline means no scan.
- Add a bounded approval-retention worker. It uses the shared deadline
coordinator and reads pending approvals directly.
- Share first-linked-issue ordering and execution visibility through
`approval-visibility.ts`, used by both the feed and worker. Preserve
actor-specific read filtering and locked archival rechecks.
- Materialize unseen approvals on demand inside manual
keep/archive/revive transactions. Use the saved activity timestamp and
existing versioned state; retain company scoping and rollback on
failure.
- Add awaited pre-write signals to approval, budget, tool, link,
visibility, company, and retention mutation owners. Preserve outer
transaction ownership; keep provider calls outside SQL wrappers.
- Narrow retention inputs to the source fields the existing service
uses. Signal materialization only for inserted or changed state so
unchanged reads converge.
- Exclude approval collection from the maintenance feed before its
query. Other source kinds remain in that sweep.
- Add the pending-only approval deadline index with concurrent,
idempotent creation. Keep the five newest migration snapshots.
- Add real-Postgres lifecycle tests and startup wiring tests. Enable
validated external test databases for existing attention and budget
suites.
- Document producer coverage, quiet conditions, recovery and remaining
scope in `doc/idle-approval-retention.md` and the stack ledger.

## Verification

Current test revision: `dd844f4e8cbbf1db681a774dc31cd552301bef48`,
followed by docs-only `3efda33812`, which received both 5/5 reviews and
passing CI. This candidate normally integrates landed Grok `2adcde162e`.
All runtime and acceptance-test sources are byte-identical to that
tested/reviewed tree; only four independent Runner E2E accounting files
and the ledger differ. Migration 0392 and the parent journal prefix are
unchanged. Local checks are reused for their unchanged source; fresh
final-head CI and reviews are required against master. This remains the
approval-retention duty.

- **315 cases pass** across eight complete suites: approval
retention/service, attention, decision retention, budgets, tool gateway,
server startup and scoped deadlines. All 30 real-PostgreSQL lifecycle
cases pass. Server production typecheck and build pass.
- **Four regressions failed before repair**: unseen young
keep/archive/revive returned 404; the manual keep rollback case could
not reach its update. They now pass. A failed keep update leaves no
retention row or audit; retry succeeds with one audit.
- Both linked-issue orders are checked against the feed: hidden
first/visible later is excluded; visible first/hidden later remains
eligible. The feed and worker use the same owner query, with actor read
filtering preserved.
- Generated concurrent/idempotent migration
`0392_small_frightful_four.sql` adds only the pending approval index.
Every historical parent SQL migration and all 390 parent journal entries
are unchanged. Five generated snapshots remain (0388–0392). Schema
drift, migration safety/numbering and actual fresh apply/replay on five
isolated PostgreSQL databases pass.
- Existing precision, transaction, drain/resume, failure fairness and
shutdown tests remain. Earlier transaction-stub, budget-import,
diagnostic-client and CI failures are recorded in the owning audit and
stack ledger; later passes do not erase them.

- Three additional real regressions failed before the wake repair:
ordinary mutations emitted tool-review signals in both resolution cases,
and sequential tenant writes caused repeated global reads. Tests now
prove terminal approve/reject/cancel signals and one aggregate
reconciliation per thirty-second window with a different company's
pending approval retained. The aggregate budget is stricter than a
separate per-company budget and matches the old default cadence; known
deadlines preempt it.
- The initial coalescing fixture expected the concrete pending-work hold
to clear before reconciliation; it now preserves that hold and measures
SQL. The first broad repair passed 313 cases but failed the existing
parked-review cancel case. The original cancel producer is restored; all
315 now pass without weakening assertions. At old head `5991f257bb`,
security was action-required and Greptile returned service error
`8fb1835c-e564-476b-b802-d5be164ea915`. Fresh repaired-head results are
required, not assumed.

Commands: server Vitest for `approval-retention-work`,
`approvals-service`, `attention-service`, `decision-retention-service`,
`budgets-service`, `tool-gateway-service`,
`server-startup-feedback-export`, `scoped-deadline-work`; server
typecheck/build; DB snapshot drift, migration checks and fresh
apply/replay. Fresh exact-head Greptile/Ponytail and CI remain merge
gates. No full local repository suite or deployed host suspension/cost
claim is made.

Earlier stack verification below is historical evidence at the
explicitly named revisions, not this head's acceptance:

Approved delegated Test-tab calls now evaluate caller and target
policies in the existing claim transaction. Real HTTP regressions first
showed a missing caller charge and shared-limit self-rejection. All 78
connection-management cases and 92 gateway cases now pass on the owning
branch, along with server production typechecking. All 78
connection-management cases also pass at combined tip
`d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover
shared charges, separate charges, rollback on denial, duplicate
approval, and rejection of a later authorized call. Intermediate
counter/identity fixture failures and a reused-database Slack collision
are recorded in [the owning
audit](https://github.com/paperclipai/paperclip/blob/20557d2c3dd0949c734eaa8095fd78b9b07d7fd5/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes).
The earlier combined checks below remain evidence for their stated
revisions. Current-head CI and review remain required.

Earlier published head: `20557d2c3dd0949c734eaa8095fd78b9b07d7fd5`. This
remains a single-duty PR stacked on #17490.

- Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5`
with normal merges. Every branch passed generated-schema checks and its
target-branch migration-order gate. Existing SQL is unchanged; the 25
stack migrations follow master’s new migration.
- Duty-specific producer coverage, lifecycle tests, and earlier failure
records:
[2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/76a49810c7ff403ae7720bef0d453747bd844731/doc/plans/2026-10-24-recurring-server-sleep-stack.md).
- Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355
startup/cancellation/recovery/dispatch/preservation tests**, **75
connection-management tests**, and production server typechecking pass.
These combined checks do not mean every intermediate PR independently
ran the full repository suite.
- Real master-to-stack database upgrade passes: **379 unchanged prior
migration receipts plus 25 new migrations**, with the gateway
responsibility fixture preserved.
- Owning-branch checks: 84 runner installation/settlement cases plus
runner typechecking and generated TypeScript/Rust profile agreement; 253
GitHub review workflow cases. Stop-attribution and held-worker readiness
regressions failed before their fixes.
- Failure accounting: the initial combined run passed 349 cases and
failed six idle assertions in a reused preservation database containing
old cancelled runs. Repeating the combined command with a fresh
preservation database passed. Earlier invalid project selection, merge
import/type errors, and the interrupted full local suite are not counted
as passing checks.
- **Latest-head CI and review remain required.** Prior CI findings
include GitHub idle observation, route teardown, attention cleanup,
touch-picker E2E, workspace health, and voice-cleanup timing; local
passes do not erase those failures. No full local repository-suite,
deployed sleep, fleet cost, or live Copilot qualification claim is made.

## Risks

- Signals cover the application database owner and its child pools.
Independent SQL writers and other active processes need an explicit wake
integration. Startup reconstructs committed durable state.
- The source row lock prevents stale approval-activity archival. It does
not add cross-table serializability for concurrent link, visibility, or
company-policy changes.
- Newly changed eligibility can wait up to the thirty-second
notification window, matching the former default maintenance cadence.
Saved deadlines still preempt; no new idle polling is added.
- Failed records retain recovery debt. Bounded cursor progress lets
later rows proceed without dropping failures.
- The shared retention transaction now groups materialization and
activity-version updates. Existing attention and retention consumers are
tested against PostgreSQL.
- Kept, archived, resolved or inactive-company approvals can become
quiet, but their supported mutations must retain their wake
notifications.
- The local scheduler cannot wake stopped compute. This PR neither adds
approval deadlines to Cloud's external alarm contract nor claims
measured host or database suspension.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 19:23:24 -05:00
DottaandPaperclip 2adcde162e fix(server): wake Grok dispatch from committed reservations (#17490)
## Thinking Path

> - Paperclip manages agent work and should become idle when no work is
owed.
> - Grok routine dispatch already persists each outbound intent before
delivery.
> - Its dispatcher starts a one-second timer whenever public MCP is
configured.
> - That timer queries settings even when Grok is disabled, and scans
dispatches when enabled but idle.
> - The reservation transaction and feature-setting transaction already
define when work can become available.
> - This PR connects those writes to the shared coordinator and retains
the Grok service's claim and delivery policy.
> - The worker becomes quiet without adding a queue table or replaying
uncertain sends.

## Linked Issues or Issue Description

Stacked on #17486. This PR owns only Grok routine dispatch. It does not
include decision retention or other remaining maintenance owners.
Related open Grok PR #17278 concerns Runner credential-lease fixtures,
not this dispatcher. The duplicate search found no overlapping
dispatcher migration.

**What existing behavior does this improve?**
Grok routine dispatch recovery in the server.

**Current behavior**
With public MCP configured, the dispatcher reads instance settings every
second, including when Grok is disabled. With Grok enabled, it scans for
prepared dispatches every second. The baseline regression observed three
queries in 3.2 seconds with Grok disabled.

**Proposed behavior**
Reconcile once at startup. Wake from committed dispatch reservations and
experimental-setting changes. Schedule another pass only while prepared
dispatches remain or reconciliation failed. Disabled and settled workers
perform no recurring SQL or provider calls.

**Reason and benefit**
Remove unnecessary tenant database activity while preserving dispatch
recovery, current authorization, provider outcome semantics, and
shutdown ownership. This follows the existing sleep-aware maintenance
direction; ROADMAP review found no competing dispatcher redesign.

**Breaking changes**
No public API or provider payload change. The internal dispatcher start
method receives the shared coordinator; that coordinator now owns
shutdown. Sending, accepted, unknown, and terminal dispatches remain
non-replayable. This does not extend Cloud wake coverage.

## What Changed

- Register the Grok worker with the existing coordinator; remove its
independent timer.
- Signal reservation and experimental-setting writes before commit.
Nested writes follow their outer transaction.
- Preserve ten-row batches, durable send claims, current authority
checks, and existing cancellation/uncertainty policy.
- Join admitted work during shutdown and leave later rows for restart.
- Add a concurrent, replay-safe partial index for prepared dispatch
lookup. Keep only the five newest generated snapshots.
- Reuse the existing real OAuth/agent/routine fixture for lifecycle
tests and add a validated external PostgreSQL test option.
- Document producers, recovery, quiet conditions, and remaining
host-sleep limitations.

## Verification

Test revision: `6d81746fe260ee879442e7e0c879cfe88c250c8e`, followed by
docs-only head `679f523d78`, which received both 5/5 reviews and passing
CI. This candidate normally integrates landed target `8dfed893d8`. Its
entire runtime and acceptance-test sources match that reviewed/tested
tree; only four independently merged Runner E2E accounting files and the
ledger differ, preserved exactly. Migration 0391 and the parent journal
prefix are unchanged. Local checks were not repeated on unchanged
runtime; fresh final-head CI and reviews are required.

- **273 cases pass** in nine complete suites: real-PostgreSQL dispatch
lifecycle, Grok qualification, server startup, scoped deadlines and all
five instance-settings suites. Server typecheck and production build
pass with the updated Runner vendor.
- The baseline disabled-feature regression failed with three SQL reads.
Existing lifecycle cases cover first reservation, restart,
non-replayable history, re-enable, outer commit/rollback, drain,
failure/retry, concurrent enqueue, paging, foreground send race, expiry,
settings commit/rollback, and shutdown.
- Generated migration `0391_third_roulette.sql` adds only the
concurrent/idempotent prepared-dispatch index. All 389 parent journal
entries are unchanged; five generated snapshots remain (0387–0391).
Migration safety/numbering, schema drift and actual fresh apply/replay
on two isolated PostgreSQL databases pass.
- The earlier 259-case selection and 40,000-row generic-plan probe
remain historical evidence: both unchanged queries used the partial
index, six shared buffers including the routine join, 0.032ms / 0.025ms
local execution. No new deployed suspension or cost claim is made.
- Earlier browser CI failed during agent-chat fixture restoration (lock
timeout/HTTP 500), followed by a test seeing the experiment left
enabled. Those failures remain recorded in the owning audit; they were
not dispatcher assertions. Later pre-reconciliation `a74305e2b4` checks
were terminal and green. Fresh CI and both 5/5 bot reviews are required
for this head.

Commands: server Vitest for `grok-dispatch-work`, `grok-bot-routine`,
`server-startup-feedback-export`, `scoped-deadline-work` and the five
`instance-settings-*` suites; server typecheck/build; DB migration
safety/numbering, snapshot drift and fresh migration apply/replay.

## Risks

- Signals are local to the shared database owner and its child pools.
Independent SQL writers and other replicas need explicit wake
integration. Startup recovers committed prepared rows.
- Disabled Grok retains prepared rows and resumes after a committed
settings change. This does not relax Cloud's binding/assignment/hold
guards.
- Existing send-error cancellation policy is preserved. Sending or
unknown outcomes are not reset or automatically replayed; this is not an
exactly-once delivery change.
- Shutdown awaits the current send within existing transport timeouts
and stops before the next row.
- The index adds write/storage cost but bounds queries independently of
completed history. Existing migration execution runs concurrent index
creation outside a transaction.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 19:07:53 -05:00
DottaandPaperclip e9e84bb8a1 fix(server): recover decision effects from saved claim deadlines (#17482)
## Thinking Path

> - Paperclip manages agent work and the decisions that authorize it.
> - A committed decision can leave effects unfinished after an
interrupted execution.
> - Recovery must preserve that debt without scanning settled history
forever.
> - Claims already store the state and time needed to schedule recovery.
> - A failed claim must get another attempt even while newer claims
arrive.
> - Malformed claims need a durable failure outcome instead of an
endless retry.
> - This PR uses committed signals, saved deadlines, and bounded
recovery rotations.

## Linked Issues or Issue Description

**What happened?**

Periodic decision maintenance queries for unfinished effects even when
none remain. A cursor that only wraps after a short page can postpone an
old failure indefinitely under continuous new work. Missing saved
deciders or chosen options can also leave non-executable claims scanning
forever.

**Expected behavior**

Open decisions and completed history produce no recurring
effect-recovery SQL. Committed claims wake recovery, and restart
reconstructs their deadlines. Genuine failures retain retries. Known
malformed outcomes settle with an audit and preserve completed effects.
Recovery must not invent authority, repeat committed effects, or lose
continuation delivery.

**Steps to reproduce**

Create an old failing claim with a one-row recovery page, then keep
adding newer due claims. Before the rotation fix, the old failure is
never revisited. Separately, recover a claim with a missing decider or
chosen option: the old implementation skips or throws on every attempt
without settling it. Creating an ordinary open decision also caused an
unnecessary effect scan.

Stacked on #17479. Other decision expiry, continuation, target and
retention duties have separate PRs in the stack.

## What Changed

- Coalesce sequential committed notifications for up to one second
without delaying already saved deadlines. This bounds instance-wide
recovery reads and adds no idle polling.

- Register effect recovery with the shared scheduler and
transaction-aware notifications. Derive deadlines from the existing
claim timestamp and recovery grace.
- Remove effect recovery from recurring aggregate maintenance when the
worker owns it. Preserve the explicit combined recovery entry point.
- Page within a frozen final timestamp/ID and retain PostgreSQL
precision at both boundaries. Failed claims and healthy later claims
both make progress.
- Load the current claim and derive recovery authority inside the shared
execution flight. Preserve foreground joining, current authorization and
per-effect locks.
- Recheck known malformed claims under a row lock. Save a failed
outcome, or partial when an executed effect receipt exists, with
`effectRecoveryError` and an atomic `decision.recovery_failed` audit.
Publish activity after commit and use normal continuation delivery.
- Avoid effect-recovery notifications for open-decision creation. Claim
and settlement writes retain their existing notification boundary.
- Add generated migration `0390_salty_killraven.sql`, the concurrent,
idempotent partial index for running-claim lookup. No queue table is
added.
- Cover quietness, transaction and lifecycle boundaries, retry fairness,
malformed outcomes, and concurrent repair. Document the contract in
`doc/idle-decision-effect-recovery.md`.

## Verification

Current candidate normally integrates landed expiry `3421366903`,
including expiry-only creation signals and all 21 independent master
changes. The test revision is
`dba8a79d1e1a5b97b8b5f6dac0f7c7837536531a`; the publication adds only
this evidence documentation.

- **143 current cases pass** across all 26 recovery, 54 decision, 20
expiry, 42 startup and one scoped-deadline cases. Server types/build
pass with the rebuilt Runner vendor (1,524 inputs and 1,563 outputs).
These results follow the landed dependency reconciliation; prior 69- and
141-case results remain historical evidence for their stated revisions.
- Generated migration `0390_salty_killraven.sql` adds the existing
running-claim partial index through the concurrent executor. All 388
parent journal entries are unchanged; five generated snapshots remain.
Schema drift, numbering/safety and actual migration apply/replay pass.
- Existing regressions exercise failed-claim rotation under sustained
arrivals, malformed decider/option outcomes, atomic audit failure,
row-lock repair races, current authority, foreground effect joining,
coalesced tenant notifications, transaction commit/rollback,
drain/resume, paging with PostgreSQL timestamp precision, and
shutdown/restart. Completed effects are not repeated.
- The first integrated run passed 140 cases and failed an overlap test
that assumed the effect deadline was the scheduler's earliest deadline.
Continuation coalescing now creates an earlier registered wake. The test
services that earlier wake before asserting the original effect grace
deadline; it still requires one shared effect flight, two lock attempts
including the test gate, one committed comment and one continuation. No
production timing was relaxed.
- Earlier red/green fairness, malformed-claim, notification
amplification and fixture failures remain recorded in [the owning
audit](doc/idle-decision-effect-recovery.md). Extra upstream
embedded-only tests attempted during parent integration could not
initialize locally; CI must exercise that environment.

- Previous candidate `f54a707d74` received both 5/5 reviews but its CI
server shard 6 was interrupted (run 40597921712). The step was
cancelled, with no failing assertion or suite summary, and the verify
gate also failed. No retry or passing claim was made for that run. The
necessary parent repair is included here; fresh CI and both reviews are
required.

Commands: server typecheck/build and Vitest for
`decision-effect-recovery-work`, `decisions-service`,
`decision-expiry-work`, `server-startup-feedback-export`, and
`scoped-deadline-work`; DB migration checks, schema snapshot drift and
real migration apply/replay. No full local repository-suite or deployed
sleep/cost claim is made. Fresh current-head CI and Greptile/Ponytail
are required before merge.

## Risks

Missing deciders and missing chosen options now produce durable failed
or partial outcomes instead of remaining running forever. Their audit
and receipts remain available; no effect executes under invented
authority. Only these known malformed states are settled this way.
Database, audit, and other execution failures retain retries.
Continuations retain their existing at-least-once delivery contract.

The supported writer model remains process-local notifications plus
restart reconstruction. Independent SQL writers and multiple replicas
gain no new wake protocol. Cloud admission and durable external alarms
are unchanged. These tests establish worker behavior, not deployed host
suspension or fleet savings.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 18:24:55 -05:00
DottaandPaperclip 3421366903 fix(server): schedule decision TTL expiry from saved deadlines (#17479)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Open decisions have a saved expiry time.
> - The main maintenance tick scans those times even when none is due.
> - Committed creation and resolution can maintain the actual next
deadline.
> - The existing scheduler can run expiry only when that deadline
arrives.
> - This pull request gives decision TTL expiry its own worker.
> - Terminal history becomes quiet while expiry and continuation
delivery remain recoverable.

## Linked Issues or Issue Description

Targets master after #17478 merged. This is slice 10 of the recurring
server work migration. It owns decision TTL expiry. Target invalidation,
interrupted effects, and retention remain separate migrations. The full
ledger is in `doc/plans/2026-10-24-recurring-server-sleep-stack.md`.

**What existing behavior does this improve?**

Time-based decision expiry in the server.

**Current behavior**

The combined decision maintenance sweep queries expired open decisions
on each tick. It mixes that query with target invalidation, effect
recovery, and continuation delivery.

**Proposed behavior**

Creation, bundle creation, and existing decision updates notify the
shared scheduler. A dedicated worker runs at the first saved open
expiry. Resolving an open decision removes its deadline. The combined
tick opts out of TTL selection while its other duties remain intact.

**Reason and benefit**

A future decision no longer requires repeated TTL queries before its
deadline. Terminal history and an empty decision table leave no
recurring TTL work. No queue table is added.

## What Changed

- Add an expiry owner for bounded due selection, next-deadline lookup,
and expiry settlement. Reuse it for expiry-on-use and the existing
target expiration transition.
- Use the central expiry-only creation notification before insert,
including bundles and caller-owned transactions. Update and terminal
transitions continue to notify expiry and continuation.
- Register the enabled-heartbeat TTL worker before decision
housekeeping. Preserve disabled-heartbeat behavior and the public
combined sweep's default batch limit.
- Honor `PAPERCLIP_DECISIONS_SWEEP_BATCH_SIZE` (default 100). Keep a
fair expiry/ID cursor across failures. Stop between candidates on
shutdown and reconstruct saved work after restart.
- Recheck open status and due TTL during settlement. Merge metadata and
persist activity atomically with expiry. Publish activity and deliver
continuation after commit. Handle exact timestamp equality.
- Continue successful overdue batches after one second through the
existing deadline adapter; preserve thirty-second failure retry and all
default callers.
- Add generated migration `0389_brave_hulk.sql`, an idempotent
concurrent partial index on open expiry and ID. Preserve five generated
snapshots.
- Add 20 real-PostgreSQL lifecycle cases and startup ownership
assertions. Document contracts and limitations in
`doc/idle-decision-expiry.md`.

## Verification

Current candidate: `aa03d2c4c25af4e6c2acf49d3160bbaf6ada030b`. This PR
targets master after #17478 merged. Creation now signals only its real
expiry obligation; terminal updates preserve continuation delivery.

- **117 current cases pass** across all 20 expiry, 54
decision/continuation, 42 startup and one scoped-deadline cases. Server
typecheck/build pass. Separate migrated PostgreSQL fixtures exercise
real transaction and notification behavior.
- Single and bundle creation each failed a regression before the
expiry-only signal repair: open rows unnecessarily notified continuation
recovery. Both now arm their saved expiry without a continuation signal.
Cancelling each row afterward still signals and delivers continuation
normally. The distinction is in the existing central notification owner;
there are no per-route flags or counters.
- **Three new real PostgreSQL regressions failed before repair:** a
configured batch of two was ignored, a successful 101-row backlog waited
for the thirty-second error retry, and an activity insert failure left
the row expired without its audit. All three pass. The audit failure
rolls back expiry; retry records one activity, preserves unrelated
metadata and wakes once.
- The shared deadline adapter has optional `continuationMs`. Only this
owner opts into one-second successful overdue continuation. Other
callers keep the previous default, and errors still retry after thirty
seconds. The archive consumer passes with the option absent.
- Migration `0389_brave_hulk.sql` is regenerated from the parent's
metadata. Parent SQL/journal entries are preserved, five generated
snapshots remain, schema drift and migration numbering/safety pass.
Fresh migration apply and replay pass.
- Lifecycle coverage includes first producer activation, future-deadline
quiet, terminal history, exact equality, commit/rollback, restart,
drain/resume, failed expiry and wake delivery, fair paging through
failed rows at PostgreSQL timestamp precision, shutdown and recovery.
- Failure accounting: the first full repair selection passed 113 cases
and failed a startup assertion that observed a pending follow-up
notification before reconciliation. The assertion now waits for the
exact saved future deadline, then still proves no early expiration.
Earlier fault-fixture and timestamp-cursor failures remain recorded in
[the behavior audit](doc/idle-decision-expiry.md). Extra upstream
embedded-only tests attempted on the parent could not initialize
locally; CI must cover that environment.

Commands: server typecheck/build and Vitest for `decision-expiry-work`,
`decisions-service`, `decision-retention-service`,
`server-startup-feedback-export` and `scoped-deadline-work`; DB
migration checks, snapshot drift and actual migration apply/replay. No
full local repository-suite or deployed sleep/cost claim is made. Fresh
current-head CI and Greptile/Ponytail remain required before merge.

## Risks

- Expiry uses `expiresAt <= now`; the old combined loop skipped exact
equality. The deadline boundary is now covered explicitly.
- TTL expiry records the TTL reason. Target-driven expiry preserves its
own reason. When both causes apply, the first terminal transition wins.
- Supported producers share the application DB signal owner. Independent
SQL writers or multiple active processes need a separate notification
contract. New deadline-edit APIs must use the existing helper.
- The index requires top-level concurrent migration execution. It is
registered with the migration owner and replayed in verification.
- Future in-process deadlines are not external Cloud alarms. Existing
sleep guards remain conservative. This PR does not claim host/database
suspension or measured savings.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 18:10:36 -05:00
DottaandPaperclip a63d19816f fix(server): wake decision continuations from committed changes (#17478)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A completed decision can owe a wake to the agent that proposed it.
> - The decision row already stores the pending continuation.
> - The main maintenance tick scans this debt even when no continuation
is ready.
> - Committed changes can activate recovery and leave settled history
quiet.
> - This pull request gives continuation delivery its own worker and
shares foreground delivery ownership.
> - Pending work stays recoverable without an unconditional continuation
scan.

## Linked Issues or Issue Description

Targets master after #17475 merged. This is slice 09 of the recurring
server work migration. Each PR owns one reason for background work. The
remaining inventory is in
`doc/plans/2026-10-24-recurring-server-sleep-stack.md`.

**What existing behavior does this improve?**

Decision continuation delivery and recovery in the server.

**Current behavior**

Foreground paths send a wake and clear a pending flag. The combined
decision sweep scans pending continuations on every maintenance tick. It
can overlap foreground delivery.

**Proposed behavior**

Terminal decision writes notify a dedicated worker through the existing
transaction boundary. Foreground and recovery calls share one in-flight
delivery per decision and DB owner. Recovery re-reads readiness, pages
past failures, and retries only unresolved debt or errors.

**Reason and benefit**

Open decisions, running effects, and delivered history no longer cause
an independent continuation scan. No queue table is added. Decision
expiry, target invalidation, interrupted effects, and retention remain
separate duties.

## What Changed

- Extract continuation readiness, delivery, and paged recovery into the
decision owner. Preserve the wake payload and at-least-once crash
behavior.
- Signal the ten decision update sites before SQL in their existing
transaction scope. Keep effects and wake calls outside those
transactions.
- Explicitly create the continuation obligation on all three
archive-proposal failure settlements, including recovered running rows
without inherited pending metadata.
- Serialize concurrent decision effect execution through final
settlement. Coalesce foreground and worker wakes. Re-read the durable
flag before sending. Fence effect completion to running state so
competing completion cannot restore acknowledged debt.
- Expose an opaque DB work-scope key. Child pools and nested
transactions share that identity; independent roots remain separate.
Existing signal consumers keep their behavior.
- Bound each recovery pass to 100 candidates, retain a fair cursor
across failures, and stop between deliveries on shutdown. Forward the
existing AbortSignal through the shared deadline adapter.
- Register enabled-heartbeat continuation recovery before decision
housekeeping. Remove only its independent scan from the main tick.
Preserve the public combined sweep and disabled-heartbeat behavior.
- Add generated migration `0388_ambitious_micromax.sql`, an idempotent
concurrent partial index for pending continuations, and keep five
generated snapshots.
- Add real producer, transaction, failure, concurrency, paging, quiet,
and shutdown tests. Document ownership and remaining limits in
`doc/idle-decision-continuations.md`.

## Verification

Current candidate: `18f682863cadb91281a86b4da2553a4579ee054a`, normally
integrated with landed archive squash `5785090ff1`. All 31 independent
incoming master files are preserved exactly, including
controller-authority recovery. All 97 decision/startup/deadline cases
plus server types/build pass after the required Runner vendor rebuild.
The preceding full five-suite selection passed 293 cases; the
continuation owner behavior is unchanged by master reconciliation.

- **97 cases pass at this master integration** across decision, startup
and scoped-deadline suites, plus server types/build. The preceding
parent candidate passed the full 293-case
decision/gateway/archive/startup/deadline selection; those results
remain evidence for that revision. All selected cases executed with
separate PostgreSQL fixtures where needed.
- Tenant-driven global scan amplification reproduced six sweeps for six
sequential real cancellations. Existing thirty-second notification
coalescing now permits one recovery sweep per window while retaining
pending work. The first green fixture wrongly expected the intent hold
to clear before coalesced reconciliation; the hold remains and quiet is
checked after settlement. Foreground delivery stays immediate.
- The archive contention test uses a signaled dedicated
single-connection read and two FIFO SQL round trips instead of a fixed
delay. Removing effect coalescing makes it fail with two contender
reads; restoring production code passes. No production test hook was
added.
- **Three new real PostgreSQL regressions failed before repair**:
recovered archive proposals with invalid signatures, mismatched
manifests or archive execution failures and no inherited pending flag.
Each now persists failure, wakes once, acknowledges continuation
delivery and remains quiet on replay. Running-state fencing is
preserved.
- The unmerged continuation index is regenerated as
`0388_ambitious_micromax.sql` after parent migrations 386/387. All
parent journal entries are unchanged, five generated snapshots remain,
numbering/safety and snapshot/schema-drift checks pass. Fresh database
migration apply and replay pass; the concurrent index is valid.
- Existing lifecycle coverage includes foreground/worker coalescing,
child pools, transaction commit/rollback, failed acknowledgement,
restart, drain/resume, shutdown and fair 100-row recovery. Earlier
bounded-pass and concurrent archive-result regressions and their repairs
remain recorded in [the owning behavior
audit](doc/idle-decision-continuations.md).
- Failure accounting: the first combined command used an incorrect
retention fixture variable and skipped 15 cases. The corrected separate
external-PostgreSQL retention run passed all 15. The historical nine
embedded-only work-signal cases were not executed by this selection.
Extra upstream embedded-only suites attempted on the parent could not
initialize locally; they remain CI verification, as recorded in the
parent PR. Earlier reused-fixture, assertion and compile failures remain
in the behavior audit and stack ledger; they are not erased by later
passing checks.

Commands: `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter
@paperclipai/server build`, server Vitest for `decisions-service`,
`decision-retention-service`, `server-startup-feedback-export` and
`scoped-deadline-work`; `pnpm --filter @paperclipai/db
check:migrations`, migration apply/replay, and the generated
snapshot-drift test. Local targeted evidence does not establish a full
local repository run, deployed sleep or fleet savings. Fresh exact-head
CI and both automated reviews remain merge gates.

## Risks

- Notification-driven recovery remains global but is coalesced to thirty
seconds. This bounds request-driven scans; it does not introduce
company-scoped selection. Foreground delivery remains immediate.
- Delivery remains at least once. A crash or acknowledgement failure
after accepting a wake can send it again. Local coalescing is not a
cross-process claim.
- Supported writers share the application DB signal owner. Independent
SQL writers or multiple active server processes need a separate wake
contract. Wake-producing service methods do not gain a new guarantee
when called inside an uncommitted caller transaction.
- The concurrent index needs top-level migration execution; it is
registered with the existing migration owner and can be replayed safely.
- No Cloud sleep blocker or external alarm coverage changes. Worker
quiet does not prove deployed app or database suspension.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 17:44:51 -05:00
DottaandPaperclip 5785090ff1 fix(server): wake archive notification delivery from committed changes (#17475)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Auto-archive writes an existing notification outbox in its archive
transaction.
> - The maintenance sweep polls that outbox even when it is empty.
> - The committing producer can wake delivery. Saved rows and claim
timestamps can restore unfinished work.
> - This PR separates delivery from retention scanning and uses the
existing coordinator for retries and deadlines.
> - Settled delivery becomes quiet while archive and notification
behavior remain intact.

## Linked Issues or Issue Description

**What happened?**

The retention maintenance sweep calls archive notification delivery on
every tick. An empty outbox and retained delivered history still cause
database work.

**What did you expect?**

Committed auto-archive should wake delivery. Pending work and existing
claims should retain retry or expiry deadlines. Settled history should
cause no recurring delivery SQL.

**Steps to reproduce**

Start a configured instance with an empty or fully delivered archive
outbox. Leave it idle. The old maintenance tick still checks the outbox.

This is slice 08 of the sleep migration. It remains stacked on
projection repair #17468. Retention eligibility and scanning, decision
TTL expiry and decision continuation recovery have separate owners and
PRs.

## What Changed

- Notify archive delivery before the existing outbox insert in the
auto-archive transaction. Preserve outer transaction commit, rollback
and uncertainty handling.
- Register a delivery worker with the existing coordinator. Remove
notification delivery from retention sweeps.
- Keep claim recovery, selection, company/agent grouping and
timestamp-fenced settlement in the delivery service. Keep the existing
service API and payload.
- Count a notified group only when it settles rows. Use one timestamp
for failure reset.
- Add generated concurrent migration
`0387_boring_professor_monster.sql`. Its partial index matches pending
order: `last_attempt_at NULLS FIRST, created_at, id`. Add no table.
- Test real startup registration and prove a retention tick does not
send notifications. Preserve the disabled-scheduler no-notification
contract.

## Verification

Current candidate: `457c70fc6acbd681fcccd90d08b6391efb202bce`, normally
merged with pinned master `8161f2a813` after projection #17468 landed.
All 39 independent files from that master integration are preserved
exactly. Archive owner behavior and migration 0387 are unchanged.

- **58 archive/startup/scoped-deadline cases pass** after master
reconciliation. Server typecheck and build pass. The preceding
projection-parent integration passed all **239**
gateway/projection/archive/startup/deadline cases.
- All 385 parent journal entries remain an exact prefix, with own
migration 0387 appended; the five newest generated snapshots remain.
Generated index apply, replay, schema drift and real pending-order query
measurements passed in the prior owner verification.
- Failure accounting: the broader local command also attempted six
upstream heartbeat-session-preparation and 55 native control-plane
cases. The embedded PostgreSQL runtime was unavailable: the six
heartbeat cases skipped, and native setup failed before its 55 cases
could execute. This is an incomplete additional local check, not a
passing full run. The owning 58 cases passed with the existing external
fixture where applicable. CI must exercise its supported embedded
environment.
- The first independent-master comparison used the shared mutable remote
ref, which another worktree advanced during verification. Pinning the
actual merged master SHA confirms all 39 independent files match
exactly. No incoming source change was dropped.

The behavior audit in `doc/idle-decision-archive-delivery.md` records
earlier fixture failures, the stale-claim regression and the index
probe. Local tests do not prove deployed suspension or fleet savings.
Fresh current-head CI and both reviews remain required.

## Risks

- Delivery remains at least once. A wake accepted before an
acknowledgement is lost can be retried.
- Each pass keeps the existing 500-row limit. Failed or remaining
pending work retains the existing thirty-second retry. Saved delivery
claims retain their five-minute deadline.
- Auto-archive is the existing outbox producer. Manual archive/revive
and reviewed-manifest actions do not gain notifications.
- Signals use the existing single-active-writer database owner.
Independent SQL or another active server needs a wake contract.
- Concurrent index creation uses the existing top-level executor and
invalid-index retry path. Existing SQL and migration receipts are
unchanged.
- Cloud sleep guards and external-wake coverage do not change. Local
worker quiet is not proof of database or host suspension.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge


The final projection efficiency parent is integrated normally. All 239
affected gateway/projection, archive, startup and scoped-deadline cases
pass, with server typecheck/build. The archive owner behavior and index
remain unchanged. Fresh reviews and CI are required on this parent
integration.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 17:26:54 -05:00
DottaandPaperclip 8161f2a813 fix(server): wake tool-review card repair from committed changes (#17468)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Tool requests and invocations store the execution outcome. Review
cards show that outcome.
> - The maintenance tick repeatedly joins these tables even when every
card matches.
> - Committed changes can wake repair. Existing rows can restore work
after restart.
> - A large company backlog also needs a bounded pass so other companies
and shutdown can proceed.
> - This PR uses the shared coordinator for scoped repair, continuation
and failure deadlines. Settled history becomes quiet.

## Linked Issues or Issue Description

**What happened?**

The aggregate maintenance tick scans tool-review projections on every
tick. Matching terminal history still causes database work. A company
with many stale cards can also hold one worker in an unlimited drain.

**What did you expect?**

Relevant writes and startup should wake repair. Failed or unresolved
cards should retain recovery. Matching history should cause no recurring
SQL. Each normal worker pass should honor drain and shutdown.

**Steps to reproduce**

Start an instance with completed tool requests and matching cards. Leave
it idle. The old maintenance tick keeps querying for mismatches. A large
mismatch backlog also drains without a row limit or cancellation check.

This is slice 07 of the sleep migration. It now targets master after
#17458 landed. Related owners are pending expiry #17453 and approved
recovery #17454. Their execution duties remain separate from card
repair. The prior open-PR search found no duplicate projection
migration.

## What Changed

- Register company-scoped projection repair with the existing delivery
coordinator. Remove its recurring aggregate-tick scan.
- Signal through the existing review-write transaction owner and linked
interaction mutations. Ordinary unlinked confirmations do not wake
repair.
- Keep interpretation, locks, current-state checks, activity and
continuation callbacks in the gateway. A repair never repeats a provider
tool call.
- Limit each normal worker pass to 500 rows. Check abort and admission
between rows. Save company progress through truncated and failed passes
so blocked early pages cannot hide later cards.
- Schedule truncated work after one second. Back off unresolved cards
from thirty seconds to five minutes; fresh committed hints reset the
delay. Clear obsolete cursors after empty reconstruction.
- Skip repeated company discovery for scoped hints and the extra
candidate query on an empty sweep with no saved cursor. Attempted or
resumed passes still verify remaining mismatches because reflection
returns no settlement result.
- Add generated migration `0386_steady_epoch.sql` for the existing
`(company_id, interaction_id)` lookup. Use the existing concurrent-index
executor. Add no table.

## Verification

Current candidate: `d4706fbf2a04a00ade30181eb1bd9322e8455818`. All 222
cases in the full gateway/projection/interruption, startup and
scoped-deadline suites pass without skips after the query/retry
efficiency repair; server types and build pass. A single committed
company hint with matching retained history and a separate empty direct
sweep each perform one candidate query. The unresolved-card regression
failed before repair and now verifies backoff through the five-minute
cap, prompt producer activation and quiet after settlement. An initial
assertion incorrectly assumed an entire approval emitted one wake; its
separate real writes require isolating the single-hint measurement.

The earlier master integration `fe6e10629b` additionally established the
following evidence before this efficiency follow-up:

- **320 cases pass** across full gateway/projection/interruption,
startup, scoped-deadline, narrowed identity-read and authenticated
gateway HTTP suites. The HTTP selection includes all 81 cases.
- **Six real PostgreSQL regressions failed before their fixes:** pass
bounds, abort, drain, progress past unprojectable pages, progress past
failed pages, and first repair after empty reconstruction. The complete
owning selection passes all 220 cases.
- Server production typechecking and build pass. Generated index
migration, snapshot drift and numbering/safety checks pass. The five
newest snapshots remain; master journal entries are unchanged.
- A real missing-link query on 30,000 history rows changed from a
sequential scan that filtered all 30,000 rows and touched 429 buffers to
the new index with two buffers. Both treatments returned no rows.
- Normal master integration preserves 35 independent performance files
exactly. The gateway also retains master's narrowed identity read.

Commands include `pnpm --filter @paperclipai/server typecheck`, `pnpm
--filter @paperclipai/server build`, and server Vitest selections for
`tool-gateway-service`, `server-startup-feedback-export`,
`scoped-deadline-work`, `run-identity-projections`, and
`connection-management.integration`, with separate PostgreSQL test
fixtures.

[The behavior and failure audit](doc/idle-tool-review-projection.md)
records the producer contract, earlier fixture failures, index
measurement, red/green regressions and integration evidence. Fresh
current-head CI and Greptile/Ponytail remain required before merge.

## Risks

- Signals use the existing single-active-writer database owner. Another
replica or independent SQL writer needs a wake contract.
- An already issued database query or reflection finishes before
cancellation takes effect. Checks between rows bound further dispatch.
- Legacy manual/global and disabled-scheduler one-shot repairs keep
their complete-sweep behavior. The normal coordinator worker always
supplies its row budget and admission controls.
- Unprojectable cards and thrown failures retain recovery with a
five-minute backoff cap. The company mismatch query still grows with
terminal history. Empty history does not retain a recurring scan.
- The index migration uses concurrent creation and the existing
invalid-index recovery path. Prior SQL and journal entries remain
unchanged.
- Cloud sleep eligibility and external-wake coverage do not change.
Local worker quiet is not proof of host or database suspension.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 17:09:31 -05:00