Commit Graph
113 Commits
Author SHA1 Message Date
DottaandClaude Fable 5.2 8ab955776d perf(github): check the requester provenance of an issue list's review waits in one batch (#20851)
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can review a GitHub pull request. After the agent publishes
the review, the task waits in review for the next GitHub reply. A later
run of the same task can inherit this wait.
> - Issue lists, issue summaries and inbox attention show these waits.
For each waiting task, the server first checks the requester provenance
of the waiting run (`chatRunSourceMessageId`). This check is an
authorization check: it walks the identity-context lineage of the run
and rejects a changed requester, another task, another company or
another provider.
> - #20673 batched the review reads and the publication reads of one
list request. It kept the provenance check as one call for each waiting
run, inside `Promise.all`.
> - Thus a request with N waits still built and sent at least 2N
statements at the same moment. The database is fast for each statement,
but the server builds each statement on its main thread.
> - On a hosted instance, a CPU profile of the server showed the main
thread at 96% of one core. SQL text building, per-statement timers,
socket writes and garbage collection had the largest shares. Connection
samples showed 235 to 292 pool connections whose last statement was the
source-message link read of this check, and 53 to 112 whose last
statement was its identity-context read.
> - This pull request checks the provenance of all the waiting runs of
one request in one batch. The runs that need the same kind of read share
one statement. Each run gets the same decision as a separate call, from
the same walk and the same reads.
> - The benefit is that a list request with 250 waits sends 17
statements, not 658, and uses about one quarter of the CPU time. The
statement count no longer follows the number of waits.

## Linked Issues or Issue Description

Refs #20673 (batched the review reads and the publication reads of the
same request; this change does the same for the provenance reads that
#20673 kept for each run).
Refs #20551 (authorizes the waits of one issue page with one read).

**What happened**

`readInheritedGitHubReviewWaits` called `chatRunSourceMessageId` one
time for each waiting run, inside `Promise.all`. Each call sent an
identity-context read and a source-message link read. A run that
continues an earlier run also sent a parent-context read and a
source-run read for each continuation in its lineage.
`listExternalReviewWaits` calls this function for the issue list, the
issue summaries and the inbox attention of a company, so each of those
requests repeated the same reads for every waiting task.

Evidence from a hosted instance (sanitized):

- CPU profile of the server process, 30 s, 5 ms samples: main thread at
96% of one core. Building SQL text for select statements: 17.7%
inclusive. The idle-timer restart that the database client does for each
statement: 7.6%. Socket writes: 6.2%. Garbage collection: 13%.
- Three samples of the pool connections (about 494 connections). 235 to
292 connections had the source-message link read of
`chatRunSourceMessageId` as their last statement. 53 to 112 connections
had its identity-context read as their last statement.
- The `DISTINCT ON` review read that #20673 added ran about 29 times
each minute. It runs one time for each list request, so each request
started hundreds of provenance reads.

**Expected behavior**

One list request sends a number of provenance statements that does not
follow the number of waiting tasks. The decision for each run does not
change.

**Steps to reproduce**

1. Put many tasks of one company in review, each with an inherited
GitHub review wait.
2. Open the issue list or the inbox.
3. Count the statements that the server sends. The identity-context read
and the source-message link read run one time for each waiting task.

The benchmark below does these steps on a local PostgreSQL.

## What Changed

- `server/src/services/chat-channels/run-origin.ts`:
- The provenance decision is now one walk (`sourceMessageWalk`). The
walk asks for one read at a time and decides on the returned rows only.
Its checks, their order and its error messages are the checks of the
previous `chatRunSourceMessageId`.
- The five reads of the walk (active identity context, source-message
link, board message, parent context, source run) are defined one time in
`provenanceReads`. Each definition takes its values either as literals
or as SQL columns.
- `chatRunSourceMessageId` (single run) drives the walk with one
statement for each read, as before. The joins, the company scope and the
filters are the same. The identity-context reads now select the seven
columns that the walk uses instead of every column.
- New `chatRunSourceMessageIds(db, runs, options)` (batch). It drives
one walk for each run. In each round, the runs that wait for the same
kind of read share one statement: `unnest(...)` of the request values,
`CROSS JOIN LATERAL` the same single-row read with the request columns
in place of the literals. Thus each run gets the row that its own single
read returns, with `LIMIT 1` and the company scope on each join. The
result for each run is `{ messageId }` or `{ forbidden }` (the 403 that
the single call throws). A database error rejects the whole batch, as
`Promise.all` did.
- `server/src/services/chat-channels/github/review-wait.ts`:
`readInheritedGitHubReviewWaits` checks the provenance of its runs with
the batch. A 403 still gives `revoked`. The single-run
`readInheritedGitHubReviewWait` does not change.
- `server/src/services/chat-channels/run-origin.test.ts`: two new tests
(see Verification).
- `server/src/__tests__/github-review-wait.test.ts`: one new regression
test for the statement count.
- `server/scripts/benchmark-review-wait-provenance.ts`: the reproduction
and the before/after measurement. It loads the pre-fix modules from
`--baseline-ref` and fails unless both versions return the same waits.
- `scripts/benchmark-github-review-wait.ts`: removed an assertion that
`run-origin.ts` is byte-identical to its baseline. Both sides of that
benchmark import the current helper, so the comparison stays fair.
Without this change, the documented command of that benchmark fails.
- `doc/DATABASE.md`: describes the batch provenance check and the new
benchmark command.

Statement count of one batch: the walk of one run does at most 3 + 4L
reads, where L is the number of continuations in its requester lineage.
The batch does the same number of rounds for its longest lineage. Each
round sends at most one statement for each kind of read (5 kinds), for
any number of runs. A request whose waiting runs accepted their GitHub
comment directly sends 2 provenance statements.

## Verification

Red/green regression test
(`server/src/__tests__/github-review-wait.test.ts`):

- `reads the waits of forty runs with the statements of four runs`.
Before the fix: `expected { few: 10, many: 82 } to deeply equal { few:
4, many: 4 }`. After the fix: passes.

Equivalence tests
(`server/src/services/chat-channels/run-origin.test.ts`), on real
PostgreSQL:

- `gives every run of a batch the decision of its own check, in every
provenance state`. 33 runs, each in a different provenance state, are
checked with the single function and with the batch for five option sets
(no options, legacy wake allowed, GitHub, GitHub with legacy wake,
Slack). The test compares the decision of each run (message, `null`, or
the 403 and its message) for: the full batch, the reversed batch, and a
batch of one run. The states include the negative cases: a run of
another company, a message that is linked only in another company, a
message that is linked to another task, a conversation that moved to
another task, a continuation on another task, a changed requester (run,
identity context, continuation, board message), a link of another
provider, a missing link, an outbound link only, a deleted message, an
agent message, a rejected or pending identity context, a missing parent,
another agent and a lineage cycle. The test also states the expected
GitHub decision for each state, so the comparison cannot pass on two
wrong answers.
- `checks the runs of a batch with statements that follow the lineage,
not the number of runs`. 4 runs and 40 runs each send 2 statements. 40
separate checks send 80. One continuation in each lineage gives 4
statements for the batch.
- A mutation check: when the batch gives a row to the wrong request,
both tests fail.

Commands:

- `pnpm vitest run server/src/services/chat-channels/run-origin.test.ts
server/src/__tests__/github-review-wait.test.ts`
- Focused run of 15 test files that import the changed modules, their
callers or the shared seed helper, including
`external-chat-wait.integration.test.ts`, `chat-github-tools.test.ts`,
`slack-authority.test.ts` and
`issue-list-review-wait-authorization.integration.test.ts`: 482 tests
passed. In `idle-completed-history.test.ts`, `afterEach` hooks (a
`TRUNCATE` of the test database) timed out at 30 s for 1 test in this
run and for 3 tests in a second run of the file, while other agents
loaded the machine. No assertion failed. The three tests of that file
that reach the changed code pass when they run alone. CI runs the full
file.
- `server/src/__tests__/chat-channels.integration.test.ts` also uses the
single check. It was not run locally (very large file). CI runs it.
- Typecheck of the changed files and of the callers of
`chatRunSourceMessageId` with a focused `tsc` program: no errors in
these files. The full server typecheck was not run locally (shared
machine). CI supplies it.
- The first CI run of this pull request failed `ci / policy`: the module
boundary check rejects an insert into the agents table outside the agent
lifecycle module, and the benchmark seeded its fixture agent that way.
The second commit seeds that agent with SQL. `pnpm
check:module-boundaries` passes locally, and the benchmark gives the
same statement counts.
- `pnpm verify` returns exit 2 (broader verification required) because
the change adds a file under `server/scripts/`. CI supplies the broad
coverage.

Before/after measurement:

```sh
pnpm --filter @paperclipai/server exec tsx scripts/benchmark-review-wait-provenance.ts --baseline-ref=e2cf9a670fc5d5f5058fd0972beb71650f77c554
```

One call of `listExternalReviewWaits` (the read of one issue list
request). Embedded PostgreSQL 18.1, Node v26.4.0, `VACUUM ANALYZE` after
seeding. One company, one agent, one GitHub repository. Every wait has a
published review. Every tenth waiting run continues three earlier runs.
3 warm-up calls, then 15 pairs in alternating order. Both versions
return the same waits and the same inherited wait for each run.

| Waits | Version | Statements | SQL text bytes | Wall ms, median
(min–max) | CPU ms, median (min–max) |
| --- | --- | --- | --- | --- | --- |
| 20 | before | 60 | 31,084 | 14.1 (12.8–19.6) | 13.7 (8.7–39.3) |
| 20 | after | 17 | 15,847 | 12.5 (9.4–22.7) | 8.8 (4.1–87.1) |
| 250 | before | 658 | 300,611 | 63.3 (58.1–89.6) | 78.1 (64.8–103.3) |
| 250 | after | 17 | 17,171 | 20.8 (18.9–29.2) | 18.7 (14.1–27.3) |
| 1000 | before | 2,608 | 1,179,710 | 256.0 (221.0–363.2) | 303.3
(262.2–507.8) |
| 1000 | after | 17 | 21,695 | 54.9 (46.7–81.7) | 51.0 (41.6–72.2) |

Of the 658 statements before the fix at 250 waits, 650 are provenance
reads: 325 identity-context reads, 250 source-message link reads and 75
source-run reads. After the fix, 9 statements do the same reads. The
other 8 statements of the request do not change. Sizes are set with
`WAITS=20` and `WAITS=1000`. Other agents used the same machine during
the measurement, so the maximum values have noise.

- The batch statements use index scans only
(`run_identity_contexts_pkey`, `run_identity_contexts_company_run_idx`,
`chat_message_links_company_comment_idx`, `chat_conversations_pkey`,
`chat_endpoints_company_id_uq`, `heartbeat_runs_pkey`). The 9 batch
statements of a request with 250 waits take about 1 ms of server
execution in total.
- Wall time and CPU time are almost equal before the fix. This shows
that the cost is in the server process, not in the database.

Not verified:

- The effect on the hosted instance. This pull request does not deploy.
The share of the main-thread CPU that this path uses there is not known;
the profile gives the share of all statement building.

## Risks

- The provenance check is an authorization check. The risk is a batch
decision that differs from the single decision. Mitigation: both paths
run the same walk and build each read from the same definition. The
batch read is the single read inside `CROSS JOIN LATERAL`, for each
request row. The equivalence test compares both paths in every state.
- A database error (for example an invalid UUID in a legacy task
binding) rejects the whole batch. The previous `Promise.all` also
rejected the whole request. The callers of the batch pass the native
task of each run.
- The statement count of the batch follows the longest lineage of its
runs. A batch with one very long lineage sends as many statements as
that run sent before, in sequence. It does not send more.
- The batch sends its statements in sequence on one connection. Before,
the reads of N runs used up to N pool connections at the same moment.
- The single-run callers (GitHub tools, review publication, response
comments, Slack authority, the finalizer, the status committer) keep one
statement for each read. Their identity-context reads select fewer
columns.
- No migration. No new index.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (code reading, editing, shell, tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 10:19:29 -06:00
DottaandClaude Fable 5.2 a4b872477e fix(chat): do not wake the assignee for a decision its own run recorded (#20842)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task can be bound to a chat conversation (Slack, Telegram).
Confirmation cards are published there
> - The assignee's running turn can record the user's chat reply as the
decision on a card (resolve-from-comment). That run already owns the
decision, and the in-app route does not wake the agent again
> - The chat continuation worker (`interaction_wakeup`) still woke the
assignee for that decision
> - The wake waited behind the running turn. When it was promoted, the
reply was already delivered, so its context kept `source:
chat:<provider>` but no chat message
> - Reviewed-chat attestation then refused the run
(`reviewed_chat_execution_binding_not_authorized`, `context_mismatch`).
The refusal is correct; the wake should not exist
> - This pull request makes the worker settle that action without a wake
> - The benefit is no failed setup run on the task, and no change to any
permission check

## Linked Issues or Issue Description

Refs #20809

**What happened?**
On staging, an automation-triggered run failed in about 5 s with
`setup_failed` and error
`reviewed_chat_execution_binding_not_authorized` (denial reason
`context_mismatch`). The task was bound to a Slack thread and was
`in_review`. The assignee's previous run had recorded the user's Slack
reply as acceptance of a plan confirmation and kept working. The chat
continuation worker then queued a second wake for the same decision.
That wake carried no undelivered chat message, so attestation found zero
admitted comment ids.

**Expected behavior**
No second wake. The responding run already acts on the decision, as in
the in-app path.

**Steps to reproduce**
1. Bind a task to a Slack conversation and publish a
`request_confirmation` card with `continuationPolicy: wake_assignee`.
2. While the assignee's run is active, the user replies in Slack. The
run calls `POST
/api/issues/:id/interactions/:interactionId/resolve-from-comment`.
3. The worker processes the `interaction_wakeup` chat action and wakes
the assignee. The wake is deferred behind the active run and loses the
delivered reply. With the task `in_review`, the next run fails
attestation.

## What Changed

- `server/src/services/chat-channels/interactions/continuations.ts`:
when the current interaction was resolved by the assignee's own run from
a comment (`resolvedByAgentId` is the assignee, `resolvedByRunId` is
set, no `resolvedByUserId`, and `result.commentId` is set; only
resolve-from-comment writes that), the worker marks the action
`processed` with `interaction_wakeup_owned_by_responding_run` and
creates no wake. Actions saved before the deploy take the same path.
- `server/src/__tests__/chat-interaction-publications.test.ts`: real
resolve-from-comment flow (no wake) and the board-accept control (one
wake).
- Reviewed-chat attestation and every authorization check are unchanged.

## Verification

- Red on master: the new test fails, `wakeup` called 1 time.
- Green: `server/src/__tests__/chat-interaction-publications.test.ts`
32/32 pass.
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- Low. Only a self-resolved, comment-recorded decision skips the wake. A
user decision through a chat button or the board still wakes the
assignee.
- Not covered here: a user decision made through a chat button while
another run of the task is active can still be deferred and lose its
chat comment id. That is a separate case and is not changed by this PR.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 09:47:43 -06:00
DottaandPaperclip aaf1337c7b fix(slack): require mentions only for channel roots (#20821)
## Thinking Path

> - Paperclip helps people manage AI agents for work.
> - Slack connects people to those agents through channels and threads.
> - A mention can start a conversation without making the bot answer all
channel traffic.
> - The current setting also requires mentions on every thread reply.
> - This interrupts an existing conversation and can cancel a queued
reply.
> - This change requires mentions on channel roots and allows replies
without a mention.
> - People can continue a thread without adding the bot name to each
message.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The Slack connection setting that requires an at-mention.

**Current behavior**
When the setting is on, a channel thread reply must mention the bot. The
delivery and execution checks reject a reply that has no mention.

**Proposed behavior**
Require a mention on a top-level channel message. Allow replies in
existing threads without a mention. Keep the current DM access controls.

**Reason and benefit**
A person can continue an existing conversation without repeating the bot
name.

**Breaking changes**
Existing connections with the setting on will accept thread replies
without a mention. The saved setting, default value, and API field stay
the same.

## What Changed

- Rename the control to **Require at-mention in channels**. Explain the
thread and DM rules in the help text and operator documentation.
- Keep the mention policy in the Slack module. Use the thread root and
message IDs to identify a reply.
- Call that policy from durable admission, wake authorization, and
queued run start. These shared paths must enforce the same setting when
a delivery resumes. Other providers keep their current behavior.
- Update PostgreSQL integration tests for thread replies, DMs, queued
deliveries, and combined wake inputs. Add tests that reject incomplete
reply identity.

## Verification

- Reproduced rejected thread replies and cancelled queued reply runs
before the fix.
- Passed 16 focused PostgreSQL integration cases after the fix.
- Passed 71 tests on revision `31b1adede2b92bad24b8fa21b8f6fffa9bd26d04`
after the rebase. This includes the full durable wakeup suite, Slack
policy tests, and eight selected Slack delivery cases.
- Command: `pnpm exec vitest run
server/src/services/chat-channels/slack/wake-authority.test.ts
server/src/__tests__/durable-chat-wakeup.test.ts
server/src/__tests__/chat-channels.integration.test.ts -t "at-mentions
by default|thread replies without mentions|rechecks queued
Slack|preserves running Slack tool authority|Slack channel mention
policy|durable inbound chat scheduler receipts"`.
- Strengthened the queued-root test to check the mention-rejection
reason and content redaction. A temporary change that disabled mention
enforcement caused this test to fail with the unrelated-message reason.
The real policy is restored.
- Removed the added source comments to follow the contribution rules.
- `node scripts/check-module-boundaries.mjs` and `git diff --check`
pass.
- `pnpm verify` returns exit 2 because the operator documentation is a
global CI input. Required CI must supply the wider coverage.
- Local preflight installed dependencies and built the plugin SDK. It
could not build Runner Rust binaries because this environment has no
`cargo` executable. The focused server tests run without those binaries.
- Live Slack and browser tests were not run. This change uses the
existing provider message and thread identities.

- Required CI passed on `31b1adede2b92bad24b8fa21b8f6fffa9bd26d04`: [PR
workflow](https://github.com/paperclipai/paperclip/actions/runs/43035551294),
including typecheck, build, server/chat tests, Runner checks, and
browser E2E. Current-head Greptile and Paperclip reviews are both 5/5
with no open findings.

## Risks

- Existing connections with mention gating on will now accept
unmentioned thread replies. This is the intended behavior change.
- A missing or invalid thread or message ID does not grant the reply
exception.
- No schema migration is required. Destination, sender, and company
access checks stay in place.
- The existing chat integration test file is large. This change updates
its current Slack cases to reuse the full delivery and authorization
fixture.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning and code execution tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-11 08:26:52 -06:00
DottaandClaude b93df4ca38 fix(chat): consume a saved chat wake whose message a completed run answered (#20689)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - External chat (for example GitHub review requests) wakes the
assigned agent with a saved chat message
> - When the agent is already running, the chat wake is saved behind
that run and released after it finishes
> - The running agent often reads the new message as part of its current
turn, so the message is already answered when the saved wake is released
> - The release kept that chat wake anyway, removed its now-delivered
message ids, and started a reviewed chat run with no message
> - A reviewed chat run with no message cannot be attested, so it failed
with `reviewed_chat_execution_binding_not_authorized` and the task
became blocked
> - This pull request treats a chat message wake like a comment wake:
when a completed run already received all its messages, the saved wake
is consumed
> - The benefit is that a message that arrives during an active run no
longer blocks the task after it was answered

## Linked Issues or Issue Description

**What happened?**
A GitHub chat message arrived for a task while the assigned agent's run
was active. The wake was saved behind that run. The run read the message
and answered it. When the run finished, the saved chat wake was still
promoted. Its message ids were removed because a completed run already
received them. The new run had source `chat:<provider>` and no message,
so reviewed chat attestation denied it with `context_mismatch`. The run
failed with `reviewed_chat_execution_binding_not_authorized`.
Review-path recovery inherited the same context and failed too, and the
task was blocked.

**Steps to reproduce**
1. Assign a task in review to an agent that is running.
2. Deliver an external chat message for the task. The wake is saved
behind the active run.
3. Let the active run receive the message as wake input and succeed.
4. Release the task's execution lock.
5. See that the saved chat wake is promoted with no message, instead of
being consumed.

**Expected behavior**
The saved chat wake is consumed with "Queued messages were already
delivered to a completed run". No new run starts. The chat does not get
a "not started" notice for a message that was answered.

**Paperclip version or commit**
`origin/master` at 6e524d8a69.

**Additional context**
This was seen on a staging deployment on five runs across three tasks in
one day. It is a message timing race, not a deploy or authorization
problem.

## What Changed

- `server/src/modules/wake-queue/adapters/postgres.ts`: a saved wake
whose reason is `External chat message received` and whose payload
mutation is `chat_message_received` is treated as message-only. If a
completed run already received all of its messages, the release consumes
it, the same as a comment wake. If any message is still unread, it is
promoted as before.
- `server/src/services/chat-inbound-wakeup-publications.ts`: a chat
receipt consumed for that reason does not publish "This follow-up was
not started". Other declines keep that notice.
- `server/src/services/queued-comment-delivery.ts`: constants for the
chat message wake reason and the delivered-message reason.
- New test
`server/src/modules/wake-queue/adapters/chat-wake-delivered.test.ts`.

## Verification

- `cd server && npx vitest run
src/modules/wake-queue/adapters/chat-wake-delivered.test.ts`
- RED on `origin/master`: the release promotes the consumed chat wake
("Unexpected promotion of a consumed chat wake"), and the receipt
resolves to a `not_started` notice.
- GREEN with this change: the wake is cancelled with the delivered
reason, no run starts, no "not started" notice, and a chat wake with an
unread message is still promoted.
- Related suites: `src/modules/wake-queue`,
`heartbeat-reviewed-chat-binding.integration.test.ts`,
`durable-chat-wakeup.test.ts`,
`heartbeat-native-status-context.test.ts`,
`external-chat-wait.integration.test.ts`,
`heartbeat-held-comment-wake.test.ts`.
- `npx tsc --noEmit -p server` passes.

## Risks

- A chat wake that merged with another kind of wake takes the merged
wake's reason and mutation. Only a wake whose current reason and
mutation are both the chat message values is treated as message-only, so
other merged intents keep promotion.
- A visible "Your follow-up is queued." notice is not followed by a "not
started" notice when the message was already answered. That matches the
existing behavior when the wake is promoted.
- Not changed: a separate case where messages from two different chat
conversations are merged into one saved wake. That case fails with
`binding_denied` and needs its own change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, read-only API access). Extended
reasoning was used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 20:15:13 -06:00
DottaandClaude Opus 5.5 e9db5e0d03 perf(github): read the inherited review waits of an issue list in one pass (#20673)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can review a GitHub pull request. After the agent publishes
the review, the task can wait in review for the next GitHub reply. A
later run of the same task can inherit this wait.
> - Issue lists and inbox attention show these waits. For each waiting
task, the server reads the inherited GitHub review wait: the source
comment, the latest review, and the review publication.
> - The server did these reads one time for each waiting task. A list
request with N waiting tasks sent N copies of the prepared review
statement and N copies of the publication read.
> - On a hosted database, the prepared review statement ran about
270,000 times in about 2.5 hours (up to 87 calls per second). The run
identity read before it ran the same number of times. The list read that
calls it ran only about 1,400 times, so each list read started about 190
review reads on average. No other caller reads the waits of many runs in
a loop.
> - This pull request reads the reviews and the publications of all
waits of one list request with one statement each. Each wait gets the
same result as before.
> - The benefit is about 200 times fewer review statements and fewer
pool connections for each issue list.

## Linked Issues or Issue Description

Refs #20400 (shares review-wait reads between the callers of one
request).
Refs #20551 (authorizes the waits of one issue page with one read; this
change does the same for the inherited GitHub review reads of that
page).

**What happened**

`listExternalReviewWaits` called `readInheritedGitHubReviewWait` one
time for each candidate wait, inside `Promise.all`. Each call sent a run
identity read, a source-message read, the prepared
`inherited_github_review_wait` statement, and a publication read. A list
with N waits sent each statement N times.

**Expected behavior**

One list request sends one review statement and one publication read for
each company. The result of each wait does not change.

**Steps to reproduce**

1. Put many tasks of one company in review with an inherited GitHub
review wait.
2. Open the issue list or the inbox.
3. Count the statements of the database. The review statement runs one
time for each waiting task.

## What Changed

- `server/src/services/chat-channels/github/review-wait.ts`:
- Split `readInheritedGitHubReviewWait` into shared steps: the source
comment read, the review checks, the publication key, and the published
result. The single-run read uses the same prepared statement and the
same checks as before.
- Add `readInheritedGitHubReviewWaits(db, runs)`. It reads the source
comment of each run with the same lineage check as before. Then, for
each company, one `DISTINCT ON` statement reads the latest review for
each (source comment, task, agent) key, with the same joins, filters and
order as the prepared statement. One read finds the processed
publications. Each run gets the same result as a separate call: `null`,
`revoked`, or `published`.
- `server/src/services/native-runtime/external-review-wait.ts`:
`listExternalReviewWaits` uses the batch read. The rest of the list and
its authorization do not change.
- `server/src/__tests__/github-review-wait.test.ts`: a new test reads 18
runs in one batch and compares each result with a separate single-run
read. The runs include each revocation case (connection disabled or
inactive, review state, review agent, repository, repository id, thread,
publication, identity, company), a newer unreviewed head, no identity,
no task, an endpoint of another agent, a run in another company and a
duplicate run. The test also makes sure that the batch sends one review
statement and one publication read for each company. The existing
revocation tests also check the batch read.
- `server/src/__tests__/helpers/github-review-wait.ts`: the seed helper
can add a wait to an existing company and agent.

## Verification

- `pnpm vitest run server/src/__tests__/github-review-wait.test.ts
server/src/__tests__/github-review-check-state.test.ts
server/src/__tests__/issue-list-review-wait-authorization.integration.test.ts
server/src/__tests__/blocked-inbox-review-wait-scope.test.ts`
- Focused local run of every test that imports the changed modules or
the seed helper (7 files, 335 tests passed), including
`external-chat-wait.integration.test.ts`, `external-review-wait.test.ts`
and `legacy-execution-recovery.test.ts`. The change to the shared seed
helper makes `pnpm verify` select the full server suite, so CI supplies
that coverage.
- Local measurement on real PostgreSQL (embedded PostgreSQL, `ANALYZE`
after seeding). One company and agent, each wait has 5 reviews and 10
outbound messages. "Before" is the previous list code: one
`readInheritedGitHubReviewWait` call for each run in `Promise.all`.
"After" is `readInheritedGitHubReviewWaits`. The runs alternate order in
each pair. Both return equal results in every sample.

| Waits | Version | Statements | Review statements | Median ms | Min–max
ms | Samples |
| --- | --- | --- | --- | --- | --- | --- |
| 200 | before | 800 | 200 | 106.8 | 98.0–117.5 | 10 |
| 200 | after | 402 | 1 | 44.3 | 41.6–50.9 | 10 |
| 20 | before | 80 | 20 | 9.8 | 8.6–14.7 | 12 |
| 20 | after | 42 | 1 | 6.3 | 5.3–7.0 | 12 |

- `EXPLAIN (ANALYZE, BUFFERS)` of the batch review statement for 200
waits: index scans on each join, 6.7 ms execution. The single statement:
0.05 ms execution and about 2.7 ms planning for each call.
- The run identity read and the source-message read still run one time
for each wait. They are cheap on the hosted database (about 0 ms each),
and they keep the lineage check for each run.

## Risks

- Low risk. The batch statement uses the same joins, filters and order
as the prepared statement, and `DISTINCT ON` keeps the first row for
each key. The array filters can also match a mix of two keys. The code
uses only rows of exact requested keys.
- The single-run read and its prepared statement do not change. The
finalizer, the status committer, and the attachment reuse paths still
use it.
- Errors do not change: a 403 from the lineage check gives `revoked`,
and other errors are thrown.
- No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (code reading, editing, shell, tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 18:35:39 -06:00
DottaandClaude Opus 5.5 81c52e3666 perf(chat): find terminal GitHub responses by issue run indexes (#20671)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel posts a terminal response when an agent run
for a GitHub request stops. A background sweep finds acknowledged
response comments (`processed`, not final) whose issue is blocked or
whose latest run is terminal
> - For each candidate, the sweep reads the status of the latest run of
the assigned agent for the issue. It matches runs with
`coalesce(native_issue_id::text, context_snapshot->>'issueId')`
> - No index has that expression. So PostgreSQL reads every run of the
agent for each candidate, and it decompresses the large legacy
`context_snapshot` of each run
> - On a hosted test database, this statement ran 13 times at a mean of
11 s in a 38-minute window (149 s in total), with single runs of 16 s
and 41 s
> - This pull request splits the run match into its two disjoint
branches. Each branch uses an existing issue index
> - The benefit is that each candidate reads a few index entries instead
of the agent's full run history. The returned rows do not change, and no
migration is necessary

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The discovery page query in `registerGitHubTerminalResponseWork` in
`server/src/services/chat-channels/github/terminal-response-work.ts`.
The query runs when the sweep reconstructs its candidates, and once or
twice for each candidate check.

**Subsystem affected**
GitHub chat channel terminal-response recovery.

**Current behavior**
The `WHERE` clause has `issues.status = 'blocked' or (select r.status
from heartbeat_runs r where r.company_id = ... and r.agent_id = ... and
coalesce(r.native_issue_id::text, r.context_snapshot->>'issueId') =
<issue id> order by r.created_at desc, r.id desc limit 1) in (<terminal
statuses>)`. The planner can only use
`heartbeat_runs_company_agent_started_idx`. It reads all runs of the
agent, then evaluates the `coalesce` on each row. That decompresses the
context of each legacy run. The cost is the number of candidates times
the number of agent runs.

**Proposed behavior**
The subquery takes the latest run from two branches and then the latest
of the two:

- runs with `native_issue_id = <issue id>`, through
`heartbeat_runs_company_native_issue_contract_id_uq`
- runs with `native_issue_id is null and context_snapshot->>'issueId' =
<issue id>`, through `heartbeat_runs_company_ctx_issue_created_idx`

The two branches are disjoint, and together they are exactly the rows
where the `coalesce` is equal to the issue ID. Both branches and the
outer query use the same order (`created_at desc, id desc`). So the
selected run is the same, including ties.

**Reason and benefit**
This statement was the largest single-statement cost in the hosted
window after the earlier database fixes. A local reproduction is about
490x faster on the default fixture, and the gain grows with the agent's
run history.

**Breaking changes**
None. The API, the candidate rules and the results do not change.

## What Changed

- Rewrite the latest-run status subquery in the GitHub terminal-response
discovery page as two index-backed branches (native binding, and legacy
context without a native binding).
- Add two integration tests for the sweep. In the first, newer runs of
the same agent that do not bind the issue (one with a native binding to
another issue, but a legacy context that names this issue) must not hide
the stopped run. In the second, a newer active run with a native binding
to the issue keeps the request waiting.
- Add `scripts/benchmark-github-terminal-response-sweep.ts`. It captures
the SQL that the real worker generates at a baseline commit and at
`HEAD`, compares all rows, and measures both.

## Verification

Cause and thesis. The cost comes from the `coalesce` run match, which
has no index and makes each candidate scan the agent's run history.
Thesis: if each precedence branch uses its own issue index, the cost of
the page no longer depends on the agent's run count, and the rows stay
identical. The benchmark confirms the thesis.

Local SQL benchmark:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-github-terminal-response-sweep.ts --baseline-ref=7b33384446
```

Fixture (synthetic): two companies, each with one GitHub endpoint and
one assigned agent. Each issue has one GitHub response comment. The
first N comments are acknowledged and not final; the others are final.
Every tenth issue is blocked. Runs alternate between native binding and
legacy context, and each run has a 4 KiB incompressible context. The
fixture also has these edge cases: newer active runs with a native
binding to another issue but a legacy context that names this issue,
active runs that tie on `created_at` with the latest run (between native
and legacy runs), and newer active runs of another agent. The benchmark
compares every column of every candidate row (the page limit is lifted
for this check). Embedded PostgreSQL 18.1 on an Apple laptop. The hosted
database is Neon with different compute, so absolute times are local SQL
measurements only. Each size has 2 warmups and then alternates the
baseline and the treatment.

| Size | Runs (both companies) | Candidate rows equal | Baseline median
(range) | Treatment median (range) | Speedup |
| --- | --- | --- | --- | --- | --- |
| Small: 200 issues, 3 runs each, 20 candidates, 7 samples | 1,372 | 26
of 26 | 18.7 ms (17.7–20.8) | 3.8 ms (3.3–4.3) | 4.9x |
| Default: 1,500 issues, 8 runs each, 150 candidates, 7 samples | 25,300
(156 MB) | 200 of 200 | 3,697 ms (2,875–4,032) | 7.6 ms (6.6–9.6) | 490x
|
| Large: 3,000 issues, 10 runs each, 400 candidates, 5 samples | 62,600
(382 MB) | 538 of 538 | 57,055 ms (48,343–66,079) | 31 ms (16–79) |
1,800x |

Sizes are set with `BENCH_ISSUES`, `BENCH_RUNS_PER_ISSUE`,
`BENCH_CANDIDATES` and `BENCH_ITERATIONS`. The timed query is the real
first page (limit 100).

Plan (default size): the baseline runs `SubPlan` 270 times. Each run is
a `Bitmap Heap Scan` on `heartbeat_runs_company_agent_started_idx` that
removes about 12,000 rows by filter, for about 8.2 million buffer reads
in total. The treatment runs an `Append` of two `Limit` index scans
(`heartbeat_runs_company_native_issue_contract_id_uq` and
`heartbeat_runs_company_ctx_issue_created_idx`), for about 3,800 buffer
reads in total.

Tests:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t "terminal GitHub"`:
13 passed, including the 2 new tests.
- Mutation check: when the legacy branch does not exclude runs with a
native binding, the first new test fails. It passes with the fix.
- `cd server && npx tsc --noEmit`: passed.

## Risks

- Low risk. The change is one SQL subquery. The match is the same set of
runs: `native_issue_id::text = <uuid text>` is equal to `native_issue_id
= <uuid>`, and a null `native_issue_id` makes the `coalesce` use the
legacy context.
- The candidate set itself is not bounded by this change. Acknowledged
responses that never become final stay in the candidate set. Each
candidate is now cheap, but the count still grows with that history.
- `readGitHubResponseRun` in `response-run-state.ts` uses the same
`coalesce` match. The sweep calls it once for each candidate row, and
other GitHub paths also call it. Each call is a separate, smaller
statement. This pull request does not change it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude
Code with tool use (shell, file edits, and local PostgreSQL benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 18:25:59 -06:00
DottaandPaperclip 12f70d0d4c Keep Slack form custody and receipt transfer durable (#19825)
## Thinking Path

> - Paperclip manages work for AI-agent companies.
> - Chat connections deliver provider messages and user answers.
> - Slack requires a response within its original three-second deadline.
> - Forms and their delivery receipts must survive retries and tenant
restarts.
> - This change uses the existing connector queue and durable form
owners.
> - Current authorization, exact receipt checks and normal ACK rules
stay in force.
> - The tenant can settle useful work while keeping the connection
enabled.

## Linked Issues or Issue Description

Refs #19019 and #19348. Companion Cloud PR:
paperclipai/paperclip-cloud#1250.

## What Changed

- Preserve form grants, sessions, encrypted evidence, terminal receipts
and company-authorized Retry.
- Check initial form enrollment before the provider send. Keep a typed
preparation wait only while an exact current enrollment obligation
exists.
- Refund the attempt only when the claimed attempt and credential lease
still match and provider invocation has not started. Possible or
accepted sends retain uncertain-delivery custody. An ordinary error with
the same text still uses the normal five-attempt limit.
- Notify the existing publication worker when normal enrollment commits.
Reconstruct pending work from durable state after a lost reply or owner
restart.
- Page settled history before proving idle safety. Keep unfinished,
malformed and foreign receipts visible beyond the first 1,000 rows.
- Keep Slack callback, answer, receipt and ACK rules in their provider
owners. Support body-bound Cloud idle capabilities when a complete
catalog exceeds the header budget.
- Add focused regression coverage, an enabled quiet/reactivation recipe
and explicit fixture port checks.
- Keep SDK evidence tests tied to a normally built reviewed reference,
with a separate strict refusal for actual unreviewed CI outputs.

## Verification

App head: `6284d6881b5dc38a9ee149bcbde35e69a67dd8ca`, including actual
master `dff8bed007e01c32f3826144b893ec80603d1335`. Companion Cloud head:
`f83b38187dc7e6b2e219fe865dc16d268cc66fdb`, including actual Cloud
master `b1dcc3421a91efffee631a61536cf48f357c22fc`. Cloud #1250 is merged
at `1d07d81f23d4627ab7b6dcf49176b4128a2f8f9b` after exact-head passing
checks and complete Greptile 5/5. Current exact-head CI typecheck,
build, tests and aggregate verification are green; the complete edited
Greptile review is 5/5 and all 13 review threads and nested comment
pages are resolved.

- Both latest enrollment findings were reproduced against the preceding
product. The pure readiness control returned no wait for the first
enrollment. The real-PostgreSQL control observed a provider call while
that enrollment was still staged.
- The repaired source passes three whole files: 21 readiness controls,
80 real-PostgreSQL milestone controls and 37 tenant-actor middleware
controls. All 138 pass with zero excluded or TODO cases. The SQL case
reconstructs the worker through seven waits, preserves the attempt
budget, recovers a controlled lost enrollment reply, and commits one
provider-port call and one exact grant. Replaying the selected attempt
does not duplicate them. Authorization, SDK and enrollment-response
ports are controlled in that SQL case.
- The same files cover ordinary five-attempt exhaustion, accepted and
possible sends, stale ownership, malformed bindings, attention states
and current actor restrictions. One synthetic mismatched-binding
parameter uses generation 2 against current generation 1; it is not
evidence for an older-generation control.
- The normal server-package typecheck passes at the measured source. It
runs the normal Runner TypeScript, protocol, SDK and declaration
prerequisites and all seven main App programs. The separate declaration
configuration is also recorded. A worker union-narrowing error was
found, fixed with an explicit callable/null branch, and verified by that
normal compiler. No cast or test relaxation was used.
- The normal merge of current master preserves all 78 owned paths and
15,783 unrelated master entries and modes. The three measured test files
and their changed feature paths are unchanged. New upstream DB code adds
only the normal nonunique concurrent native-session index and its
migration classification. Shared and adapter-utils changes alter
unrelated prompt or denial wording. The 138-case proof carries for those
unchanged behaviors; it is not reported as a rerun after the merge. The
normal Shared/SDK ensure, DB build, adapter-utils build and complete
server-package check also pass at the current merged head. Their fresh
inventory contains 8,143 outputs; canonical SQL and journal copies
match. This build did not apply the new index migration.
- Earlier real-PostgreSQL retained-history coverage reproduced the
missing proof with 1,005 settled rows. Its repair retains the existing
two-second/64-query inspection budget, current endpoint bound and exact
owner snapshot. The 46 source-bound controls cover unfinished and
invalid tails, exact cross-page action references, Retry, rollback and
lost receipt responses.
- Earlier enabled quiet/reactivation coverage produced ordinary useful
work, kept configuration enabled, accepted fresh signed work plus a
duplicate, and produced exactly one second useful reply. Its two
App-worker windows lasted at least 30 seconds with zero measured PG
writes, connection or outbound activity after settlement. That evidence
stays tied to its measured revision. A missing worker EXIT marker limits
that earlier proof; parent evidence does not replace it.
- Earlier three-second callback, terminal ACK, current-question race and
complete body-size controls remain source-bound. The
500-owner/1,000-binding body measured 313,875 bytes with a 618-byte JWT;
signature tampering failed closed.
- Final managed-provider acceptance now passes all five fresh Node
boots: A/A, A/AB, B/A rejection, B/AB recovery, B/B final. Two
independently configured, challenged, verified, linked and enrolled
Slack identities exercise ACK and retirement debt separately. Normal
provider factory/materialization delivers the configuration through a
controlled external stack-provider boundary. The registered opening
returns HTTP 200 in 21 ms within the original 3-second deadline. Two
answers persist, normal ACK is accepted, exact retirement and lost
acceptance/GC replies recover, and the original canonical snapshot
remains unchanged without duplicate provider sends or submissions.
- The separate complete committed-preflight case passes. A committed
`update_attempted` response is lost before SDK entry; normal
service-owner reconstruction after the original lease/deadline recovers
the same useful update and one answer without a second trigger. This
reconstructs service owners within one App process; the separate five
fresh Node boots cover process restart and managed signing-key rotation.
- Assertion-time PG socket reports balance 15 opens/15 closes across the
five boots and 5/5 in preflight, peak 2 and zero live sockets or
unexpected attempts. Worker terminal EXIT markers were missing and are
not replaced by parent EXIT markers. These are domain/recovery and
assertion-time connection proofs, not a new whole-worker terminal quiet
measurement. All known owners/groups closed; independent SQL audits were
empty; only successful fixture databases were removed. Earlier failed
fixture databases/logs remain retained. No product guard, deadline or
assertion was weakened. Exact runtime source and normal outputs are
unchanged by the three test files in the current follow-up, so these
source-bound proofs carry without repeating those journeys.
- The preceding App CI ran 11,312 support cases: 11,289 passed, three
SDK-evidence cases failed, and 20 external cases skipped. Normal CI
dependency setup resolved six stale Slack patch-hash references and
produced an unreviewed lock. The public SDK proof correctly refused that
lock; the tests incorrectly assumed the CI output was a reviewed
reference. The three-file test repair keeps the protected lock policy,
source/compiler/parser/Zod pins, scanner and redaction rules unchanged.
It builds an owned reviewed reference using the existing normal
Shared/SDK producers for the positive seal test, while a separate
negative requires actual unreviewed-build privacy refusal and no
retained SDK bytes. It does not relabel the CI build as certified or
admit a new lock hash.
- All four whole SDK support files pass locally: public type 30,
evidence 16, public-input synchronizer 8 and lock preparation 10,
totaling 64 with zero excluded/TODO cases in 28.73 seconds. The genuine
fresh reference build/positive seal and separate invalidated-reference
privacy refusal passed within their original 120-second bounds. Original
installed dependencies, SDK outputs/cache and runtime source stayed
unchanged; normal global setup produced only its own Runner JavaScript
outputs. PostgreSQL was denied throughout, all four worker markers were
observed, and the owned process group/private test root closed. The
local canonical-lock negative mutates only its own reference; fresh
remote CI also passes the actual resolved-CI-lock branch: all 16
evidence and 30 public-type tests pass, including the genuine
reviewed-reference positive and the strict actual-unreviewed-build
refusal. The upstream synchronizer file runs its current 10 cases in CI.
- The first complete CI attempt on this head also exposed two separate
30-second timeouts: one real reporter-artifact test and the
`agent-test-environment` module-loading hook. Full original logs remain
preserved. Their exact test, sanitizer and evidence source blobs are
unchanged from the preceding head; inspection found no deterministic
setup cause. After the original run finished, one retry of only the
failed jobs passed on the same synthetic merge
`168851e210671ebe97e7a778e8cf891807e80502`: Runner support reports
11,304 passes and 20 external skips; serialized shard 6 reports all 395
passes, including all 67 route bodies. No source, timeout or assertion
changed. This proves recovery; the original causes remain unexplained.
- The final pre-merge tree audit uses actual master
`17136a6c25ef6cb7c816a29140a4c48c2c75fb31` (refreshed after the earlier
`85dec046` audit). The expected normal squash preserves all 81 owned
paths and 15,816 unrelated master entries and modes. Only OpenAPI has
edits from both sides, and its complete content composition preserves
the new project route and token help alongside this PR's API entries.
The shared inbound selection changes preserve the same candidate set;
new concurrent indexes, Native/workspace behavior and shutdown changes
remain upstream behavior. The scoped form proof uses process agents and
does not qualify the new Native/sandbox branches. The latest
question-schema change still accepts the explicit version-1 payload
exercised here. The last master delta also changes managed AI/Native
credential refresh and writeback; those branches are outside the
measured process-agent form paths. Existing 8,143 local outputs remain
tied to their measured revision; no new whole-tree output equality or
deployed suspension is claimed.
- Reproduce the public path with
`doc/testing/slack-cloud-qualification.md`, the companion Cloud
checkout, Node 24/22 and fresh owned fixture databases. Aggregate local
`pnpm -r typecheck`, `pnpm test:run` and `pnpm build` were not rerun as
one command set. Current exact-head required CI and the complete
Greptile 5/5 review now pass; the full edited summaries and all
paginated threads were read.

Local worker quiet does not prove deployed host or Neon suspension.
Photon and Discord remain separate work; this PR completes the described
Slack slice.

## Risks

Use compatible App and Cloud readers before transferring custody.
Signatures prove authenticity; current authorization and exact ownership
still control execution. Preserve old targets, secrets and terminal
facts until their normal owners release them. Keep the managed seal
identity stable while custody remains. The required Slack adapter patch
remains pinned. Upstream's concurrent index uses the normal top-level
migration path.

## Model Used

OpenAI `gpt-6.1-sol` implemented and tested the changes. A GPT-6-based
Codex managing agent reviewed and integrated them. The host does not
expose the managing agent's exact model ID or context-window sizes. The
agents used code execution and test tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used, its version and tool capabilities
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have linked public related issues or PRs
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run the local tests listed above and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green for the current head
- [x] Greptile is 5/5 with no open findings for the current head
- [x] I will address all reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 12:17:50 -06:00
DottaandClaude Opus 5.5 faaae003a9 perf(chat): find the conversation head from open delivery work (#20564)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - External chat messages go into a durable delivery queue. Each
conversation drains in provider order.
> - Each drain step reads the head of one conversation. The head query
filters one endpoint's deliveries by state and thread.
> - On a hosted instance, this query ran about 10,000 times at about 300
ms each. The database was busy, and request reads waited behind
background work.
> - PostgreSQL read every processed delivery of the endpoint and its
`normalized_event` value to find a few open rows.
> - This pull request finds the endpoint's open delivery IDs first and
fetches only those rows.
> - The benefit is less background database load. The selected head and
the delivery order do not change.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The inbound conversation head query
(`earliestOpenConversationDelivery`).

**Subsystem affected**
Server chat inbound delivery queue.

**Current behavior**
The query has `endpoint_id = $1 and (state in ('received', 'retry',
'processing') or (state = 'processed' and <pending inbound wake>))`, a
thread filter on `normalized_event`, and a sort on `normalized_event`
keys. No index matches the `or`. PostgreSQL scans all deliveries of the
endpoint. For each row it reads `normalized_event` to test the thread.
It also reads all of `chat_actions` to build the pending-wake set. A
busy endpoint keeps all of its processed history, so the cost grows with
that history.

**Proposed behavior**
The query selects `id in (<open delivery IDs of the endpoint>)`. That
set is a `union all` of two small reads:
- received, retry and processing deliveries of the endpoint, through the
existing `chat_deliveries_work_idx`;
- processed deliveries that an issued or processing `inbound_wakeup`
action of the same endpoint (and company) still owns.

PostgreSQL fetches only those rows by primary key, then applies the same
reaction, thread and order rules.

**Reason and benefit**
The cost no longer grows with processed history. On a local fixture with
60,000 processed deliveries, the head query reads 1,309 shared buffers
instead of 262,630.

**Breaking changes**
None. There is no schema change, migration, API change, timer or poll
change.

Related work: #17971 changed the pending-wake predicate of the same
queue. #20400 added the index for the other chat delivery statement in
the same report (`select state ...
normalized_event->'githubAutomatic'->'context'->>'deliveryId'`). That
statement already uses the #20400 index on master, so this pull request
does not change it.

## What Changed

- `delivery-ledger.ts`: add `openConversationDeliveryIds(endpointId)`.
It returns the endpoint's open delivery IDs. It selects the same set as
the old state predicate for that endpoint.
- `conversation-queue.ts`: the conversation head query uses `id in
(openConversationDeliveryIds(endpointId))` in place of the state `or`.
The thread filter, reaction exclusion, order, limit and lifecycle
dependency read do not change.
- `inbound-queue-discovery.test.ts`: add a real PostgreSQL test. It
compares the new set with the old predicate on three endpoints. It
covers each state, issued and processing wakes, duplicate wakes, wakes
of non-processed deliveries, processed, other-kind and null-delivery
actions, an action on a different endpoint of the same company, and
reactions. It also checks that the real captured head query returns the
oracle's first row.
- `benchmark-inbound-queue-discovery.ts`: add
`--endpoint-history=<rows>` (a busy endpoint with 4 KiB payloads) and
`--only=<fixture>`. Add a work budget: the treatment head query must
visit at most 100 delivery rows on the history fixtures.
- `doc/DATABASE.md`: describe the head query and the benchmark command.

## Verification

Local SQL measurements. PostgreSQL 17.10 (Postgres.app, macOS arm64),
disposable database, synthetic data, default settings with 256 MB shared
buffers. The host was under heavy load from other work, so timings are
noisy. Shared buffers and visited rows are stable. Each arm has two
warmups and five alternating samples (median [min..max]). Baseline:
`origin/master` at `8d2bf78323`.

| Fixture | Query | Baseline | Treatment | Shared buffers | Delivery
rows visited | Output |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| 60,000 processed, 4 KiB | head | 290.09 ms [220.21..554.19] | 7.34 ms
[5.85..7.68] | 262,630 → 1,309 | 60,002 → 4 | Equal |
| 20,000 processed, 4 KiB | head | 276.02 ms [211.77..322.12] | 34.61 ms
[4.45..194.15] | 67,618 → 486 | 20,002 → 4 | Equal |
| 1,500 processed, 32 KiB | head | 15.88 ms [13.66..21.79] | 2.29 ms
[1.64..7.75] | 10,602 → 80 | 1,502 → 4 | Equal |
| 20 processed | head | 1.49 ms [0.92..3.66] | 1.85 ms [1.01..1.97] | 2
→ 45 | 22 → 968 | Equal |

- The deadline and ready selectors do not change. Their buffers are
identical in both arms; their timings move only with host noise.
- The small control is neutral. On a near-empty table, PostgreSQL picks
a sequential scan for the open-state branch. This costs about 45
buffers.
- The 60,000-row run passes `--require-work-budget`. The baseline
exceeds the budget (`baselineExceedsWorkBudget: true`). This is the
red/green proof. There is no timing threshold in CI.
- On the treatment, the remaining head cost is the scan of
`chat_actions` for pending wakes of the endpoint. The baseline also
reads all of `chat_actions` for its hashed subplan.
- `pnpm exec vitest run src/__tests__/inbound-queue-discovery.test.ts`
(server): 6 tests pass, including the new equivalence test.
- `chat-channels.integration.test.ts` filtered with `-t
"reorder|orders|FIFO|conversation drain|lease renewal|receipt
order|edit|lifecycle"`: 148 assembled chat tests pass. They cover Slack,
Teams, Telegram, Discord and GitHub provider-order drains, head-of-line
barriers, lease loss, lifecycle edits and wake retries.
- `pnpm exec tsc --noEmit -p tsconfig.json` (server) passes.

Reproduce:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-inbound-queue-discovery.ts --baseline-ref=8d2bf78323 --endpoint-history=60000 --only=endpoint-history --require-work-budget
```

Set `PAPERCLIP_TEST_POSTGRES_ADMIN_URL` to use an existing disposable
PostgreSQL server. The benchmark uses synthetic data only.

## Risks

- Query plans depend on data and settings. These local numbers do not
prove hosted request latency. A follow-up statistics window after deploy
must confirm the gain.
- The pending-wake branch still scans the endpoint's `chat_actions`. A
partial index on pending `inbound_wakeup` actions would remove that
scan. I did not add it, to avoid a migration.
- Low behavior risk. The new set equals the old predicate for each
endpoint, and the order expression is unchanged. Tests compare both on
the same rows.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edit) in Claude Code. Extended reasoning was used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 10:15:29 -06:00
DottaandClaude 1e13c58a4e fix(heartbeat): name the cause of generic adapter failures and keep SQL text off the board (#20483)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Heartbeat run settlement records why a run failed, and the board
shows that record.
> - Runs that fail without a named code settle as `adapter_failed` (or
`setup_failed`) with no cause.
> - A user, an operator, and the restart chaos harness cannot tell what
went wrong from such a run.
> - Some run errors also show raw SQL statement text to users.
> - This pull request records a bounded `failureCause` code for generic
failures, shows it on the board, and keeps SQL text out of run errors.
> - The benefit is that a generic failure now names its cause, for
example "adapter failed (40001)", without leaking messages, SQL, or
secrets.

## Linked Issues or Issue Description

Refs #20385 (the nested-cause extraction this change reuses). Related:
#20421 (restart chaos universal checks U1-U4).

**What happened?**

The restart chaos audit of 79 saved waves found 30 failed runs with
`errorCode: "adapter_failed"` and a null or generic error. Every U3
("diagnosable") warning was `adapter_failed`. Nobody could tell what
failed. Some board run errors also showed raw SQL statement text from
wrapped database errors (Drizzle `Failed query: ... params: ...`).

**Expected behavior**

A run that fails with a generic code names a short, safe cause. The
board shows the cause next to the stop reason. Run error text never
shows SQL statements or their parameters.

**Steps to reproduce**

Make an adapter throw a wrapped database error (an `Error` whose nested
`cause` has `code: "40001"` and whose message has a Drizzle `Failed
query:` statement). Before this change, the run settles as
`adapter_failed` with no cause, and `heartbeat_runs.error` contains the
SQL statement and parameters. The new tests in
`server/src/services/heartbeat/run-completion.test.ts` reproduce this.

## What Changed

- Runner: move the #20385 event-commit cause extraction into
`safeErrorCauseCode`
(`packages/paperclip-runner/src/control-plane/safe-error-cause.ts`) and
export it. The Runner's own diagnostics use the same function, unchanged
in behavior.
- Shared: add the `failureCause` contract
(`packages/shared/src/run-failure-cause.ts`): one `[A-Za-z0-9_.:-]`
token of at most 80 characters, `normalizeRunFailureCause`,
`readRunFailureCause`, and `withoutSqlStatementText`.
- Server: add `server/src/services/heartbeat/run-failure-cause.ts`.
Settlement records `resultJson.failureCause` only when the code is
generic (`adapter_failed`, `setup_failed`). `errorCode` stays the same.
- Thrown errors (`failRun`, `failRunSetup`): `provider_http_<status>`,
else `provider_<category>`, else the first safe code in the error and up
to three nested causes (`40001`, `ECONNRESET`), else the error class
name, else `unknown_error`.
- Adapter results without a code (`completeRun`): the error family,
`signal_<name>`, `process_cancellation_failed`, `exit_code_<n>`,
`adapter_reported_error`, or `unknown_error`.
- The failure log lines (`heartbeat execution failed`, `heartbeat
execution setup failed`, and a new `heartbeat run failed` warning for
generic adapter results) include `failureCause` and `errorCode`.
- Server: replace SQL statement text in run error text with "Database
query failed (<cause>)" at all three settlement paths, including the
cancelled branch. The execution blocker's `runError`, the attention
inbox (failed-run excerpt and metadata), and continuation summaries
apply the same replacement on read, so older rows also stop showing SQL
there.
- Server: Retry admission and failed-preparation replay accept only a
closed set of `resultJson` keys. `failureCause` is added to each of
those lists (`task-retry.ts`, `chat-channels/failed-run-retry.ts`,
`cancelled-native-startup.ts`, `fresh-workspace-continuation.ts`). It is
a diagnostic code with no execution or replay authority, so a
failed-preparation run stays retryable.
- Server: `runsForIssue` returns a validated top-level `failureCause`.
It is null for older rows, non-string JSON values, and malformed values.
- UI: `RunForIssue.failureCause`; `IssueRunLedger` shows "adapter failed
(40001)". Older runs keep "adapter failed". The Storybook run-ledger
fixture has a failed run with a cause.
- Restart chaos harness: run samples keep `failureCause`. U3 treats a
generic code with a specific `failureCause` as diagnosable. A bare
`Error`, `unknown_error`, or the no-code fallback
`adapter_reported_error` still counts as vague. U3 stays a warning.
- Docs: `doc/run-log-events.md` and the harness README describe the
field and the SQL replacement.

## Verification

- Red before the fix: the four new `run-completion.test.ts` cases failed
against `origin/master`'s `run-completion.ts` (no `failureCause`, SQL
text in `error`).
- Green:
- `server/src/services/heartbeat/run-completion.test.ts`: all tests
passed (real embedded PostgreSQL). New cases cover a nested `40001`
three levels deep for execution and setup failures, SQL and a secret
parameter absent from `error` and `resultJson`, the bound (an
81-character code and a SQL "code" fall back to the class name),
`exit_code_137` for a code-less adapter result, a cancelled result with
SQL text, no cause for a named code, and the log line. On this loaded
host the suite's `afterAll` database teardown hit its 30 s hook limit
after every test passed (the known local teardown limit, also seen in
#20385). CI runs the suite.
- `server/src/services/heartbeat/run-failure-cause.test.ts` (4),
`server/src/__tests__/activity-service.test.ts` (recorded, legacy,
malformed, overlong, boolean, and number cause projections),
`server/src/__tests__/issue-continuation-summary.test.ts` (legacy SQL
row).
- `packages/shared/src/run-failure-cause.test.ts` (6),
`ui/src/components/IssueRunLedger.test.tsx` (21), restart chaos
`universal.test.ts` and `observer.test.ts`.
-
`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts`:
207 passed (the stall-diagnostic cases still read `40001` and
`ECONNRESET`).
- Typecheck: shared, runner, server (all layers), UI, restart chaos
harness. `pnpm check:token-gates`: all gates clean.
- `pnpm verify` returned exit 2 (global doc input changed); required CI
supplies broad coverage.
- CI on `6295f8d` found that the new key made failed-preparation runs
ineligible for Retry (`chat-channels.integration.test.ts` saved
board-message Retry got 409;
`workspace-recovery-preparation-dispatch.test.ts` dispatched 0 Retries).
Both reproduced locally and pass after the allowlist fix, with the other
459 Retry cases in `chat-channels.integration.test.ts` and all 25 in
`workspace-recovery-preparation-dispatch.test.ts`.
- No chaos waves were run. Storybook was not opened in a browser; the
fixture typechecks and the label is covered by the UI test.

## Risks

- Low risk. `errorCode` values do not change, so retry, recovery, and
comparison logic is unaffected.
- No schema change and no migration. `failureCause` is a new optional
`resultJson` key. Rows saved by the previous release have no key and
render as before; a new row read by an old server is ignored.
- The SQL replacement only acts on a Drizzle `Failed query:` marker, or
on `$n` placeholder text with a statement shape. Ordinary error text
does not change. When it acts, it also drops the text after the
statement.
- Code that checks `resultJson` against a closed key list must allow
`failureCause`. I found and updated the four lists on the Retry path; a
list I missed would refuse a Retry for a run with a generic failure.
- This is the run log / instance database path. It adds no Telemetry
event.
- The server failure log line still logs the raw error message
(unchanged behavior); only the board-facing text is changed.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file editing) and code execution for
tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 02:14:53 -06:00
DottaandClaude Opus 5.5 347d85fef3 perf(chat): index GitHub review lookups and share review-wait reads (#20400)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - GitHub chat endpoints let agents review pull requests. The server
reads review state when it lists issues and when it updates GitHub
checks.
> - On a hosted instance, two GitHub review queries used the most total
database time. One query ran about 660,000 times at about 33 ms. The
other query ran about 10,000 times at about 1 s.
> - The first query finds a source comment's chat link. No index covers
`chat_message_links.comment_id`, so each call reads the full table.
Issue lists also read the same review waits two times per request.
> - The second query joins deliveries, links and runs on a JSON key. The
planner walks the agent's run history in creation order and reads each
run's context.
> - This pull request adds four concurrent indexes, splits the run
lookup into two small reads, and shares review-wait reads in one
issue-list request.
> - The benefit is less background database load, so request reads wait
less. The results stay the same.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
GitHub review state reads. These are the inherited review-wait lookup by
source comment, and the review-check lookup of the run that a delivery's
comment woke.

**Subsystem affected**
Server chat channels (GitHub review waits and checks), issue list
attention maps, and the database schema.

**Current behavior**
- `readInheritedGitHubReviewWait` filters `chat_message_links` by
`company_id`, `comment_id` and `direction`. No index has `comment_id`,
so PostgreSQL reads the full table on each call. The run-origin check
does the same lookup first.
- An issue list calls the external review-wait read from the review
attention map and again from the blocked-inbox attention map. The two
maps read the same in-review issues.
- `githubReviewCheckState` joins deliveries, links and runs with
`context_snapshot->>'wakeCommentId' = comment_id::text`, sorts by
`created_at desc` and takes one row. The planner reads the agent's runs
in creation order and reads each run context. If no run matches yet, it
reads all of them. The delivery filter on `normalized_event` and the
link join on `delivery_id` also have no index.

**Proposed behavior**
The same rows come back from small index reads. Each issue's review wait
is read one time per issue-list request.

**Reason and benefit**
These two statement families used about 33,000 s of database time in one
cumulative statistics window. The database had about 50 concurrent
active queries, and request reads waited behind this background work.

**Breaking changes**
None. The migration adds indexes only.

## What Changed

- Migration `0456_chat_review_lookup_indexes` builds four indexes with
`CREATE INDEX CONCURRENTLY IF NOT EXISTS`. It runs through the existing
top-level statement path, which repairs an invalid index on retry:
- `chat_message_links_company_comment_idx` on `(company_id, comment_id)`
where `comment_id is not null`.
- `chat_message_links_delivery_idx` on `(delivery_id)` where
`delivery_id is not null`.
- `chat_deliveries_github_automatic_delivery_idx` on `(endpoint_id,
normalized_event->'githubAutomatic'->'context'->>'deliveryId')` where
that value is not null.
- `heartbeat_runs_company_wake_comment_idx` on `(company_id,
context_snapshot->>'wakeCommentId')`. This index is not partial. The
planner does not use expression statistics from a partial index. Without
these statistics, it estimates many matches and keeps the creation-order
walk.
- `githubReviewCheckState` reads the delivery's comment IDs first. If
there is a comment, it reads the newest same-agent run with one of these
`wakeCommentId` values. If there is no comment, it skips the run read.
The comparison stays text equality on `comment_id::text`.
- `createExternalReviewWaitReader` shares one read per issue between the
review and blocked-inbox attention maps of one issue-list request. The
reader exists only for that request.
- `doc/DATABASE.md` describes the indexes and the new read order.
- `scripts/benchmark-review-lookup-indexes.ts` compares the pre-fix and
new code with and without the indexes.
`scripts/benchmark-review-lookup-index-migration.ts` times the migration
on a large synthetic database.

## Verification

Local SQL measurements on a synthetic fixture. They are not production
timings. Environment: embedded PostgreSQL 18.1, baseline revision
`02bfe71a6b`. Fixture: 50,000 links, 5,000 deliveries with about 2 KiB
events, and 4,000 runs with about 12 KiB incompressible contexts. The
wake run is in the middle of the run history. Each value is the median
server execution time from `EXPLAIN ANALYZE` (2 warmups, 9 samples). The
order was: no indexes, indexes, then no indexes again.

| Caller and statement | Before (no indexes, two passes) | After |
Shared buffers before → after |
| --- | --- | --- | --- |
| Inherited review wait, link by source comment (same SQL) | 2.72 / 2.36
ms | 0.076 ms | 929 → 12 |
| Review check, wake run found | 31.9 / 21.9 ms | 0.29 ms (two
statements) | 25,776 → 62 |
| Review check, no wake run yet | 71.0 / 47.5 ms | 0.06 ms (two
statements) | 34,861 → 12 |

- Indexes alone do not fix the review check. The old SQL with the new
indexes took 8.6 ms (run found) and 29.7 ms (no run), because it still
walked the run history. The new SQL without the new indexes was neutral
(23.5 and 35.1 ms).
- Service wall times on this shared machine were noisy, so the table
reports server execution time.
- Correctness: the benchmark fails unless the inherited statement
returns the same positional rows in each index state. The check service
result and the final wake-run rows of the new code must also match the
pre-fix module for both cases.
- Migration timing, normal migration runner, synthetic data: 200,000
runs (1.8 GB with 8 KiB incompressible contexts), 500,000 links and
100,000 deliveries. First run 4.2 s; rerun 0.38 s. Separate builds:
links by comment 1.5 s, run wake comment 1.1 s, deliveries 0.7 s, links
by delivery 0.3 s. All indexes were valid. The run index was 8.3 MB.
- New tests:
- `server/src/__tests__/github-review-check-state.test.ts` checks that
the newest same-agent run is selected. It also checks that other agents,
other companies, uppercase IDs, non-string values, list-only
`wakeCommentIds`, comment-less links and unknown deliveries do not
match. It checks that the generated statements can use the new indexes.
- `server/src/services/native-runtime/external-review-wait.test.ts`
checks that the reader reads each issue one time, keeps concurrent
callers correct, and rejects all waiting callers when a read fails.
- Red/green: with the pre-fix `review-check-state.ts`, the behavior test
passes and the index test fails. With this change, both pass.
- Focused runs pass: `issue-blocker-attention`,
`heartbeat-issue-liveness-escalation`, review-wait, review-check-state,
external-review-wait, review-repair (106 tests), and
`external-chat-wait.integration` (285 tests). `pnpm --filter
@paperclipai/server typecheck` and `pnpm --filter @paperclipai/db
check:migrations` pass.

Commands:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-review-lookup-indexes.ts --baseline-ref=02bfe71a6b
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-review-lookup-index-migration.ts
```

## Risks

- The migration builds four indexes concurrently. It does not block
writes. On a large database, the run index reads every run context one
time. The local run took 1.1 s for 1.8 GB from a warm cache. Cold
storage on smaller compute can be slower.
- The run index adds one entry for each run insert and for each run
update that is not a heap-only update. Many runs have no
`wakeCommentId`, so these entries hold a null key.
- The review check now sends two statements instead of one when a
delivery has a comment. Each statement reads only a few index entries.
- When two matching runs have the same `created_at`, either can be
selected. The previous query had the same tie behavior.
- This change does not change the number of in-review issues that one
issue list reads. The blocked-inbox map still reads all in-review issues
of the company. A smaller scope needs a separate change to the liveness
classifier.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code with
tool use (shell, file edits, local PostgreSQL benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 18:47:34 -06:00
Dotta a3e7e63330 Guard Native recovery with process-space proof and explicit containment (#19813)
## Thinking Path

> - Paperclip manages agents and their work.
> - Recovery must preserve execution ownership and accepted results.
> - A saved PID proves nothing outside its recorded process namespace.
> - Uncertain unfinished work must stay held until its original owners
stop.
> - Operators need a source-bound containment action and a separate
ordinary Retry.
> - Accepted results must use the existing finalizer without another
provider turn.
> - This PR keeps those paths safe, recoverable, and quiet between real
obligations.

## Linked Issues or Issue Description

Refs: #19713

Refs: #19591

Recovery could use a local PID to release or replace a controller from
another process namespace. Connected-chat Retry also rejected successful
containment on a task in `todo`. The new containment action used a
broader permission than the existing owner-stop action. A later review
found that result-less guards also rejected the existing accepted-result
confirmation action.

## What Changed

- Record and verify the controller's kernel/PID namespace. Keep unknown
and foreign unfinished ownership held. Preserve independent durable stop
evidence and the existing remote lease fences.
- Use the shared accepted-result ownership reconciler. Bind settlement
to the observed lease token, generation, boot, PID, namespace and
process birth. Retain finalizer, cleanup, current-controller, live-owner
and remote exclusions.
- Share the remote-owner reader across recovery consumers. Scope runner
leases to the same company, task and run. Do not make local PID probes
for remote accepted-result ownership.
- Gate original-host containment on `runtime:confirm_owner_stop`. Bind
it to the current task, source, assignee, original expiry, lease and
Stop state. Audit separate controller/provider containment and reviewed
action outcomes.
- Keep containment separate from ordinary Retry. Use one canonical
committed receipt and the normal admission, budget, workspace and
lineage checks. Preserve the recorded external-action outcomes.
- Keep the controller-only confirmation actionable for both result-less
work and accepted results. Result-less work gets its next source-bound
attention in the same transaction. Accepted results keep their original
result and use normal shared settlement/finalization.
- Keep identity inspection quiet. Preserve the original foreign-owner
expiry and use the existing scoped finalizer in accepted-result recovery
controls.
- Update the recovery card, validators, execution projection, docs and
access maps. Reuse existing canonical columns; add no migration or
ownership backfill.

## Verification

Current publication head: `71ea76b568de09d6313d8992e803131d9d5c54f3`.

**The current fix has a causal red/green check.** On the test-only
revision, both foreign-local stopped/fenced controller cases failed
because recovery probed saved runner, process-group and provider PIDs in
this host's namespace. This measured the wrong probes; it did not run
duplicate providers. After the fix, the same two cases passed every
no-probe, no-claim, no-controller-stamp, no-stop-receipt and
unchanged-authority assertion. Three existing remote preparation, grace
and fenced-controller progress cases also passed. Total: **5 passed, 0
failed;170 filtered rows excluded**. All three normal affected App
TypeScript programs passed on this head (660,681 and428 roots).

The fix separates a controller wait from execution stop proof.
Independent live/grace evidence still schedules recovery. A
stopped/fenced foreign controller can pass this namespace guard only for
remote execution, which keeps its later checkpoint and never-launched
checks. Foreign local execution stays held before runner or provider PID
inspection.

Earlier source-scoped results remain evidence, not fresh executions on
the current head:

- On `c86dd741`, the complete failed-owner file passed **25/25**, with
no skips or filters. Empty scans wrote no controller stamps. Foreign
ownership kept its original expiry. Normal returned-claim settlement and
attention at the exact deadline passed.
- On `c86dd741`, **15 selected accepted-result/restart controls
passed**. The81 filtered rows are excluded. Both completion cases used
the existing scoped finalizer and kept their accepted result without
another provider turn.
- Trusted CI at synthetic merge `50d072a5` (actual master `1b7fe334`
plus `c86dd741`) passed the complete **17-case maintenance-server
file**. This includes the original90-real-second quiet observation and
two qualified hourly windows. The quiet case took118,918ms. The four
jobs that failed on the earlier publication all passed on that synthetic
merge.
- On `6243b32c`, the complete ownership-report file passed **29/29**.
Native NULL namespace stayed unprovable and generic compatibility
remained. Three selected recovery controls passed;170 filtered rows are
excluded.
- On `f81f2c3c`,356 business cases passed: whole ownership-route221,
recoveryUI70, owner-proof38 and controller-confirmation8; selected
accepted-result authority18; and one real connected-chat containment,
separate authorized Retry, useful Native result and downstream reply
case. The2,024 filtered rows are excluded. A separate earlier
accepted-result causal case kept the original result and completed after
restart without replay.

Normal Shared exports, App programs, UI typecheck, token gates and map
checks passed on their recorded revisions. Unchanged DB, adapter, Eval
and Runner inputs/outputs have explicit carry proofs. The clean normal
merge with fetched master `8fe524f7` preserves all31 exclusive owned
paths and14,827 unrelated master entries. The nine overlapping paths
were reviewed, including the upstream unidentified-run stop action.
Fresh CI must validate that actual merge composition.

PG cases used existing external PG17, fresh owned databases, pools at
most4, one worker and no SQL observer during cases. Registered clients
and recorded process groups closed. Independent final SQL audits found
zero campaign sessions before normal database cleanup. Original failing
fixtures remain retained.

Original failures are preserved. One new test first failed compilation
due to a missing getter import; an import-only correction passed its
full owning programs before the causal red/green check. A local quiet
attempt stopped in the normal prerequisite hook before any business case
and its collector rejected two normal build markers. The later17-case CI
pass establishes quiet recovery; it does not turn that failed local
collector into a pass. An earlier useful case passed before its private
cleanup collector failed on an owned read-only directory; normal cleanup
completed after all owners closed. Earlier runner-shutdown cancellation
and a separate private whole-suite failure remain unexplained.

Current-head checks are complete:57 success,1 neutral and3 skipped,
including the successful aggregate CI gate. The legacy status passed.
Greptile is5/5 and Ponytail gives an explicit5/5 Ship verdict for this
head. All seven review threads are resolved. Full recursive typecheck,
the full test suite and release build were not repeated locally. No
result here proves deployed suspension, a live provider or termination
on another host.

## Risks

- Unknown physical ownership can require an owner/admin action. A false
containment statement can be unsafe.
- New work, reassignment, Stop, renewed authority, accepted results or
pending cleanup can invalidate an action. The operator must refresh.
- Controller confirmation does not prove provider/tool containment.
Completed external actions must not be replayed.
- Generic legacy and child-process compatibility remains separate from
the strict Native controller policy.
- Existing saved state and canonical schema are preserved. Permanent
safety refusals remain held with their normal named recovery path.

## Model Used

OpenAI `gpt-6.1-sol` agents implemented and tested the changes with
reasoning, repository tools and code execution. A GPT-6-based Codex
managing agent reviewed and integrated them. The host does not expose
the manager's exact build or the context-window sizes.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-09 12:06:11 -06:00
DottaandClaude Opus 5.5 dc0534d1b5 feat(recovery): destroy a retained orphan sandbox once nothing references it (#19976)
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs use remote sandboxes held by environment leases
> - #19964 settles an orphaned active lease when its holder is proven
gone. It stops and keeps the sandbox, because recovery may still need
its files
> - Nothing destroyed that retained orphan later, so a stopped sandbox
could stay at the provider indefinitely
> - This pull request destroys a retained orphan once no recovery,
preservation or retention obligation references it, from the reconciler
pass only
> - The benefit is that orphaned sandboxes do not leak, and a sandbox
that recovery still wants is never destroyed

## Linked Issues or Issue Description

Refs #19964
Refs #19933

**What happened?**
After #19964, an orphaned sandbox is stopped and kept (`released` with a
stop-and-retain receipt). No path destroyed it when recovery no longer
needed it.

**Expected behavior**
A retained orphan is destroyed once database facts show that no
obligation references it. It is never kept past the environment's
retention window (the provider-attested lease `expiresAt`) only because
of a soft reference. A sandbox that an active operation or a live
recovery path still needs is never destroyed.

**Steps to reproduce**
1. Let a run's server die after the run is terminal and before its
sandbox lease is released.
2. Let the ownership settle stop and keep the sandbox.
3. Resolve the run's recovery. The stopped sandbox stays at the
provider.

## What Changed

- `sandbox-stop-and-retain.ts`: an ownership settle's stop intent also
writes a top-level `orphanedRunLeaseRecovery` marker. The marker stays
after the stop intent is removed.
- `environment-retained-orphan-settlement.ts` (new): the settle for
`environment_retained_orphan_sandbox`. It locks the lease row and checks
database facts:
  - the run is terminal;
- always held: native export resume, stop-and-retain intent, in-flight
cleanup claim, workspace restore recovery, workspace recovery cleanup,
bound source-recovery consumer, pending legacy shutdown fence, or
another live lease on the same provider resource;
- held while the stop receipt is a usable source (`expiresAt` not
passed): pending native export or preservation fence, open recovery
action on the issue, or a queued, scheduled or running run on the issue.
- Then it moves the lease to `pending_cleanup` (ephemeral) with a
compare-and-swap. The pending-cleanup sweep destroys it from the
recorded lease. The settle never calls a provider.
- `recovery/ownership-registry.ts`: the kind lists only releasable
retained orphans (an SQL mirror of the checks), so a held orphan is
never polled. It reports `nextDueAt` = the earliest retention-window end
of a softly held orphan.
- `recovery/ownership-invariant-report.ts`: an `OwnableKind` may report
`nextDueAt`; the pass folds it into the report's next due time, so the
reconciler wakes when the window ends.
- `environment-cleanup-state.ts`: the first destroy attempt of a
released orphan is due at once.

## Verification

- `cd server && npx vitest run
src/__tests__/environment-retained-orphan-sandbox.test.ts
src/__tests__/ownership-reconciliation-report.test.ts
src/__tests__/environment-run-lease-ownership.test.ts
src/__tests__/heartbeat-orphaned-active-lease-sweep.test.ts
src/__tests__/heartbeat-pending-cleanup-sweep.test.ts` (embedded
Postgres).
- Red on master, green here (tests written after the code): the settle
marks the lease; an unreferenced retained orphan is destroyed through
the cleanup sweep; a softly held orphan waits until its window ends and
the pass reports that due time.
- Counter-cases (green): never destroyed with an open recovery action, a
queued run, a scheduled retry, a stop-and-retain intent, a native export
resume, a workspace restore recovery, an in-flight claim, a pending
legacy shutdown fence, another live lease on the resource, or an active
run; a read-only pass changes nothing; a retained sandbox without the
orphan marker is left alone.
- Full typecheck is left to CI (local memory limit).

## Risks

- A retained orphan saved by #19964 before this change has no marker, so
it is not destroyed by this path. The window was short (since #19964
merged).
- A held orphan with no `expiresAt` is released on the first reconciler
pass after its last reference clears. If that clearing writes no signal
the reconciler wakes on, the release waits for the next pass.
- The SQL listing mirrors the settle's checks except the pending native
export, which the settle checks in code. Such an orphan stays on the
settle watch until that export settles.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit, embedded Postgres tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 01:38:22 -06:00
DottaandPaperclip 2e52dfa864 fix(slack): retain DM thread tasks after completion (#19486)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Slack conversations store that work in tasks and comments.
> - A native thread must keep its task when work finishes.
> - The server treated all DMs as linear conversations.
> - A reply in a completed DM thread therefore started a new task
without its earlier comments.
> - This pull request uses the native thread identity for task
continuity.
> - People can continue the same work with its saved context.

## Linked Issues or Issue Description

**What happened?**

A reply in a Slack DM thread created a new task after the preceding task
was Done or Cancelled. It also created a new task when the saved
conversation was completed. The new task lacked earlier comments.

**Expected behavior**

A native Slack thread keeps one task in channels and DMs. Later replies
retain earlier user and agent comments. Top-level DMs keep their
existing task generation rules.

**Steps to reproduce**

1. Start work in a Slack DM thread.
2. Mark its task Done.
3. Reply in the same thread.
4. Before this fix, the reply creates another task.

**Paperclip version or commit**

Reproduced from master at eb5c24b61c.

**Deployment mode**

Built from source. Tests use local PostgreSQL and provider fixtures.

Related PR: #16151 covers the DM mention exemption. This fix preserves
that behavior. GitHub searches found no duplicate fix for completion
routing.

## What Changed

- Detect Slack native thread roots in the Slack module.
- Classify native DM threads as persistent threads for task routing.
- Select the resource type from the actual DM destination. Keep DM
access checks.
- Preserve DM task controls. Select the latest DM task even after
completion. Require a delivered control receipt before an explicit new
task generation.
- Add completion, duplicate, saved-history, separate-root, Idle, and
explicit-control regressions.
- Document the task boundary. Correct only the inaccurate DM-thread
statement in the existing feature recipe.

## Verification

- Before the production edit, the new PostgreSQL regression failed in
all three threaded-DM completion cases: two conversations existed
instead of one. The channel-thread and top-level-DM controls passed.
- The initial eight routing cases passed. CI then found five existing
slash-control failures. All five reproduced locally. Greptile reported
the same regression at 4/5.
- The first correction passed all 16 focused PostgreSQL cases. Secondary
review then found that slash commands skipped Done, Cancelled, and
completed DM tasks. Four new cases failed before the lookup change.
Commit 23ff4f04f passed all 20 focused cases, including status, explicit
new/close, delivered receipts, and a new task generation. The original
same-thread continuity cases still pass.
- Command: `pnpm exec vitest run
server/src/__tests__/chat-channels.integration.test.ts -t 'preserves
Slack thread history across completion|keeps separate Slack DM thread
roots|returns two answered Slack turns to Idle|binds Slack DM slash
controls|treats D-prefixed Slack status|serializes duplicate Slack
admission with an owned provider-reply settlement|suppresses a queued
provider reply when destination reach|continues to accept authenticated
Slack slash close/new'`.
- Map validation and `git diff --check` passed. Map validation proves
structure only.
- All 107 neighboring Slack adapter, SDK runtime, and receipt tests
passed across three files.
- The initial complete server typecheck passed with
`PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm --filter @paperclipai/server
typecheck`, including all three test-tree shards.
- Full local recursive typecheck and build stopped because this
workspace has no `cargo`. A separate default server typecheck was killed
under concurrent memory load; its lower-memory rerun passed. The broad
local test run was interrupted to free memory, so no full-local-suite
pass is claimed. The complete server typecheck passed again for
correction 367b6f9b3, including all three test-tree shards. The [full CI
pipeline](https://github.com/paperclipai/paperclip/actions/runs/42250458200)
passed for exact head 367b6f9b3f. This
includes typecheck, build, all chat and other test lanes, browser E2E,
Runner, and aggregate verify. The first CI run passed broad typecheck,
build, E2E, Runner, and other test groups, but its chat failures remain
recorded above.
- The initial regression launch could not load the unbuilt plugin SDK.
It ran after building the required workspace dependencies. A test-only
resource assertion was corrected to read persisted DM resources, which
the settings list omits. The completed-conversation case preserves an
existing review status.
- The initial correction test used a follow-up timestamp older than the
delivered control. The chronology gate correctly rejected it. A current
provider timestamp now verifies the new generation without weakening
that gate.
- Greptile's map finding was addressed by removing recipe expansions and
all agent-access map changes. Both review threads have replies and are
resolved. Greptile reviewed exact head
367b6f9b3f at [5/5 with zero new
findings](https://github.com/paperclipai/paperclip/pull/19486#issuecomment-6689117840).
The secondary Paperclip Review rated that head 4/5 and found the
completed-task slash lookup error. Commit 23ff4f04f fixes this finding.
Greptile and Paperclip Review both rate head
23ff4f04f6 at 5/5. All three review
threads are resolved. The [full CI
pipeline](https://github.com/paperclipai/paperclip/actions/runs/42255926149)
now passes on this exact head. This includes typecheck, build, all test
lanes, browser E2E, Runner checks, Canary Dry Run, and aggregate verify.
All checks are terminal: 54 success, 1 neutral, and 3 skipped. Snyk also
passes. The PR has no merge conflict.
- The complete server typecheck passed again on 23ff4f04f, including all
three test-tree shards.
- The new test first asserted the conversation-list state. That API
displays terminal tasks as completed before the control is sent. The
test now reads stored binding state to check that completion waits for
the delivered receipt.
- No live Slack send or deployment was performed.

## Risks

- Native DM threads now reopen the existing task after Done or
Cancelled. Explicit new/close controls still release the binding after
their response is delivered. Top-level DMs still start new generations
after completion.
- Existing split tasks stay separate. Future replies use the latest
saved binding. This change does not merge old tasks or copy surrounding
DM history into a newly created thread.
- No schema, credential, provider scope, or permission changes are
required.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, repository
editing, and code execution. The runtime does not expose a more precise
serving model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-07 08:35:29 -06:00
DottaandClaude Opus 5.5 9a52a2dfeb fix(native): retry ACPX session.open startup timeouts and give cold Claude a 60 s bound (#19661)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs start an ACPX provider through Runner. `session.open`
opens the provider session before any prompt.
> - On a loaded host, a cold ACPX Claude `session.open` took longer than
its 30-second bound. The controller reported
`runner_local_connect_failed: PRP command session.open timed out`.
> - The server classified this as the generic
`native_session_interrupted` with `ambiguous_state`, so the task was
blocked at once. A person had to click Retry, sometimes several times.
In one restart-chaos storm, one task needed three Retry clicks, and on
another task both Retry runs timed out too.
> - The server already has a bounded same-run bootstrap retry for
failures that come before provider work (#18854). It did not recognize
this timeout. The 30-second bound also cut off normal slow startups.
> - This pull request gives cold ACPX Claude a 60-second `session.open`
bound, like Pi and OpenCode (#19487). It also retries a startup
`session.open` timeout automatically on the same run, from clean state.
> - The benefit is that a slow cold start on a busy host completes, or
retries itself, instead of waiting for a person.

## Linked Issues or Issue Description

Refs #18854 (bounded bootstrap retry for definite `session.open`
failures), #19487 (60-second cold OpenCode admission), #19665 (the
bootstrap proof that this change reuses).

**What happened?**

ACPX Claude runs failed with `runner_local_connect_failed: PRP command
session.open timed out` about 35 to 40 seconds after they started. The
coordinator recorded `native_session_interrupted` with `recoveryMode:
"ambiguous_state"` and `providerSessionEstablished: false`. The task was
blocked with a board-owned action. No provider event and no checkpoint
existed. The host 1-minute load was 17 to 44, sometimes with no restart
nearby.

**Expected behavior**

A normal slow cold start completes. A `session.open` that still times
out during runner startup, with no provider event, no checkpoint and no
turn, retries on the same run with backoff. At most two automatic
attempts follow the first. A run where provider work may have started is
never retried automatically.

**Steps to reproduce**

1. Run ACPX Claude tasks on a host with a 1-minute load of about 25 or
more.
2. Observe a run fail after about 35 to 40 seconds with `PRP command
session.open timed out`.
3. Observe the task blocked with `ambiguous_state` and a manual Retry.

**Paperclip version or commit**

`master` at `6c7cc2c5d6`.

## What Changed

The bound (evidence below):

- `coldAdmissionTimeoutMs` gives cold ACPX Claude 60 seconds (live
adoption keeps 30 seconds). The runner's Rust sidecar
`session_open_timeout` and the sidecar's provider handshake
(`acpxSessionHandshakeTimeoutMs`) give Claude and Pi 60 seconds too. All
three bounds must agree, or the shortest one fails first. Claude's warm
run attachment uses the same cold budget, like Pi.

The automatic retry:

- `nativeSessionFailureSourceCode` returns
`provider_session_open_timeout` only for exactly `<startup failure
code>: PRP command session.open timed out`. The startup failure codes
are `runner_local_connect_failed`, `runner_direct_wss_failed` and
`runner_ingress_unavailable`. The transport uses them only before
startup completes. After startup the code is
`provider_transport_failed`, which is unchanged.
- `nativeProviderRecoveryEvidence` treats the cause as pre-session.
Durable evidence still decides: a bootstrap retry needs no provider
event (`session.started`, `turn.started` and the others) and no
checkpoint. The existing budget applies: three attempts with backoff of
about 30 seconds, then about 60 seconds. When the budget is used, the
task is blocked with the existing "Provider startup failed on 3 attempts
before any provider work began" action.
- A timeout is indeterminate: the runner can open a provider session
after the controller stops waiting. Live runs showed both leftovers, so
the retry now starts from clean state:
- `prepareNativeSessionOpenTimeoutRetry` (server) runs when the failure
settles. It quarantines a local runner root only when the
restart-recovery proof (`runnerdStateProvesIncompleteBootstrap`, #19665)
shows that no turn began and every recorded process is gone. Any other
root, and any remote runner, stays `ambiguous_state` as before.
- `resetAbandonedBootstrapRecord` (runner ACPX adapter) runs on a fresh
open (no runner-recorded identity). If ACPX has a saved record for the
session key with no messages, no prompt and no events, it is marked
`reset_on_next_ensure`. Claude saves nothing before the first prompt, so
reusing such a record fails the first turn with
`session_resume_required`. A record that holds any conversation is
unchanged.
- A chat retry of an exhausted `provider_session_open_timeout` run is
accepted like the other pre-provider causes.
- `doc/execution-semantics.md` and the Runner README describe the cause
and the bound.

## Verification

**Measured startup distribution** (ACPX Claude
`runner.session.bootstrap` / `runner.session.startup` spans from
restart-chaos run events, 1-minute load 8 to 44):

| Source | Bound | Successful cold bootstraps (s) | Failures |
| --- | --- | --- | --- |
| Earlier storm evidence | 30 s | 2.9, 4.7, 7.4, 8.6, 8.7, 26.2, 27.4,
27.6, 29.2 | startups ended at 34.9 to 40.3 s (the 30 s `session.open`
bound plus spawn and `run.prepare`) |
| This branch, wave s3601 | 60 s | 6.3, 7.8, 8.1, 13.4, 14.4, 22.3,
22.8, 22.9, **37.3** | 0 `session.open` timeouts |
| This branch, wave s3602 | 60 s | 13 samples, p50 10.5, p90 17.9, max
25.7 | 0 `session.open` timeouts |
| This branch, wave s3610 (final head, load 28 to 42) | 60 s | 10.4,
11.3, 11.7, 12.0, 13.3, 14.2, 29.6, **30.4**, **30.9** | 0
`session.open` timeouts |

Under load, successes clustered just under 30 s and failures sat just
above it, so the 30 s bound was cutting the distribution off. With 60 s,
cold starts of 30.4, 30.9 and 37.3 s succeeded that the old bound would
have failed. 60 s is about 1.6 times the largest observed success, and
it matches Pi and OpenCode.

**Live restart-chaos waves** (local, ACPX Claude only, 4 in flight,
`--rc06-masked`, real provider, K1/K2 restarts):

- s3601 (8 tasks, load 19 to 32): 7/8 PASS. No `session.open` timeout.
The one failure is an R3 Retry signature after K1
(`native_restart_recovery_blocked`), not this cause.
- s3602 (8 tasks, load 17 to 39): 8/8 PASS. No `session.open` timeout.
Three Retry clicks were for `native_provider_terminal_failed`
(`provider_turn_lost_on_restore`), a different restart signature.
- Forced timeout, s3609 (4 tasks, B5, K2): a local, uncommitted fault
injection gave every other cold startup a 1.5 s `session.open` deadline.
Result 4/4 PASS. Run 4fe6537a timed out, retried itself on the same run,
and then succeeded through four K2 restarts with no person involved. Run
02799a85 hit the injection three times in a row; the budget stopped it
after attempt 3, and one Retry completed it.
- Before the two clean-state fixes, the same forced wave (s3607 and
s3608) showed the retry reusing leftover state: `turn.start` failed with
`session_resume_required`, and then "ACPX session is not open". Those
runs are the red evidence for the two clean-state fixes.
- Final head, s3610 (8 tasks, K1/K2, no injection, load 28 to 42): **8/8
PASS**, 0 `session.open` timeouts. The one Retry click (t06) was for
`provider_turn_lost_on_restore`, a different restart signature.

**Tests:**

- Red before the fix: `native-startup-retry.integration.test.ts -t
timeout` (embedded PostgreSQL, real executor, real resume claim). The 3
retry cases failed with `native_session_interrupted` /
`ambiguous_state`.
- Green: `native-startup-retry.integration.test.ts` passes 36/36. New
cases:
- for each startup failure code, a timeout retries on the same run with
backoff, keeps the task `in_progress` with an agent-owned
`resume_native_run` wake, and blocks after attempt 3 with a board-owned
action;
- no automatic retry when `session.started` or `turn.started` is
durable, when the code is `provider_transport_failed`, or when another
command timed out;
- a real runner root with an opened session and no turn is quarantined;
a root with a turn command stays `ambiguous_state` and intact; a remote
runner is never prepared.
- `codex-runtime-adapter.test.ts`: 113 pass. New cases: a 45 s delayed
handshake is admitted for Claude and rejected for Codex; a
never-prompted record is reset only on a fresh open; records with
messages, a prompt or events are kept.
- `abandoned-bootstrap-record.test.ts` (real `createAcpRuntime` with
`createRuntimeStore` and a controlled ACP peer that, like Claude, cannot
load a never-prompted session): without the reset, a fresh open reuses
the record and the first turn fails with SESSION_RESUME_REQUIRED after
`session/load`. With the reset, the provider receives `session/new` and
the turn completes.
- `vitest run src/drivers/acpx src/cli` in the runner: 107 files, 1,935
tests pass.
- `native-session-executor.test.ts -t "session.open|startup|bootstrap"`:
144 pass. `runnerd-codex-transport.test.ts -t "cold process admission"`:
4 pass. `cargo test -p paperclip-runner-core --lib admission_budget`
passes.
- `npx tsc --noEmit -p server` and the runner `tsc -p
tsconfig.test.json` pass.

## Risks

- **This changes Runner Rust code. The Daytona runner image must be
rebuilt after merge** for remote sandboxes to get the 60-second sidecar
bound. Until then, a remote Claude `session.open` still fails at 30
seconds in the sidecar (as a definite failure, which already retries). A
remote timeout stays operator-owned, because a remote root cannot be
proven clean here.
- A cold ACPX Claude start that hangs now fails after about 60 seconds
instead of 30 seconds.
- A timed-out `session.open` may have created an empty provider session.
No prompt was sent. The retry quarantines that runner root and asks ACPX
for a new session.
- Same-run retries of a fresh-attempt Retry run can still fail with
`isolated_task_directory_binding_mismatch`. This is RC-35 (workspace
re-preparation), which has its own fix in progress.
- No schema or API change. The new error code appears on runs and in run
error codes.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with
tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 19:37:11 -06:00
DottaandPaperclip 0819f4ea78 fix(native): verify controller ownership before retrying saved turns (#19638)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native task turns can keep a provider conversation from an earlier
run.
> - A copied conversation checkpoint names the new run before its
controller has attached.
> - A failed attachment can make recovery treat that copied identity as
current execution authority.
> - The current run needs its own durable controller checkpoint, or a
valid staged attachment, before exact resume.
> - This pull request checks that ownership and uses the normal Retry
path to continue a saved user turn after verified cleanup.
> - The task can continue once with its saved work and history.

## Linked Issues or Issue Description

Refs #19399 and #18860.

**What happened?**

A new user comment resumed a completed native conversation. The retained
runner process was gone before the new controller attached. The new run
still had a copied provider session ID. Recovery selected exact resume
for that new run and then failed its unchanged owner and lease checks.
Ordinary first-attempt failure settlement also omitted the saved-user
Retry handoff that restart recovery already uses.

**Expected behavior**

Exact resume requires current controller evidence. A saved user turn can
request one normal Retry after this infrastructure failure. The normal
path must verify provider stop, saved files, current user authority and
all execution gates before it starts a fresh session.

**Steps to reproduce**

1. Finish a native warm run and retain its provider checkpoint.
2. Admit a user comment as a new run and copy the provider checkpoint
through the real resume producer.
3. Stop the retained runner before attachment. The real remote process
check reports that the process is not running.
4. Observe that the copied provider ID selects exact current-run resume.
Observe that ordinary failure settlement saves no Retry request.

## What Changed

- Check a bounded encrypted controller header for fenced remote Runner
recovery. Keep local Runner recovery on its existing disk-backed path.
Match the run, conversation and runner identity. Preserve the existing
staged-attachment path.
- Share the existing saved-user Retry admission between restart recovery
and ordinary terminal native settlement. Keep its exact infrastructure
errors, user wake, comment, provider-turn and retry-chain checks.
- Add real PostgreSQL and stopped-process regressions. Verify malformed
and foreign checkpoint holds, current-owner resume, user authority and
Stop. Verify saved-file recovery creates one fresh-session successor.
- Document the ownership and Retry contract.

## Verification

- RED: the real checkpoint rebind producer and stopped-process check
selected exact current-run resume without a current controller
checkpoint.
- RED: ordinary settlement created zero Retry receipts for the real
native failure code; expected one.
- Review RED: a real current local controller file existed with no
database checkpoint rows. The initial gate returned ambiguous state. The
fix takes a required trusted execution-target kind and applies the
database requirement only to the remote persistence path.
- Initial local revision: `3662055880ccf8e09c3c50a340897e16c78627e7`,
macOS ARM64, Node 26.4.0, pnpm 9.15.4, embedded PostgreSQL.
- Initial server Vitest: 42 selected cases passed across controller
recovery integration, remote restart adoption, ordinary run completion
and retained workspace recovery. The cases cover the stopped-process
producer, current/foreign/malformed ownership, automatic user
continuation, Stop, unsettled turns, system wakes and a manual Retry on
first use.
- `pnpm --filter @paperclipai/server typecheck`: passed with this
worktree's Runner, shared and SDK builds and all seven server layers.
- `node scripts/check-module-boundaries.mjs`, whitespace checks and the
added-code-comment audit: passed.
- Review correction: `8c795e2e0e44b5209da3e228593c5cd35e948cae`. The
real local/remote controller case passed against a fresh PostgreSQL
cluster. It verifies local disk recovery without database rows, remote
current-owner resume, and remote missing/foreign/malformed holds. Fresh
`node scripts/typecheck-layers.mjs` passed all seven server layers. The
existing offline chat reader now requests metadata only and retains its
full independent native-state safety check.
- A broader local recheck hit PostgreSQL setup/cleanup hook timeouts.
These are retained as failed setup attempts; the fresh-cluster
regression above passed. Required CI supplies broad final-head coverage.
- Settlement-order correction:
`dafb721c3bbb7ae86fbc3c9c6c953ace29ca088b`. The real PostgreSQL
regression first proved both failures: Stop after an already-failed row
created one Retry instead of zero; a DB-native promotion with an old
legacy caller snapshot created zero instead of one. All six settlement
scenarios now pass. The hook requires the local failed outcome and the
canonical settled native row. Fresh seven-layer server typecheck passed
again.
- Required shared fixture base integrated at
`d22d8f9d5ee3dc36550a94d55c1d8376f3b9aab2`. The six runner-skills
fixture cases and all seven server typecheck layers passed. Required CI
passed for this head:
[workflow](https://github.com/paperclipai/paperclip/actions/runs/42309419592),
including all server layers, tests, build, Runner verification and E2E.
The final snapshot has 57 successful checks, three intended skips and
one neutral result. Current-head Greptile is 5/5; both concrete review
threads are resolved. The independent re-review ended without a final
response; its prior Stop finding has a real RED/GREEN repair.

The focused command was run from `server/`:

```sh
../node_modules/.bin/vitest run --config vitest.config.ts src/services/heartbeat/run-completion.test.ts src/services/native-runtime/native-controller-recovery.integration.test.ts src/services/native-runtime/native-session-executor.test.ts src/services/workspace-recovery.test.ts -t 'replays saved user intent|does not assign current-run resume authority|remote controller restart adoption|continues a user turn once|does not continue automatically'
```

## Risks

- A terminal failure of a fenced Runner adds one exact-run header read.
The read selects one encrypted header and limits inspection to 1 MiB.
Invalid or missing ownership evidence stays held.
- Saved user input can request one automatic Retry. It uses normal
preservation, current authorization, budget, approval, Stop and cleanup
gates. Unsettled provider turns and system wakes do not start an
automatic successor.
- This PR changes no provider process, credential, model or lease
authority checks.

## Model Used

OpenAI GPT-6 family through Codex, with reasoning, tool use and code
execution. The model is inherited; no override was used. The exact
serving model ID and context-window size are not exposed by this
runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 17:31:23 -06:00
2e35730637 feat(chat): receive sealed Slack events through the current SDK owner (#19019)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Slack connectors turn authenticated messages into company work.
> - Direct callbacks need an awake tenant.
> - Cloud can retain authenticated asynchronous events until the tenant
wakes.
> - Replay must preserve the original callback proof and current
endpoint authority.
> - Configuration changes must retain registration responsibility
through failures.
> - This PR enrolls verified Slack identities and drains sealed events
through the normal SDK owner.

## Linked Issues or Issue Description

**Subsystem affected**

Server chat connectors and signed Cloud connector delivery.

**Problem or motivation**

Direct Slack Events API delivery depends on an awake tenant. Broker
outages, lost responses and configuration changes must preserve delivery
and registration responsibility. Enrollment failures must also leave
other connector events able to drain.

**Proposed solution**

Stage nonsecret enrollment or retirement intent inside the local
configuration transaction. Let the existing connection-event owner
perform broker requests and retries. Replay sealed events through the
current Slack SDK with their original authentication evidence.

**Alternatives considered**

An early acknowledgment for every callback changes command and modal
semantics. A second delivery ledger duplicates existing ownership.
Reusing a new stack identity to retire old registrations crosses an
authorization boundary.

A companion Cloud broker change is required. Related setup fixes: Refs
#18837, Refs #18878. Kickoff searches found no competing async
enrollment consumer. The ROADMAP's connector setup and access controls
stay in place.

## What Changed

- Persist enrollment and credential-free retirement intent before local
configuration commits. Broker failure does not make a completed pause or
removal look unsuccessful.
- Bind company, endpoint, connection, public ID, generation and
authorized actor. Preserve lost-response responsibility and recover only
the supported same-instance configuration outage.
- Permit an explicit move to a new authorized stack only after confirmed
old retirement. Keep the old identity until the new broker receipt
matches the current intent and generation.
- Show permanent signing-credential failures as attention. Preserve
transient retries and reject uncertain inbound authentication before SDK
admission.
- Drain events and acknowledgments before enrollment reconciliation.
Persist a bounded retry on the exact busy intent so other due endpoints
can enroll or retire. Preserve current custody under concurrent changes.
- Stage retirement in the Apps removal transaction too. Archive the
endpoint and connection, retain the original subject and instance, then
revoke credentials. Retirement does not need the deleted Slack
credential.
- Acknowledge a malformed sealed Slack receipt as invalid once. Keep
actual broker outages on the retry path.
- Run the real configuration, SDK ingress, generation fences and public
Apps removal paths in the default SQL suite. Keep Vitest transport
helpers outside production services.
- Preserve original callback bytes, signature, time and public URL
evidence. Use the existing transactional chat-delivery ledger for
redelivery.
- Provide a checked-in prewire guard and launcher using the current Node
executable. Add broker, worker, SDK, SQL and controlled-process
acceptance.
- Stabilize AI account summaries by ID and grant ID. This fixes a CI
migration-repeat equality failure without changing SQL, authorization or
reconnect behavior.
- Open the four configured database connections before the enrollment
test freezes its clock. Keep all 35 original assertions and all
timeouts.
- Measure status-card dispatch entry before its mock does database work.
Keep the original strict 500 ms bound and check any still-armed earlier
deadline. Bound that fixture's pool to four connections.

## Verification

- Candidate: `54fa701d41b4ced62f4ff338d39b946015670ae3`. Normal merges
preserve master `47027e7` and all owned changes. A read-only merge
assessment against fetched master `cb5d7e44` is conflict-free. The only
overlapping file has separate AI test additions. The branch has not
merged that later master; fresh remote checks must validate the current
PR test tree.
- At this head, normal server production typechecking and declaration
emission passed in 35.24s. The affected actual CI test shard passed in
6.27s. Other unchanged CI layers, 92 affected pure controls and 12 guard
controls retain their source-matched results.
- All **15 default real-PostgreSQL cases passed in 34.66s**, with no
skips or filtering. They cover normal configure and SDK ingress,
rollback, stale generations, malformed receipts, busy-lease fairness,
concurrent intent custody, unknown errors, and public Apps removal.
Removal tests check actual credential revocation, retained retirement
through outage and repeat removal, one ordinary retry and confirmed
retirement, transaction rollback and foreign-company refusal.
- The public removal counterfactual failed before the fix: archival and
credential revocation passed, but the inactive intent was absent. The
same test source passed after the staging call. Separate counterfactuals
also reproduced busy-endpoint starvation and malformed-receipt retries.
Original failures remain retained.
- The paired broker/SDK/process journey passed in 31.98s at ancestor
`0324c011`. It preserves callback proof, lost-ACK redelivery, exactly
one comment, one delivery and one successful controlled process. Its
test, producer, SDK, guard and runtime bytes are unchanged. The new
public Apps archive branch is not called by that journey; its evidence
carries on that checked basis.
- The new SQL cases need no companion Cloud checkout. Normal CI runs
them with the repository fixture. For an existing dedicated PostgreSQL
fixture, set `PAPERCLIP_SLACK_OWNER_TEST_DATABASE_URL` and
`PAPERCLIP_TEST_POSTGRES_ADMIN_URL`, then run `node
server/scripts/test-slack-cloud-owner.mjs` with Node24. The optional
paired launcher still requires the normally built companion Cloud
checkout.
- Every pool was bounded to four or fewer connections. The final 15-case
run sampled three runtime clients plus one observer. The independent
final audit found zero clients and all five recorded processes absent.
Earlier closing-socket observations are kept separate from final
cleanup.
- Earlier unrelated CI failures are documented: [enrollment
timeout](https://github.com/paperclipai/paperclip/actions/runs/41944373937/job/126973483754)
and [status-card observation
bound](https://github.com/paperclipai/paperclip/actions/runs/41944373937/job/126973483843).
The enrollment test retains all 35 assertions and its original timeouts.
A controlled connection probe established the frozen-clock mechanism;
the original remote access history remains unknown. The status-card
fixture measures dispatch entry with the original strict 500ms bound. A
real delayed-dispatch control fails that bound.
- Full local workspace typecheck, build and test inventory was not run
because its embedded-cluster and wider resource paths exceed this local
lease. Exact-head remote CI now passes: 54 successful checks, three
skips and Ponytail's neutral check, with aggregate verification green
and the legacy security context independently confirmed successful. Both
complete bot reviews are 5/5, with all 13 paginated review threads
resolved. Seven execution jobs originally reported runner shutdown; one
diagnostic rerun recovered them and the dependent aggregate check.
Original logs remain retained, and the underlying shutdown cause is
unknown. This is recovery evidence, not a product-fix claim.
- Provider responses are controlled. This proves local SQL, broker, SDK
and process behavior. It does not prove live Slack, agent tools,
synchronous callbacks or deployed suspension.

## Risks

The companion broker is required. Outage or conflicting authority can
leave explicit pending or attention responsibility. A move between
stacks requires confirmation from the old authorized owner first.
Synchronous commands, actions, modal validation, options and challenges
retain their deadlines and endpoint blocker. This slice grants no full
Slack endpoint sleep exemption. No deployed suspension or savings claim
is made.

## Model Used

OpenAI `gpt-6.1-sol` workers through Codex, with reasoning, repository
tools and code execution. The served backend revision and context limit
are not exposed. A GPT-6-family coordinator performed technical review
and integration.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-11-06 15:22:51 -06:00
DottaandClaude Opus 5.5 e680fa6814 fix(native): retry startup interrupted by a closed runtime API listener (#19462)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs prepare a remote workspace, then start a runner and a
provider session
> - A graceful server shutdown closes the HTTP listener before it drains
runs
> - A run that is still preparing then asks for the local runtime API
origin, and the lookup fails with
`local_runtime_api_listener_unavailable`
> - Recovery did not know this code. It treated the failure as an
ambiguous provider state and blocked the task behind a board-owned
action, although no runner or provider process existed
> - This pull request classifies the closed listener as a definite
pre-session failure
> - The benefit is that a deploy no longer strands a preparing task. The
same run gets a bounded bootstrap retry that the next server process
claims

## Linked Issues or Issue Description

**What happened**
A native Codex task on a remote sandbox spent about ten minutes in
workspace preparation. A deploy started a graceful shutdown during that
time. When preparation finished, the executor called
`resolveLocalRuntimeApiUrl()` to build the tool authority. The listener
was already closed, so it threw
`local_runtime_api_listener_unavailable`. No runner backend or provider
session had started. The failure classifier returned the generic
`native_session_interrupted` code. `nativeProviderRecoveryEvidence` then
returned `ambiguous_state`. The run became a terminal failure, the task
became blocked, and the recovery action told the board to resolve an
ambiguous session and to not open a replacement session. The same
signature occurred on three other runs during an earlier deploy.

**Expected behavior**
A failure that occurs before any runner or provider process exists is
safe to retry on the same run. The task must stay in progress with a
scheduled, bounded bootstrap retry. If a provider event or checkpoint
exists, recovery must stay operator-owned.

**Steps to reproduce**
Start a native run. Close the server's HTTP listener before the executor
creates the runner backend. Observe a `terminal_failure` with
`recoveryMode: "ambiguous_state"` and a blocked task. The new
integration test reproduces this with the real
`resolveLocalRuntimeApiUrl()` and a real Postgres database.

**Environment**
Server native runtime, remote sandbox target, `codex_app_server` driver.
Related recovery work: #19249, #19304, #19305.

## What Changed

- `nativeSessionFailureSourceCode` returns a new
`local_runtime_api_listener_unavailable` source code for this error.
- `nativeProviderRecoveryEvidence` treats that code as definitely
pre-session. The existing guard stays: the run must have no checkpoint
and no durable provider event.
- The chat failed-run retry accepts the code in its list of ordinary
pre-session failures, so an exhausted startup retry can still be retried
from chat.
- New integration tests in `native-startup-retry.integration.test.ts`
cover the retry path and the operator-owned path when a provider event
exists.

## Verification

- Red before the fix: the new test failed with `phase:
"terminal_failure"`, `recoveryMode: "ambiguous_state"`, `recoverable:
false` and a board-owned recovery. This matches the failed run on the
deployed instance.
- Green after the fix: `cd server && npx vitest run
src/services/native-runtime/native-startup-retry.integration.test.ts`
passes 26 of 26 tests. The retry test asserts `retryable_failure`,
`bootstrap_retry`, backoff timing, an agent-owned `resume_native_run`
wake, and a claim through `claimNativeSessionResumptions`.
- Safety case: when a durable `turn.started` event exists, the same
error stays `terminal_failure` with `ambiguous_state` and a blocked
task.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"listener|classif|pre-session"` passes.
- Not run: a live deploy that interrupts a preparing sandbox run. The
test uses the real listener lookup and the real recovery tables, but not
a real sandbox.

## Risks

- Low risk. The change only affects runs that fail with this exact error
and have no checkpoint and no provider event.
- The retry uses the existing bootstrap budget of three attempts with
backoff. Repeated failures still stop and hand the task to the board.
- The GitHub callback bridge that started in the sandbox before the
failure is a per-attempt helper. The existing cleanup path releases it.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use, code execution and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 08:27:59 -06:00
c2cc69d1e4 fix(recovery): resume and settle preserved native results (#19034)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native execution can save an accepted result before workspace export
fails.
> - Recover must preserve the files and settle that same result without
another agent turn.
> - The export failure can own the task's Blocked transition and its
recovery notice.
> - Recovery must prove preservation, stop, source identity and current
authority before settlement.
> - Recover can also transfer repair ownership to the system, which must
keep explicit board export repair available.
> - This change restores those paths while preserving later task state,
live owners and unknown action outcomes.

## Linked Issues or Issue Description

Related #17916 protects unfinished exports. Related #19004 preserves
deliberate Retry authority. This change repairs accepted-result
finalization and its recovery notice.

**What happened?**

A native run accepted its result, then exhausted workspace export.
Normal Recover could save the files but leave the result and old
recovery notice unsettled. A saved Retry could also use a different
generation from preservation. Separately, Recover changed the action
owner from board to system. Explicit export Retry then rejected that
same source as stale, even when its accepted result and stopped
allocation still matched.

**Expected behavior**

Preserve files, finish the original accepted result and close only its
obsolete notice after exact proof. Keep explicit board export repair
available after system adoption. Preserve current authority, Stop, later
task changes, newer execution and every physical ownership fence.

**Steps to reproduce**

1. Accept a native result and exhaust its workspace export.
2. Request ordinary Recover, optionally with an existing saved Retry.
3. Let file preservation complete. Inspect the task and source recovery
notice.
4. To reproduce the explicit-repair defect, call export Retry after
Recover adopts the action. Before this repair it rejects the valid
system-owned source before any provider call.

## What Changed

- Record the exact accepted result, contract, task status version and
prior decision when export exhaustion creates Blocked.
- Bind passive finalization to complete, source-bound preservation and
the current chat/review authority. Recheck the proof under status commit
locks.
- Keep the saved Retry identity separate from its preservation
generation. Retain coalescing and reject malformed or mismatched
history.
- Preserve an existing published GitHub assessment for the
no-publication status-unproven wait. Keep all semantic, result,
authority and publication checks.
- Close an old export notice only after its own accepted result, export
or preservation receipts, effects, projection and old Runner ownership
settle. Keep newer work and files unchanged.
- Permit explicit board export Retry after Recover transfers the exact
source to the system. Require valid system and workspace recovery
records for that same run, plus every existing result, stop, allocation,
task and shared physical-lock check. Reject missing proof, another
source and agent ownership before provider access.
- Merge current master while retaining the stronger Stop, source and
verified preservation checks.
- Update the execution contract. No schema, endpoint, Runner protocol,
model, trust or provider artifact change.

## Verification

- Current head: `9be5a73489801095c9f39c9bfcb285a22b055912`.
- The explicit-repair regression fails before the new guard change:
ordinary Recover adopts the real database action, then export Retry
rejects it as stale. It passes after the repair.
- On the merged source, three owning PostgreSQL/file suites pass **220
tests**: `preserved-native-response-wait.integration.test.ts`,
`native-workspace-export-retry.test.ts` and
`idle-completed-history.test.ts`. The export suite includes four
unrelated-owner/source rejection controls and both board-owned and
system-adopted active-exporter controls. Results and run counts prove
that repair creates no provider turn.
- Server layered typecheck, module boundaries, `git diff --check`, and
the added-line secret/private-incident scan pass.
- Full workspace typecheck (`pnpm -r typecheck`) and build (`pnpm
build`) pass. The complete local `pnpm test:run` is still running; its
completed serialized chat partitions pass. Current-head CI is green (53
successful checks, 3 expected skips). Greptile reviewed
`9be5a73489801095c9f39c9bfcb285a22b055912` at 2026-11-05 00:58 UTC with
confidence **5/5**; all review threads are resolved.
- Original qualification remains supporting evidence: the
accepted-result fixture exercises HTTP Recover, real files, the
finalizer, saved Retry coalescing, eight commit races, current
principal/resource/head changes, Stop, later task changes, notice
closure and repeated delivery. No accepted response, provider turn or
publication is replayed.
- Earlier failures remain recorded. Original red controls exposed
wrong-action preservation, unrelated finalization barriers, generation
mismatch and a lost GitHub assessment; the original repair added
controls for them. The prior CI run
[41908572248](https://github.com/paperclipai/paperclip/actions/runs/41908572248)
failed an unchanged completed-history runtime fixture because its
loopback service refused connections through the original ten-second
readiness window. That owning suite passes unchanged in the 220-case run
above. The earlier readiness failure's cause remains unproven; no wait
or assertion was weakened.
- Local and CI tests do not establish hosted task recovery. The serving
revision and original-task outcomes require separate verification after
rollout.

## Risks

- Historical failures without the exact export-owned Blocked receipt
keep their current task status. A saved result does not grant execution
or publication authority.
- Explicit export repair still resumes only the same retained allocation
under the shared physical lock. It cannot take over an active exporter,
create another agent turn, choose a replacement sandbox or discard saved
files.
- Retiring a source notice does not release physical resources or modify
a newer task owner, later Done state or saved snapshot history.
- The proof adds bounded reads only for concrete recovery obligations.
It adds no recurring worker or polling timer.

## Model Used

OpenAI GPT-6 through Codex for the original repair. OpenAI GPT-6.1 Sol
(`gpt-6.1-sol`), xhigh reasoning, for the merge, integration
verification and system-adopted export repair. Both used repository
tools, code execution and PostgreSQL tests. The runtime does not expose
a context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and searched for duplicate or related
work
- [x] I have described the issue and linked related public PRs
- [x] I have included no internal task identifiers or private incident
links
- [x] I have added behavioral and safety regression tests
- [x] I have documented verification limits and risks
- [x] I have updated the execution contract
- [x] All Paperclip CI gates are green on the current head
- [x] Greptile is 5/5 on the current head with no unresolved comments
- [x] I will address review comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Devin Foley <devin@paperclip.ing>
2026-11-04 17:16:17 -08:00
3634c4a845 fix(chat): log the underlying error when a publication becomes delivery_unknown (#18875)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat connectors (Slack and others) publish agent messages to
external conversations through the publication worker
> - When the provider accepts a message but the local commit after the
send fails, the publication becomes `delivery_unknown`
> - A `delivery_unknown` row blocks the later publications in the same
conversation until an operator resolves it
> - The log line for this case contains only the generic reason. It
drops the error that the post-send commit threw
> - Operators cannot see why the commit failed, so they cannot tell a
race from a lease loss or another cause
> - This pull request adds the underlying error, redacted and
length-limited, to the existing warning for this case only
> - The benefit is that operators and maintainers can find the real
cause of stalled chat threads

## Linked Issues or Issue Description

Refs #18874

I searched open and closed issues and pull requests for
`delivery_unknown`, Slack publication failures and "could not confirm
its durable result". I found no duplicate. #18874 describes the bug and
the proposed behavior fix. This pull request does not change behavior.
It only makes the cause visible.

## What Changed

- `server/src/services/chat-channels/publications/failures.ts`: when
`providerAccepted` is true, the `chat publication failed` warning gets a
`cause` field. The field contains the underlying error message, redacted
with `redactSensitiveText` and limited to `MAX_ERROR_TEXT`, like the
existing `error` field.
- The stored `redactedError`, the disposition and the operator
resolution flow do not change.
- `server/src/__tests__/chat-run-milestone-work.test.ts`: the existing
`rechecks resolved prompt ownership after delivery ...` test now spies
on `logger.warn`. It asserts that `delivery_unknown` logs the `cause`,
and that `retry` and `failed` do not log it.

## Verification

- Ran in a `node:24-bookworm-slim` container on this branch:
- `npx vitest run src/__tests__/chat-run-milestone-work.test.ts` (in
`server/`): 72 passed.
  - `pnpm run typecheck` (in `server/`): passed.
- Negative check: I removed only the new `cause` field and ran the test
again. The two `delivery_unknown` cases failed. The `retry` and `failed`
cases passed.

## Risks

- Low risk. The change adds one field to one warning log line, only for
publications that the provider already accepted.
- The new field uses the same redaction and length limit as the existing
`error` field, so it does not add a new path for secrets into the log.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), 1M context window, in
Claude Code with tool use (shell, file edits, code search). A human
reviewed and requested the change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: lyuboslavmedarov <285605601+lyuboslavmedarov@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 17:38:39 -06:00
DottaandPaperclip 84e9c8f364 fix(retry): wait for admitted source cleanup before dispatch (#18898)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connected-task Retry keeps the original request and its current
authority.
> - One durable Retry can already own a queued successor.
> - The old native environment can still have pending cleanup before
that successor starts.
> - The task execution lock points to the successor, so the old cleanup
can escape the queue claim check.
> - Dispatch then refuses the source state and fails the new run before
provider work.
> - This pull request keeps that same run queued and wakes dispatch when
cleanup changes.

## Linked Issues or Issue Description

Refs #18845, #18853, #18860, #18877, #18882 and #18932. These repairs
cover preservation binding, interrupted preparation, old native
ownership, manual native holds or other setup guards. This change covers
an already admitted connected-task Retry whose native predecessor still
owes cleanup. It uses the shared cleanup predicate so the manual-hold
repair remains with its existing owner. Search found no duplicate
queued-source cleanup repair.

**What happened?**

A board Retry saved its exact chat request and admitted one queued
successor. The source lease was still pending cleanup when that
successor reached dispatch. A retained stop receipt did not settle the
full cleanup contract. The queue checked the task's current execution
owner, which was already the successor. The native authorization guard
then rejected the predecessor state and failed the new run before
provider dispatch.

**Expected behavior**

Keep the same canonical Retry and queued run while the exact source
cleanup is pending. After authoritative settlement, submit that run
through normal dispatch once. Do not infer safe execution from a
stop-and-retain receipt alone.

**Steps to reproduce**

1. Retry an authorized connected request from a failed native run.
2. Admit one successor while capacity keeps it queued.
3. Leave or restore the source lease to pending cleanup with a matching
retained stop receipt.
4. Release capacity and resume the ordinary queue.
5. The old claim path starts the successor and the native source guard
refuses it.

**Paperclip version or commit**

Reproduced against master at `f77499283270d2e966aabd58493f9d35c6bbdf5f`.

**Deployment mode**

The regression uses the real HTTP route, chat authorization, queue
claims, disposable PostgreSQL and the production work coordinator on
macOS arm64. A controlled adapter supplies the final result. No hosted
provider replay or deployment is claimed.

## What Changed

- Validate the exact canonical Retry, chat action, company, task, agent,
source and successor before testing source cleanup.
- Keep the successor queued under the existing issue claim lock while
the shared cleanup predicate remains pending.
- Wake only the scoped queued-dispatch worker after cleanup or exact
workspace-owner release commits. Add no timer or background scan.
- Verify pending cleanup at intake, later cleanup invalidation, one
final execution, idempotency, Stop, changed input, unknown stop
receipts, a different live controller and signed sandbox absence
followed by owner release.
- Document the queued-wait contract. No schema, endpoint, credential,
model or provider policy changes are required.
- Align five native preparation acceptance fixtures with the existing
workspace publication contract. Use the real reservation and publication
owner. Retain a raw-root refusal case with no provider launch and
preserved files.

## Verification

- Latest master sync at `e26cad76b8f20ba823d702ed60e7b7294f8f5a7c`
includes master `396b7fc62a0a4b1d8aadbda8221c05879d54177c`. The shared
test helper preserves #18944's canonical retained-directory default, the
real publisher for the five fresh-workspace cases, and the explicit
raw-root refusal. Independent remerge review found no dropped upstream
behavior, guard or assertion.
- The latest head passes the focused six cases, all 580 complete
workspace/continuation tests, all 38 selected HTTP Retry cases, fresh
full recursive Node 24.22.2 types and full build. No affected tests
skipped. An initial cached test-layer attempt read old budget
declarations; regenerating them resolved all seven errors without
test/mock or production-export changes.
- All current-head CI is green at
`e26cad76b8f20ba823d702ed60e7b7294f8f5a7c`, including required [ci /
verify](https://github.com/paperclipai/paperclip/actions/runs/41846304017/job/126636751314)
and [ci /
e2e](https://github.com/paperclipai/paperclip/actions/runs/41846304017/job/126636638086).
Greptile and Paperclip Review both rate this exact head 5/5, with no
findings or open review threads.
- Master sync at `3e4f266c2da172a84afd0b9cb4a91ea3b95f3c0d` merges
master `b891e69bd23d6dab8ce536569e15bc4a7a31c35c`. The only conflicts
were adjacent queue imports and two HTTP Retry test matrices. Both
imports and both complete matrices remain unchanged. Independent review
found no relaxed guard or assertion.
- Node 24.22.2 and pnpm 9.15.4: all 38 selected real HTTP Retry cases,
194 queue/cleanup/finalizer cases, module boundaries, recursive types
and build pass at `3e4f266c2da172a84afd0b9cb4a91ea3b95f3c0d`.
- That sync head's [server
check](https://github.com/paperclipai/paperclip/actions/runs/41838741852/job/126604742761)
failed five native acceptance cases in CI and locally with
`remote_workspace_published_path_invalid`. The test fixture supplied a
raw directory instead of a fresh publisher-generated workspace. The
guard from #18932 remains unchanged. The same five cases passed earlier
Linux CI before that guard was present.
- The test-only alignment at `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8`
obtains the saved authority, path and run reference from real
`reserveRemoteWorkspace` and `prepareRemoteWorkspace`. All five original
cases and a raw-root no-launch/no-result/files-preserved control pass.
Their native result, file, catalog and single-execution assertions
remain intact. An initial setup attempt was refused for a noncanonical
allocation root; the fixture now resolves that owned root before
publication.
- At previous fixture head `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8`,
the generated server test-layer typecheck passed. All 580 tests in the
complete workspace/continuation family pass, with no skips. That
test-only commit left production files byte-identical to the sync head.
- Previous published-head CI was green at
`dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8`, including required `ci /
verify` and `ci / e2e`. The [Linux Node 24.22.2 server
shard](https://github.com/paperclipai/paperclip/actions/runs/41842665765/job/126618334638)
executed all five original acceptance cases and the new raw-root refusal
case successfully. Its whole workspace recovery file passed all 245
tests.
- Greptile reviewed previous published head
`dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8` at 5/5, with no findings or
open review threads. No unchanged-job rerun was used for the reproduced
sync-head failure.
- Causal red: the real HTTP regression started and failed the successor
at the native source-state guard while cleanup remained pending. A
second red proved that signed sandbox absence followed by copyback-owner
release lacked a queued-dispatch wake.
- Green: seven HTTP/PostgreSQL cleanup cases, the exact confirmed-owner
release notification control and seven existing retry lineage controls
pass. The real coordinator resumes from committed cleanup or
workspace-owner release, without a manual queue restart. One controlled
adapter execution produces the result after settlement.
- Earlier branch verification passed full repository `pnpm -r
typecheck`, `pnpm build`, module boundaries and `git diff --check`.
- The full stale-queue file control passed all 64 cases with unchanged
observer/hook limits. The initial broader affected run had 192 passes
and one non-assignee native-owner case miss its existing five-second
adapter observer while already running with no error. The isolated case
passed; the unchanged base passed all 64 cases but hit its existing
30-second teardown timeout. Those original results are retained. The new
queue hint has no dispatch consumer in that fixture.
- Earlier-head CI was green: 54 success, three skipped and two neutral
checks at `6adb9f1ba682282c512d5e751ba2aa663f614ca2`. One chat job was
cancelled by an external runner shutdown; its single unchanged-job rerun
passed. No source was changed to obtain that result.
- Greptile and Paperclip Review both rated earlier head
`6adb9f1ba682282c512d5e751ba2aa663f614ca2` 5/5 with no findings or
unresolved threads. These results do not qualify the master-sync head.
- At initial verification, a complete local `pnpm test:run` was not
repeated while the related setup repair's full local run was active. The
full earlier-head CI suites passed. A full local suite has not been
repeated for the mechanical master sync. Local verification uses the
affected HTTP, producer, lineage and queue suites. No full local-suite
pass is claimed.
- The fresh clone's missing plugin SDK links were repaired with the
repository's own postinstall helper before verification. The embedded
PostgreSQL package's own symlink helper was also required.
- No staging task was submitted again with this repair, and no
deployment is claimed.

## Risks

- A pending source cleanup can keep the queued run waiting. Its cleanup
owner still owns reconciliation and its existing limits.
- Stop, changed source, unknown termination and live ownership retain
their existing authority checks. A cleanup notification grants no
execution authority.
- A new cleanup change after claim still faces the existing final
dispatch guard.
- Cleanup notifications can change dispatch timing in a running
controller. The real coordinator cases verify commit-bound wakeups; the
broader current queue-file control passed all 64 cases. The earlier
loaded-run and baseline teardown failures are retained in verification.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, code editing, tool use,
PostgreSQL and browser verification. This session does not expose a more
specific deployment model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 08:33:39 -06:00
DottaandPaperclip 2cbfc35a07 perf(github): reuse the inherited review wait query template (#18965)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - External review waits check current GitHub review and publication
bindings.
> - The GitHub service rebuilds the same seven-table Drizzle query on
each check.
> - A sampled production window attributed 6.3% of SQL serialization CPU
to this query shape.
> - This pull request prepares that query once per exact database or
transaction object.
> - Each call still reads current provenance, review, connection and
publication state.
> - The local service benchmark reduces median client CPU per read by
24.35%.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Repeated `readInheritedGitHubReviewWait` calls from external review wait
reads and attention retention.

**Subsystem affected**

Server GitHub chat review waits.

**Current behavior**

Every call constructs and serializes the same joined SELECT. The
database driver already supports prepared statements. That setting does
not reuse the JavaScript query template.

**Proposed behavior**

Compile the existing selector once for each supplied database,
transaction or savepoint object. Bind company, issue, agent and comment
IDs on each execution. Keep every current database read and check.

**Reason and benefit**

Remove repeated query construction and serialization CPU. Do not cache
rows or authorization decisions. Exact open-PR searches found no
duplicate query-template change.

## What Changed

- Add an exact-handle `WeakMap` for the GitHub review selector and
execute it with fresh placeholders.
- Preserve the SQL projection, joins, JSON types, latest-review order
and all current authority checks.
- Extend real PostgreSQL tests for new bindings, warm-cache revocation,
latest heads and transaction/savepoint rollback.
- Capture prepared executions in the existing test helper.
- Add an actual-service benchmark and a report with every timing sample
and setup failure.

## Verification

- `github-review-wait.test.ts`: 16 tests passed on real PostgreSQL.
- `external-chat-wait.integration.test.ts -t 'reads small current
external review waits without saved prompts'`: four affected cases
passed; 269 unrelated cases skipped.
- Pinned baseline: `e54f9fbf828e06e385e0c62dc381ccb46190aa30`. Final
source SHA256:
`fe647e7fe2aed82609e8f1d66a0109e7a10ee28313ae2ef218ba3ab39b5cd105`.
- Exclusive-window benchmark: 500 warmups per version, nine alternating
pairs of 100 complete service reads on disposable PostgreSQL. Median
client CPU: 1.00336 → 0.75908 ms/read (24.35% less). Median wall time:
2.87089 → 2.38731 ms/read (16.84% less). CPU excludes the PostgreSQL
process.
- Every generated SQL statement, parameter, query count and complete
result matched. Warm joined-selector compilations per ten service reads:
10 → 0.
- See `doc/performance/2026-11-04-prepared-github-review-wait.md` for
reproduction, all samples and measurement limits.
- Setup record: loopback required execution outside the sandbox. The
fresh embedded-Postgres dependency lacked macOS dylib aliases; aliases
to existing identical bytes restored the fixture. The SDK import needed
a current workspace SDK/shared build. A broad external-chat suite run
was interrupted and replaced by the affected test filter. Those attempts
are not counted as passing suites.
- Greptile: 5/5 on current head
`d2e6ae8814c3a56c380bd5d60c47e6f98642f085`, with no actionable issues or
inline comments.
- Current-head CI is green at
`d2e6ae8814c3a56c380bd5d60c47e6f98642f085`: 52 successful checks, three
intentional skips and the successful Snyk status. This includes
typecheck, build, the full test matrix, end-to-end shards and the
aggregate gate.
- Combined local validation at `4ab58ff70b` includes both measured
production templates and final test fixes. Full recursive `pnpm -r
typecheck` passed in 39.864 seconds. Three real PostgreSQL files passed
35/35 tests (five chat cases, 16 GitHub review cases and 14 shared
prepared-query cases). The combined build passed earlier with the same
production template bytes.
- The initial combined typecheck failed on a separate chat test typing
error and build setup. The later passing run resolves that attempt. The
original full local `pnpm test:run` was deliberately interrupted after
six chat shards passed and partial general tests ran; it is not a
passing full local suite. Each PR has independent current-head CI for
broad coverage. The initial loopback sandbox restriction was resolved by
an escalated run of the same source.
- Local benchmark/test dependencies were locked at the baseline; the
rebase adds the upstream lockfile refresh and UI paste fix only.

## Risks

- A prepared query captures its session. The exact supplied handle is
the cache key so transactions and savepoints do not borrow the root
session.
- Placeholder mistakes could cross bindings. Tests alternate independent
company/issue/agent/comment fixtures on the same handle.
- The benchmark has one synthetic eligible fixture with warm sequential
reads. It does not predict production endpoint latency or whole-server
CPU savings.
- No schema, index, database driver setting or public API changes.

## Model Used

OpenAI `gpt-6.1-sol` through Codex. The context-window size is not
exposed by the session. Used reasoning, tool calls, source editing and
code execution. Independent Codex review found no correctness issues.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 08:29:59 -06:00
DottaandPaperclip b6352578c5 fix(task-retry): retain chat Retry during identity contention (#18896)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task Retry must preserve a person's request and enforce the current
chat authority.
> - Chat retry preparation takes nonblocking source locks before it can
authorize the saved source.
> - Sustained lock contention could discard the Retry request before it
was saved.
> - This pull request saves one unadmitted request in the existing chat
retry queue.
> - The worker checks the source and current access before normal
admission. The task shows a scheduled retry while it waits.

## Linked Issues or Issue Description

**What happened?**

Task Retry returned HTTP 500 when another transaction held the saved
chat principal's identity lock or the endpoint authorization row. The
source preparation error rolled back the canonical Retry intent. The
same gap existed when Retry had to restore a certified input batch that
had never reached a provider. With the bounded synchronous retries in
#18895, sustained contention returns 409 without a saved intent.

**Expected behavior**

HTTP 202 retains one Retry request. Acceptance does not grant source
authority or mean that a provider started. The existing worker prepares
the request if the source locks become available before its saved
deadline. It must reject changed input, revoked access, Stop, or
superseding work before input adoption, publication, or provider work.

**Steps to reproduce**

1. Prepare a failed task with its original admitted GitHub chat source.
2. Hold that principal's `chat-identity` advisory lock or the endpoint
row on a second PostgreSQL connection.
3. Send the normal task Retry HTTP request. Check the response and
persisted intent.
4. Release the lock. Process the existing chat retry queue and check
that one authorized successor is queued.

**Paperclip version or commit**

Reproduced on master `8771606fd87f69ed4057779e3dce3a9bca8088e6`.

**Deployment mode**

Source checkout, real Express Retry route and embedded PostgreSQL.
External chat transport is a test fixture. The live hosted fix has not
been deployed or verified.

Related: #18895 handles brief admission contention with bounded
synchronous attempts and returns 409 if it remains busy. This change
retains a durable request through sustained identity contention. It uses
the existing background retry owner and adds real HTTP, restored-input,
Stop, publication, restart, and task-presentation coverage.

## What Changed

- Preserve a strict pending-source action only when the identity
try-lock reports contention or the endpoint authorization query reports
lock-unavailable through an explicit Retry callback. Bind it to the
canonical task Retry, company, task, agent, endpoint, conversation, and
principal. Other database and authorization errors keep their existing
behavior.
- Revalidate the canonical intent, current board permission, saved
source, and latest execution before the existing worker can prepare that
action. Ordinary authorization rejects an unprepared action.
- Keep certified input restoration inside source preparation. Recheck
the saved deadline before the prepared payload write. Expiry, a nested
savepoint, or outer rollback cannot commit partial restoration.
- Give source authorization contention one saved 60-second deadline and
a separate counter. Keep its waiting phase until current source
authorization succeeds or the action terminates. It does not spend the
eight delivery attempts. Duplicate clicks, worker restart, and
background sweeps do not renew either limit. Reuse the existing work
notifications and outbox deadlines.
- Persist completion of the authorization wait under the current claim
before delivery. A restart revalidates current authority. Generic
delivery failure preserves the stored phase even when a commit
acknowledgment is lost; it does not grant a wake or renew the saved
deadline, counters, or claim.
- Project pending preparation and prepared but unsettled actions as the
existing scheduled-retry state with the actual outbox deadline. Keep the
exact task, intent, agent, and source binding. Active execution keeps
its normal working state. No UI or API enum change is needed.
- Add regressions and document the admission boundary.

## Verification

- RED: the original real HTTP route returned 500 under a held PostgreSQL
identity lock. The restored-input case also returned 500 while its 60
sibling source and budget cases passed. The current endpoint-row case
returned 409 without a saved intent under the bounded synchronous
fallback.
- RED: the worker failed a valid Retry during an actual nine-second
PostgreSQL lock. Removing only the latest-run guard makes the new
queued-run negative fail. Source preparation could also commit restored
input after its deadline, and an already prepared action could process
after a late lock release. The prepared but unsettled task view lost its
scheduled state.
- GREEN: all 91 selected real PostgreSQL integration cases passed.
Coverage includes the 62-case saved-input matrix; coalescing; restart;
one real heartbeat successor; source, actor, Stop, newer-work and
publication refusals; a real nine-second lock; fixed-deadline
exhaustion; the separate eight delivery attempts; certified restoration
rollback on expiry; prepared-action early and late release; a real
conversation-row wait that crosses the deadline during current
authorization; endpoint-row contention; unknown source; an
already-revoked actor; unchanged unrelated ingress errors; durable-work
idle settlement; actual outbox presentation deadlines; and recovery
after authorization commits, with Stop and a lost commit acknowledgment.
- GREEN: all 75 cases in `task-retry.test.ts`,
`task-interruption-presentation.test.ts`,
`chat-failed-run-retry-boundary.test.ts`, and `chat-action-work.test.ts`
passed without a name filter.
- RED → GREEN: stopping after source authorization committed left a
stale waiting flag, so a fresh worker failed a previously authorized
Retry after the deadline. The fix saves that phase in the same
transaction. A real PostgreSQL commit followed by an injected lost
acknowledgment also reproduced generic settlement restoring the stale
flag. The corrected settlement preserves the stored phase. No wake is
sent on the ambiguous response. Recovery queues once after current
checks; Stop still prevents a successor. The deadline, contention count,
original input, failed run, and publications stay unchanged.
- Module boundary checks pass. Full workspace typecheck and the full
build passed at `d8a4a1a739aa6825714b04691d4166a00991808b`. At this
exact head, both required checks (`ci / verify` and `ci / e2e`) pass.
The inventory has 53 successful, three skipped, and one neutral check,
plus the automatic Paperclip Review in progress. The legacy Snyk status
passes. Greptile is 5/5 on this head, with no new comments or unresolved
threads. A current Paperclip assessment has not been published yet.
- The full local `pnpm test:run` did not pass: its second chat shard hit
a 15-second timeout in the HTTP/restart test after 348 and 293 passing
cases. One bounded diagnostic of the same 642-case shard pair passed,
with that case completing in 606 ms. The matching Linux CI pair passed
too. The original timeout remains unexplained and nonreproduced. No
timeout changed. The earlier stopped run was incomplete, and both
conflicting documentation additions are retained.
- CI chat 3/3 exposed a separate fixture assumption: restoration can
commit before a durable wake receipt exists. One known pre-receipt
delivery failure reproduces the same missing-run error. The corrected
fixture drives the existing worker until the exact canonical intent and
wake agree. It proves one real wake, one successor, exact source/actor
binding, and one restoration. The exact failed shard pair now passes
locally: 635 cases. Workspace typecheck also passes at test-only head
`94d691c9df61b05019b96900d5800a8f1c484e9a`. Production source is
unchanged from the previous verified build. That head completed all
registered checks: 53 successful, two neutral, three skipped, plus the
successful legacy status. Greptile scored 5/5. The subsequent Paperclip
4/5 persistence finding is addressed by the source repair above; its new
exact-head review is pending.
- Compatibility: 111 selected cases passed with #18895 head
`9b5d877161ffcbf16c0fb6920e8155a3620c85b0` in a disposable worktree. The
branch is now synchronized with master `4a83ba4050`, which includes that
change.
- Preserve the old CI failure: at head
`37954951fb66f3b50d0f2a3bec78d6993c479e87`, server shard 6/12 initially
failed the empty organization-audience replacement versus member-removal
assertion in `tool-access-service.test.ts`. The exact race family passed
on unchanged base and head; all 501 file tests and two 30-case controls
passed on base. The one Linux replay passed 119 files and 2,833 cases,
with three skipped. No guard or assertion changed. The initial failure
remains unexplained and nonreproduced; this PR does not claim to fix it.
- The prior external Superagent scan could not start because its
Security Graph returned HTTP 503. Its neutral check was not a completed
security assessment.
- These tests exercise the real Express route, PostgreSQL transactions,
and heartbeat queue. External chat transport is a fixture. They do not
run a live model or prove recovery of an original hosted task.

## Risks

- A still-pending source authorization wait fails after its fixed
60-second contention deadline. A completed wait is saved, but new
contention keeps the original deadline. A healthy lock can exceed that
limit. Other delivery errors retain the eight-attempt budget. Current
authority checks remain required before admission. Inspection does not
reset either limit.
- The new pending phase must never be treated as an authorized chat
source. Normal authorization rejects it until the worker prepares and
binds the exact source.
- The task projection reads the existing linked outbox deadline. It adds
a scoped join and does not create a timer or change a business deadline.
- No schema migration, provider identity change, or new delivery queue
is included.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository editing, tool
use, and test execution. The host did not expose an exact deployment
snapshot ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] Required Paperclip CI gates are green (`ci / verify` and `ci /
e2e`; automatic review progress is recorded above)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 07:18:48 -06:00
DottaandPaperclip c5251b4c0e fix(chat): retry fully recorded setup failures before dispatch (#18882)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connected tasks retain their exact external request across failures.
> - Task Retry must preserve that request and prove that old execution
cannot continue.
> - A workspace setup failure can settle before any provider dispatch.
> - The chat retry guard accepted one specific validation failure but
rejected other fully recorded setup failures.
> - This pull request reuses the same strict server settlement proof for
those failures.
> - The benefit is one safe continuation through the existing queue,
with all source and execution gates intact.

## Linked Issues or Issue Description

Refs #18803, #18845, #18852, #18853 and #18860. These changes preserve
workspace writers, bind completed preservation, or repair other
preparation and native continuation paths. This change repairs the
separate connected-chat guard for fully recorded legacy setup failures.
Search also found #18876, which repairs issue scope during deferred wake
promotion and does not repair this guard.

**What happened?**

A connected task failed during workspace reservation before provider
dispatch. Its source remained authorized. Task Retry saved an intent and
chat action, but the chat worker rejected that request as
`failed_chat_retry_not_authorized`. No successor was created. The old
failure had a bootstrap marker, a dispatch-only Runner profile, the
matching server error and durable measured settlement events, and no
runtime, process, output, result or environment lease.

**Expected behavior**

A deliberate board Retry can continue the exact saved input after the
complete never-dispatched proof passes. An unfinished earlier writer,
changed source, accepted result or unknown execution state must still
prevent execution.

**Steps to reproduce**

1. Admit an authorized request through a restricted chat endpoint.
2. Settle setup before provider dispatch, for example while an earlier
workspace writer is unfinished.
3. Keep the exact server error and durable measurement events, a
dispatch-only Runner profile, and no execution resources.
4. Press task Retry after the independent writer and ordinary admission
gates permit work.
5. The original guard saves the Retry action but refuses to create a
successor.

**Paperclip version or commit**

Reproduced on master at `a00109aa35`. The branch is rebased on current
master.

**Deployment mode**

Hosted task diagnosis. Local proof uses real Express HTTP routes, chat
authorization, heartbeat queue admission and disposable embedded
PostgreSQL on macOS arm64, Node 26.4.0. Provider work is deliberately
prevented by occupied fixture capacity.

## What Changed

- Extend only canonical task Retry's existing strict pre-provider
preparation proof to generic server-settled setup failures.
- Keep exact result shape, dispatch profile, controller, runtime,
process, output, event and resource checks. No diagnostic message
allowlist is added.
- Exercise GitHub and Telegram HTTP Retry with the original sender and
saved input. Keep the failed run unchanged and create at most one queued
successor.
- Cover provider events, retained native state/results, modified source,
unknown result metadata, live controllers, process identity, source
leases and an independent unfinished writer.
- Document the guarded contract and correct the obsolete access-map
sentence. No new endpoint or agent permission is added.

## Verification

- Red before repair: both setup causes failed through the real
HTTP/chat/queue path because the durable Retry had no successor. An
initial retained-native negative fixture had an invalid foreign-key
setup; that fixture was corrected before the final run.
- Green: 37 focused integration cases pass, including 22 new
GitHub/Telegram cases and 15 existing pre-provider source and execution
guards.
- Current head `1bf69d20e02a049248ec378f087505563daeb9a0`: all 37
focused integration cases and full repository `pnpm -r typecheck` pass
after the master rebase. The full `pnpm build` also passes.
- Map structure, module boundaries and `git diff --check` pass.
- Full local `pnpm test:run` is running; all six chat shards passed and
the remaining server suites are progressing. This is not a full-suite
pass claim.
- Greptile and Paperclip Review both reviewed this exact head at 5/5
with no actionable findings or review threads.
- Initial CI passed all jobs except the unchanged Runner package suite.
Its `runtime-host.test.ts` approve-reads case failed before its intended
assertions because a managed credential home could not acquire its
loopback-port quorum. The exact case passes locally. A wider 121-case
check hit the same lease error in a different Pi case (120 passed); all
121 pass with one worker. The exact competing port owner is unknown. One
unchanged retry of only failed CI jobs passed, including the Runner
package suite and aggregate `ci / verify`. All current-head required CI
gates are green; the original failure remains in workflow history.
- The dependency setup initially missed the embedded PostgreSQL
package's dylib symlinks. Running its own postinstall helper fixed the
local setup before assertions. This was not a product test failure.
- This change has not been deployed. The original hosted task has not
been replayed with the fix.

## Risks

- Never-dispatched proof must remain strict. A bootstrap marker or empty
process fields alone cannot authorize continuation. The regression cases
retain contrary-state holds.
- The inherited session display hint remains history and supplies no
resume authority.
- Current connection, principal, source integrity, pause, budget,
approval, dependency and workspace-owner gates remain in their existing
owners.
- No schema, provider profile, trust policy, model or credential
permission changes are required.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, code editing, tool use and
PostgreSQL/browser verification. This session does not expose a more
specific deployment model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 07:18:40 -06:00
DottaandPaperclip 0dedbf23c3 fix(chat): retain receiver ownership during runtime construction (#18930)
## Thinking Path

> - Paperclip manages agents and their work.
> - Chat receivers hold a durable lease before they accept messages.
> - Runtime construction can outlast a lease renewal interval.
> - Renewal previously required an attached runtime before construction
had returned.
> - That check could revoke the same receiver that was still starting.
> - This change uses the exact pending initialization as proof during
construction.
> - The existing durable token and endpoint checks still fence every
renewal.

## Linked Issues or Issue Description

Related: #18899 exposed the existing receiver failure in its chat CI
shard. This is a separate lifecycle repair. #18795 provides the relevant
production and sleep fixture context. Searches found no overlapping open
initialization repair.

**What happened?**

The receiver acquired its lease and started its renewal timer. The
service then awaited credentials, command preparation and runtime
construction before attaching the runtime to its context. A renewal
during that interval treated the missing attachment as lost ownership.
Configuration later failed with a Gateway ownership error, although its
original durable token was still current.

**Expected behavior**

The exact pending initialization can retain its own lease while
construction runs. Missing attachment alone must grant no ownership. A
changed token, paused or removed endpoint, changed generation or
credentials, and shutdown must still fence the receiver.

**Steps to reproduce**

1. Use the real configure and PostgreSQL ownership producers with a
60-second lease and 50-millisecond renewal.
2. Hold the actual runtime construction call for 200 milliseconds.
3. Release construction. Untouched code returns the Gateway ownership
error.
4. With the same construction delay and a five-second renewal control,
configuration succeeds.

**Paperclip version or commit**

The causal regression fails on unchanged production source at
`e7549ab613`. The original CI error was observed in the chat shard of
#18899. That CI run did not record the precise setup and renewal order,
so this local proof does not establish its historical scheduling.

## What Changed

- Store the runtime context identity in the existing pending
initialization entry.
- Allow pre-attachment renewal only for that exact tracked
initialization and current local ownership.
- Recheck local ownership inside the existing renewal transaction.
Preserve company, endpoint, generation, credential and original-token
checks.
- Before SDK listener initialization, wait for any older renewal,
recheck local ownership, then obtain fresh durable proof. Preserve the
existing checks after initialization.
- Keep attached runtime identity checks, lease deadlines, renewal
intervals, shutdown and listener retirement unchanged.
- Add actual configure and database race controls. Document the shared
Discord and Photon receiver contract.

## Verification

- Untouched production red: the actual construction gate crosses a
renewal and returns the same Gateway ownership error. The otherwise
identical slower-renewal control passes. No lease, exception or provider
outcome is injected in these positive cases.
- A subsequent review found a second real race: final runtime removal
did not prove that SDK initialization had never started. Four actual
initialization-hook controls failed before the pre-initialize fence. The
final matrix asserts zero hook calls for lost or changed authority,
including a foreign-token control with no periodic renewal due. An
attempted cross-company endpoint move is rejected by the existing
database foreign key; the valid original initialization still succeeds.
- A third actual-producer red holds completion of a real committed
renewal, changes the durable token, and proves that joining that old
promise could incorrectly accept configuration. The corrected startup
check waits for that renewal and then obtains fresh proof; it cannot
credit the earlier result alone.
- Final relevant full Discord registration group, takeover/callback
cases, buffered host-pause recovery, failed construction,
credential-fenced pause and shared Photon consumers: 34 passed across
two files; 1,840 other cases were deselected. Four executed cases cover
Photon transport recovery, failed authentication, duplicate allocation
ownership and checkpoint fencing. The tests use the real
configure/database producers with fake provider transports and observe
SDK initialization; they do not prove physical network behavior.
- All server type layers, the module-boundary gate, and production
server/vendored Runner build pass at exact final
`38cf82ec114bd12c0aa7bbc125ed72b1ae941b4e`. The production build stamp
records this exact commit.
- Earlier fixture mistakes are preserved separately: a test initially
changed every lease key, including the credential lease, and another
attempted an invalid cross-company row move. Neither is counted as a
production behavior red. The corrected controls target the Gateway key
and assert the existing company constraint.
- An initial isolated dependency graph resolved an older shared
telemetry build. Correct workspace links restored the all-layer
typecheck without changing telemetry source.
- Full repository checks were not run locally for this narrow slice.
Fresh current-head CI and substantive reviews remain required. No hosted
task, provider work, fleet action or deployment was performed.

## Risks

The ownership helper is shared by Discord and Photon. The selected tests
cover both consumers. The pending entry grants only permission to renew
the original token; it grants no message, callback or publication
authority and adopts no listener. Missing or changed initialization
identity fails closed. There is no schema change, historical backfill,
new worker or additional idle scan.

## Model Used

OpenAI `gpt-6.1-sol` in Codex, with reasoning, repository inspection,
source editing, shell execution and test tools. The host did not report
the context-window size. A separate coordinator reviews the source
before delivery.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 22:57:39 -06:00
Devin FoleyandPaperclip 909d86b59a fix(native): retry runs that fail before provider work (#18854)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs start the provider through Runner. `session.open` starts
the provider session before any prompt.
> - When `session.open` fails, the server records the generic
`native_session_interrupted`. Recovery evidence then reports
`ambiguous_state`, so the first failure blocks the task with no
automatic retry.
> - The failure is often temporary. On one instance an agent's provider
account hit its usage limit. For about 14 hours every later run failed
at `session.open`, and 47 tasks stayed blocked. Outside limit windows,
the same error also occurs once and a later retry succeeds.
> - The server already has a bounded same-run bootstrap retry for
failures that are proven to come before provider work. It did not
recognize this failure, and it did not consider a known provider limit.
> - This pull request classifies a definite `session.open` failure as a
pre-session failure. Bootstrap retries back off and wait out a known
provider limit. The task is blocked only after the shared budget is
used, with a clear reason.
> - The benefit is that a temporary startup failure or a provider limit
no longer strands tasks on the first failure, and every existing replay
guard stays in force.

## Linked Issues or Issue Description

Refs #18847. That pull request keeps the provider cause on
`session.open` rejections and names typed categories (for example
`classification=provider_rate_limit`). This change does not depend on
it. With #18847, a typed limit at `session.open` uses the same cooldown
path, because the server reads the cause with
`providerFailureFromError`. Refs #18622, which added the
`ProviderFailure` cause. Refs #13993, an open change about legacy
adapter startup backoff; it does not touch the native recovery path.

**What happened?**

A native run (`paperclip_runner`) failed at `session.open` before any
prompt: `ACPX sidecar command session.open was rejected
(retryable=false, classification=session_ensure_failed)`. The server
recorded `native_session_interrupted`. Recovery evidence had no
checkpoint and no provider event, but the cause was not in the
pre-session list, so the mode was `ambiguous_state`. The coordinator
went to `terminal_failure`, a board-owned `active_run_watchdog` action
was created, and the task was set to `blocked` on the first failure.
During a provider usage limit, every later run of the agent failed the
same way, so each task was blocked.

**Expected behavior**

A run that provably failed before provider work retries automatically,
with backoff and a bounded budget. If the agent has a known provider
limit, the retry waits for the cooldown. The task is blocked only when
the budget is used, and the recovery action explains why. A run where
provider work may have started is never replayed.

**Steps to reproduce**

1. Make Runner reject `session.open` for a native run (for example a
provider at its usage limit).
2. Observe the coordinator: `terminal_failure`, `recoveryMode:
"ambiguous_state"`, `providerSessionEstablished: false`. The task is
`blocked`.
3. The new `native-startup-retry.integration.test.ts` reproduces this
with a real database and the executor.

**Paperclip version or commit**

`master` at `5877d751c0e`.

**Deployment mode**

Native Runner with an ACPX Claude provider. The fix is in the server and
is not specific to one deployment mode.

## What Changed

- `nativeSessionFailureSourceCode` returns a new cause,
`provider_session_open_rejected`, when Runner reports that
`session.open` failed or was rejected. Those outcomes are definite, and
the command runs before any prompt. An `indeterminate` outcome stays
`native_session_interrupted`. Earlier, more specific causes (for example
a missing runtime package) keep their classification.
- `nativeProviderRecoveryEvidence` treats the new cause as pre-session.
The durable evidence still decides. A bootstrap retry happens only with
no provider session, no provider event and no checkpoint. Otherwise the
mode stays `ambiguous_state` and the task is blocked as before.
- New module `native-runtime/native-startup-retry.ts` plans each
bootstrap retry. It is called only for `bootstrap_retry` mode.
- Backoff: the existing `infrastructureRetryDelayMs` (about 30 s, then
about 60 s, plus or minus 20 %). The jitter derives from the run and
attempt, and it is persisted with `nextAttemptAt`.
- Limit cooldown: evidence is a `rate_limit` or `quota_exhausted`
provider failure on the current failure (the #18847 shape), or on an
agent run that started in the last 24 hours. A
`native_provider_usage_limit` or `provider_quota` error code also
counts. The cooldown is the provider's `retryAt` (or `retryNotBefore`),
or one hour after the failure when no reset time is known. A later
successful run of the agent clears older evidence. The cooldown is
capped at 24 hours. The retry runs after the cooldown plus the backoff
delay.
- A failed evidence read is logged and falls back to plain backoff, so
it cannot stop the failure from settling.
- The budget does not change: three provider attempts, shared with
checkpoint resumes and replacements. When it is used, the coordinator
records `native_session_retry_exhausted`. The task is blocked, and the
board-owned action says: "Provider startup failed on 3 attempts before
any provider work began. Automatic retries are stopped." When a limit is
known, the message includes the expected reset time.
- While a retry waits, the coordinator detail, the recovery action
evidence and the scheduled wake record `providerCooldownAt`. The next
action names the reset time, and the task projection shows "Retry
scheduled".
- `PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS` (one hour) moves to
`system-recovery-state.ts`. The legacy provider-quota recovery default
now refers to it. The value does not change.
- A chat retry of an exhausted `provider_session_open_rejected` run is
accepted like the other pre-provider causes. The existing proofs still
apply.
- `doc/execution-semantics.md` describes the backoff, the cooldown and
the exhausted state.

## Verification

- Red: before the fix, the first new integration case failed. The
coordinator was `terminal_failure` with `recoveryMode:
"ambiguous_state"`, `originalFailureCode: "native_session_interrupted"`
and `providerSessionEstablished: false`, and the task was `blocked`.
Four more cases failed the same way. The two never-retry cases passed
before and after the fix.
- Green: `npx vitest run
src/services/native-runtime/native-startup-retry.integration.test.ts` in
`server` (embedded PostgreSQL, the real executor, and the real resume
claim). 8 cases pass:
- a rejected `session.open` is retried with backoff (attempt 1 in 24 to
36 s, attempt 2 in 48 to 72 s), and the task stays `in_progress` with an
agent-owned action and a `resume_native_run` wake;
- after three attempts the task is blocked, the action is board-owned
with the clear reason, and the resume claim refuses another attempt;
- a recent limit failure holds the retry until one hour after that
failure, and the claim refuses it before the cooldown;
- a provider reset time from an earlier run, or from the startup failure
itself, sets the cooldown;
  - a distant reset time holds the retry for at most one day;
- a later success, another agent's limit, and an expired limit do not
delay the retry;
- a durable provider event, or an indeterminate `session.open` outcome,
still blocks the task with `ambiguous_state` and no retry.
- `native-session-executor.test.ts`: all 1,087 tests pass, including a
new classification case for `failed`, `rejected`, `indeterminate` and a
different command.
- `chat-channels.integration.test.ts -t "requires terminal-safe native
ownership for an exact chat retry"`: 23 pass, including the new
`session_open` case. That case fails without the list change.
- Related suites pass: native session resumption, native provider
failure, capacity and overload integration tests, provider failure
classification, native status arbiter corpus, native restart recovery.
- `npx tsc --noEmit -p server` passes. `git diff --check` passes.
- [GitHub
CI](https://github.com/paperclipai/paperclip/actions/runs/41713988964)
is green on head `32561aa335d`. This includes `ci / verify`, `ci / e2e`,
typecheck, build, all server, chat and workspace groups, all serialized
server suites, and all Runner checks, which run the native-runtime
server tests. In the first attempt, eleven jobs stopped because their
runners received a shutdown signal; no test reported a failure. A rerun
of those jobs passed.
- A local run of the whole `server/src/services/native-runtime`
directory was stopped before completion because it runs serially for a
long time. CI ran those files.
- Runtime dependency: the new work is one indexed query (`company_id`,
`agent_id`, `started_at`) with `LIMIT 20`. It runs only when a bootstrap
retry is planned or exhausted, not on the normal path. No new timer or
polling: the wait uses the existing persisted `nextAttemptAt` deadline
and the existing resume dispatch.
- Not verified: a live provider account at its usage limit, and a live
remote sandbox that waits through a full cooldown. The full repository
suite was not run locally; CI ran it.

## Risks

- A task that waits for a cooldown stays `in_progress` with its
execution lock, for up to one day. This is the existing same-run retry
state. Stop, reassignment, pause and an explicit Retry keep their
current behavior through the existing claim checks.
- During the wait the failed run keeps its environment lease in the same
way as any same-run retry. A remote lease with an unexported prepared
workspace is retained by the orphaned-lease sweep until the retry runs.
Other active leases of the failed run are released after the usual five
minutes, and the retry acquires a new one.
- A `rate_limit` failure without a reset time (for example a short HTTP
429) delays only startup-failure retries of that agent, by up to one
hour. Normal runs are not delayed, and a later success clears the
evidence.
- A startup failure that needs user action (for example a sign-in
failure) now uses two automatic retries, about 90 seconds in total,
before the task is blocked. Before, it was blocked at once with an
ambiguous reason.
- The agent's other queued runs are not held during a cooldown. Each one
fails at `session.open` without provider work and then waits for the
same cooldown. An agent-wide admission hold has no clean extension point
in the current queue, so it is not part of this change.
- Re-arming consumed scheduled monitors is out of scope. This change
makes it matter less: a run that consumed a monitor wake now retries
itself on the same run. A monitor that a run consumed before its retry
budget was used stays consumed.
- No schema or API change. The new error code appears on runs and in run
error codes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with
extended reasoning and tool use (shell, file editing, test execution).
The host does not expose the context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 13:24:59 -08:00
DottaandPaperclip ccc63ea458 fix(server): prove completed history before idle sleep (#18646)
Reviewed and verified at 592bce69c6.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-03 12:37:25 -06:00
DottaandPaperclip a92f616cd0 fix(db): attribute bounded captures to execution owners (#18634)
## Thinking Path

> - Paperclip manages agent work through shared server and database
services.
> - High query volume can spend main-thread time on SQL construction and
result decoding.
> - The existing bounded response capture counts rows and bytes, but it
cannot name the operation that executed a query.
> - Bundled callers can be missing, and measuring every parsed JSON
value adds work to the diagnostic.
> - This pull request carries a semantic execution owner into each
measured query and bounds response sizing.
> - Operators can rank query volume by owner before choosing a
performance fix.

## Linked Issues or Issue Description

Refs #17504, #17522, #17537, #17564.

**What existing behavior does this improve?**

The opt-in `PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW` diagnostic.

**Subsystem affected**

Database diagnostics and existing server execution boundaries.

**Current behavior**

The capture groups by table, operation, and builder caller. It cannot
distinguish execution owners that reuse a prepared query. Bundled
deployments need source maps for caller locations. The capture visits
every returned value and serializes parsed JSON again to estimate its
size.

**Proposed behavior**

Version 2 captures use fixed semantic owners for HTTP requests, delivery
queues, execution phases, and budget work. They count completed calls,
returned rows, errors, and completion wall time. They sample successful
results 1, 17, 33, and so on, with fixed cell and string/byte limits.
Complex values remain explicitly unmeasured. Semantic ownership works
without source maps.

**Reason and benefit**

A short census can identify which operations issue the most queries or
return the most rows. It preserves each query's execution owner through
pooled connections, concurrent operations, and prepared-query reuse. It
reduces diagnostic work without claiming per-query CPU attribution.

**Breaking changes**

Opt-in capture output moves from `entry.v1`/`summary.v1` to
`entry.v2`/`summary.v2`. Sampled primitive byte lower bounds replace
all-result byte estimates. Consumers must use the new fields and must
not extrapolate them into a total. There is no database migration or
application API change.

## What Changed

- Added `withDatabaseOperation(owner, action)` to the DB package. The
server owns semantic labels. Queries retain execution ownership in their
completion closure.
- Added fixed owners at existing HTTP, delivery, budget, and execution
timing boundaries. Delivery work replaces the context of the producer
that woke it.
- Added complete call/row/error counters and completion wall durations
within the existing bounded window. Durations include waits and are not
CPU measurements.
- Sampled and bounded primitive-value sizing. Complex and oversized
values remain unmeasured. Known owners skip repeated stack extraction.
- Preserved the deadline, group limit, paced output, query outcomes, and
existing coverage exclusions. Updated the operator documentation.
- Added real PostgreSQL owner-isolation and prepared-query tests, server
scope tests, and plain bundled-runtime proof.
- Moved projection-test and benchmark full-response sizing into an
explicit test helper. Existing payload-size thresholds remain unchanged;
these synthetic checks do not use the production sampler.

## Verification

- Final-commit DB capture suites: 43 tests passed in 13.86 seconds
against the actual PostgreSQL driver, including concurrent owners,
prepared reuse, transaction failure, a dedicated client, sampling, and a
query crossing the deadline. DB typecheck passed.
- Server attribution and existing delivery/execution/budget suites: 53
tests passed. Final concurrent HTTP GET/JSON POST scope refinement: 5
tests passed.
- Runtime bundle suite: 8 existing tests passed. The new no-source-map
owner test passed after its fixture was corrected to use a driver
thenable. An ordinary native Promise can bypass an overridden `.then`;
the real PostgreSQL tests separately cover driver integration.
- Observer-only experiment: three alternating pairs per case on Node
24.22.2, macOS arm64, 3,000 calls per run after warm-up. For an
already-materialized synthetic JSON result, median capture CPU fell from
82.37 to 1.87 microseconds/call. For a 100-row primitive result, it fell
from 8.10 to 4.40 microseconds/call. Every result retained identity and
all calls were counted. The new diagnostic intentionally measures less
byte data; these are observer costs, not database latency or a hosted
application speedup. Disabled-case timings were below one
microsecond/call and too small for a performance conclusion.
- Final lifecycle regression: 28 unit tests and DB typecheck passed.
Closing the last capture disables and replaces owner storage;
overlapping captures retain it.
- Final-commit full recursive typecheck passed in 38.38 seconds and full
build passed in 40.81 seconds. The build stamp matches
`83ed4ba9cebd68d48929c46f15ab32d5c0b9687b`. Focused server attribution
tests also passed 5/5 before the test-helper-only migration. Full test
acceptance will use the repository's parallel CI gates; the unsharded
local command takes hours.
- Verification failures retained: initial sandbox runs could not open
local IPC/PostgreSQL listeners; permitted reruns exercised the real
services. One old test expected JSON serialization and was updated to
assert explicit unmeasured complex values. The first bundle fixture used
the wrong promise boundary, as described above. A retained-callback
regression also demonstrated that disabling and reusing the same
AsyncLocalStorage instance could resurrect an old owner; replacing the
instance after the last capture closes fixes that case.
- The first CI run found older projection tests consuming the previous
full-response fields. Those consumers now use the full-response test
helper, preserving their byte thresholds. The two complete affected
route suites plus helper tests passed 115/115 in 73.08 seconds. A
temporary full-row query negative control then failed the unchanged
2,048-byte limit at 5,000,812 bytes and was restored; five other service
suites passed 34/34 and the affected chat cases passed 3/3. All three
previously failing CI shards passed on the corrected head. A local
skill-list benchmark smoke check completed 255 captured queries with no
in-flight queries or overflow; this was integration verification under
concurrent test load, not a timing benchmark.
- Two final-head runner jobs lost communication with GitHub before
reporting a test result. One targeted retry of those jobs and their
dependent aggregate passed. All 53 final-head checks passed, with 3
intentional skips; previously passing suites were preserved.
- Hosted capture remains pending until the reviewed revision is deployed
and a bounded window is explicitly configured.

Reproduce the focused checks with Node 24 and the repository pnpm
version:

```sh
pnpm exec vitest run packages/db/src/response-capture.test.ts packages/db/src/response-capture-postgres.test.ts
pnpm --dir server exec vitest run --config vitest.config.ts src/__tests__/database-operation-attribution.test.ts scripts/bundle-runtime.test.mjs
pnpm -r typecheck
pnpm build
```

## Risks

- Owner context is active while a valid future capture window is armed,
up to 30 minutes before accounting starts. No-window and expired calls
take the direct path. Diagnostics still add bounded work while enabled.
- A nearest nested owner is reported. Broad `http.request` and
unattributed groups remain visible; this does not identify every service
callsite or SQL shape.
- The existing census excludes tagged/raw explicit-execute/streaming
paths and internal protocol commands. Its counts are not PostgreSQL
transaction totals.
- Completion wall time includes queue/database/network/scheduling waits.
Concurrent durations overlap. Sampled bytes exclude complex values,
large strings, and values beyond the sizing limits.
- Activation is startup-only. Operators must choose the window after
artifact preparation, allow rollout/warm-up time, and remove the expired
setting afterward.

## Model Used

OpenAI Codex, GPT-6 family, with code execution, repository inspection,
and parallel agents. The exact deployment model ID and context-window
size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 12:12:21 -06:00
DottaandPaperclip 9ca9af544a Drive remaining live run and resource recovery from durable work (#18419)
Reviewed and verified at 6a4c193906.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 16:00:35 -05:00
DottaandPaperclip eb12cfc96b perf(github): activate connection work from events and deadlines (#18395)
Reviewed and verified at f77620b9a4.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 15:38:43 -05:00
DottaandPaperclip b6b57079b9 fix: evaluate status cards on committed changes and deadlines (#18401)
Reviewed and verified at 21795e222a.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 11:41:59 -05:00
DottaandPaperclip 3ee4c5b613 Make saved execution receipt recovery react to durable work (#18409)
Reviewed and verified at 8f306e0a6e.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 10:03:59 -05:00
DottaandPaperclip 42fd568cf8 Drive issue recovery from committed changes and exact deadlines (#18402)
Reviewed and verified at 11b49125db.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 09:41:27 -05:00
DottaandPaperclip 5c807d11fa fix(github): prevent duplicate review score headings (#18321)
Clarify the shared GitHub review submission contract for new and existing bots.
Remove copied score headings while preserving the structured score and saved assessment.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-30 10:19:31 -05:00
ec3b558277 fix(server): schedule accepted native response recovery (#17696)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs store accepted results and their finalization decisions.
> - A completed run can still owe its public answer after an
interruption.
> - Periodic recovery scans for that answer even when no new work
exists.
> - This pull request wakes recovery from committed changes and saved
retry deadlines.
> - The existing owner still verifies the result, source, and current
publication authority.
> - Settled or dormant response work no longer needs its own periodic
scan.

## Linked Issues or Issue Description

Refs #17690 (merged). This is slice 48 of the sleep migration stack, now
based on master. Related: #17661 handles GitHub terminal publication;
this change handles native accepted-response presentation.
Governed-review presentation remains a separate duty.

**What happened?**

Native finalization recovery repeatedly checks already accepted results
for a missing answer. Removing that check without complete notifications
would lose responses when evidence arrives late, an agent resumes, or
admission settles after execution.

**Expected behavior**

Recover the answer from its saved result when a supported change makes
it eligible. Retain real retry deadlines. Stop querying when the answer
is selected or cannot currently be published. Reconstruct work after
restart.

**Steps to reproduce**

1. Commit a native accepted result with no selected response comment.
2. Run finalization recovery while its source is unavailable or locked.
3. Restore that source or release the lock. The answer must appear once
without repeating provider execution.

## What Changed

- Honor recorded non-create decisions and skip-comment policy in both
discovery and recovery. Apply durable conversation/result/continuation
guards before recovery locks. Preserve genuine unanswered and legacy
recovery.
- Scope routine response wakes to the affected issue/run. Coalesced
input names its active run; pending-effect settlement names its issue.
Sixteen existing global authority/configuration hooks use one explicit
company invalidation boundary. No new transaction wrappers or tables are
added.
- Add an accepted-response worker using the existing delivery
coordinator and scoped deadline helper. Preserve saved retry deadlines
and pending-effect fences under the owner's lock.
- Share the finalization notification boundary. Wake on finalization
settlement, exact accepted-result/final-message evidence, agent
availability, retry admission, and existing chat authority changes.
- Keep lifecycle policy in services. A narrow pre-write status hook lets
lifecycle retain ownership of its transaction and status transitions.
- Register the worker during enabled maintenance startup. Remove its
authority subscription on shutdown. Periodic replay skips this duty on
both committed and fresh-commit paths; ordinary finalization and
explicit recovery retain their behavior.
- Add a partial index and expression statistics for unpresented native
runs. Migration 0412 supports concurrent replay and adds no queue table.
- Add real PostgreSQL and lifecycle coverage. Preserve the
implementation and producer audit in the plan document.

## Verification

- Additional real Photon callback acceptance on the published production
code disproves the security warning: ten active assertions cause zero
response-recovery hints/SELECTs, while three concurrent assertions
during an actual attention-to-active recovery cause exactly one
wake/SELECT. The existing locked status guard already enforces this
boundary. No production change was needed; the resolved finding now has
a successful current-head security check. The regression will be
retained with the next slice.
- Eligibility repair source `892f36e5f66ba1af123034936a7160c64c3ab883`
passes **1489 distinct server cases**: 1139 in six complete
recovery/finalizer/monitor/summary/workspace suites (81.02 seconds),
plus 350 selected chat cases (297.68s). Another 1463 chat cases are
outside the pattern; the complete 1813-case file is not claimed. All
eleven new regressions are included in that count. Five dedicated fresh
PostgreSQL 17 databases migrate through unchanged 0412, with no
fixture-source adaptations. Database build, server types/build,
boundaries and full-SHA migration ordering pass. All seven test source
hashes match.
- The exact published baseline first fails eight behavioral assertions
while the three genuine-answer cases pass. The same corrected test
source passes all eleven after repair: recorded none/reuse remain
suppressed; five ineligible history families take zero recovery row
locks across repeated issue hints; missing/legacy/create-null answers
still publish once; fresh periodic finalization defers presentation to
its worker. Earlier fixture setup and variable/compiler failures remain
explained in the audit, rather than erased by later success.
- Repair source `2fc8b0614aaf9ac7337045de4ced7325f95edc45` passes **1821
server cases**: 1496 in 13 complete producer/lifecycle/authority suites
(110.63 seconds), plus 325 selected chat cases (278.99 seconds).
Coverage includes all native-session cases, queue admission, status
decisions, membership/board access, the complete GitHub workflow,
accepted-response lifecycle and real authority restoration. The other
1477 chat cases are outside this selection; the complete 1802-case file
is not claimed for this repair run.
- Actual PostgreSQL regressions first reproduce a coalesced-input hint
missing its run scope and two routine hint paths producing 12
response-history SELECTs each after three notifications. The same
assertions pass with run-scoped coalescing and zero response-history
SELECTs for unscoped routine hints. Actual membership restoration still
publishes the waiting response once. Assertion time limits are
unchanged; this measures real driver SQL and persisted producer work.
- Eleven dedicated fresh PostgreSQL 17 databases migrate through 0412.
One embedded-only wake-queue fixture import temporarily uses its own
external database and restores byte for byte. All 14 selected source
files match pre-run hashes. This qualifies external-database behavior
rather than reproducing embedded CI locally.
- Master advanced independently with #18126's Pi archive retry fix.
Normal merge `2a535bc61259ef99387369e8b32498f774e5ba2c` preserves both
complete binary patches. All accepted server/database/UI/test/CLI trees
stay identical; the three independent runner installer/manifest files
are retained. All 13 installer tests pass, including stalled header/body
cancellation. Combined server types/build, boundaries and full-SHA
migration order pass. A subsequent normal merge of #18128 at
`e8b78cce5bb557b086bee9ca930eeaab1d09b244` retains its three test-only
file changes with both complete patches preserved. The changed HTTPS
backfill case passes on a fresh PostgreSQL fixture (1 selected case; 178
outside the pattern), and combined types/build, boundaries and migration
order pass again. All 14 repair source hashes remain unchanged. The
independent browser fixture changes are preserved; no local browser run
is claimed.
- The earlier source `9feec170ef8174011fc3c5d3d6d76f0a0f39fea0` passed
the 4171-case full baseline before this scope repair. That scope is
preserved in the audit rather than represented as a current full rerun.
All 0412 database bytes remain unchanged: original concurrent
index/statistics/ANALYZE, prior executable history, journal entries and
retained snapshots. Five snapshots 0408 through 0412 remain. The
populated 0411-to0412 upgrade preserved 410 prior receipts and 38
companies, appended one receipt and none on replay; index/statistic
checks passed.
- The earlier P2 security failure, Greptile's former conditional
dependency on #17702, and the subsequent actual Ponytail eligibility
findings are recorded in
`doc/plans/2026-10-25-native-response-presentation-sleep-audit.md`. This
PR now includes the issue-scoped pending-effect notification fix itself,
so it is independently correct before #17702. Fresh published-head CI,
Greptile 5/5 and a clean security scan remain required. No
deployed-sleep, external-wake or fleet-savings claim is made.


## Risks

- The writer contract is process-local. Independent SQL writers and
other replicas are not covered by these notifications.
- Unpresented backlog reconstruction still has memory and time cost
proportional to candidate count. Pages contain at most 100 rows;
execution uses the shared fair batch. Selected history is excluded.
Health-only signals do not trigger reconstruction.
- Publication policy remains with its current owner. Digest, company/run
binding, source authorization, selected-answer suppression, and
ownership fences remain required.
- The rest of native recovery and governed-review presentation still run
separately. This PR does not relax Cloud sleep eligibility or add
external wake coverage.
- The stack parent has merged and its actual ancestry is reconciled.
This publication still needs fresh current-head CI and review.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell
execution, and PostgreSQL tests. The runtime does not expose a more
specific model identifier or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 02:31:20 -05:00
DottaandPaperclip 7afee8f062 fix(server): schedule direct GitHub webhook recovery from saved deadlines (#17666)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/218/changes/paperclip/218
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - GitHub callbacks can fail before the app records an inbox row.
> - Direct GitHub integrations need bounded recovery from provider
delivery history.
> - The chat timer checks for due recovery every second, even without an
eligible integration.
> - Existing recovery checkpoints already store the next scan time and
runtime epoch.
> - This pull request schedules those saved obligations and wakes on
committed configuration changes.
> - It removes the final chat maintenance interval while preserving
authenticated recovery.

## Linked Issues or Issue Description

**What happened?**

The app queries due GitHub recovery endpoints every second. This still
runs when no direct GitHub integration needs recovery. Other chat
maintenance duties already have independent workers in the preceding
stack.

**What did you expect?**

Reconstruct existing recovery deadlines at startup and after committed
changes. Run only when an eligible endpoint reaches its saved deadline.
Make no recurring SQL or HTTP calls for unconfigured, disabled, or
Cloud-relayed integrations.

**Steps to reproduce**

Start an instance without an active direct GitHub integration. Leave it
idle. The chat maintenance interval continues to call
processFailedGitHubWebhookDeliveries each second.

Follows merged #17664. Related #15981 owns the original GitHub webhook
ingress/recovery separation. #17662 preserves review-check recovery,
which is a separate obligation. No duplicate scheduling migration was
found in the related-PR search.

## What Changed

- Put deadline reconstruction, scopes, and registration in the GitHub
provider owner. Read existing recovery checkpoints in pages of 100.
Reload current eligibility before each dispatch.
- Dispatch batches of five through the existing shared helper. Preserve
company fairness and the default limit of 25 for other consumers.
- Notify inside configuration, credential, checkpoint,
installation-lifecycle, and Cloud-registration transactions. A signed
installation deletion reproduced a missing wake before the fix.
- Detect credential-only epoch changes during a future backoff. Preserve
the existing floor reset without adopting old events into new
credentials.
- Gate every new recovery HTTP call on dispatch admission and shutdown.
Join in-flight work and retain its existing outcome/uncertainty
settlement. Other queues remain independent.
- Remove the final chat reconciliation interval and its unused
coordinator. Update the old coordinator fixtures; preserve the voice
retention worker acceptance.
- Restore known-unsent redelivery claims after drain/shutdown, guarded
by exact attempt identity. Preserve uncertain outcomes after transport
starts and never overwrite a newer receipt.
- Keep all existing redelivery, source, authority, lease, receipt, and
callback checks. Add no table, migration, or external wake contract.

## Verification

Current source: `ab00fc5b30d51d6c85891536599e7bdfd0c822fe`.

- **751 distinct cases pass** at
`3d7f163bc5259b60f9bd19b7fc90b402f0a6e925`: 274 complete GitHub review
cases, 294 other GitHub-named cases, 30 publication cases, 145
voice/retention cases, and eight startup cases.
- All 19 new recovery worker cases and all six stop-before-transport
variants pass. Coverage includes configured eligibility, callback-only
admission, committed configuration changes, outer transaction
settlement, restart/drain/standby, saved deadlines, fair batches,
independent queues, held HTTP, joined shutdown, known-unsent
restoration, and newer-receipt fencing.
- The first workflow filter omitted the separate recovery group. The
saved case list exposed the gap. The second selection ran the remaining
GitHub cases in a new database. Passed-name sets are disjoint. The two
commands filtered 1,504 and 1,484 unrelated cases respectively. No full
chat or repository-suite claim is made.
- Database build, server typecheck/build, module boundaries, and
migration order pass. Five fresh PostgreSQL fixtures receive the
unchanged parent migration history through 0403. This slice adds no
database migration or table.
- The parent browser-fixture repair is carried with both complete
patches preserved. At predecessor
`836acef0370d42323fdfdbd6d62a805b2469f1d4`, entire server, package and
UI source trees and owned hashes match the 751-case acceptance. The
current follow-up adds three test cases and audit evidence; production
source remains unchanged. The parent has merged.
- [The owning
audit](https://github.com/paperclipai/paperclip/blob/ab00fc5b30d51d6c85891536599e7bdfd0c822fe/doc/plans/2026-10-25-github-webhook-recovery-sleep-audit.md)
preserves earlier failures and exact acceptance scope. Fresh exact-head
Greptile and all CI gates remain required.

## Risks

- Direct GitHub endpoints still require the existing one-minute external
history cadence, with five-minute failure/claim backoff and provider
backoff up to one day. Cloud relay endpoints are excluded. This does not
promise network-idle direct integrations.
- The existing history, envelope, attempt, and lease limits remain.
Provider history never substitutes for a genuine authenticated callback.
- Supported producers share the current database owner's process-local
signals. Independent SQL writers and other server processes need their
own integration. Unknown transaction outcomes remain held by the
existing reconciliation protocol.
- Already-started HTTP keeps its existing timeout and is joined during
shutdown. No later request starts after drain or stop; durable attempt
uncertainty is preserved.
- Host-sleep eligibility and Cloud alarm coverage stay unchanged. Local
worker tests do not prove host/database suspension or fleet savings.
- Each remaining sleep slice needs its own current review and CI checks
before merge.

## Model Used

OpenAI Codex (GPT-6), with reasoning and code execution. The exact
serving model ID and context-window size were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge


Parent startup-test correction: #17664 fixes its legacy coordinator
boundary marker. This slice removes that coordinator and timer entirely,
so its existing absence assertions supersede that obsolete inspection.
All other parent and upstream changes are preserved. At predecessor
`836acef0370d42323fdfdbd6d62a805b2469f1d4`, complete server, package and
UI trees match the 751-case acceptance; all eight startup cases also
pass after the explicit conflict resolution. Fresh review and CI are
required for this new head.


## Required review finding: orphan assignment investigation

The reported orphan endpoint cannot be committed through the existing
schema: assigned_agent_id is NOT NULL, its agent FK is ON DELETE
RESTRICT, and a validated nondeferrable composite FK enforces the same
company. A first reproduction failed at fixture mutation with PostgreSQL
23502 before reaching the worker; that failure remains recorded. Three
new real-PostgreSQL cases verify NULL/missing/deleted assignments are
rejected, then prove the preserved valid endpoint recovers once, saves
its next deadline, and makes no SQL reads during a 15-second
pre-deadline interval. All 22 recovery-scheduling cases pass (19
previous plus three new); unselected cases are not counted. The
preceding style-only commit preserved production behavior and
non-comment tokens, and the prior 751-case evidence retains its original
scope (754 distinct cases across both selections). A new review must
assess this evidence before merge; the old conditional verdict is not
waived. Actual parent squash ancestry is carried with whole-tree
identity before this test-only follow-up.


Repository style follow-up removes seven new code-comment lines from the
recovery ingress owner and constraint tests. Their explanations remain
in the audit. Both files retain identical non-comment TypeScript token
streams, preserving all guards, assertions and deadlines. Production
eligibility SQL is unchanged. Fresh current-head CI and reviews remain
required.


## CI test corrections

The original strict deletion assertion reproduced CI's PostgreSQL-18
failure locally: the forbidden delete reports 23001 (restrict
violation), versus 23503 on PostgreSQL 17. The corrected assertion
accepts only these integrity codes for deletion, retains exact codes for
NULL/missing assignments, and verifies the original agent and assignment
remain. All 22 recovery cases pass on embedded PostgreSQL 18; the same
three integrity probes pass on PostgreSQL 17.

A delayed real announcement response reproduced the attachment E2E's
filechooser timeout: the dismiss handler tried to click outside an
already-open modal menu. It now closes the menu, dismisses through the
UI, and restores that menu so the original action resumes. Controlled
late-announcement acceptance passes; the temporary fault is removed. All
11 complete attachment receipt journeys pass, retries zero and original
assertions/budgets unchanged, using a fresh supported external
PostgreSQL-17 database with canonical isolated
onboarding/server/UI/storage. An earlier embedded-cluster startup
failure occurred before tests and remains recorded. Production server
typecheck passes; production source and schema are unchanged. Fresh
current-head CI and review are required.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 20:05:31 -05:00
DottaandPaperclip 8bc051245f fix(server): wake stale GitHub review repair from committed changes (#17664)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/215/changes/paperclip/215
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - GitHub checks and task admission can finish in either order.
> - A check can reject an old head before admission creates its review
row.
> - The chat maintenance loop repeatedly scans for that late review.
> - Existing cancellation and delivery receipts already identify the
repair.
> - This pull request wakes repair from committed changes and leaves
settled history quiet.

## Linked Issues or Issue Description

**What happened?**

GitHub chat maintenance repeatedly scans all review projections. Empty
instances and retained history incur database work even when no repair
is owed.

**What did you expect?**

A committed stale-head cancellation or admitted review should wake
repair. Only the exact delivery's unassessed queued/running review
should become superseded. When no repair remains, this duty should stop
making database calls.

**Steps to reproduce**

Configure a GitHub chat endpoint and leave it idle after all reviews
settle. The previous maintenance callback continues to call
repairSupersededReviews. A rapid push can also cancel a check before
admission inserts its review, which explains why recovery must remain.

Refs #17662, the merged review-check publication parent. Related #17661
and #17656 migrate terminal responses and acknowledgements. This slice
owns only late stale-head projection repair.

## What Changed

- Move the repair predicate and execution into the GitHub provider
owner. Match the original admitted delivery and cancellation receipt.
Recheck state and assessment in the update.
- Register a scoped worker with the existing coordinator. Use keyset
pages and fair bounded dispatch. Keep retries only for unresolved repair
or failed reconstruction.
- Signal before cancellation, automatic receipt insertion,
automatic-review binding, and check-driven review-state writes. Preserve
existing transaction ownership and outer commit behavior.
- Remove the periodic provider wrapper and publication-maintenance
callback. Preserve explicit manual recovery and independent
missed-webhook recovery.
- Add two concurrent, idempotent partial indexes. Use a guarded delivery
UUID lookup and a bounded exact-receipt lookup. Add no tables.
- Add ten real-database lifecycle and producer cases. Document the
producer audit, failure history, and remaining limits.

- Keep YAML browser reads scoped to the editor after focus changes, with
the original journey assertions and deadlines.

## Verification

Current source: `0886a0229b1c4f68d2f582da86aa1f248e9124f3`.

- At `dd6bca0485c3444d6e3d0ee10ec4aa651e801392`, **287 distinct cases
pass**: 274 complete GitHub workflow cases and 13 startup cases. The
GitHub command filters out 1,485 unrelated cases. The workflow suite
uses a new, pre-migrated PostgreSQL database.
- Coverage includes both orders of cancellation and admission, outer
commit and rollback, restart, drain, standby, failed-update retry,
concurrent assessment under an observed row lock, shutdown during that
wait, more than one candidate page, manual history, and settled-history
quiet.
- Database build, server typecheck/build, module boundaries, and target
migration order pass.
- Canonical migration `0403_natural_cerise.sql` preserves both original
index definitions and concurrent idempotent creation. All 401 parent
migration receipts remain unchanged during upgrade. One new receipt is
added. Replay adds none. Both indexes are ready and valid on fresh and
upgraded databases. The five newest snapshots remain.
- The normal parent merge preserved the prepared tree. The later browser
fixture repair changes only its central read helper and verification
note. Entire server, package, and UI trees remain identical to the
accepted source.
- [The owning
audit](https://github.com/paperclipai/paperclip/blob/0886a0229b1c4f68d2f582da86aa1f248e9124f3/doc/plans/2026-10-25-github-review-projection-sleep-audit.md)
records the exact evidence and earlier failures. Earlier query-plan
measurements retain their historical revision scope.
- Fresh current-head CI and Greptile review remain required. No deployed
sleep, full repository-suite, or fleet-savings claim is made.

Failure accounting: browser shard 7 failed twice on
`1453a4447c0d4d33f713ae3e00734c7950f95426` because the YAML helper read
whole-page clipboard text while main was focused. Three isolated
unchanged local passes did not resolve that failure. A controlled native
focus-loss probe reproduced the same error before the repair and passed
after it. The helper now clears stale clipboard data, focuses the
editor, and waits within the existing expectation budget for a fresh
YAML copy. All six complete YAML journeys pass with zero retries in 50.9
seconds. Journey assertions and deadlines are unchanged. The temporary
fault is removed. The exact CI focus-change trigger remains unknown.
[Both CI
attempts](https://github.com/paperclipai/paperclip/actions/runs/40777754306)
remain part of the verification record. Fresh new-head checks are
required.

Ponytail follow-up: the 4/5 review of `b3c81ba` was conditional on
fixing a stale startup-test slice marker, so the merge was held.
Boundary assertions reproduced the invalid -1 end index. The test now
uses the existing timer declaration and checks both bounds; all 13
startup/hostname cases pass. The optional no-op coordinator cleanup is
already in #17666. The six YAML journeys and all production code remain
unchanged. Fresh review and CI are required for this corrected head.

## Risks

- Missing a future producer signal could delay discovery until restart.
Supported writers share the current database owner's process-local
notification scope. Independent SQL writers and other server processes
need their own integration.
- The migration builds indexes concurrently. The existing migration
runner handles interruption and invalid index recovery.
- Direct GitHub missed-webhook discovery remains periodic. This change
does not make the whole chat service idle.
- Existing Cloud wake coverage and sleep blockers remain unchanged.
These tests do not prove deployed host or database suspension.
- Each remaining sleep slice needs its own review and CI checks before
merge.

## Model Used

OpenAI Codex (GPT-6), with reasoning and code execution. The exact
serving model ID and context-window size were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 18:42:54 -05:00
DottaandPaperclip 65fa1cd617 fix(server): schedule GitHub review checks from committed changes (#17662)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/213/changes/paperclip/213
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - GitHub review checks report the state of admitted agent work.
> - The chat maintenance loop searches for check work even when none
remains.
> - Existing action rows already record unfinished publication and retry
deadlines.
> - Committed changes can wake those rows through the shared
coordinator.
> - An unresolved check must still detect a remote head change when a
webhook is missed.
> - This pull request schedules only outstanding checks and leaves
settled history quiet.

## Linked Issues or Issue Description

**What happened?**

The GitHub chat maintenance callback repeatedly queries review-check
actions. Empty queues and completed history still incur database reads.
Local state changes wait for the maintenance callback.

**Expected behavior**

Committed check creation, review binding, run changes, and
publication-lease release wake the worker. Existing retry and
remote-head deadlines apply only to unfinished checks. A settled worker
makes no recurring database or provider calls.

**Steps to reproduce**

Configure a GitHub bot with no outstanding checks, or finish all of its
checks. Leave the instance idle. The old maintenance callback continues
to call the check service.

Refs #17661, the merged terminal-response parent. Related #17656
migrates response acknowledgements. Related #16385 changes branch-policy
admission; this PR preserves that separate policy boundary. Late
stale-head projection repair remains a separate follow-up slice.

## What Changed

- Move the existing check-state reader into the GitHub provider owner.
Preserve current authority, run selection, assessment ownership, and
check replacement behavior.
- Register a scoped worker with the existing coordinator and scheduler.
Existing received and retryable-failed actions remain durable truth. No
new table is added.
- Signal check enqueue, task review receipts, manual review
insertion/binding, run lifecycle changes, and publication-lease release
through existing transaction ownership.
- Preserve the saved failure backoff, eight-attempt policy, and
competing lease expiry. Keep five-second remote-head reconciliation only
while a received check remains unresolved.
- Check admission before each HTTP request. Shutdown leaves durable work
recoverable without spending a provider failure attempt.
- Remove check dispatch from the periodic GitHub callback. Keep the
separate late-projection repair and the manual combined recovery method
compatible.
- Add an idempotent concurrent partial index for unfinished check
selection. Retain the newest five generated snapshots.
- Add eleven real-database worker cases and startup wiring coverage.
Document producer coverage and the external reconciliation limit.

## Verification

Current reviewed source: `2bba08403b195bf646eda48992b46d6f5cebf756`.

- Normal merges preserve the complete review-check patch, the landed
terminal-response parent, and current master changes. No landed SQL
migration or journal entry changes.
- At `6995f84b240a14a7b8660e1cf095b3aa1e1e3168`, **331 distinct tests
pass**: 264 complete GitHub workflow cases, 30 completion cases, eight
issue-tree producer cases, 16 login-cleanup cases, and 13 startup cases.
The GitHub command filters out 1,485 unrelated cases. Login cleanup uses
its real embedded PostgreSQL fixture.
- Combined-master acceptance: **277 distinct tests pass** at
`9c538aed1251fd7f7cecb829a0ecbce6110fb1ed`: 264 complete GitHub workflow
cases and 13 startup cases. The GitHub command filters out 1,485
unrelated cases. A new PostgreSQL database receives all migrations
through 0402..
- Migration `0402_ancient_malice.sql` preserves the original partial
index definition and concurrent idempotent creation. All 400 prior
migration receipts stay unchanged during upgrade. One new receipt is
added. Replay adds no receipt. The index is ready and valid on fresh and
upgraded PostgreSQL databases. The five newest snapshots are retained.
- A rollback-only query probe adds 10,000 processed actions and one
pending action. The actual parameterized candidate predicate selects the
one pending row through the partial index, with two shared buffers and
0.007 ms execution time. This measures candidate discovery, not all
lease or provenance reads.
- Database build, server typecheck and build, module boundaries,
migration safety and migration order pass. Audit: [GitHub review-check
scheduling](https://github.com/paperclipai/paperclip/blob/2bba08403b195bf646eda48992b46d6f5cebf756/doc/plans/2026-10-25-github-review-check-sleep-audit.md).
- Fresh checks and Greptile review are required for this head. Prior
scores do not count. These checks do not prove deployed host suspension
or fleet savings.

## Risks

- Unresolved queued/running checks still make remote-head requests at
the existing five-second deadline. GitHub webhook receipt does not prove
future pushes cannot be missed. This PR does not claim active reviews
become network-idle.
- Run and configuration hints rebuild outstanding checks for the
affected company. Enqueue, review, and lease hints narrow to the
affected endpoint/repository/PR. Startup and scope reconstruction read
existing pending rows; they are not constant-cost operations.
- Delivery admission failures remain discoverable through the unresolved
check's external reconciliation deadline. Not every shared delivery
mutation gains a new immediate signal.
- Notifications use the existing process-local database ownership
contract. Independent SQL writers and other replicas need their own
supported integration.
- Provider markers, publication leases, and saved actions preserve
recovery after uncertainty. This change does not expand Cloud alarm
coverage or prove deployed host/database suspension.
- Each remaining sleep slice needs its own review and CI checks before
merge.

## Model Used

OpenAI Codex (GPT-6), with reasoning and code execution. The exact
serving model ID and context-window size were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 17:08:27 -05:00
DottaandPaperclip f8226b0515 fix(server): wake GitHub terminal responses from committed changes (#17661)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/210/changes/paperclip/210
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip coordinates agent work and presents results in chat.
> - GitHub acknowledgements already persist the response action and
provider receipt.
> - A terminal run can still owe a final response after acknowledgement.
> - Periodic maintenance searches those rows even when no relevant state
changed.
> - Committed run, issue, presentation, lease and publication changes
identify when recovery can advance.
> - The owner should recover from those events and real deadlines, then
become quiet.

## Linked Issues or Issue Description

Refs #17656, which is merged. This existing terminal-response slice
targets master.

**Problem:** A saved GitHub response can remain nonfinal while its run,
native presentation, publication or source is unresolved. Removing
maintenance without covering those producers loses recovery. Polling
settled history keeps the tenant active.

**Expected behavior:** Committed supported changes wake the terminal
owner. It reconstructs existing actions, preserves provider authority
and pending publication ownership, waits at saved deadlines where
needed, and becomes quiet after settlement.

## What Changed

- Register terminal GitHub response work with the shared scoped deadline
coordinator, using existing action/delivery/publication/run rows.
- Notify response availability from supported run milestones, native
identity, recovery resolution, issue updates and tree/workspace
restoration writes. Preserve outer transaction commit/rollback.
- Keep GitHub response interpretation in its platform owner. Use the
bounded run projection for both direct authorization and scheduling,
including native presentation guards and deliberate passive outcomes.
- Retain the existing publication lease, markers, source/principal
checks and final receipt precedence. A publication owner or uncertain
outcome keeps its existing authority; the worker does not replay it.
- Remove only recurring terminal-response recovery, preserving the
manual path and neighbouring duties. Add no table or scheduler service.
- Regenerate the original sparse terminal-work index as
`0401_yummy_ultimates.sql`, concurrently and idempotently through the
existing top-level migration runner. Preserve all prior SQL/journal
entries and the five newest snapshots.

## Verification

Published head: `88a209e28b969b5229e9a621aeddfd3cbae9b0cd`, integrated
with landed #17656 (`ba5bba86dcdde32b14d4d7dec1b51898e5f6f22d`) and
master `52b86c2032a189b0ea259c5b0cb7158b07ef788c`.

- **74 distinct current cases pass:** 17 terminal-response
lifecycle/producer cases (1,721 other chat cases skipped), all six
queued-comment consumption regressions for the new upstream query
optimization, all 30 heartbeat completion cases, all eight real/unit
issue-tree control cases and all 13 startup cases. Database build,
server typecheck/build, module boundaries and four fresh migrations
through 0401 pass.
- The earlier **304-case** prepared-parent matrix passes at
`aa72f32641b2e2c680342cd49477272349aca9db`: all 253 GitHub workflow
cases, 30 completion cases, eight issue-tree cases and 13 startup cases.
It retains that revision scope. Repeated cases are not added to the
current 74.
- The terminal matrix covers finalized-history quiet, deleted Board
sources, pending presentation/passive outcomes, outer rollback/commit,
paused/new runs, restart/drain/standby, committed native identity, lease
waits, full-page unrelated history, uncertain publication ownership and
tree/workspace/exhausted-recovery producers.
- A populated 0400 database upgrades to 0401 with all 399 prior
migration receipts identical, exactly one added, and none added on
replay. The terminal index is valid/ready on fresh and upgraded
databases. Its SQL matches the original draft after whitespace
normalization; generation keeps snapshots 0397–0401.
- Normal integration `3c8339a541bf68bc9957b9d5a8df80d3a7c3452d`
preserves the complete 24-path owned binary patch and all five new
master paths. The landed parent tree also equals the previously
published parent tree byte for byte. Both normal merge parents and the
exact composition tree are checked. No force push or rebase occurs.
- [The owner
audit](https://github.com/paperclipai/paperclip/blob/88a209e28b969b5229e9a621aeddfd3cbae9b0cd/doc/plans/2026-10-25-github-terminal-response-sleep-audit.md#acceptance-on-the-landed-github-parent-and-current-master)
preserves the original producer, quietness, failure-isolation,
skipped-suite and presentation-wait results with their scopes. Current
passes do not erase earlier failures or turn skips into passes.
- The audit-only follow-up leaves all owned source/test/migration hashes
unchanged. **Fresh exact-head CI and Greptile 5/5 with no actionable
findings remain required.** No full local repository suite, live GitHub
tenant or deployed sleep result is claimed.

## Risks

- Process-local notifications cover supported application writers and
shared pools. Independent SQL writers and other replicas gain no new
wake guarantee.
- Native terminal status alone does not authorize fallback publication.
Pending presentation and existing publication ownership remain separate
obligations.
- Stop can leave an in-flight provider request settling; admission
fences stop a subsequent request. Unknown outcomes retain their existing
recovery rules.
- Index creation can take time on large histories. Concurrent/idempotent
migration and existing replay recovery remain in charge.
- Worker quiet does not establish host/database suspension. Cloud alarm
coverage and sleep admission remain unchanged.

## Model Used

OpenAI Codex, using reasoning, repository inspection, code execution and
real PostgreSQL tests. The exact model identifier and context-window
size are not exposed in this session.

## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 16:51:57 -05:00
DottaandPaperclip ba5bba86dc fix(server): wake GitHub acknowledgements from committed changes (#17656)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/206/changes/paperclip/206
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip coordinates agent work and presents its progress in chat.
> - GitHub deliveries save acknowledgement actions and provider
receipts.
> - Periodic maintenance currently searches for unfinished
acknowledgements even when idle.
> - Committed delivery, lease, response and presentation changes
identify when work can advance.
> - The existing owner should wake on those changes and recover only at
saved deadlines.
> - Completed history should become quiet without losing unfinished
provider effects.

## Linked Issues or Issue Description

Refs #17653, which is merged. This existing GitHub acknowledgement slice
targets master.

**Problem:** A saved GitHub acknowledgement can wait for a publication
lease, fast-response result or native presentation decision. Polling
empty or unchanged work keeps the tenant active. Removing maintenance
without covering these producers can strand the acknowledgement.

**Expected behavior:** Committed changes wake the scoped owner. It
reconstructs existing actions on startup, retries at saved deadlines,
parks unchanged presentation waits and becomes quiet when settled.

## What Changed

- Register GitHub acknowledgement work with the shared scoped deadline
coordinator. Reuse existing action, delivery, lease and fast-response
rows.
- Add transaction-aware response-availability notifications at supported
writes. Keep GitHub protocol details in its platform owner and shared
notifications limited to IDs.
- Share the bounded latest-request/run projection between scheduling and
direct response authorization. An absent native presentation decision
parks the action until its real writer commits.
- Retain publication leases, source/principal authority, version
fencing, response markers and final receipt precedence. Check admission
before each new provider request; let in-flight work settle. Stop does
not consume the provider retry budget.
- Remove only recurring acknowledgement recovery and keep its manual
path. Add no work table or scheduler service.
- Regenerate the three original indexes as
`0400_absent_phantom_reporter.sql`: pending acknowledgement IDs,
fast-response delivery lookup and publication comment/destination
lookup. Each uses concurrent/idempotent creation and the existing
top-level migration runner. Preserve all landed SQL/journal entries and
the five newest snapshots.

## Verification

Published head: `09ca00b9f029838434ec042cda691345d67b493e`, on landed
continuation parent `063cfd43813cabf18417605fb9eefb9aacf95c8d`.

- **87 distinct current cases pass:** 18 real PostgreSQL acknowledgement
lifecycle/presentation cases, all 26 fast-response service cases, all 30
heartbeat completion cases and all 13 startup cases. Database build,
server typechecking/build, module boundaries and three fresh migrations
through 0400 pass. The current Runner bundle rebuild also passes after
upstream provider-event changes.
- The lifecycle matrix covers empty/history quiet, lease release,
restart, drain/standby, fast-response failure/expiry/publication/unknown
outcome, streaming quarantine, outer rollback/commit, saved provider
retry, stop between requests and receipt state. The real presentation
writer proves quiet while pending, rollback, committed reactivation and
final quiet.
- The earlier **305-case** matrix (236 full GitHub workflow cases, 26
fast-response service cases, 30 completion cases and 13 startup cases)
passes at `337f272853d4f8a51b5b9300900a46eb4e13f698`. It retains its
prepared-parent scope. Repeated cases are not added to current 87.
- A populated 0399 database upgrades to 0400 with all 398 prior
migration receipts unchanged, one new receipt and no additional receipt
on replay. All three indexes are valid/ready on fresh and upgraded
databases. Their SQL definitions match the original draft after
whitespace normalization; migration history and retained snapshot bytes
are preserved.
- Normal integration `9b3607e159153e00ccdfe538d52b591bd38d6499`
preserves the full 28-path owned binary patch and all 20 unrelated
master paths, including native retry, provider-event and evaluation
workflow changes. Both normal merge parents and the exact tree are
checked. No force push or rebase occurs.
- [The owner
audit](https://github.com/paperclipai/paperclip/blob/09ca00b9f029838434ec042cda691345d67b493e/doc/plans/2026-10-25-github-chat-maintenance-sleep-audit.md#acceptance-on-the-actual-landed-continuation-parent)
retains the original missing-notification, JSON precedence, receipt
lifecycle, external-database fixture and migration-registry failures.
Earlier 69/228/broad counts keep their original scope. Skips and failed
revisions are not relabelled as current passes.
- The final audit-only edit leaves all owned source/test/migration
hashes unchanged. **Fresh exact-head CI and Greptile 5/5 with no
actionable findings remain required.** No full local repository, live
GitHub tenant or deployed sleep result is claimed.

## Risks

- Process-local transaction hints cover supported application writers
and shared pools. Independent SQL writers and other replicas do not gain
a new wake guarantee.
- Provider effects remain guarded by the existing publication lease and
current source/principal authority. Unknown outcomes retain their
existing recovery semantics.
- Presentation waits depend on their supported milestone/transaction
producers. A missing producer can defer recovery until another supported
change or restart.
- Index creation can take time on large histories and runs concurrently
outside a transaction. Existing migration recovery handles replay; prior
SQL is unchanged.
- Worker quiet is distinct from host/database suspension. Cloud wake
coverage and sleep admission remain unchanged; later maintenance duties
are separate slices.

## Model Used

OpenAI Codex, using reasoning, repository inspection, code execution and
real PostgreSQL tests. The exact model identifier and context-window
size are not exposed in this session.

## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 16:06:42 -05:00
DottaandPaperclip 063cfd4381 fix(server): wake interaction continuations from committed responses (#17653)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/204/changes/paperclip/204
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip coordinates agent work and human approvals.
> - Human decisions already persist the continuation action that resumes
an agent.
> - A response can commit before its callback or publication finishes.
> - Periodic maintenance repeatedly discovers those rows, including when
idle.
> - Committed writes should wake the existing owner; saved retries and
claims supply real deadlines.
> - Startup reconstructs durable work, and settled history becomes
quiet.

## Linked Issues or Issue Description

Refs #17651, which is merged. This existing continuation slice targets
master and removes its duty from periodic chat maintenance.

**Problem:** A saved `interaction_wakeup` action can survive a committed
human response whose callback crashes. Without producer coverage,
removing maintenance strands the continuation. Repeated empty
maintenance also keeps the database active.

**Expected behavior:** The outer transaction's settlement wakes the
continuation owner. It handles ready conversation publications,
dispatches under existing authority, preserves interrupted/retry work,
and becomes quiet after settlement.

## What Changed

- Register continuation recovery with the shared scoped deadline
coordinator. Reconstruct existing issued/processing actions and retain
the existing retry and stale-claim deadlines.
- Notify from both enqueue producers and owner-local claim/settlement
writes. Preserve outer commit/rollback and avoid moving provider or
heartbeat calls into database transactions.
- Give the existing publication queue an optional
company/endpoint/conversation scope. The continuation owner gives ready
cards in that conversation a bounded delivery opportunity before
claiming the action; FIFO, capacity, ambiguity and provider authority
remain with the queue.
- Carry admission/cancellation through publication preparation and check
before every new send. Join in-flight requests and retain pending
actions/publications through drain, standby, shutdown and resume.
- Remove only recurring continuation recovery; keep the manual API. Add
no table. A concurrent partial index on existing pending continuation
actions and matching company/id keyset paging prevent retained history
from being scanned. Parent #17651's indexed confirmation discovery and
current master are retained.

## Verification

Published head: `a1d16f84b3fae361d5d648670a5964486370c3a3`, on actual
landed terminal-confirmation parent
`ee67d2ece9bda157dfad9bdf25175b1d004ed341`.

- **75 distinct current cases pass:** 32 selected real PostgreSQL chat
cases (14 continuation lifecycle/producer/stop cases, one new
continuation query-cost case, and 17 parent confirmation/query-cost
cases; 1,671 skipped), and all 43 cases in complete
interaction-publication and app/startup suites. Database build, server
typecheck/build, module boundaries and fresh migrations through 0399
pass. The original landed-parent matrix passed 74 cases before the
additional query-cost repair; those are not added again.
- The new real-SQL regression seeds 20,000 completed/unrelated actions
and one pending processing claim. Before repair it examines 20,345 rows
and hits 20,421 buffers. The final startup and company-scoped queries
must examine at most two action rows and fewer than 20 action buffers;
empty scoped recovery must also stay below 20 buffers. Planner settings
are unchanged. Migration `0399_ambitious_onslaught.sql` is
concurrent/idempotent and uses the existing top-level migration runner;
prior SQL/journal entries and the five newest snapshots are preserved.
- The matrix covers callback recovery, concurrent responses, malformed
prefixes, more than one page, saved retries/stale claims, outer
rollback/commit, deferred/future/paused publication states, restart,
drain and stop before each new provider request. Quiet assertions
inspect real selects/transactions across virtual idle time.
- Normal integration `db5a84c0f8b73a0ce863f4d2fbaee0a115cf044f`
preserves the complete 12-path owned binary patch and all 12 unrelated
current-master paths, including managed Cloud sandbox changes. A
verified semantic base resolves squash-ancestry conflicts; both normal
parents and the exact tree are checked. The subsequent index/keyset
repair is a deliberate additional change, tested separately; the final
patch is not claimed identical to the original 12-file implementation.
No force push or rebase occurs.
- The same 74 pre-index cases passed on the earlier prepared and actual
landed parents. The new query-cost regression failed before its repair,
then passes on the repaired source. Earlier 57-case and broad acceptance
retain their original revision scopes; repeated cases are not added to
the current count. The audit-only follow-up leaves all owned server
source/test hashes unchanged.
- [The owner
audit](https://github.com/paperclipai/paperclip/blob/a1d16f84b3fae361d5d648670a5964486370c3a3/doc/plans/2026-10-25-interaction-continuation-sleep-audit.md#acceptance-on-the-actual-landed-terminal-confirmation-parent)
retains the original missing-signal regression, the skipped standalone
publication suite, and the review's stop-before-send failures with their
corrections. Skips and earlier failing revisions are not relabelled as
current passes.
- **Fresh exact-head CI and Greptile 5/5 without unresolved actionable
findings remain required.** The former remote head's reviews are stale.
No full local repository suite, live provider or deployed sleep result
is claimed.

## Risks

- Process-local transaction hints cover supported application writers
and shared pools. Independent SQL writers and other replicas do not gain
a new wake guarantee.
- The publication queue's existing FIFO, capacity, retry/ambiguity and
authorization contracts still govern effects. In-flight provider
requests can settle after Stop; the fence prevents a subsequent request
from starting.
- Missing a supported producer can defer recovery until another change
or startup. Producer and rollback tests cover this slice's supported
paths.
- Worker quiet is separate from host/database suspension. External alarm
coverage and Cloud sleep gates are unchanged.

## Model Used

OpenAI Codex, using repository inspection, code execution and real
PostgreSQL tests. The exact model identifier and context-window size are
not exposed in this session.

## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 15:45:18 -05:00
DottaandPaperclip ee67d2ece9 fix(server): wake terminal confirmation recovery from committed changes (#17651)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/202/changes/paperclip/202
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip persists confirmation clicks and interaction decisions
before dispatch completes.
> - A crash after a decision commits can leave a processing confirmation
action.
> - Periodic publication maintenance repeatedly scans those actions even
when idle.
> - Saved decisions and the supported board/admin writers already
identify when settlement can change.
> - This owner reconstructs durable actions, reconciles real changes and
then becomes quiet.

## Linked Issues or Issue Description

This existing terminal-confirmation slice follows merged #17649. It
removes only terminal-confirmation recovery from periodic chat
maintenance while retaining the explicit manual API and existing durable
action/interaction/publication rows.

**Problem:** An accepted response can commit before its dispatch
settles. Removing periodic recovery without covering board decisions,
expiry and ownership changes would strand processing actions. Pending
human decisions must remain recoverable without polling indefinitely.

**Expected behavior:** Committed supported changes wake the scoped
owner. It reconciles terminal responses once, parks unresolved human
decisions, preserves concurrent commits/restart/drain and becomes quiet
when settled.

## What Changed

- Register terminal-confirmation work with the existing delivery
coordinator and scoped lifecycle. Reconstruct processing action IDs,
bound selection pages, preserve concurrent wakes and keep current
authorization in the existing interaction owner.
- Add the existing transaction-aware confirmation notification topic at
real resolution writes and callback settlement. Reuse milestone signals
instead of adding per-caller recovery wrappers.
- Cover ordinary and checkbox confirmations, suggestions and connection
intents through their owning service. Cover board accept/reject,
cancellation/withdrawal, opt-in comment supersession, expiry,
reassignment and already-satisfied connection resolution.
- Preserve task/actor authority, one-use callback tokens and immutable
provider publication semantics. Explicitly fail malformed missing
bindings instead of silently retaining unrecoverable processing actions.
- Remove only this duty from recurring publication maintenance. Add no
table or scheduler infrastructure. A concurrent partial index on
existing action rows prevents recovery discovery from scanning completed
history.

## Verification

Published repair head: `a35e78d076c1b618bc643391b7520fa6952fc996`,
targeting master after normal integration of
`644937c93150c423f12674992c7ea1e8b5dbc98c`. [Detailed acceptance and
preserved
failures](https://github.com/paperclipai/paperclip/blob/a35e78d076c1b618bc643391b7520fa6952fc996/doc/plans/2026-10-25-terminal-confirmation-sleep-audit.md#bound-retained-history-reads--october-28-review-repair).

- **New query-cost regression:** Capture the actual SQL emitted by
startup and company-scoped confirmation discovery. Seed 20,000
completed/unrelated action rows and one pending human decision. Before
the index, the regression fails because recovery examines 20,009 action
rows. After the partial index, startup and scoped discovery must examine
at most two action rows and fewer than 20 table/index buffers; the
no-pending scoped path must also stay below 20 buffers. Planner settings
are unchanged, and returned IDs remain correct.
- Match startup keyset ordering to `(company_id, id)`, the partial index
order. The index alone still let PostgreSQL choose the primary key and
scan 20,206 rows for `ORDER BY id`; the final query fixes this without
forcing a planner setting or adding a second index. Fixed kind/status
predicates stay literal for prepared-plan index eligibility.
- Add the generated `0398_yielding_random.sql` migration, using `CREATE
INDEX CONCURRENTLY IF NOT EXISTS` and the existing top-level migration
runner. Keep the five newest generated snapshots and preserve every
prior SQL/journal entry. No work table is introduced.
- **110 distinct cases pass:** 17 final confirmation
lifecycle/query-cost cases (1,671 skipped), seven real Runner descriptor
variants (295 skipped), and all 86
app/startup/bundled-plugin/plugin-database cases. Server/database
typechecking and builds, migration safety/numbering, module boundaries
and a fresh PostgreSQL migration through 0398 pass. The final test-only
vacuum precondition is followed by another 17-case confirmation pass and
server typecheck. Counts are not added across repeated runs. Normal
integration of master `644937c93150c423f12674992c7ea1e8b5dbc98c`
preserves the complete owned patch and all 18 unrelated master paths.
- Previous landed-parent acceptance: 29 distinct cases (16 confirmation
lifecycle/producer cases with 1,671 skipped, plus all 13 startup cases)
passed on `32e64a7e9ea5b457e54ca7b268be90e1e155b451`; the earlier
80-case matrix remains scoped to
`c1e8542b588a4fa59b62f9829541d3daea4f1ddc`. Counts are not added across
repeated runs.
- Actual producer cases prove parking/quiet, caller-owned outer
transaction rollback, commit-driven automatic settlement, restart/drain
and concurrent commits. Checkbox, suggestion, connection-intent and
Telegram/Slack callback paths retain existing authority and immutable
publication semantics.
- **Fresh exact-head CI and Greptile 5/5 with no unresolved actionable
findings are required.** The preceding head's Runner CI failed its
timeout-attachment durable-suspension case (1 failed/1,955 passed/4
skipped); that failure is retained and not counted as a pass. The new
head must pass this job. No full local repository, live provider or
physical suspension claim is made.

The owner audit retains the earlier missing-board-producer regressions
and their corrections, including the invalid supersession fixture that
assumed a new checkbox expires an older one. Existing behavior was
preserved by using the supported opt-in human-comment path. Prior
scheduler/backoff, route and plugin test failures remain at their
original revisions; no old pass is substituted for current remote gates.

## Risks

- Process-local notifications cover supported application writers and
their shared pools, not independent SQL writers or other replicas.
- Processing actions waiting on a human response are deliberately
parked. Supported interaction resolution/expiry writers must use the
notification boundary; a missing producer delays settlement until
restart.
- Invalid bindings have explicit failure evidence. Source, actor, task
and publication authority remain checked by their existing owners.
- Following continuation/GitHub duties remain separate. Cloud sleep
gates and external alarms are unchanged; local worker quiet does not
establish physical host/database suspension or fleet savings.

## Model Used

OpenAI GPT-6 / Codex, with reasoning, repository inspection, code
execution, and PostgreSQL tests. The exact runtime model identifier and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 15:00:31 -05:00
DottaandPaperclip f4f02880ca fix(server): wake failed retry notices from committed changes (#17649)
## Thinking Path

> - Paperclip keeps exact failed chat retries in durable action and wake
receipts.
> - Deferred and declined retries owe safe notices to their original
conversation.
> - Periodic publication maintenance repeatedly inspects those receipts
when idle.
> - Committing receipt, authority, lifecycle and lease changes already
identifies when eligibility can change.
> - A scoped owner reconstructs durable obligations, drains bounded
candidates, retains contention, then becomes quiet.

## Linked Issues or Issue Description

This existing failed-retry notice slice follows #17647. It removes the
failed-retry notice duty from periodic publication maintenance while
preserving the explicit manual API. Existing action, receipt,
publication and source rows remain authoritative; no new table or
scheduler is added.

**Problem:** Removing the scan alone loses notices after real admission,
cancellation, early refusal, requester or agent restoration, and
relevant lease settlement. Millisecond cursor truncation can also repeat
denied history.

**Expected behavior:** Committed supported writes wake the scoped notice
owner, which uses current authorization and precise cursors. Real
retries survive failure, contention, restart and drain; settled work has
no recurring SQL.

## What Changed

- Extract failed-retry projection under the existing authorization
owner. Preserve current receipt/source authority and provider transport
authorization.
- Register bounded scoped work using the existing delivery
coordinator/scheduler. Preserve exact PostgreSQL cursor precision,
contention and a wake arriving during reconciliation.
- Extend the shared transaction-aware notification boundary to
failed-retry notices. Cover admission, conditional denial, early
refusal, requester changes, and agent lifecycle transitions to an
invokable status. Two shared receipt hooks and their budget fixture
correction already landed in the preceding inbound slice; preserve them
once.
- Notify when relevant native lease cleanup completes or a reusable
lease is retired, preserving claim fencing and caller-owned
transactions.
- Avoid the discovery scan identified in #17647: known task scopes
require zero discovery SQL; company/startup discovery uses a single
DISTINCT projection from the same candidate predicate. Bounded current
authorization still runs afterward.
- Remove only this duty from periodic publication maintenance, retain
manual behavior and startup reconstruction, and add real
producer/lifecycle/query-budget regressions with an audit.

## Verification

Published head: `fa95a1f2fb9d546aa62dbd3b43aa714f5c88bd35`, targeting
master after #17647 merged at
`1aa3a21ac41f602977638cf0be875b01a243e69f`.

- **42 distinct current combined-master cases pass:** 21
retry-notice/native-lease/restored-input producer cases (1,651 skipped),
plus all 21 app/startup and task-drain admission-release cases. Server
typechecking, build and module boundaries pass. The final follow-up adds
acceptance documentation only; all 17 owned server source/test hashes
are unchanged.
- The prior **404-case** matrix passes at prepared revision
`c2c53b839497be6f5e28aa2cd5ba50cfd1d76d4a`: 259 across six complete
consumer/lifecycle/environment suites, 20 notice/lease cases, 53
ordinary-notice and 72 Retry/budget cases. Those chat selections skip
1,651/1,618/1,599 cases. A subsequent 21-case combined producer check
passes after the parent restoration repair. These earlier runs retain
their revision scope.
- The normal two-parent integration
`43d0f84ea56845b8fc4946a7533ecbf7f07b679a` preserves the complete
19-path owned binary patch and all 23 unrelated master paths/deletions.
The semantic merge base is the verified landed parent head
`41f7e2c9e351efa6cfa5ef699c0e69df7425012b`; squash ancestry otherwise
causes historical conflicts. Both parent commits and the exact resolved
tree are verified; no rebase or force push occurs.
- Three scope-discovery regressions fail before the fix. The first SQL
projection fails with PostgreSQL 42702 (ambiguous company column), then
passes after using a distinct company alias. Known task hints have zero
discovery SQL; company/startup discovery has one query returning only
distinct task scopes. Existing bounded projection/current authorization
remain intact.
- Current producer tests cover commit/rollback, parked authority
restoration, real agent resume, early refusal, requester changes, native
lease cleanup/replacement, drain/restart/shutdown, precise cursors and
contention. Parent shared admission/denial/restored-input notifications
are preserved once. No polling backstop or storage is added.
- [The owner
audit](https://github.com/paperclipai/paperclip/blob/fa95a1f2fb9d546aa62dbd3b43aa714f5c88bd35/doc/plans/2026-10-25-failed-retry-notices-sleep-audit.md#final-landed-parent-acceptance--october-28)
records current commands, fixtures, failure history and limits. Fresh
exact-head remote checks and Greptile 5/5 are pending; old-head reviews
do not count.

The owner audit preserves prior failures and their original revision
scope. The old remote head's Runner receipt timeout, cancelled gate, and
review error are not current passes; fresh exact-head checks and
Greptile 5/5 are required. No full local repository suite, live provider
tenant, external wake alarm, physical suspension or savings claim is
made.

### Final remote gates

Exact published head `fa95a1f2fb9d546aa62dbd3b43aa714f5c88bd35` has
Greptile 5/5 with no actionable findings, successful current main CI and
ready-triggered security scan. The final dependency/quality review
workflow `40747725096` passed after waiting for a GitHub-hosted runner.
No Ponytail verdict is present.

Two automatic main CI requests and two identical-head quality-review
requests were created one second apart. The older main run was
automatically cancelled; only the proven duplicate, unstarted quality
request was cancelled manually. Their same-head, same-workflow
replacements both finish successfully. The merge guard verifies these
replacements, all current checks and current required-status integration
rules before excluding the superseded cancellation contexts. No actual
failing review or current check is waived and no protection bypass is
used.

## Risks

- Process-local signals cover supported writers using this boundary, not
independent SQL or other replicas.
- Lease completion now reads/locks the exact row before updating.
Reusable lease retirement wakes company scope because its previous issue
can differ. Claims and transaction ownership remain enforced.
- Broad discovery can inspect history inside PostgreSQL; DISTINCT
reduces returned rows and round trips, without claiming constant
database cost. Task hints are wake requests, so bounded reconciliation
handles already exhausted scopes.
- Native filesystem activation belongs to execution authorization and is
not an eligibility input for this notice projection. Other
notice/maintenance duties and Cloud sleep gates remain separate.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, local code inspection,
shell execution, and PostgreSQL testing. The exact serving model
identifier and context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 14:14:28 -05:00
DottaandPaperclip 1aa3a21ac4 fix(server): wake inbound queue notices from committed changes (#17647)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Chat users receive a notice when accepted input waits for execution.
> - Periodic publication maintenance scans those receipts even when
nothing changes.
> - Admission, cancellation, source removal, and authority changes
already define when a notice can change.
> - Existing action, receipt, publication, and provider-link rows
preserve that work across restart.
> - This PR gives ordinary inbound queue notices a commit-driven owner
using the shared scheduler.
> - Settled notices stop querying while unfinished batches and
contention retain recovery.

## Linked Issues or Issue Description

This ordinary inbound-notice slice follows merged #17895 and now targets
master. Related extraction: #16187. Failed-run retry notice projection
remains a separate following slice (#17649); its two shared
receipt-writer notifications move forward here because ordinary input
can attach to those receipt owners.

**What happened?**

Publication maintenance scans accepted input to project queued,
not-started, and removed-source notices. Empty scans keep touching the
database. Its cursor also loses PostgreSQL timestamp precision, so a
full page of suppressed candidates can repeat.

**Expected behavior**

Committed changes should wake the notice owner. The worker should
reconstruct saved candidates after restart and stop scanning when
settled. It must retain a wake across drain, failed selection,
contention, and a commit during reconciliation.

**Steps to reproduce**

Start an otherwise idle instance with chat history. Observe the ordinary
inbound notice scan in periodic publication maintenance. For the cursor
defect, retain a full page of denied candidates with microsecond
timestamps before an eligible notice.

## What Changed

- Register an ordinary inbound-notice owner with the existing delivery
coordinator and scheduler. Bound each pass to 100 candidates, preserve
precise continuation cursors and earlier-page contention, and reset
scoped discovery after committed changes.
- Notify from real admission, failed action settlement, source removal,
task/tree/agent/budget/company cancellation, signed Slack Stop, and
deferred queue cancellation/failure. Preserve pending-status and
absent-run guards so concurrent promotion wins.
- Cover task privacy/grants, project membership/visibility and current
authority through their persistence owners. Keep receipt ownership,
current authorization, provider policy and closed notice text in the
existing chat service.
- Cover shared retry-owned receipts through the existing durable
authorization callback and a notifying transaction around conditional
denial retirement. These small producer fixes already exist in #17649;
no per-caller recovery wrappers are added. Retain its matching
budget-refusal fixture correction.
- Remove only ordinary notices from periodic maintenance. Preserve the
separate failed-retry pass at 50 and keep the explicit combined manual
API.
- Preserve timestamp precision, fairness, restart, drain/resume,
rollback and quiet behavior. Add two real producer transaction
regressions and update the audit. Add no storage or scheduler
infrastructure.

## Verification

Current publication head: `41f7e2c9e351efa6cfa5ef699c0e69df7425012b`,
after normal merge of master `d30c8f42b842327ee1c18c41a2efd6ba623da5cd`
(#17960). The integration preserves both complete added/removed patches,
all non-overlap upstream blobs/deletions, and task-before-comment lock
order before the notice source lock/intent. Full binary identity is not
claimed for the overlapping files. The final follow-up changes
acceptance documentation only.

- **202 distinct combined-master cases pass on
`1720b685d9d5fb9cb4377223fdaa0a9fae2aff9c`:** 148 across the real
PostgreSQL receipt concurrency and two comment route suites, plus 54
notice/FIFO/producer cases (1,598 skipped). Current server
typechecking/build and module boundaries pass; all 21 owned source/test
hashes are unchanged after the documentation follow-up.
- The earlier CI voice-reconstruction quiet assertion failed once after
105 fetches. All 15 voice lifecycle cases pass unchanged locally; no
assertions or production code were relaxed and no CI retry was used. The
original failure remains in the audit, and current-head full CI must
pass.
- **Earlier restoration repair (head
`28c898e93504318056d7a6ab2bbc5eafa6b639ae`): 126 distinct selected cases
pass (54 ordinary-notice/producer cases with 1,598 skipped, plus 72
separate Retry/budget cases with 1,580 skipped).** Fix the blocking
[restored input stays
silent](https://github.com/paperclipai/paperclip/pull/17647#discussion_r4404007181)
finding. A failed Retry's earlier receipt wake can drain before
`releaseUnadmittedTaskRetryInput` restores ordinary receipts in a later
transaction. The restoring transaction now awaits the existing
company/task notification before its guarded write.
- The new causal regression builds a durable unadmitted failed-Retry
owner, uses the real input-claim owner, consumes an earlier committed
wake, and proves two minutes of worker quiet with no notice. The real
release transaction then restores the ordinary row, emits intent/commit
settlement, and reactivates the coordinator to create its queued notice.
It fails before the hook with the restored row visible but no intent.
This is real PostgreSQL producer/consumer boundary proof with seeded
Retry-refusal state; the separate compatibility selection exercises
existing supported GitHub Retry flows. It is not a claimed complete
public-request failure walkthrough.

- **Earlier discovery-repair acceptance (revision
`4a713edfa06b9d3dcc0e81e900006fbdb973b2d9`): 267 distinct selected cases
pass:** 142 complete cancellation/app/admission consumer cases, 53
ordinary-notice cases, and 72 separate retry/budget cases. The
selections skip 1,598 and 1,579 other cases. The final master
integration leaves this complete patch byte-identical; the new master
paths concern provider workspace/session reuse and E2E reports.
- Fix the valid P2 [discovery rereads denied
history](https://github.com/paperclipai/paperclip/pull/17647#discussion_r4386889825),
despite the previous head's 5/5 summary. Task hints now need zero
discovery SQL; company/startup discovery returns DISTINCT company/task
identifiers with one query. Bounded projection still checks current
authority and candidates. Three real database regressions with 250
denied actions and a second company fail before the fix and pass
afterward, checking actual query and returned-row counts. Broad SQL may
still inspect candidate history; no constant database-cost claim is
made.
- Both new producer tests fail before their fixes and pass afterward.
One observes intent before a callback-owned receipt write and settlement
after outer commit. The other uses real heartbeat admission/recovery and
a PostgreSQL row lock to prove exact company/task intent exists before
conditional denial retirement; the outside connection still sees
deferred status until commit.
- The reader can select a retry-owned deferred receipt for later
ordinary input, and receipt resolution follows its owner. Disabling
coalescing on the retry's own request does not exclude that later path.
The two transferred notifications close the actual writer gaps; current
authorization remains enforced by existing consumers. This is
producer/transaction proof, not a claimed complete mixed-ingress
walkthrough.
- The moved callback exposes an old test mock that authorized against
the root DB. The existing #17649 fixture correction authorizes and
writes in one transaction; original requester, comment, skipped-status
and budget assertions remain. Both the initial invalid lock fixture and
the corrected missing-intent failures are preserved in the audit.
- Updated shared dependency build, server typechecking/build and
module-boundary checks pass. Because master also changed agent
configuration, upgrade admission and runner contracts, the same 264
cases were rerun on combined revision
`875a6ee8d211cba7f0499aace7fcea7f6880c4a2` with new isolated databases.
Repeated checks are not 528 different tests. Tests under `src/__tests__`
were executed separately from production typechecking.
- [Current acceptance and preserved failure
ledger](https://github.com/paperclipai/paperclip/blob/41f7e2c9e351efa6cfa5ef699c0e69df7425012b/doc/plans/2026-10-25-inbound-wake-notices-sleep-audit.md#shared-retry-receipt-producer-gap-resolved-october-28)
records current scope, exact commands, actual fixtures, source hashes
and remaining limits. Earlier baseline/full-suite counts retain their
original revision scope.
- Server typechecking/build and module boundaries pass after the current
restoration hook. The earlier combined-master compilation at
`661f2f6da8c6c22cdcc25edccd57502ee6fa8c4b` retains its original scope.
- **Current exact-head CI is green and Greptile is 5/5.** The old head's
review error is not a pass. No full local repository matrix, live
provider tenant, external alarm, physical suspension or savings claim is
made. Following notice duties retain their separate owners.

### Final merge verification

- Reviewed PR head remains `41f7e2c9e351efa6cfa5ef699c0e69df7425012b`:
exact-head Greptile 5/5, all 55 check contexts clean, no unresolved
review threads, and no Ponytail verdict present.
- One Runner shard initially failed descriptor suspension and a
five-second OpenCode fixture deadline. The same package passed on the
preceding head. The first local attempt was invalid because required
Rust fixture binaries were absent; after building them, all eight
relevant cases pass with unchanged timeouts. One failed-job retry then
passes 1,956 cases (four skipped). No source/assertion/time-budget
change or second CI retry was used.
- Current master `0e365768f032a472807b5720da8431f92d2f07ce` adds manual
monitor and task-drain fixes. Its routes file overlaps but merges
cleanly. Local normal-merge validation commit
`2720b79c769cb609ad4d392512331e67b910ff3f` preserves both complete
added/removed patches and every unrelated master blob/deletion. This is
not raw binary identity for overlapping files.
- **156 distinct cases pass on that exact combined tree:** all 102
monitor/drain/policy cases plus 54 notice cases (1,598 skipped). Types
and build pass. An initial competing-cold-check run failed setup hooks
(38 passed, two hook failures, 62 skipped); the same four suites pass
alone with all original hook/test timeouts and assertions intact. The
failed run remains recorded, not counted as full acceptance.
- This merge proof changes no published source or review head. The
protected pinned-head merge guard recomputes both patches and requires
the exact tested master/tree/head, successful validation receipts and
unchanged live merge gates. Previous acceptance counts retain their
original revision scope; no local full-repository or deployed-sleep
claim is made.

Master then advanced to `75754ffb7af2b7ee7dce1e915a145faa82a8fb2a` with
#17976 (five UI/e2e/map files only). The entire `server` and `packages`
trees are identical to the validated composition. Both complete patches
and all newly landed UI files are preserved in the final clean merge
tree; the protected gate checks these exact source-root hashes and
changed paths. No additional UI-test result is claimed.

## Risks

- Missing a supported producer could delay a notice until restart. Tests
cover real service and API producers; durable rows remain authoritative.
Notifications cover supported application writers in the owning process,
not arbitrary external SQL writers.
- Privacy invalidation can affect descendants, so those changes use
company scope. Ordinary admission and publication changes use task
scope. Settled history causes no recurring scan.
- Provider output remains subject to current authority and receipt
ownership. A cancelled request without a visible queued notice does not
acquire a new failure message.
- This establishes worker quiet, not host or database suspension. Cloud
alarms and conservative sleep gates do not change. Other maintenance
remains in place.
- Remaining stacked slices still need their own integration, review, and
CI. This slice merges only under the user’s explicit campaign
authorization after current-head gates pass.

## Model Used

OpenAI Codex, GPT-6 family, with repository inspection and code
execution tools. The exact serving variant and context-window size are
not exposed in this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 13:34:38 -05:00
DottaandPaperclip 1ed0af4186 fix(server): wake native chat progress from committed changes (#17639)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Chat users receive coarse progress updates while native runs
execute.
> - A recurring chat pass scans saved run events even when no progress
changes.
> - This traffic prevents an idle database from staying idle.
> - Events, lineage changes, and publication state already identify when
projection can change.
> - This PR gives native progress its own commit-driven worker and
schedules only unfinished work.
> - Idle progress stops scanning while restart and recovery preserve
delivery.

## Linked Issues or Issue Description

Refs #17606. This is slice 29 of the recurring-server-work stack, based
on `codex/sleep-28-chat-run-milestones`. It owns native chat progress
only. Related historical optimization: #16845. The parent and this PR
must land in stack order.

**What happened?**

Native chat progress is still called by the one-second chat
reconciliation loop. It repeatedly selects run events with no new work.
Its cursor also truncates PostgreSQL timestamp precision, which can
repeat a rejected full page forever.

**Expected behavior**

Committed events and eligibility changes should wake the projection
owner. Existing rows should reconstruct work after restart. A settled
worker should have no recurring database activity. Full batches, lock
contention, and errors must retain recovery.

**Steps to reproduce**

Start an otherwise idle instance with chat history. Observe the native
progress lane in the recurring chat coordinator. For the cursor defect,
create a full page of suppressed candidates sharing a microsecond
timestamp and one eligible later candidate. The old cursor fails to
advance.

## What Changed

- Extract native progress from lifecycle milestones into its owning
service. Keep the public combined API and budget compatible.
- Register a scoped worker with the existing delivery coordinator and
scheduler. Remove native progress from the recurring chat coordinator.
- Notify before allowlisted event writes in both persistence owners.
Preserve outer commit, rollback, duplicate detection, and the ordinary
non-progress log fast path.
- Extend the existing lineage/authority notification boundary once.
Failed-publication replay and operator retry carry their issue scope.
First working publications signal progress so commit ordering cannot
strand an already-inspected event.
- Limit notification payloads to company/task IDs so wider caller inputs
never serialize database handles or private fields. Load task-binding
fields only for progress event kinds; other native/recovery events
retain their narrow lock query.
- Preserve current safe phrases, cadence, phase deduplication, provider
policy, authorization, and final-output precedence.
- Retry concrete issue/run lock contention. Disarm deleted or terminal
candidates. Preserve scoped dirty work across batches, failures, drain
and shutdown.
- Preserve PostgreSQL timestamp precision in pagination. Add a bounded
regression for the former infinite rejected-page scan.
- Document the producer audit, lifecycle contract, failures, and
remaining limits in
`doc/plans/2026-10-25-native-chat-progress-sleep-audit.md`.

## Verification

Published head: `b7ee820e49fd06c780b36b79d9b1c0883557fb73`, targeting
master after #17606 merged at
`5f0da562dbfdf3229f3cd4bb3dc742ab541986b4`. A normal ancestry merge
preserves the complete 18-path native-progress patch byte-for-byte
relative to its verified parent, with all six unrelated current master
paths intact. No new production change was introduced during final
reconciliation.

- **163 distinct selected cases pass on the verified unchanged patch:**
137 across eight complete suites (native progress, atomic event append,
all five milestone worker suites, and app admission wiring), plus 26
real chat integration cases selected by `native progress`. The other
1,595 chat integration cases were skipped, not passed.
- The full worker suites cover restart, committed events, trace
reprojection, duplicate/no-progress events, first-working publication
ordering, concrete lock contention, issue scope, backlog, drain/resume,
failures and shutdown. Settled history leaves no recurring SQL. The real
chat cases verify operator retry and replay, microsecond pagination,
authority, cadence, phase deduplication, final/question precedence and
redaction.
- The sole earlier parent conflict preserves both this slice's
issue-scoped retry wake and the exact voice conversation scope. The
current parent's milestone producers, onboarding completion guard,
identity query/read-scope coverage, migration 0397 and voice
cleanup/retention repairs remain present.
- Server production typechecking, build and module-boundary checks pass.
The selected server tests live under the excluded `src/__tests__` path
and were executed separately. Each database owner uses a new isolated
app-migrated PostgreSQL fixture.
- [Current-parent acceptance and failure
ledger](https://github.com/paperclipai/paperclip/blob/b7ee820e49fd06c780b36b79d9b1c0883557fb73/doc/plans/2026-10-25-native-chat-progress-sleep-audit.md#current-milestone-parent-acceptance-october-28)
records current counts and scope. Older combined checks and failures
retain their original revision scope and are not added to 163. No new
table, dependency, workflow, migration, or scheduler contract is
introduced.
- **Exact-head Greptile is 5/5 and all 55 current checks are clean.** No
Ponytail review is present, which is an explicitly authorized
nonblocker; no failing verdict is waived. Older head/parent reviews are
not current passes. Actual failing Ponytail findings must be fixed.
Missing/stopped/paused turns without a verdict are authorized
nonblockers; active unfinished reviews retain the original per-head
timer with a 25-minute monitor cutoff respecting the 30-minute maximum.
- No complete local repository matrix, live provider tenant, external
wake, physical host/database suspension or savings claim is made.
Following stack duties remain separate.

## Risks

- Signals are process-local under the existing single active writer
contract. Independent SQL writers or another active process need an
explicit wake integration.
- Allowlisted direct log writes now use a transaction. Native bound runs
register transaction intent; unrelated direct event types retain their
atomic-statement path.
- The shared eligibility boundary wakes both lifecycle and native
projection. It does not decide provider policy or copy event payloads.
- The existing 24-hour selection window, event-time cadence and
authority checks remain. A suppressed event is not retried just because
wall-clock time passes.
- This establishes worker quiet, not whole-instance suspension. Cloud
alarm coverage and conservative sleep gates do not change. Other chat
maintenance remains on its existing cadence.
- This actual-master reconciliation still requires fresh exact-head
review and CI before protected merge. Subsequent slices retain their own
gates.

## Model Used

OpenAI Codex, GPT-6 family, with repository inspection, code execution
and browser-test tools. The exact serving variant and context-window
size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 11:22:29 -05:00
DottaandPaperclip 5f0da562db fix(server): wake chat lifecycle milestones from committed changes (#17606)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connected chats show safe notices when a run starts, waits,
finishes, or fails.
> - A recurring scan projected these notices even when no eligible work
existed.
> - Eligibility depends on committed run state, conversation lineage,
and current authority.
> - Existing database rows can reconstruct that work after restart.
> - This PR wakes a scoped worker from those writes and removes
lifecycle polling.
> - Native progress events keep their separate existing owner until the
next migration.

## Linked Issues or Issue Description

**What happened?**

The chat maintenance timer repeatedly scanned run lifecycle state. Empty
and settled instances still queried the database. A run-status event
alone cannot replace the scan: prompt receipts, continuation receipts,
identity changes, and restored authority can make a run eligible later.

**What did you expect?**

Committed changes wake projection. Startup reconstructs eligible work. A
completed pass stops until another relevant change arrives. Failed work
retains recovery without scanning settled history forever.

**Steps to reproduce**

Configure an automatic chat endpoint. Finish its work, or leave it
unused. Observe recurring lifecycle projection queries from the chat
maintenance callback.

Stacked on #17592. Related foundation: #15625. This PR owns lifecycle
milestones only.

## What Changed

- Preserve upstream coalesced Cloud membership writes together with
locked membership restoration and its committed chat notification.

- Signal milestone eligibility from the real Slack account-link
transaction.

- Register the lifecycle worker with the existing delivery coordinator.
Remove lifecycle projection from the periodic chat callback. Keep native
progress in a separately named lane and preserve the explicit combined
API.
- Reconstruct company/issue scopes from the existing selector. Drain
bounded batches with company fairness. Retry failures, retain writes
through drain/resume, and disarm settled work.
- Signal from supported run, presentation, interaction, receipt,
lineage, conversation, recovery, and authority writers. Preserve their
transaction ownership, including nested savepoints and uncertain
settlement. No new table is added.
- Preserve provider authorization, authored-final suppression,
responsible-user continuity, deduplication, the 24-hour selection
window, and raw-output restrictions. Keep provider interpretation in its
owner.
- Preserve PostgreSQL timestamp precision in lifecycle pagination.
Rejected submillisecond candidates must not repeat the same page and
starve later notices.
- Document producer coverage and alternate-writer dispositions in
`doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md`.

## Verification

Published head: `1d056f68c5b1005933b9e12de8745bf1c4cbeeca`, targeting
master after #17592 merged at
`7d75ae0569e41786640cf09647ae8849411d2f22`. A normal merge integrates
the actual landed parent. The entire 77-path milestone patch is
binary-identical to the locally verified preparation across that
ancestry reconciliation; both unrelated master paths are preserved. The
four earlier producer/test integration conflicts retain scoped milestone
and voice notices, current onboarding completion admission, and expanded
identity query/read-scope coverage.

- **330 distinct selected cases pass on the verified unchanged patch:**
305 across 14 complete suites (all five milestone worker suites, run
identity, board claim, Cloud provisioning, auth session routes, Cloud
actor authentication, stale-lock recovery, GitHub credentials, project
Git authorization, and app admission wiring), plus 25 selected real chat
producer cases. The remaining 1,590 chat integration cases were skipped,
not passed.
- The real producer cases cover saved milestone discovery, GitHub and
Slack identity changes, account-link rollback/reconfirmation, ordinary
ingress reactivation, post-close action/delivery receipt settlement, all
ten authority-restoration paths including Cloud synchronization, and
changed native review metadata.
- Worker coverage verifies outer commit, rollback, uncertain-write
recovery, restart, bounded batching/fairness, timestamp precision,
retries, drain/resume, and settled SQL quiet. Authorization and existing
publication deduplication remain with their owning services.
- Server production typechecking, build, and module-boundary checks
pass. The tests above were executed separately. Production typechecking
excludes `src/__tests__` and does not replace those runs. Each database
owner uses its own newly created and app-migrated PostgreSQL fixture.
- [Current-parent acceptance and preserved failure
ledger](https://github.com/paperclipai/paperclip/blob/1d056f68c5b1005933b9e12de8745bf1c4cbeeca/doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md#current-parent-integration-and-acceptance-october-28)
records commands, counts, producer scope, conflict dispositions, and
limits. Earlier combined-stack counts and failures keep their original
revision scope; local acceptance does not erase old CI failures.
- No new migration, queue table, dependency, workflow, or shared
scheduler contract is introduced. Parent migration 0397, bounded consent
recovery, voice cleanup/retention repairs, and runner checkpoint
restoration remain present.
- **Exact-head Greptile is 5/5 and all 55 current checks are clean.** No
Ponytail review is present, which is an explicitly authorized
nonblocker; no failing review is waived. Older reviews and parent checks
are not current passes. Actual failing Ponytail findings must be fixed;
an absent/stopped/paused turn without a verdict is an authorized
nonblocker. Active unfinished Ponytail uses its original request/start
timer, with a 25-minute monitor cutoff respecting the user's 30-minute
maximum.
- No complete local repository matrix, live provider tenant, external
alarm, physical host/database suspension, or fleet savings claim is
made. Native progress remains a separate following slice.

## Risks

- A future eligibility writer must notify the existing projection
boundary. Independent SQL writers or another active server process are
outside this process-local signal contract.
- Authority and lineage remain sensitive. The worker uses the existing
selectors and publication policy; it does not grant execution or
disclosure authority.
- Native progress and other publication maintenance still have recurring
work. This PR removes one reason, not the entire chat maintenance timer.
- Worker quiet does not prove database suspension, host sleep, or
external alarm coverage. Cloud sleep blockers remain unchanged.
- Merge eligibility requires passing current-head CI and Greptile 5/5
after this actual-master reconciliation. Subsequent stack slices retain
their own gates.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and
context-window size are not exposed. Used repository inspection,
reasoning, shell execution, PostgreSQL tests, browser E2Es and GitHub
tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details available in this session)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run the documented focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 11:07:54 -05:00
DottaandPaperclip 7d75ae0569 fix(server): recover Teams attempts from saved claim deadlines (#17592)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Teams file delivery saves a claim before each provider operation.
> - A crashed operation can leave that claim unfinished.
> - Periodic maintenance scans for abandoned claims even when none
exist.
> - This PR wakes recovery from committed changes and the saved claim
deadline.
> - Empty work becomes quiet while unknown effects remain safe to
review.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Teams abandoned file-transfer recovery. This is slice 27, stacked on
consent expiry #17590.

**Current behavior**
The remaining Teams maintenance callback scans every 30 seconds for
expired transport claims. It runs from shared publication maintenance.

**Proposed behavior**
The Teams owner reconstructs existing claim deadlines at startup and
reloads a transfer after its committed update. At expiry it clears the
claim and preserves the existing unknown-outcome or conflict state.
Normal receipt settlement removes the deadline.

**Reason and benefit**
Remove the last periodic Teams maintenance scan without resending a
possibly delivered card or file.

**Breaking changes**
No schema or public API change. Explicit diagnostic recovery methods
remain compatible. Consent expiry stays with its separate parent worker.

## What Changed

- Add a Teams-owned worker for consent_sending, uploading,
file_info_sending and conflict rows with a saved attempt expiry.
- Notify from the existing versioned update before writes. Preserve
transaction ownership and shared uncertain-write reconciliation.
- Expose a narrow per-transfer recovery method that uses the existing
lock order, integrity checks and atomic projection.
- Remove the periodic Teams maintenance callback and its
publication-maintenance invocation.
- Add real claim, receipt, conflict, backlog, restart, precision,
failure, drain and shutdown tests. Wire the owner through existing app
admission gates.
- Retain the existing signoff cancelled-run selection test and GitHub
inline retry-window race assertion.
- Document supported writers, remaining sleep limits and the failure
ledger in `doc/idle-teams-attempt-recovery.md` and its linked audit.

## Verification

Published head: `24c54399909105984e2a10f5869064984f69c472`, targeting
master after consent #17590 merged at
`4a89456bb3f9aef87cfe0e922e8b9d1a777a6c7e`. A normal ancestry merge
reconciles the landed squash with the previously verified child. The
complete 13-path binary patch is identical before and after
reconciliation, and all 30 unrelated master paths are preserved. This
adds no new worker behavior; the 239-case local evidence below belongs
to the verified unchanged patch. All 58 current-head checks are now
clean, including the complete required CI run. Greptile is 5/5 at this
exact head. The current Ponytail turn is paused without a verdict, which
is an explicitly authorized nonblocker; no failing review is waived.

The complete 13-path slice preserves the reviewed attempt implementation
and inherits the parent's report migration 0397, bounded consent worker
and retention CI test repair. No new schema, queue table, shared
scheduler contract, workflow or lockfile change is added by this duty.

- **239 distinct selected cases pass:** 221 across five complete
transfer/publication/consent/runtime/app-wiring suites, 16 selected real
Board/Teams-authority/GitHub-retry integration cases, and two delegated
Slack policy cases. Other chat integration cases (1,579) and
connection-management cases (81) were skipped, not passed. Each database
owner uses a new isolated PostgreSQL database.
- Real Board publication saves an active attempt deadline; its confirmed
receipt retires recovery, and SQL stays quiet afterward. The inherited
consent producer and authority cases still pass. Lifecycle coverage
exercises claim stages, unknown-outcome quarantine without repeating
provider I/O, conflicts, rollback, restart, precision, backlog/fairness,
failure recovery, drain and shutdown. The changed GitHub metadata/retry
race passes without duplicate assessments.
- The old server10 CI failure, run `40557802358` job `122390238707`, was
a delegated Slack invitation rate-window assertion (2,122 passed/one
failed/15 skipped in that shard). The verified parent already contains
its Date-only fixture fix. Both linked and invitation cases now pass
their full caller/target/shared-policy scenarios on a fresh server
fixture; no duplicate policy change or new owned file was added. That
old failure is preserved rather than called a pass.
- Server production typecheck, build and module-boundary checks pass.
Test files were executed; production typechecking does not statically
check them. The existing test-only signoff cancelled-run selection fix
is retained but was not executed locally in this acceptance; fresh
remote E2E gates remain required.
- [Current-parent acceptance and preserved failure
ledger](https://github.com/paperclipai/paperclip/blob/24c54399909105984e2a10f5869064984f69c472/doc/plans/2026-10-25-teams-attempt-recovery-sleep-audit.md#current-consent-parent-preparation-and-acceptance-october-28)
records current scope, commands, counts and limits. Earlier 235-case and
combined-stack counts belong to their stated revisions and are not added
to 239. The [stack
ledger](https://github.com/paperclipai/paperclip/blob/24c54399909105984e2a10f5869064984f69c472/doc/plans/2026-10-24-recurring-server-sleep-stack.md)
retains inherited failure history. No complete local repository suite,
live Teams/Slack tenant, external wake, physical suspension or savings
claim is made.
- **Exact-head Greptile 5/5 and all 58 checks are clean.** Actual
failing Ponytail findings must be repaired; a missing/stopped/paused
turn without a verdict is an authorized nonblocker. An active no-verdict
turn waits at most 30 minutes from its original request/start (25-minute
monitor cutoff). Older reviews and parent checks are not treated as
current passes.

## Risks

- Recovery must not infer that an expired claim means a provider effect
did not happen. It retains unknown outcomes and does not resend.
- Late receipts must still match current claim identity. Current-state
rechecks and existing protocol tests preserve this fence.
- Supported deployment uses one active writer process through its owning
DB client. Independent SQL and additional processes need a separate wake
contract.
- Cloud external alarm coverage and sleep eligibility are unchanged.
Other chat duties still run periodically until their own migrations.
- Voice-report parent review repairs are propagated: explicit
single-active-writer scope and an indexed report window. Thirty
report-worker/startup tests pass on the combined child. Fresh remote
checks, older stack findings and final master reconciliation remain
required.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, tool use and
code execution. The exact serving model ID and context-window size are
not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 10:51:16 -05:00
DottaandPaperclip 4a89456bb3 fix(server): schedule Teams consent expiry from saved deadlines (#17590)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Teams file delivery saves consent and transfer state in the
database.
> - Unused consent must expire even when no new message arrives.
> - The maintenance loop currently checks this state every 30 seconds.
> - This PR schedules each existing consent deadline and wakes on
committed changes.
> - Empty or settled consent work stops querying while expiry remains
recoverable.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Teams unused file-consent expiry. This is slice 26 of the
recurring-server migration stack, following merged #17586 and targeting
master.

**Current behavior**
The Teams maintenance callback scans for expired unused consent and
abandoned transport attempts together, even when consent work is empty.

**Proposed behavior**
The Teams owner reconstructs unused consent at startup and reloads one
transfer after a committed write. The shared scheduler runs its saved
expiry. Abandoned attempts keep their current recovery until a separate
slice migrates them.

**Reason and benefit**
Remove unnecessary consent queries without losing expiry, atomic
publication projection, or recovery after restart.

**Breaking changes**
No schema or API change. The explicit combined recovery method remains
available. Cloud sleep eligibility and external alarms remain unchanged.

## What Changed

- Add a Teams-owned scoped deadline worker for the three unused consent
phases. Registered reconstruction yields after one 100-row page per
scope refresh and resets cursors on registration; the explicit
diagnostic inventory still returns every eligible row.
- Signal before insertion and the existing versioned update inside the
owning transaction. Nested writes wait for outer settlement.
- Extract the existing locked per-transfer recovery operation. Preserve
authority checks, atomic projection, bounded lock waits and
unknown-outcome quarantine.
- Remove unused consent from periodic maintenance selection and
execution. Keep abandoned attempts as a separate obligation.
- Register the worker through existing app lifecycle gates. Add
lifecycle, producer, production wiring and precision regressions.
- Document supported writers, remaining sleep limits and the failure
ledger in `doc/idle-teams-consent-expiry.md` and its linked audit.

## Verification

Published head: `9e84cfab1f795279c8cec413a946631f1d781bd2`. This
follow-up changes only the inherited voice-retention test and its CI
failure ledger; all production behavior is unchanged from `36f43922`. It
normally merges actual landed master/report parent
`8049fbf51d0701ca6b242e4b1198e5449f7661b2`. The verified 12-path Teams
patch was byte-identical across the actual-parent ancestry integration.
This test-only CI follow-up adds one existing retention test path,
bringing the final scope to 13 paths. Every unrelated master change,
report documentation fix, and migration 0397 is preserved. No database,
shared-worker, workflow or lockfile change belongs to this slice.

- **223 distinct selected cases pass:** 209 across five complete
transfer/publication/consent/runtime/app-wiring suites, plus 14 selected
real Board-producer and Teams-authority integration cases. The other
1,580 integration cases were skipped, not passed. The protocol and
production consumers use separate new PostgreSQL databases migrated
through the application migrator.
- A real 105-transfer regression failed against the original worker
because one registered startup pass selected 105 rows instead of
yielding at 100. The corrected worker uses the existing paged shared
contract. Restarting the same service midway through discovery selects
the first 100 again, restores all 105 saved deadlines and expires them
at the original deadline. SQL is quiet before the deadline and after
settlement, with no consent-card or upload requests. The one-shot
inventory still returns all 105 rows.
- A guarded mutation removing only cursor reset fails at five rows
instead of 100 after restart. The original production source was
restored before the complete suites. The initial regression fixture
incorrectly faked network timers before its first SQL connection and
timed out; that setup failure is retained separately from the corrected
105-versus-100 regression. No assertion or timeout was weakened.
- The real Board producer registers the worker before publication,
commits the transfer wake, preserves the saved expiry, remains SQL-quiet
until due and atomically projects expiry to a cancelled publication
without another provider request. Existing authority and consent
transitions remain covered.
- Complete lifecycle coverage retains all three consent phases,
precision rounding, rollback and held outer commit, exact-row hints,
expiry recheck, failed projection retry, corrupt-row fairness,
drain/resume, shutdown and separation from abandoned-attempt
maintenance. Production server typecheck, build and module boundaries
pass. Server production typechecking does not statically check test
files; those files were executed.
- [Current bounded reconstruction and failure
ledger](https://github.com/paperclipai/paperclip/blob/36f43922b0a408f76adb7de31280812e9b752019/doc/plans/2026-10-25-teams-consent-expiry-sleep-audit.md#bounded-reconstruction-and-current-parent-acceptance-october-28)
records exact scope, counts, guarded mutation, corrected timer setup,
earlier fixture failures and limitations. Earlier 222-case and
combined-stack results belong to their stated historical heads; they are
not added to 223 or claimed rerun here. The [stack
ledger](https://github.com/paperclipai/paperclip/blob/36f43922b0a408f76adb7de31280812e9b752019/doc/plans/2026-10-24-recurring-server-sleep-stack.md)
retains inherited failure history. No complete local repository-suite,
live Teams tenant, Cloud wake, host/database suspension or savings claim
is made.
- **Fresh exact-head Greptile 5/5 and all CI are required.** Actual
failing Ponytail findings must be repaired. A missing/stopped/paused
turn without a verdict is an authorized nonblocker; an active no-verdict
review waits at most 30 minutes from its original request/start
(25-minute monitor cutoff). Older scores and parent checks are not
accepted as current passes.

- **Preserved inherited CI failure:** prior head `36f43922` had
exact-head Greptile/Ponytail 5/5 but CI run `40708566195` server6 job
`122887395220` failed the retention fairness assertion (2,291 passed/one
failed/one skipped in that shard). A fixed final UUID reproduces the
wrong expectation that an undiscovered page-two row is already deleted.
The corrected test controls time, expects that row to remain before
discovery and requires its deletion in the next fair batch, then all
remaining history and idle SQL. The existing five-second wait budget and
global timeout are unchanged. All 13 retention lifecycle cases pass;
bypassing company rotation fails the corrected assertion, and the
original shared helper is restored byte-for-byte. Earlier 223-case Teams
acceptance remains valid for unchanged Teams source and is not relabeled
as a new complete repository run. [Full CI repair
ledger](https://github.com/paperclipai/paperclip/blob/9e84cfab1f795279c8cec413a946631f1d781bd2/doc/plans/2026-10-25-teams-consent-expiry-sleep-audit.md#inherited-retention-ci-fairness-repair-october-28).
Fresh CI and exact-head reviews are required for this test-only
follow-up; old scores are stale.

- **Current merge gates:** all 58 checks are now clean for this exact
head and Greptile is 5/5 with no actionable findings. The current
Ponytail turn is paused without a verdict, an authorized nonblocking
state. Workflow `40712464720` attempt 1 independently hit npm
`ECONNRESET` during the serialized8 job's tool self-installer, before
its tests ran. Exactly one targeted job retry was requested after the
original workflow finished; attempt 2 completed successfully on the
unchanged head. No code, dependencies, workflow, timeout or review timer
changed for that retry. The original setup failure is retained in its
[job
log](https://github.com/paperclipai/paperclip/actions/runs/40712464720/job/122901243926);
the successful replacement is [job
122910350074](https://github.com/paperclipai/paperclip/actions/runs/40712464720/job/122910350074).

## Risks

- Missed supported producer signals could delay expiry until restart.
Real insertion, cancellation, consent transitions, caller-owned
transactions and Board publication are covered.
- Corrupt or locked rows must retain retry without starving healthy
work. Paging, fairness, atomic rollback, redaction, drain/resume and
shutdown tests cover these paths.
- Independent SQL writers and other processes need their own
notification contract. No persistent DB listener is added.
- Attempt recovery still scans periodically until the next slice. A
local deadline is not a durable Cloud alarm. This PR does not claim
whole-instance sleep.
- This slice is reconciled with its landed parent; current-head remote
checks/review and live-master composition proof still gate merge. Later
stack duties remain separate PRs.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, tool use and
code execution. The exact serving model ID and context-window size are
not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 10:32:30 -05:00
DottaandPaperclip 71b448b9cb fix(server): schedule inbound voice cleanup from durable obligations (#17580)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Incoming voice calls need approval, expiry, and safe provider
cleanup.
> - The remaining voice maintenance tick scans admissions every second,
including idle endpoints.
> - Admission status does not prove that the provider call has ended.
> - Existing admission rows can retain the cleanup owner, confirmation,
and retry deadline.
> - This pull request replaces the cleanup scan with committed changes
and actual deadlines.
> - Idle cleanup stops querying while unfinished calls keep a
recoverable path to termination.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Inbound voice admission cleanup. Its reply-delivery parent #17578 has
landed; this PR targets master. Live-session recovery is in #17576. It
preserves separate ownership for those services.

**Current behavior**

The voice maintenance tick scans expired admissions even when no cleanup
is due. A session can commit before the separate approved-admission
write. Expiry can then terminate that authorized call. A rejected guest
call can be labeled ended before provider termination succeeds.
Seven-day retention can delete that unfinished cleanup.

**Proposed behavior**

Wake cleanup from committed admission and session changes. Schedule the
original expiry or saved retry. Claim current state under the admission
lock before provider I/O. Commit approval with session reservation. Keep
unconfirmed termination durable and transfer bound calls to live-session
recovery.

**Reason and benefit**

Configured idle endpoints and confirmed history cause no recurring
admission cleanup SQL or provider requests. Approval, retries, crash
recovery, and cleanup remain supported. History retention and call
reports stay separate migration items.

## What Changed

- Recover terminal and rejected legacy admissions whose session binding
was lost. Copy known termination or bind a proven matching live session
for cleanup without granting approval.
- Register inbound cleanup with the shared coordinator and scoped
deadline worker. Yield after each 100-row scope page and reset cursors
on registration. Remove only cleanup from periodic voice maintenance.
Keep complete manual reconciliation.
- Add commit-aware notifications at admission, approval, rejection,
guest binding, pending-tool expiry, lifecycle confirmation, and session
termination writes.
- Commit authorized session binding and admission approval together.
Recheck cleanup ownership under the same admission lock. Recover older
interrupted bindings only from matching session evidence.
- Add nullable cleanup confirmation, next-attempt, and provider-account
fields to the existing admission table. No new queue table is added.
Generated migration 0396 backfills only provable historical termination
in committed 1,000-row keyset batches and adds a concurrent partial
index.
- Save the retry before provider contact. An unconfirmed end receipt
retains cleanup. Bound calls remain owned by live-session recovery.
- Verify the signed provider account before using a rotated key for
unbound cleanup. A different account cannot inspect or end the saved
call. Legacy admissions require their original credential fingerprint.
- Prevent history pruning and company deletion from erasing unfinished
cleanup. Company deletion locks endpoints before the voice owner's
check, matching ingress serialization.
- Document producer coverage, ownership, deadlines, verification, and
remaining sleep limits.

## Verification

Published head: `a4e7ccf976fc5793a8914e4e4861aa080d43a4da`. The reply
parent #17578 has landed. A normal merge integrates actual master
`e7ea2e2961863171956e70fb05d3d1f3790f7909`, preserves its unrelated
changes, and regenerates this slice’s migration after landed migration
0395. Fresh exact-head reviews and CI are required.

- The previous Ponytail 3/5 finding is repaired: legacy terminal
admissions no longer enter approval-only repair, and rejected admissions
with valid matching live sessions retain their rejection and persist the
binding needed for session-scoped cleanup confirmation. Five regressions
failed with ownership conflicts before the repair. Five
conflicting-session regressions prevent an unrelated leg from being
claimed. No cleanup operation grants approval or starts a provider call.
- **24 complete cleanup lifecycle cases and 122 complete voice-store
integration cases pass** on the repair. After actual-master integration
and migration regeneration, the complete 24-case cleanup suite passes
again on a new empty PostgreSQL 17 database, and the database migration
integration test passes separately. The cleanup owner, voice-store test
and admission schema are byte-identical to that tested repair; the
cleanup fixture only changes the migration filename. Migration
numbering/safety, database build, production server typecheck and build
pass.
- Migration `0396_tan_paladin.sql` was generated with `pnpm
db:generate`. Every landed SQL migration and the complete parent journal
prefix are unchanged. Generated snapshots retain exactly 0392–0396,
pruning 0391. The reviewed cleanup SQL, concurrent idempotent index and
committed 1,000-row historical backfill are retained byte for byte under
the new generated filename, registered for top-level execution. The real
application migrator boots a fresh database, repairs an interrupted
concurrent index, serializes replays and verifies eligible-history
backfill across actual transactions.
- The preceding published head
`48d8356e73308579470c182e759eead6e83173d2` received Greptile 5/5 but
failed CI server shard 9/12: one existing connected MCP rate-limit test
expected two real-clock calls to share a fixed 60-second bucket. A
controlled window rollover reproduced its exact failure. That existing
fixture now freezes only `Date` around the same-window 429 assertion,
verifies acceptance after advancing one window, and restores the real
clock in `finally`. All 92 gateway acceptance cases pass. Production
rate limiting, timeouts and assertions are unchanged. This adds one
existing CI fixture to the slice; the original failed run remains
recorded.
- The first regression invocation skipped the suite because it used the
wrong database variable; that is not a pass. The corrected invocation
reproduced five ownership conflicts. The first fixed cleanup run passed
22 and failed two new low-level deletion test calls on a
retained-session foreign key; the regressions now check the cleanup
deletion guard at its owning boundary, with production deletion ordering
unchanged. The final full suite passes all 24.
- The first complete voice-store invocation passed 121 and timed out on
one transcript case at the unchanged 15-second limit. That case passed
in isolation on a fresh database. One subsequent complete invocation,
with builds finished, passed all 122 without source, assertion or
timeout changes. The underlying timeout cause is unproven; the original
failure remains recorded. A new PostgreSQL startup also failed from
exhausted host semaphore capacity; fresh databases on the already-owned
server were used without removing shared IPC resources.
- Prior source `b8b12199b9` passed 266 distinct cases and carried three
guarded mutation proofs; those older results are not added to this
repair’s case count. Its CI run 40641168596 failed E2E shard 3 when
legacy cancellation hit a one-second PostgreSQL lock timeout. GitHub
initially rejected a retry while the run was active, then accepted one
targeted retry after completion; attempt 2 passed on that same old head.
The lock holder and underlying cause remain unproven. The unchanged Node
policy checker’s older failure on ignored generated Copilot manifests
and passing tracked-input mirror are also retained in the audit; no
workflow, checker, vendor manifest or ignore rule was modified.
- [Current repair and failure
record](https://github.com/paperclipai/paperclip/blob/a4e7ccf976fc5793a8914e4e4861aa080d43a4da/doc/plans/2026-10-25-voice-inbound-cleanup-sleep-audit.md#legacy-terminal-admissions-missing-their-session-binding--october-28).
Current-head CI and Greptile 5/5 gate merge. A completed failing
Ponytail verdict must be fixed; missing or interrupted reviews do not
block, and active pending review waits are capped at 30 minutes under
the maintainer’s explicit instruction. No new full local repository
suite, real provider, Cloud alarm, host/database suspension or savings
claim is made.

## Risks

- The migration distinguishes admission status from physical
termination. Legacy ended guest rejections without proof remain cleanup
obligations. Missing original credentials can keep those old obligations
unresolved.
- Future writers must use the existing notification boundary.
Independent SQL writers and multiple processes need their own wake
contract.
- Pending inbound cleanup now blocks company deletion instead of
allowing cascade deletion to lose the provider obligation. Existing
voice-history deletion restrictions remain. This deliberately includes
an active anonymous inbound ring and an old call whose termination
cannot be verified. An operator must stop ingress and settle the call
before deletion; this slice adds no abandonment override and never
treats missing credentials as proof of termination. This is the retained
cleanup ownership policy, not a claim that every deletion attempt can
complete immediately.
- History retention and report processing still run in the remaining
maintenance lane. This PR does not extend Cloud wake coverage or claim
measured host/database suspension.

## Model Used

OpenAI Codex (GPT-6), with reasoning and code execution. The exact
serving model ID and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 08:30:15 -05:00
DottaandPaperclip e93732c3d4 fix(server): deliver voice replies from commits and saved deadlines (#17578)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Phone calls receive approved answers from the existing chat
publication queue.
> - The chat maintenance timer checks for voice replies every second,
including when no reply is ready.
> - A reply can be saved before its publication commits. An early wake
alone can miss that final transition.
> - Existing session, reply, publication, and action rows already record
delivery and recovery work.
> - This pull request uses committed changes and saved deadlines to
drive phone reply delivery.
> - Empty reply queues become quiet while authorization, ordering, and
uncertain-send protection remain in place.

## Linked Issues or Issue Description

Stacked on #17576. Related delivery infrastructure: #15625. Related
voice connection work: #11418.

**What existing behavior does this improve?**

Background phone reply delivery and recovery of abandoned dispatch
intents.

**Current behavior**

Chat maintenance calls voice reply delivery every second. It also scans
old dispatch intents. Publication transport starts a direct push before
the final publication commit. The timer masks that timing gap.

**Proposed behavior**

Committed publication, reply, session, and configuration changes wake
the shared coordinator. Ready replies drain in bounded passes. Saved
rate-limit and dispatch-expiry deadlines retain recovery. An empty queue
or a permanent delivery barrier has no recurring reply scan.

**Reason and benefit**

Remove recurring SQL from idle phone reply delivery without dropping
replies, replaying uncertain sends, or adding a queue table.

**Breaking changes**

No API or schema change. Explicit manual recovery remains available.
Inbound admission cleanup, history retention, and reports remain on
their current paths for separate migrations.

## What Changed

- Extract ordered reply state and multipart policy within the voice
owner. Retain current authorization and atomic dispatch claims.
- Register a scoped reply worker with startup reconstruction, fair work
batches, failure backoff, drain/resume handling and shutdown ownership.
- Extend the existing shared dispatcher with normalized partial-page
results to preserve the bounded parent recovery contract. Existing
complete-array consumers retain their calls.
- Signal before relevant writes in their existing transactions. Both
normal publication completion and operator resolution wake the actual
conversation after commit.
- Bound reply inventory to 100 total headers per refresh. Preserve
partial scans, concurrent changes and newer denied-call restoration
obligations across reconstruction.
- Track abandoned dispatch actions independently of live calls.
Quarantine expired intents without resending them.
- Use existing active-call authorization checks to reactivate replies
that were previously denied. Healthy call inspections do not rescan
empty reply queues.
- Remove the periodic reply scan and the early direct publication push.
Keep provider requests outside database transactions.

## Verification

Published head: `1435cab10be3f95c38ab6672d2a890ae49e9d300`. Voice
recovery #17576 has landed at master
`e1ba77f06771a27114b3999004b0ca5e3c172dd4`. A normal merge reconciles
squash ancestry while preserving the complete reviewed 31-file reply
patch byte for byte and all unrelated master changes. The PR now targets
master; fresh exact-head CI and Greptile are required. The prior
449-case/typecheck/build results below cover the identical owned source,
not a new full-suite run.

- **449 distinct targeted cases pass**: 111 complete voice-store/reply;
19 complete parent recovery; and 319 complete tool-gateway,
secret-proposal, shared dispatcher/coordinator/batching/startup/hostname
cases. Production server typecheck and build pass. The policy checker
and its Node tests pass. Later edits record evidence only.
- Two real normal-publication/operator-resolution tests seed 105
unrelated calls and 105 future dispatches. Five publications selected
**1,065/1,060 inventory rows before repair**; both paths now pass the
same **125-row total budget** and deliver every answer once. This counts
selected SQL results, not physical tuple visits or host savings.
- Publication commits now signal their actual conversation. The voice
reader returns at most 100 total session/action headers per refresh,
preserves phase/cursor state and retries, and resets on worker
registration. Current-state checks and dispatch claims still own
provider effects. The 105-action case measures the page cap, restarts
the same owner during reconstruction, waits at saved future deadlines
and quarantines every abandoned action without resending.
- A real 403 fixture proves that an older broad scan cannot erase a
denied call first observed behind its cursor. Its owner callback still
emits the committed restoration wake after scan completion. The existing
actual-service authority-restoration and final-publication timing tests
remain in the full suite. They are not replaced by direct helper calls.
- Guarded removal of conversation scoping, cursor reset or the denial
revision fence fails the intended budget, 100-versus-5 restart or
missing committed-wake assertion; source is restored. An initial
shadowed fixture name and an inadequate due-action reset probe are
retained as failed attempts in the [current
audit](https://github.com/paperclipai/paperclip/blob/d2c0a7de8c3e9cac8382b1268f3f5d883a2067b3/doc/plans/2026-10-25-voice-reply-delivery-sleep-audit.md#publication-inventory-and-paging-acceptance-october-28).
- Parent recovery still passes paging, restart, behind-cursor insertion,
feature changes, actual disablement, quiet health observations, current
authority, exact-scope failure fences and wildcard-failure isolation
through the shared worker. Four existing array-based worker consumers
retain their API and pass their full suites.
- The two voice fixtures self-migrate separate empty databases. Legacy
tool-review/secrets suites use separate migrated clones. Earlier
full-local attempts remain limited by embedded PostgreSQL setup and are
not passing full-matrix claims. Older counts apply only to their stated
revisions. Fresh exact-head Greptile, Ponytail if present, security and
all CI remain merge gates. No live-provider, multi-writer,
external-wake, deployed-sleep or savings claim is made.

## Risks

- Missing a supported producer can strand work. Tests cover real
enqueue, retrieval, normal publication, and operator publication paths.
Startup reconstructs existing records.
- Signals cover the process-local database owner. Independent SQL
writers or other replicas need their own wake contract.
- Authority restoration relies on the existing live-call recovery check.
This dependency must be preserved if that owner changes later.
- A provider acceptance receipt does not prove speech. Unknown outcomes
remain blocked from automatic resend.
- This change does not extend Cloud alarms, allow sleep during live
calls, or prove deployed suspension or cost savings.

## Model Used

OpenAI GPT-6 via Codex, with repository inspection, reasoning, code
execution and test tools. The runtime does not expose a more specific
deployment identifier or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used with the available version and
capability details
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
ticket identifier
- [x] I have run the selected tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 07:50:27 -05:00
DottaandPaperclip e1ba77f067 fix(server): schedule live voice recovery from session changes and deadlines (#17576)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Voice calls retain durable session state and provider identities.
> - The chat maintenance timer scans live voice sessions even when no
call exists.
> - Session writes, saved expiry and active-call inspection deadlines
identify real work.
> - This pull request moves live-session recovery to the shared
coordinator.
> - Idle session recovery becomes quiet without losing exact-call
cleanup or current authorization checks.

## Linked Issues or Issue Description

Refs #17572. This PR is stacked on receiver discovery. It owns only live
voice session recovery. Reply delivery, inbound admission cleanup,
inbound history retention and call reports remain separate migration
slices.

**What happened?**
The one-second chat maintenance lane reads voice settings and scans live
sessions even when an instance has no calls or only terminal history.

**What did you expect?**
Reconstruct live calls on startup. Wake on committed session or
configuration changes. Inspect active calls and retry cleanup at actual
deadlines. Stop querying when no live obligation remains.

**Steps to reproduce**
Start an instance with no live voice sessions. Let startup settle. The
old maintenance method continues to read settings and session inventory
every chat maintenance pass.

## What Changed

- Add a voice-owned bounded deadline reader and scoped worker on the
existing coordinator. Retain concurrent writes across pages and reset
partial scans on restart. No tables or migrations are added.
- Separate generic connection health observations from configuration
changes. Existing receiver and command consumers retain their
notifications. Voice eligibility and provider operations remain in the
voice owner.
- Signal existing session transactions and configuration/feature
changes. Keep provider calls outside database transactions. Standalone
session writes use one small transaction helper.
- Preserve saved expiry, the two-minute unknown-creation grace,
active-call authorization and missed-hangup inspection, and
exact-provider cleanup. Recovery never creates a provider call.
- Retry transient authority lookup failures instead of treating them as
confirmed revocation.
- Fence expiration against a provider identity bound after discovery. A
late known call remains eligible for cleanup.
- Preserve work through rollback, drain, restart, concurrent refresh,
backoff and shutdown. Page inventory and rotate bounded batches across
companies.
- Leave the four other voice maintenance duties in the periodic lane.
- Document producer coverage and the supported writer contract in
`doc/idle-voice-session-recovery.md` and the dated audit.

## Verification

Published head: `f0b2e3cc22a18e7d08b10d4fc02aae4a19a52f30`. Receiver
#17572 has landed in master at
`dce004259394486f6361d0c0dd5a33a5272acc0a`. This normal merge reconciles
squash ancestry; the complete 22-file voice recovery patch is
byte-for-byte identical to the reviewed and tested `ad79c7e805` patch,
while all unrelated master changes are preserved. The PR now targets
master. Fresh exact-head CI and Greptile are running; interrupted
Ponytail turns are not completed verdicts under the maintainer’s
clarified merge policy, and an actual failing verdict still blocks.

- **210 distinct targeted cases pass on the current repair**: 19
complete recovery lifecycle cases and 191 complete voice-store,
coordinator, batching, startup and hostname cases. Production server
typecheck and build pass. Later edits record evidence only.
- CI `40633667141` on prior head `1b6323c164` failed the feature-change
assertion with `34050 < 25000`; its verification aggregate also failed.
The test now disables automatically advancing fake time, drives 20
controlled scheduler turns and requires all 205 calls to close before an
unrelated inspection deadline can mask a lost wake. Removing generation
replay fails with missing closures. The repair does not benchmark
database/host speed or raise the elapsed-time bound.
- A real committed exact-call regression reproduces wildcard-backoff
starvation. The worker keeps the broad retry but processes independent
exact hints and cached due calls. Each check reloads current state and
authority. Failed exact scopes remain fenced. Removing wildcard
isolation fails the intended assertion; a successful broad retry then
leaves the worker SQL-quiet. Full-row and lean queries now share the
state/scope/cursor predicate.
- Earlier acceptance on `712f3862b5` passed 725 distinct targeted cases.
Its measurements remain useful history: five real metadata writes and
one health check over 205 future calls selected 1,230 rows before the
inventory repair and zero afterward, without changing the saved
deadline. Those older test counts are not added to the current 210-case
count.
- Both external voice fixtures self-migrate; the current full
voice-store run uses a separate initially empty database. Earlier
fixture/method/observer failures, source mutations and partial edit
attempts remain in the [current
audit](https://github.com/paperclipai/paperclip/blob/ad79c7e805367fc7e2df3ea53c942ca6c7d27365/doc/plans/2026-10-25-voice-maintenance-sleep-audit.md#wildcard-failure-and-deterministic-generation-acceptance-october-28).
Source is restored after guarded mutations.
- Older CI `40632057899` failed because its trusted policy script was
absent from the checkout, while build separately received a runner
shutdown signal. E2E shards succeeded but their aggregate failed on
policy. The current checkout includes the landed policy tooling and its
checker/tests pass locally. The shutdown cause is unproven; neither old
failed run is counted as passing. Earlier full-local attempts stopped on
embedded PostgreSQL setup failures and did not pass.
- Fresh exact-head Greptile, security and all CI must pass before merge.
A completed failing Ponytail verdict blocks; an interrupted turn without
a verdict does not. No full current local repository matrix, live
provider acceptance, multi-writer support, Cloud wake, host suspension
or savings is claimed.

## Risks

- Process-local hints cover the existing active writer and shared
clients. Independent SQL writers or another replica need explicit wake
integration.
- Membership, task binding, phone preferences and other authority inputs
remain checked on the existing 30-second active-call inspection deadline
and on use. This is not an instant cleanup guarantee for every authority
mutation.
- Unknown creation retains its existing maximum-duration grace. Cleanup
failures retain retries. A provider request is never replayed merely
because its creation response is unknown.
- Genuine configuration changes retain bounded reconstruction. Metadata
writes can probe parent eligibility, while health-only observations
leave voice recovery quiet. Paging state is process-local and
reconstructs from durable rows after restart.
- Active calls, the other voice maintenance owners and connector
transports remain real obligations. This does not expand Cloud alarm
coverage, relax sleep eligibility or prove deployed suspension.
- The receiver and Discord parents have landed. Current-master
compatibility and exact-head CI and Greptile remain merge gates.

## Model Used

OpenAI GPT-6 via Codex, with repository inspection, reasoning, code
execution and test tools. The runtime does not expose a more specific
deployment identifier or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket identifier
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 07:38:48 -05:00