Commit Graph
51 Commits
Author SHA1 Message Date
DottaandClaude c14f2fe45d fix(native): read the scheduled-resume evidence the resume reconciler claimed (#20656)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #20645 lets a local same-run scheduled resume send the one
same-conversation continuation for a turn lost on restore. It applies
only when durable records prove three facts: the previous runner
stopped, the attempt was admitted within the budget, and it is the same
checkpointed conversation.
> - It read the third proof from the coordinator's saved
`failure_detail` while the phase was `retryable_failure`.
> - In production, the resume reconciler
(`claimNativeSessionResumptions`) claims the run first. It moves the
coordinator to `observed` and clears `failure_detail`. So the executor
never saw that evidence, and #20645 refused every real scheduled resume.
> - The restart-chaos L7 wave showed this live. Codex B5 TM2 hit the
exact path (`runner.drain` timeout, scheduled resume,
`provider_turn_lost_on_restore`), sent no continuation, and needed
Retry.
> - This pull request makes the reconciler record that evidence durably
when it claims the run, and makes the executor read it from there.
> - The benefit is that #20645 works on the real path. The proof rule is
unchanged.

## Linked Issues or Issue Description

Refs #20645

**What happened?**
Wave `w-20261110T212307-s9107` (L7, seed 9107, local, master plus
#20645), task codex B5 TM2:
- `native_session_recovery_failed: runner_local_connect_failed: PRP
command runner.drain timed out`;
- `retryable_failure` with `recoveryMode: "exact_checkpoint_resume"`,
`recoverable`, `providerSessionEstablished` and `checkpointExists` all
set;
- `native.local_process_stopped`;
- attempt 2 started;
- `turn.failed` with `provider_turn_lost_on_restore` for the same turn
ID that the checkpoint was running.

No continuation was sent. The run failed with
`native_provider_terminal_failed` and was preserved with
`uncertain_provider_action`.

**Expected behavior**
The proven scheduled resume sends the single continuation.

**Steps to reproduce**
The new integration test below. The live path is timing-dependent (the
L7 command in #20645).

## What Changed

- `native-finalization-reconciler.ts`: when it claims a same-run resume,
the reconciler appends one `same_run_resume` entry to
`recovery_history`. The append is bounded like other history writes, and
it runs in the same update that clears the failure. The entry copies
`recoveryMode`, `recoverable`, `providerSessionEstablished`,
`checkpointExists`, `successorRunId` and `replacementDenied` from the
saved failure, with the controller boot and provider attempt.
- `scheduled-resume-continuation.ts`: the evidence for the third proof
now comes from either of two durable records:
- the saved failure, when the executor claims a `retryable_failure`
coordinator directly (unchanged); or
- a reconciler claim: an `observed` coordinator whose last history entry
is a `same_run_resume` entry naming the executor's own controller boot.
The two other proofs are unchanged: the stop receipt, and an admitted
attempt from 1 to 3.
- `native-session-executor.ts`: passes the coordinator's last history
entry and the controller boot to the decision.
- `doc/execution-semantics.md`: describes the recorded evidence.

Readers of `recovery_history` that look at the last entry do so only for
restart claims, Stop recovery and shutdown suspension. Restart
recovery's previous-claim lookup skips non-claim dispositions, so a
`same_run_resume` entry does not change those paths.

**Still refused** (tested):
- every #20645 case;
- the last history entry is not the resume claim, or there is no
history;
- the resume claim was recorded by another boot;
- the claimed evidence is a bootstrap retry, has no checkpoint, or
records a replacement;
- a reconciler-claimed resume without a stop receipt.

## Verification

- `native-scheduled-resume-continuation.integration.test.ts` now also
runs through the real reconciler claim (`claimNativeSessionResumptions`)
before `executePaperclipNativeSession`. On master the reconciler-claimed
allowed case fails (`resumeInterruptedTurn` is false). With this change
it is true, and the claim admits attempt 2. A reconciler-claimed resume
without a stop receipt still passes false.
- `scheduled-resume-continuation.test.ts`: decision cases for the
reconciler-claim entry.
- `native-session-resumption.test.ts` passes. 54 tests pass across the
three files.
- `cd server && npx tsc --noEmit` passes.
- Live: not yet. The failing wave above is the reproduction. A rerun is
timing-dependent.

## Risks

- One more `recovery_history` entry per scheduled resume, inside the
existing 20-entry bound.
- The proof rule itself is unchanged. This only reads the same evidence
from the record where it now survives.
- No schema change. A run claimed by an earlier build has no
`same_run_resume` entry, so it keeps the existing failure: refused, with
Retry offered as today.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs). Extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 18:21:25 -06:00
DottaandClaude Opus 5.5 1ca3e8ea2a fix(native): let a resume claimed after a restart write the run's existing log (#20427)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a native run is interrupted, Paperclip can persist a same-run
resume: a later attempt continues the same run from its exact checkpoint
> - If the server restarts before that resume is dispatched, the new
server process claims it. The resume then reopens the run's existing
log, and only the current controller may write it
> - The resume claim took the lease owner but left the dead process's
controller identity on the lease. The log writer check compares the boot
id, so it refused, and the exact checkpoint resume failed as
`run_log_writer_authority_lost`
> - This pull request stamps the claiming process's identity on the
resume claim, as restart recovery's own claim already does
> - The benefit is that a resume persisted before a restart continues
after it, instead of failing and needing a manual Retry

## Linked Issues or Issue Description

Refs #19831. Storage chaos wave `w-20261110T003423-s9903` (local ST2,
seed 9903), task t01 (codex C1, K1). The wave went 7/8; t01 was the one
task that needed a Retry.

- Run 279772b3 was SIGKILLed during its first turn. The restarted server
resumed it, and the Codex snapshot failed ("cannot safely recover an
ambiguous Codex turn start"). A same-run resume was persisted.
- A second SIGKILL hit (00:35:37), and the next process dispatched the
scheduled resume at 00:35:56.
- The resume's run-log claim failed in `createRunLogWriterAuthority`:
the coordinator lease still carried the killed process's
`controllerBootId`. The exact checkpoint resume was refused
(`run_log_writer_authority_lost`), and the run ended `adapter_failed`,
needing a Retry.

The wave had S3 latency injected, but the failure does not depend on it.
The new test reproduces it on master with no storage faults.

## What Changed

-
`server/src/services/native-runtime/native-finalization-reconciler.ts`:
`dispatchNativeSessionResumptions` writes the claiming process's
controller identity (boot id, pid, host hash, process start) along with
its lease owner and expiry. This matches the restart-recovery claim, so
ownership proofs and the run-log writer check see the process that
actually owns the resume.
- `server/src/__tests__/native-session-resumption.test.ts`: a regression
test for this case.

## Verification

- `native-session-resumption.test.ts`:
- A coordinator persisted for resume by a previous process (stale boot
id), on a run with an existing object-store log, is dispatched.
- The run-log writer check accepts the new claim, and the lease carries
this process's identity.
  - The test fails on master with `run_log_writer_authority_lost`.
- Regression: the native resumption, binding, recovery-deadline,
restart-recovery, scheduled-resume, owner-release, runner-ownership and
recovery suites pass locally. `tsc --noEmit` passes for `server`.

## Risks

- The resume claim now also refreshes the controller pid, host hash and
process start on the coordinator row. These describe the process that
owns the lease, exactly as after a restart-recovery claim. A claim by
the same process writes the same values as before.
- No schema migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 19:57:24 -06:00
DottaandClaude Opus 5.5 a9cf4ee731 fix(outbox): deliver a user comment whose wake was lost on a task its agent then closed (#20066)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A user's comment on a task with a live run must reach the agent: it
is steered into the run or delivered in a follow-up run. Per-comment
exactly-once delivery is plan invariant 4
> - A comment commits before its wake is written. A control-plane
restart in between loses the wake, and the reconciler's lost-wake rule
(chaos s9401/t69) saves the missing outbox entry
> - That rule only looked at `todo` and `in_progress` tasks. In the
restart chaos Daytona wave s9501, task t06 got a comment 10 ms before a
K1 kill. The live run then finished, its agent closed the task, and the
comment was never read
> - This pull request extends the rule to a task its agent closed, and
reopens that task for the user's message, as the release drain already
does
> - The benefit is that a message sent while the agent works is never
silently dropped by a restart

## Linked Issues or Issue Description

Refs #19831 (plan, section E: work-intent outbox, invariant 4), and
#19992 (the s9401/t69 and t72 rules this extends). Found by the restart
chaos campaign, wave w-20261108T142906-s9501, task t06:

- The comment was saved at 14:40:17.048.
- The K1 kill came at .058.
- No wake request ever carried the comment, and no run event shows it
steered in.
- The run that started before the comment finished at 14:44:22 and
closed the task.

## What Changed

- `recovery/ownership-invariant-report.ts`
(`checkUserCommentsWithoutWake`): also reports a comment on a `done`
task its agent closed (`taskClosedByAgentSql`), when no wake carries it,
no run started after it, and no run of the task logged it as a steered
`userMessage` item.
- `work-intent-outbox.ts` (`saveUndeliveredUserComments`): a new
`reopenAgentClosedDone` option. In the same transaction that locks the
issue, re-checks the agent close and saves the receipt, it reopens the
task to `todo` through the issues service. It also logs `issue.updated`
(reopened from done, source `undelivered_comment_backstop`).
- `native-finalization-reconciler.ts` (`applyOwnershipReportBackstops`):
the lost-wake backstop allows `done` and passes `reopenAgentClosedDone`.
The receipt keeps its `wake_lost` / `reconciler_backstop` wait, so the
existing release path delivers it once.

## Verification

- `heartbeat-held-comment-wake.test.ts`, new test "delivers once a
comment whose wake a restart lost mid-run on a task the run then
closed". It fails on `master` and passes here. The violation is found,
the backstop saves and reopens, the comment is delivered exactly once,
and a second pass and backstop deliver nothing more. All 25 tests in the
file pass.
- `ownership-reconciliation-report.test.ts`, new test "finds a user
comment that a restart left without a wake on a task its agent then
closed". It fails on `master` and passes here: only the lost comment is
reported, not one steered into the run and not one on a task a person
closed. All 26 pass.
- The server package typecheck passes.

## Risks

- **Reopening a done task.** This only happens when the latest recorded
close was not made by a person, the comment is a direct human comment,
and no run received it or had it steered in. That's the same rule the
t72 stale-continuation path and the release drain already apply.
- **The steer check** joins the task's runs to their `item.started`
events by comment ID. It runs only for candidate comments within the
lookback window.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 10:08:00 -06:00
DottaandClaude Opus 5.5 8475826d2d fix(wake-queue): deliver user comments whose wake a restart lost or a stale continuation dropped (#19992)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A user's follow-up comment on a task must reach the agent exactly
once, through restarts (invariant 4)
> - The final unmasked local chaos storm (96 tasks, 31 restarts) passed
94 tasks. The 2 failures were both lost user comments
> - In t69, a restart killed the server between saving the comment and
writing its wake. In t72, a comment's own run was cancelled as stale,
because the earlier run completed the task first without reading the
comment
> - This pull request adds a reconciler rule that gives a lost comment
its wake, and saves the comment before a stale cancellation
> - The benefit is that a user's message is never silently dropped when
a restart or a completion race hits at the wrong moment

## Linked Issues or Issue Description

Refs #19956 (work-intent outbox), #19944, #19831 (plan, invariant 4).

**What happened?**
In chaos wave `w-20261108T055444-s9401` (local, unmasked, codex and
claude, 30-min K13 storm, master `46bf913fb9`), 94 of 96 tasks passed.
Two failed:
- **t69, R1 `rested_not_done`.** The user's comment was saved at
06:14:59.358 and reopened the task to `todo`. A K1 SIGKILL fired 28 ms
later, before the wake was written. No wake row exists. Recovery skips a
`todo` task whose last run succeeded, so the task sat with no run until
the wave ended.
- **t72, R4 `concurrent_runs`.** A comment posted during a run queued
its own run. That run was claimed at 06:16:04.874. The first run's
completion marked the task `done` 2 ms later. The comment's run was then
cancelled with `continuation_task_ownership_changed` before dispatch,
and the comment was lost.

**Expected behavior**
Each accepted user comment is delivered once, unless the task is
cancelled or reassigned, or a person closes it.

**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts
--replay w-20261108T055444-s9401/t69` (or `.../t72`).

## What Changed

- `recovery/ownership-invariant-report.ts`: a new reconciler-owned I4
rule, `user_comment_without_wake`. It finds a user comment that meets
all of these conditions:
  - it is newer than every run of the task;
- the task is assigned, is not a conversation, has no live run or lock,
and has at least one earlier run;
  - no wake request carries the comment.

A comment younger than 30 s is not reported yet, because it may still
get its normal wake. Its end-of-grace time is returned as `nextDueAt`.
- `native-runtime/native-finalization-reconciler.ts`: the backstop step
is now exported as `applyOwnershipReportBackstops`. For each such
comment it saves the missing outbox receipt (`executionWait: wake_lost`)
and promotes it.
- `work-intent-outbox.ts`: new `saveUndeliveredUserComments`, which the
reconciler rule and the stale-run path both use. It saves only live user
comments that no completed run received. It writes nothing for a task
that is reassigned, is a conversation, or has a status the caller does
not allow.
- `heartbeat/queue.ts`: the saved-receipt pass that already sends
pause-saved messages also sends `wake_lost` receipts. It uses the same
per-comment consumption check and the same chat-input exclusion.
- `heartbeat/stale-comment-continuation.ts` and `heartbeat.ts`: before a
stale continuation is cancelled, its undelivered user comments are saved
as a deferred receipt. The release drain then reopens the `done` task
and delivers them once. This applies only when an agent run that started
before the comment completed the task after it. A task that a person
marks done is left closed.
- `tests/restart-chaos/grader.ts`: R4 ignores runs cancelled before
dispatch (`continuation_task_ownership_changed`) when it looks for
overlapping execution. Those runs never executed. Comment consumption
still counts them.

## Verification

Each focused file ran alone in one process (`--maxWorkers=1`):
- `heartbeat-held-comment-wake.test.ts` (22 pass). This includes two new
cases:
- t69 shape: the reconciler rule fires, the backstop delivers the
comment once, and a second pass does nothing;
- t72 shape: the stale run is cancelled, the task reopens, and the
comment is delivered once.

  Both fail with the rule or the save disabled, and pass with them.
- `ownership-reconciliation-report.test.ts` (23 pass). It includes the
rule's boundaries: a carried comment, a too-young comment (deadline
only) and an already-read comment.
- `tests/restart-chaos/grader.test.ts` (17 pass). The new R4 case fails
with the old grader. A cancelled run that is not a pre-dispatch
cancellation still fails R4.
- Unchanged and passing: comment wake batching (27), refused assignment
wake (3), wake-queue postgres adapter (103), stale queue invalidation
(68), deferred receipt work (45). The first version also reopened a task
that a person had marked done, which broke one deferred-receipt test.
That case is now excluded.
- The server package typecheck and the restart-chaos harness typecheck
pass.
- Same-session chaos context on master: K6 crash loop
`w-20261108T063326-s9402`: 6 of 6 pass. The K4 long-hold wave is still
running. After merge, I'll replay only the targeted fault shapes (t69
and t72).

## Risks

- The reconciler rule runs on each report pass. It is bounded by the
1-hour lookback and the per-kind row cap, and it excludes conversation
tasks, so chat input keeps its own authorization.
- A comment younger than 30 s waits up to the end of its grace period
before the backstop acts.
- The stale-run save reopens a `done` task only for a comment that the
completing run could not have read.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 02:14:45 -06:00
DottaandClaude Opus 5.5 d33b3ac263 fix(wake-queue): save temporarily refused work intents and promote them through one path (#19956)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Users send work to agents as assignments, comments and Retries. The
wake queue delivers each one as a run
> - When an admission check refused a wake for a temporary reason, the
wake often ended as a terminal skip, so the work was lost
> - Restart-chaos waves found dropped comments and Retries (RC-37,
s7532) and double delivery (RC-53). Each fix covered one hold type
> - The durable ownership plan (section E) asks for one outbox rule: a
closed set of terminal outcomes, a saved deferral for every temporary
refusal, and one promotion path
> - This pull request adds that rule. It saves messages refused by a
pause, promotes deferred intents from every hold-release event through
one function, and adds a reconciler backstop
> - The benefit is that an accepted message is delivered exactly once
after a pause lifts, and the reconciler reports any intent that is stuck
or dropped

## Linked Issues or Issue Description

Refs #19831 (plan, section E), #19838 (inventory, wake-queue rows and
section 1b), #19927 (Phase 2 data: the I4 false positive), #19702,
#19741, #19829, #19826. Builds on #19944 (section D, merged).

**What happened?**
- A user commented while the agent was paused, or while the company was
paused. Admission refused the wake. Nothing saved the message as
pending, so when the pause lifted, nothing delivered it.
- Hold-release events woke different subsets of workers. A saved intent
then waited out a 30-second refusal backoff after its hold cleared.
- The I4 reconciler check matched free text with a regular expression.
Phase 2 found a false positive: a delivered retry whose error said
"retrying once".

**Expected behavior**
An accepted work intent ends only as delivered, consumed by a later run,
task cancelled, or hard stop. Every temporary refusal saves the intent
as `deferred` with its reason and its release trigger. Every
hold-release event promotes deferred intents through one function
(invariant 4).

**Steps to reproduce**
Pause an agent. Post a user comment on its task. Resume the agent. On
`master` the comment is never delivered as a wake.

## What Changed

- New `server/src/services/work-intent-outbox.ts`:
- `WORK_INTENT_TERMINAL_OUTCOMES`: the closed set of terminal outcomes.
- `TEMPORARY_REFUSAL_TRIGGERS`: maps each temporary refusal code to its
release trigger.
- `promoteDeferredWorkIntents`: the one promotion function. It removes
the 30-second refusal backoff from the matching deferred intents. It
then wakes run dispatch, the saved-receipt worker and stranded-work
recovery, and hints each saved intent of the issue.
- `heartbeat/queue.ts`: the #19829 save-as-deferred block is now a
shared helper. A user's message refused by a paused agent, a paused
company, or a budget pause that is not a reached hard stop is saved with
`executionWait: { reason, trigger, message }`. The caller still gets a
409. The message is saved only when the task's latest run is terminal,
because the release drain promotes a saved message behind that run. A
task that never ran keeps the message in the task, and stranded-work
recovery starts the task once the pause lifts. A `pending_approval`
agent and a task-tree hold are not changed here: the release drain still
fails a promoted wake for a pending approval, and tree holds have their
own resume path.
- Hold-release events call `promoteDeferredWorkIntents`:
  - accounting settled (`run-cost-accounting.ts`);
  - environment lease released (`environments.ts`, two sites);
  - native run projected terminal (`native-run-projection.ts`);
  - budget pause lifted (`budgets.ts`);
  - company reactivated (`companies.ts`);
  - task-tree hold released (`issue-tree-control.ts`);
  - agent lifecycle ready (`agent-lifecycle-driver.ts`).
- Reconciler (`recovery/ownership-registry.ts`,
`ownership-invariant-report.ts`, `native-finalization-reconciler.ts`):
- registers three new kinds: `wake_request_deferred`, `queued_run` and
`scheduled_retry_run`;
- I4 now reads the closed refusal code (the wake `reason`) of accepted
intents, not free text;
- a new I4 check reports a deferred pause refusal whose pause has lifted
for more than 5 minutes. The worker then promotes it as a backstop;
- a lifted pause inside its 5-minute grace window returns its deadline
through `nextDueAt`, and the worker also wakes on agent-lifecycle
signals (company and agent pause changes). So a missed release event is
still checked.
- `doc/SPEC-implementation.md`: adds the rule "Accepted work intents are
delivered exactly once".

## Verification

Each focused file ran alone in one vitest process (`--maxWorkers=1`) on
embedded Postgres. Results at `2875c85fd2`:

| Question | Loop | Elapsed | Result |
|---|---|---|---|
| Is a comment refused by a pause saved, then delivered once after the
pause lifts? | `heartbeat-held-comment-wake.test.ts -t "pause lifts"` |
8 s | 2 pass. With `queue.ts` from `master`, both fail: no intent is
saved |
| Does I4 use the closed code, and does it catch a stuck deferral? |
`ownership-reconciliation-report.test.ts` | 45 s | 20 pass. This
includes the Phase 2 "retrying once" false positive and the case
"paused, then lifted" |
| No regression on the paths that changed | held-comment (15),
archived-company guard (8), undispatched execution wait (32), deferred
receipt work (45), recovery pause durability (16), native run projection
work (55), tree-control routes and services (19, 1, 7) | 8–334 s each |
All pass |

- Exactly-once: each pause test runs two promotions (a release event,
then the reconciler backstop) and many dispatch passes. It asserts one
delivery. The existing consumption tests (RC-53) still pass.
- Server package typecheck passes.
- Speed: nothing new runs before `turn.start`. The new admission work
happens only on a refusal. Promotion runs in the release transaction and
in the background reconciler.
- Not run here: chaos waves. The coordinating session schedules them on
the shared harness.

## Risks

- `promoteDeferredWorkIntents` moves `updated_at` back for deferred
intents that were refused in the last 30 seconds. A busy company with
many release events can retry refused intents more often. Each retry
re-runs normal admission, and the refusal resets the backoff.
- More release sites now wake run dispatch and stranded-work recovery.
These notifications are coalesced per company.
- Lifting a pause is now the operator's explicit authorization to send
what the pause refused, as Send is after a Stop. When the pause stopped
the task's run, the lifted pause issues the task's next explicit wake,
which adopts the saved message once. Other messages queued behind an
acknowledged Stop keep the existing rule. If Stop recovery still holds
the task, the message waits for that recovery.
- An agent pause's stop sweep keeps the messages that the pause deferred
(`preservePauseDeferred`). Terminate and cleanup stops still end them.
- The backstop check reads only pause refusals (`company.inactive`,
`agent.not_invokable`), because those holds can be checked exactly.
Other deferred kinds keep their existing promotion paths.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 23:43:17 -06:00
DottaandClaude Opus 5.5 c6e4ed1547 feat(recovery): settle a claimed wake whose run finished without settling it (#19934)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A wake request is claimed by the run that handles it, and that run
marks it finished.
> - In a K4 chaos wave (#19927), native restart recovery blocked a run
and ended it, but left its assignment wake `claimed`. Nothing ever
released it.
> - The durable-ownership plan settles such records on one shared path,
by proof, never by age.
> - This pull request registers a settle for `wake_request_claimed`. It
is Phase 2's second kind and is stacked on the accepted-result lease PR.
> - The benefit is that no run-ending path can strand a claimed wake.

## Linked Issues or Issue Description

Refs #19831 (plan), #19861 (registry), #19927 (the data that found
this). Stacked on #19933 (it adds the `settle` hook).

**What happened?**
Run `7c45860b…` failed with `native_restart_recovery_blocked`
(`execution_recovery_budget_exhausted`) at 23:19:41. Its wake
`6b3c4cce…` stayed `claimed` until the end of the wave (23:44 and
later). Rule I2 flagged it.

**What did you expect?**
A claimed wake ends when its run ends, with the run's outcome.

**Steps to reproduce**
1. `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts
--env local --faults K4 --k4-hold-max-seconds 600 --tasks 6
--concurrency 3 --seed 9203 --keep`
2. Look for `agent_wakeup_requests` rows with status `claimed` whose run
is terminal.

## What Changed

- New `recovery/wake-request-claimed-settlement.ts`: when the owner run
is terminal, and for a native run its coordinator is `committed` or
`terminal_failure`, the claim gets the run's outcome. That is
`completed` for a succeeded run and otherwise the run's status, as run
completion and native terminal projection already do. It is
compare-and-swap on the claim (id, run, status `claimed`).
- `ownership-registry.ts`: `wake_request_claimed` registers that settle.
- `native-finalization-reconciler.ts`: the pass log line includes the
settled count per kind.

## Verification

- `cd server && npx vitest run
src/__tests__/ownership-reconciliation-report.test.ts`: 17 pass,
including the new case. It covers a blocked native run, a native run
whose projection is still pending (kept), a succeeded legacy run, a live
run (kept), and a read-only pass (no change).
- Red/green (test written after the code): the new test fails on master
and passes here.
- Server typecheck passes.

## Risks

- No persisted shape changes, so there is no previous-release story
beyond this: claims saved by an older release are settled the same way.
- Race with the run's own settle: it waits for a native coordinator to
be final. For a legacy run, both write the same status, so the race is
idempotent.
- No change adds work before `turn.start`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 21:18:22 -06:00
DottaandClaude Opus 5.5 f5400e2594 feat(recovery): settle the accepted-result lease on the shared ownership path (#19933)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A control-plane restart can kill the controller after an agent's
result is accepted. The result then stays leased to a dead process.
> - Today a private sweep (`native-dead-controller-result-lease.ts`,
#19713) proves that controller dead with its own liveness rules.
> - The durable-ownership plan (#19831) replaces private proofs with one
prover (`proveOwnerGone`, #19853) and one registry (#19861).
> - This pull request moves the accepted-result executor lease to that
shared path and deletes the private sweep. It is Phase 2's first kind.
> - The benefit is one tested rule for when a dead owner's result lease
is released, and the reconciler now repairs a lease that restart
recovery missed.

## Linked Issues or Issue Description

Refs #19831 (plan), #19853 (prover), #19861 (registry), #19713 (old
sweep), #19883 (legacy host fallback), #19927 (Phase 2 data).

**What happened?**
The accepted-result lease had its own liveness proof, separate from the
shared prover. The reconciler could only report a dead owner, not settle
it. When restart recovery missed a lease, it waited out the 20-minute
lease.

**What did you expect?**
One prover decides, and both restart recovery and the periodic
reconciler pass settle a lease whose controller is proven gone.

**Steps to reproduce**
1. Accept a native run result.
2. Kill the controller (SIGKILL) and start a new server.
3. If the restart sweep does not see the row, the result stays leased
until the lease expires.

## What Changed

- `ownership-registry.ts`: a kind may register `settle` and
`onUnprovable`. `native_accepted_result_lease` registers both. Other
kinds stay report-only.
- New `native-accepted-result-lease-settlement.ts`: a compare-and-swap
settle that expires exactly the proven owner's lease. It keeps the old
guards: the run must still be `running`, remote copyback stays fenced,
and a lease taken over since the proof is left alone.
- `ownership-invariant-report.ts`: the pass calls `settle` on `gone` and
`onUnprovable` on `unprovable`. A read-only pass (`record: false`) never
settles. New `settleOwnableKind()` for restart recovery.
- `heartbeat/recovery.ts`: restart recovery calls
`settleOwnableKind("native_accepted_result_lease")` instead of the old
sweep.
- Deleted `native-dead-controller-result-lease.ts`.
- `owner-proof.ts`: accepts a board attestation (`operator_attested`).
It also keeps the one-release local fallback for rows saved before their
PID space was recorded, on single-host servers only, and flags each use
so it is logged (`legacy_host_unknown_dead_pid_proof`, #19883). An
ephemeral server returns `unprovable` (`process_space_unrecorded`) and
the existing "confirm the previous server stopped" board action is
raised.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-native-restart-suspension.test.ts`: 27 pass, 7
of them new.
- `npx vitest run src/services/owner-proof.test.ts
src/__tests__/ownership-reconciliation-report.test.ts`: pass.
- Red/green (tests written after the code): 5 new behaviour tests fail
on master and pass here. They cover the ephemeral board action, the
reconciler-pass settle, the live-owner counter-case in that pass, the
legacy marker, and operator attestation. The stamped-dead-controller and
foreign-PID-space cases pass on both, as the old path behaved the same.
- Server typecheck passes (`npx tsc --noEmit -p server/tsconfig.json`).
- Not run locally: the full server suite (`pnpm verify` asks for broad
coverage because a file was deleted). Required CI covers it.

## Risks

- Previous-release story (rules/upgrade-safe-run-state.mdc): NRF rows
saved before #19868 have no `controller_host_hash`. They keep the same
one-release local proof as before on single-host servers, and each use
is logged. On ephemeral servers they stay unprovable and ask the board,
as before. Rows on another PID space keep their lease (the fence), as
before.
- Fail-open risk: settle is compare-and-swap on the lease token, boot,
PID, host hash and start time of the proven owner. An `unprovable` owner
is never released. Live-owner and foreign-space cases are tested.
- No change adds work before `turn.start`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 20:58:04 -06:00
DottaandClaude Opus 5.5 988a4b6749 feat(recovery): ownable-kind registry and report-only ownership reconciler (#19861)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A control-plane restart leaves in-progress records owned by dead
processes, and about 31 per-subsystem proofs decide when to release them
> - The durable ownership plan (section C) asks for one registry of
ownable record kinds and one reconciler
> - Phase 1 must measure first: how many in-progress records exist, who
owns them, and which invariants they break, with no behavior change
> - This pull request registers 46 kinds from the Phase 0 inventory's
section C and adds a report-only pass to the existing native
finalization reconciler
> - The benefit is per-kind invariant-violation counts from chaos waves
and staging. That data unlocks Phase 2 (moving kinds onto one settle
path) and the removal of the company-wide admission gate

## Linked Issues or Issue Description

Refs #19831 (plan, section C and Phase 1). Refs #19846 (Phase 0
inventory, section C). Stacked on #19853 (the prover) and #19851 (one
process identity).

**Problem**
No single place lists the control plane's in-progress records or checks
the plan's invariants. Each subsystem releases its own records by its
own rules, so nobody can count how many records break "owned", "settled"
or "visible next action".

**Expected behavior**
One report-only pass lists every registered kind, classifies each owner
as gone, alive or unprovable, and records each invariant violation once.
It changes no record and stays quiet when nothing is in progress.

**Impact**
Phase 2 moves kinds onto the shared settle path in the order these
counts show. Phase 3 removes the admission gate once the I3 data exists.

## What Changed

- New `server/src/services/recovery/ownership-registry.ts`: 46
`OwnableKind` entries (`id`, `obligation`, `storage`, `listInProgress`,
`settle: null`, `invariantChecks`). Each record names its owner as a
process stamp, a fence token, a run, an operator, or none. The
skill-cache lock is registered with a named coverage gap because its
lock files live in per-skill roots.
- New `server/src/services/recovery/ownership-invariant-report.ts`: one
pass classifies every owner with `proveOwnerGone` (or the owning run's
status), then runs:
  - I1 owned: an in-progress record with no owner;
- I2 settled: an owner proven gone on two consecutive passes while the
record is still in progress;
- I3 admission independence: terminal runs whose accounting outlives the
10 min closure, which hold company admission;
- I4 exactly-once intents: wake requests ended `skipped`/`failed` on a
temporary refusal (heuristic until section E);
- I7 visible next action: an agent-assigned `in_progress` issue with no
live run, no pending wake and no open recovery action for 5 min.
- I5, I6 and I8 are reported as named coverage gaps; I8 comes in the
speed-budget PR.
- Violations are written once per process to the activity log as
`recovery.invariant_violation` (`actorId: ownership-reconciler`,
`details.mode: report_only`), insert-only with no live-event or plugin
fan-out. Nothing goes to Telemetry (AGENTS.md §5.7).
- `native-finalization-reconciler.ts`:
`registerOwnershipReconciliationWork` runs the pass in the shared
delivery-work scheduler (new queue `ownership-invariant-report`). There
is no second loop.
- `index.ts` registers it after residual recovery and does not await its
first pass, so readiness is not delayed.

### Sleep-aware trigger (rules/sleep-aware-background-work.mdc)

- **Trigger:** one pass at registration (startup reconstruction from
durable truth). After that, a settled `active-execution-recovery`
signal, which run lifecycle, controller renewal and process-evidence
writes already emit. Signals are coalesced to at most one pass per 30 s.
No producer gained a write.
- **Pending predicate and next deadline:** a 30 s follow-up only while
the last pass saw a changing in-progress set, or an owner proven gone
that needs the second I2 pass.
- **Quiet:** an empty set or the same stuck set twice sets no deadline.
The test asserts `nextWakeAt() === null` after an idle pass.
- **Turn path:** the pass never runs before `turn.start`. Stamps read
only cached identity.

## Verification

- `npx vitest run src/__tests__/ownership-reconciliation-report.test.ts`
(embedded Postgres, in `server/`): 4 tests pass:
  - every kind is registered once with no settle path;
- verdicts per class: alive (this process's claim), gone (lease of a
failed run), unprovable (no owner; another PID space);
- each rule fires on its fixture, and I2 fires only on the second pass;
- every touched table is byte-identical before and after (report-only);
- activity rows are written once per violation, and none are written
with `record: false`;
- the worker is quiet when idle and schedules a follow-up only while
records change.
- `npx vitest run src/__tests__/delivery-work-coordinator.test.ts
src/services/owner-proof.test.ts
src/__tests__/server-startup-feedback-export.test.ts`: pass.
- `npx tsc --noEmit -p server`: no errors.

## Risks

- No record changes, so no settle risk. The risk is load: one pass runs
about 46 small indexed or `limit 201` queries, at most once per 30 s
while runs are active, and none when idle.
- I4 matches refusal text with a heuristic. It can over- or under-count
until section E gives wake requests closed terminal reasons. It is
labelled as such in the code.
- Violations dedupe per process, so a restart writes a still-open
violation again (once per boot). Chaos reports count distinct `(rule,
entityId)`.
- Previous-release story (`rules/upgrade-safe-run-state.mdc`): no
persisted run state changes. The pass reads rows of any age, and a
missing field classifies as unprovable.
- `heartbeat_runs` and `native_run_finalizations` record no host or PID
space, so foreign-boot owners there classify as unprovable
(`missing_identity`). This is the inventory's finding 2, and the report
makes it countable.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:51:28 -06:00
DottaandClaude Opus 5.5 dc1adb7209 fix(tasks): advertise a native resume only while the scheduler would claim it (#19523)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a native run cannot reattach to its provider after a
control-plane restart, the native coordinator resumes the same run by
itself after about 30 seconds, up to three attempts
> - The run projection shows this correctly as "Retry scheduled" with
only `inspect_run` permitted
> - The task interruption shown on the issue read only system-recovery
evidence and `scheduledRetryAt`. It never read the coordinator's
`nextAttemptAt`, so it showed `state: failed, canRetry: true` and
offered a manual Retry while the automatic resume was still pending
> - The two surfaces disagree, and a user click can race the automatic
resume. That is one way a Retry gets parked behind "execution checks"
> - #19503 made the task interruption read the coordinator's pending
resume time. Review found that the coordinator also pushes that time
forward while it defers an ineligible candidate, so a reassigned or
blocked task could hide Retry for good
> - This pull request advertises the resume only while the scheduler's
synchronous admission checks pass
> - The benefit is one consistent story: while the automatic resume is
pending, the task shows it with its time and no competing Retry. If
recovery ends, exactly one Retry returns

## Linked Issues or Issue Description

Follow-up to #19503. Refs #19487, #19497.

**What happened?**
A native run failed with `native_session_interrupted` after a restart.
Its finalization row had `phase: retryable_failure` and `nextAttemptAt`
30 seconds later. The run projection said "Retry scheduled". The issue
interruption said `state: failed, canRetry: true, nextRetryAt: null`.

**What did you expect?**
While the automatic resume is pending, the task shows that it is
resuming, with the time, and no Retry. When the resume fails for good or
never runs, exactly one Retry that works on the first click.

**Steps to reproduce**
1. Run a native task and restart the control plane so the native attach
fails.
2. Read `GET /api/issues/:id` while the coordinator's resume is pending.
3. Observe `interruption.canRetry: true` while `execution.phase` is
`retry_scheduled`.

## What Changed

- `task-interruption-presentation.ts`: the pending-resume read now also
requires the task to be in a resumable status, assigned to the run's
agent, not owned by another execution or checkout run, the run's
question not cancelled, and the agent invokable (not paused, terminated
or pending approval). These mirror `readNativeSessionResumptionAgent`.
- `native-session-resumption-statuses.ts`: one shared status set used by
both the scheduler admission and the read model.
- No time cap: provider cooldowns and remote backoff schedules stay
visible as a pending resume.

## Verification

- `task-interruption-query.test.ts` (embedded Postgres): a reassigned
task and a blocked task show one Retry; a paused agent asks to be
resumed; in each case `readNativeSessionResumptionAgent` also refuses
the candidate, so the read model and the scheduler agree. A resume an
hour out (provider cooldown) stays `retry_scheduled` with no Retry. Red
on master for the reassigned case.
- All 57 presentation and query tests pass. Server typecheck passes.

## Risks

- Low risk. Read model only.
- A review-assignment resume (reviewer is not the assignee) shows Retry
instead of the pending resume. That is the safe fallback.
- Deferrals for a pause hold, dependency readiness or a competing run
are not mirrored. Those gates also hold a manual Retry, so no competing
action appears.
- The `POST /issues/:id/retry` endpoint is unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file editing, embedded Postgres tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 14:38:29 -06:00
Nicky LeachandClaude Opus 5.5 026dc6d135 fix(native): release a session-resume owner when the resumed run fails before its provider claim (#19371)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a finalization row. Its lease owner tells the
control plane that an execution may still be live.
> - Pausing an agent waits for `agentExecutionsHaveStopped`. That check
refuses to finish while any run of the agent has a lease owner, unless
the run was cancelled and has stop proof.
> - A same-run session resume claims that lease before it dispatches the
run. If the run then fails before the native executor takes its own
provider claim, nothing releases the resume owner.
> - The leaked owner blocks pause and resume for good, and the task
keeps looking like it has a live recovery path.
> - This pull request settles those rows at the source, and it adds a
stop-proof-gated release for rows that already leaked.
> - The benefit is that agents can pause and resume after an ordinary
resume failure, without database edits, and the cancelled-run stop-proof
rule stays as it is.

## Linked Issues or Issue Description

**What happened?**

An agent stayed in `pausing` with `hostComplete=false` and climbing
attempts. Its runs had `native_run_finalizations` rows with `phase:
observed`, `recovery_state: resuming_session`, a `lease_owner` like
`default:resume:<uuid>`, and no `result_id`. Each heartbeat run had
ended `failed`: either `setup_failed` (`Failed to acquire lease for
environment … Sandbox creation has failed`,
`native_remote_recovery_lease_mismatch`) or `adapter_failed`
(`runner_state_identity_mismatch: …`). The lease expired about 20
minutes after the run finished. Nothing settled the row. The only fix
was to set `lease_owner` to null by hand.

**Expected behavior**

When a resumed run fails before any provider turn starts, its
finalization row goes to `terminal_failure` and its lease owner is
released, as the native executor does for failures it sees. A row that
leaked before this fix is released in the background once its lease
expires and the run has stop proof. Pause then completes.

**Steps to reproduce**

1. Leave a native run in `retryable_failure` with a persisted
checkpoint, on a task that already has an active recovery incident.
2. Let session resume claim it (`claimNativeSessionResumptions` sets
`phase: observed` and a `…:resume:…` owner).
3. Make the resumed run fail before the executor claims it, for example
when the environment driver cannot create the sandbox.
4. The heartbeat failure path marks the run `failed`. The issue release
path does not settle the row, because it settles only when it opens a
new incident. The owner stays set, and `agentExecutionsHaveStopped`
returns false indefinitely.

**Paperclip version or commit**

`master` at 5ac05871b5.

**Additional context**

Root cause: two writers can end a resumed run without the executor.
`failRun` and the setup-failure path in `heartbeat/run-completion.ts`
never touch the coordinator. `recordNativeTerminalRecoveryIfNeeded` in
the wake-queue adapter settles the coordinator only when no recovery
incident exists yet. Agents that need their computer connection paired
again usually already have an incident, so the owner leaks.

Related, but different: #18905, #18866, #18072, and #17692 change how
the stop check judges runs. This pull request does not change the stop
check.

## What Changed

-
`server/src/services/native-runtime/native-failed-run-owner-release.ts`
(new): one settlement for a result-less `observed` owner on a run that
ended `failed`, `timed_out`, or `interrupted`. It writes
`terminal_failure`, null lease fields, `recoveryState: "blocked"`, the
run's failure code, and a `releasedOwner` record. It sets `nativePhase`,
sends the same finalization and execution work notifications as the
executor's terminal path, and logs `native.failed_run_owner_released`
activity. Locks follow the issue, coordinator, run order.
- `settleUnstartedNativeSessionResume`: runs after a dispatched session
resume returns. It matches only the exact claim owner and the attempt
the claim saw. The executor increments the attempt when it takes
provider ownership, so an unchanged attempt proves that no provider turn
started under this owner. The executor's own failures no longer match.
- `releaseAbandonedNativeOwners`: the background release for rows that
already leaked. It requires an expired lease and all of the stop proof
that `agentExecutionsHaveStopped` checks: settled environment leases, no
live run process or process group, no live controller lease, and no live
controller process. This process answers from its in-memory execution
registries. It skips cancelled runs, pending native Stops, and pinned
workspace restores, because those have their own stop-proof owners.
- `native-finalization-reconciler.ts`: `NativeSessionResumeClaim` now
carries `attempt`.
- `heartbeat.ts` and `heartbeat/recovery.ts`: both session-resume
dispatch sites settle the claim after `executeRun` returns. This is one
`.finally` per site.
- `execution-control-reconciliation.ts` and `execution-control-work.ts`:
the execution-finalization queue runs the background release. Its
deadline includes the earliest lease expiry among eligible rows, so the
release does not poll an idle instance. Rows with unreleased sandboxes
are excluded from the deadline.
- `agent-execution-stop.ts` is unchanged. The cancelled-only release
keeps its stop-proof rule.

## Verification

- `npx vitest run
src/__tests__/heartbeat-native-resume-lease-failure.test.ts` drives the
real heartbeat resume dispatch (`reapOrphanedRuns`). A sandbox driver's
`acquireRunLease` throws `Sandbox creation has failed`, and the task has
a recovery incident. The run fails with `Failed to acquire lease for
environment "Managed computer" (sandbox): Sandbox creation has failed`,
and the coordinator ends `terminal_failure` with no lease owner. I
removed the `.finally` wiring and ran the test again. It then failed,
and the row stayed `observed` with a `default:resume:` owner and a
20-minute expiry, which is the production state.
- `npx vitest run
src/services/native-runtime/native-failed-run-owner-release.test.ts` (10
tests):
- A claimed resume that fails at lease acquisition is settled.
`agentExecutionsHaveStopped` changes from false to true. A repeated
settlement does nothing.
- An owner the executor claimed (incremented attempt) and a run that is
still running are not changed.
- A leaked `runner_state_identity_mismatch` row is released by
`reconcileAbandonedExecutionControl`, and then
`agentExecutionsHaveStopped` returns true.
- An unexpired owner still blocks the stop check. Its expiry becomes the
finalization queue deadline.
- An active environment lease and a live controller process block
release. After both stop, the row is released.
  - Cancelled runs and pending native Stops are not changed.
  - Added during review:
    - Releasing an owner wakes the cost-accounting worker.
- Releasing a run's environment lease wakes the finalization worker, and
the owner is then released.
- A recorded PID and group that now belong to an unrelated process do
not hold the owner.
- Bounded passes rotate past a held row (`[0, 1, 1]` with page size 1),
then wrap back to it.
- The immediate settle keeps the owner while the run's recorded process
is alive.
- Suites that cover the changed files pass together: the two new files,
`native-session-resumption.test.ts` (with the claim shape updated),
`execution-control-reconciliation.test.ts`,
`execution-control-work.test.ts`, and
`agent-lifecycle/__tests__/lifecycle.test.ts`. With
`environment-cleanup-work.test.ts` added, that is 7 files and 156 tests.
I did not run the full `heartbeat-process-recovery.test.ts` suite
locally. CI runs it.
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- Behaviour change: a resumed run that fails during setup now ends with
a terminal coordinator in every case. Before, this happened only when
the issue release path opened a new incident. After a restart, restart
recovery no longer sees such a row as `observed`. This matches what the
release path already does when no incident exists.
- The background release trusts an expired lease on a run that is
already terminal, together with the same process and lease evidence that
the stop check uses. A controller on another host cannot be probed by
PID. The stop check has the same limit today. A live owner still renews
its lease, and lease-gated writes from an expired owner are already
refused.
- If an expired row's process or controller still looks live, the
release retries at the shared 15-second finalization cadence until that
process exits. Each pass handles one bounded page, and a rotating cursor
moves to the next page on each pass.
- A run-scoped environment cleanup notification now also wakes the
execution-finalization queue. Passes are coalesced, and a pass is a
cheap deadline query when nothing is eligible.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use
(shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 07:19:09 -08:00
DottaandPaperclip d8c1b9d4b5 fix(runtime): reclaim failed remote checkpoint retries (#18877)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native recovery must preserve the original execution and provider
state.
> - A failed remote run can retain its provider PID after heartbeat
releases the task execution lock.
> - Normal resumption excludes that PID, while restart recovery rejects
the released lock.
> - Recovery must reclaim that exact execution only after current
admission and durable ownership checks pass.
> - Failed retries and admission holds need separate finite budgets.
Neither can prove a provider stopped.

## Linked Issues or Issue Description

**What happened?**

A failed remote native run remains in retryable failure without a task
execution owner. Normal resumption and restart recovery both reject it
before inspecting the preserved remote provider. The current remote
workspace reference format also misses the due-retry routing path.

**Expected behavior**

A due exact-checkpoint retry reclaims its original task execution only
when current ownership and safety gates permit it. Newer task input
prevents an old claim. Lost or ambiguous provider state still prevents
replay or replacement. Ineligible recovery eventually becomes a quiet
manual reconciliation obligation.

**Steps to reproduce**

1. Start a remote native run and persist its provider checkpoint and
encrypted Runner authority.
2. Fail with exact-checkpoint recovery and a retained remote PID.
3. Release the task execution lock at heartbeat completion.
4. Run ordinary resumption and restart recovery after the retry
deadline.
5. Observe that both reject the same eligible run before remote
inspection.

**Paperclip version or commit**

Reproduced from source at `d59791828482`; integrated with upstream
`db9be4c5a6`.

**Deployment mode**

Remote workspaces with PostgreSQL. Local regressions use disposable
PostgreSQL and encrypted checkpoint persistence.

Related public repairs #18846, #18854, #18860 and #18841 cover finished
per-turn owners, pre-provider failures, unstarted follow-ups and storage
migration. They do not reclaim this failed run's released execution
lock. Merged #18879 also adds scheduled-resume lock retention and
reclaim. This PR keeps that compatibility and requires encrypted Runner
authority, canonical source ordering and a charged failed retry before
reclaim. Its scheduled-resume regression now uses a real encrypted
authority and retains a snapshot-only refusal control.

## What Changed

- Share normal admission checks and one guarded execution-binding
restore/event helper between normal and remote resumption.
- Keep agent-name normalization in a module with no imports. Preserve
heartbeat's existing export. Native admission no longer loads heartbeat
orchestration for a string key.
- Recheck company, assignment, agent eligibility, Stop, pause, approval,
dependencies, execution and checkout ownership before reclaim.
- Reject newer task input using PostgreSQL creation-time and ID
ordering, preserving timestamp precision.
- Require the recorded remote target, encrypted Runner authority, exact
checkpoint identity and existing controller fence.
- Commit task binding, controller lease, run status and recovery event
atomically. Recognize both remote workspace reference formats.
- Charge failed reattachments against the existing
three-provider-attempt budget, including retained-lock crash windows.
Active controller adoption preserves its charged attempt.
- Use the existing five-retry/30-minute system-recovery policy for
admission holds. Ended tasks and newer sources exhaust it immediately.
Preserve PID, checkpoint and task bindings, suppress resource
settlement, and assign manual reconciliation with executor stop/state
explicitly unverified.
- Register recovery intent before deadline writes; preserve live/future
deadlines quietly and roll back zero-row writes. Record changed hold
reasons once and terminal dispositions separately.
- Keep historical provider-budget exhaustion behind current
admission/source/controller/authority checks. Make an existing
orphan-cleanup fixture explicitly due.
- Include exact plugin task leases that have no workspace reference.
Reuse every reclaim and retry-budget guard, with a bounded
company/task/run lease projection.
- Share strict manual ownership proof across stale-lock cleanup,
adoption, queue release and wake admission. Require termination receipts
for every exact remote lease before normal cleanup can release either
binding.
- Preserve governing Stop through the existing audited cancellation
path. Its acknowledgement does not release an unverified provider.
Ordinary Retry can wait for exact Stop and file preservation, then queue
one continuation.

## Verification

- Current head is `5d7baf74aa89d00d7ac6e1493d38a6ce2ae7b7b5`. Its
[current CI
matrix](https://github.com/paperclipai/paperclip/actions/runs/41782198858)
is terminal and clean: 54 successful, three skipped and two neutral
contexts. Required `ci / verify` and `ci / e2e` pass. Greptile is 5/5 on
this exact head with no open findings. No fresh Paperclip 5/5 assessment
is claimed.
- The [prior complete CI
matrix](https://github.com/paperclipai/paperclip/actions/runs/41754365388)
passed on `3e641466d3edcaa6dd5021d979627082d388eb05`, with Greptile and
Paperclip both 5/5 and no open threads.
- All 233 tests across the six affected heartbeat, restart-isolation,
remote-admission, scheduled-resume and session-binding families pass on
production-identical `a5104fa89f9cb493091e000bf21ed3858871ae46` under
Node 24.22.2 and pnpm 9.15.4. They cover both workspace formats and
plugin task leases, competing controllers, exact rollback, Stop,
assignment/company boundaries, attempts, work-intent ordering, finite
backoff and quiet held state.
- The merge preserves #18931's per-run audit-failure isolation and this
PR's unchanged-transition hold. Two initial composition cases failed
because their setup or expected result described the older contract. The
rollback test now expects the exact per-run deferral while retaining
full coordinator equality and no work intent. The routine fixture writes
a real encrypted checkpoint through the normal owner before reaching the
actual routine unique constraint. No production admission guard or
rollback assertion was weakened. A separate agent reviewed this exact
merge and fixture delta without finding a blocker.
- Full recursive `pnpm -r typecheck` and `pnpm build` pass on production
merge `89c616fe45ceff75db450f67fd34224b4df7441d`. Subsequent commits
change only three test files. Both the two-file `a5104fa` and final
`5d7baf` server test-layer typechecks pass.
- Four production-identical `a5104fa` probes pass with the rebuilt
actual Rust Runner, a scripted ACPX subprocess and encrypted PostgreSQL
checkpoints under Node 24.22.2. Workspace-reference and plugin-lease
admission each cover surviving and lost providers. Replacement preserves
one provider launch, one original side effect, the charged attempt and
restored task execution binding. Lost providers reject snapshot recovery
without replacement. This is protocol-boundary evidence, not a live
Claude or hosted provisioning claim.
- Initial `a5104fa` CI found one additional routine integration fixture
missing the persisted `exact_checkpoint_resume` mode. Its encrypted
checkpoint was already present. The exact case fails locally before
repair and passes after recording that mode and expecting the charged
provider attempt. Every exact routine conflict, rollback, checkpoint,
task binding, successor-count and sibling-startup assertion remains.
Final head changes only that fixture. The correct native-server group
passes all nine files: 175 executed cases and 14 container-only skips,
matching the now-successful native-server CI gate.
- The earlier canonical HTTP Retry acceptance passed on the prior
reviewed head with real PostgreSQL, the recovery worker and real files.
Pending Stop protects both bindings and starts no successor. Exact
termination and copyback queue one preserved-workspace continuation. No
browser or live-model outcome is inferred from that test.
- The predecessor's full local `pnpm test:run` failed in an unchanged
Discord Gateway fixture with a 200 ms lease and a 422 initial ownership
error. Focused predecessor and untouched-base controls pass; that
full-run cause remains unproved. The required command was also started
on production-identical `a5104fa`. All six local chat shards passed,
covering 1,850 cases. The duplicate local invocation was interrupted
with SIGINT while the general-server group was still running after this
head's complete CI and the combined repair qualification had passed. The
wrapper exited 1 on cancellation, no process retained its test log, and
no full local pass is claimed. An earlier attempt to select the native
lane through the CI wrapper instead selected local package sharding:
2,631 cases passed and 13 failed. Those separate failures are retained
without causal attribution or unrelated source changes; that run is not
counted as native-server qualification. The direct native group and
current complete CI matrix pass.
- Live Claude Opus 5.5 continuation, hosted plugin provisioning,
deployment and recovery of historical tasks remain unverified. No
original task was retried or mutated for this verification.

## Risks

- Restoring a task binding grants remote inspection; it does not prove
that the provider survived.
- Unknown effects, lost providers and missing semantic results retain
their existing reconciliation requirements.
- Exhausted admission holds preserve execution bindings and resource
evidence for manual reconciliation. They do not authorize a fresh
executor or provider/resource release.
- The source guard is conservative: newer task input prevents recovery
of an older failed source.
- No database migration or provider profile change is required.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, code editing and
shell execution. The runtime does not expose the exact model snapshot or
context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 07:18:07 -06:00
DottaandPaperclip 0ab3ce7a03 fix(workspaces): preserve legacy recovery across storage migration (#18841)
## Thinking Path

> - Paperclip manages durable tasks and agent work through the control
plane.
> - An ephemeral web server needs existing workspace files in shared
storage.
> - Failed historical runs can still own recoverable sandbox files or
unfinished coordinator state.
> - Moving local files must preserve those obligations without replaying
the old agent turn.
> - Migration journals the exact source and fences old copyback; the
recovery owner saves verified sandbox files separately.
> - New user input can continue only after matching preservation
receipts and independent execution-stop checks.
> - Accepted results retain their finalization owner and must have its
durable workspace receipt before storage moves.

## Linked Issues or Issue Description

Builds on merged #18826 and #18835.

**What happened?**

Existing workspaces with failed native copyback could not migrate to
remote storage. Simply changing their storage authority would let later
recovery target a retired local directory, lose child-task attachments,
or leave a recovered task unable to accept a new turn.

**Expected behavior**

Preserve local files, exact sandbox provenance, task bindings,
checkpoints, and unresolved history. Recover surviving sandbox files
into immutable task-scoped attachments after migration, and let a new
authorized user turn use those files without replaying the old
execution.

**Steps to reproduce**

Create a terminal native run with a prepared copyback descriptor and a
retained sandbox. Migrate its local workspace, remove the old host
filesystem, recover the sandbox, and submit a new user turn. Also
exercise a child task reusing its ancestor's workspace and an expired
observed coordinator.

## What Changed

- Journal exact legacy run, task, descriptor, and unfinished coordinator
bindings during offline migration. Descriptor-free runs retain the
complete declared lease/allocation/directory identities; publication
locks and revalidates them.
- Publish a run-scoped fence that prevents automatic restart or copyback
into the retired local authority.
- Release only the exact journaled expired coordinator lease under the
offline deployment fence and locked revalidation, recording the prior
owner in recovery history. Preserve phase, checkpoint, result, and run
status.
- Save recovered sandbox files through the existing immutable snapshot
owner, retaining allocation and provider-stop checks.
- Attach recovered files using the current task's identity when a child
reuses an ancestor workspace.
- Accept matching migration and snapshot receipts for workspace-repair
admission and explicit new-user continuation. Changed coordinator
bindings and invalid receipts remain blocked.
- Require the finalizer's successful workspace_finalize operation before
moving a workspace whose accepted result has finalized copyback; the
existing owner can then finish assessment/status effects.
- Document unresolved workspace-policy, task-ownership, and
missing-provenance cases as separate owner actions that block
detachment.

## Verification

- 569 tests across migration, retained recovery, and explicit
continuation passed on ffcde13f57.
- The final saved-result follow-up passes all 75 migration tests. Four
receipt cases cover prepared copyback, a successful finalization
receipt, a missing receipt, and a failed receipt. The successful case
invokes the actual finalizer after migration and completes the existing
saved operation.
- Five real-file journeys cover a retained host, removed host,
child-task reuse, an expired observed coordinator, and a descriptor-free
run with two separate sandbox allocations. They recover sandbox files,
clear the repair blocker, admit an explicit /new user comment, reserve a
new remote workspace, and verify recovered attachment contents.
- Negative cases reject changed coordinator phase/owner, invalid
migration fences, changed source sets and allocations, mismatched
task/allocation/action receipts, malformed archives, and unfinished
snapshot operations. Descriptor-free continuation requires every
journaled source; one missing archive keeps the hold.
- Twelve PostgreSQL/S3 and maintenance-service tests pass on the exact
final rebuilt runtime, including a plain-Node bundled process,
authenticated original-key export, overlapping task-folder restore, and
complete predecessor filesystem removal. The final server and runtime
bundle builds also pass.
- Head 9ab6a38fa4 passed all 573
migration/recovery/continuation tests, server typecheck, server build,
runtime bundle, and 12 PostgreSQL/S3/bundled-maintenance tests.
- Verification caught an expired-owner continuation rejection (160
recovery cases passed, one failed); the admission fix and subsequent
569-test run passed. Earlier fixture failures and the parent PR's
unrelated full-suite failure remain documented in their verification
history.
- Greptile and Paperclip Review both scored final head
fa58d4a5ce 5/5, with no remaining
findings.
- CI exposed 15 blocker-read projection regressions: descriptor-free
checks performed an extra full run read for ordinary unmigrated tasks.
The follow-up gates the lookup on the migration marker, includes that
marker in the bounded blocker projection, and selects only recovery
fields. All 31 blocker-projection and real-file journey cases plus
server typecheck pass. Final-head CI passed all 52 successful checks,
including repository typecheck, build, server tests, and browser suites.
- Preserved CI failure: the unchanged Runner package had one Cursor
Cloud Stop/checkpoint timing failure (4,903 cases passed). All four
focused local cases passed. Retrying only the failed job and its
verification gate passed without source changes.
- This PR does not claim tenant enrollment or a hosted rolling-deploy
result.

## Risks

Cloud must stop every predecessor deployment before enabling the offline
migration contract. Migration preserves uncertain work; it does not
certify provider termination, invent a completed result, or authorize
sandbox deletion. Missing or ambiguous sources remain held. Preserved
attachments are separate files rather than an automatic merge into newer
task edits. Parallel shared-write policy, stale task ownership, missing
task provenance, and accepted results without a successful finalization
receipt remain enrollment blockers. Original folders and the original
volume remain retained.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. This session does not expose a more specific serving model ID
or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used with available version and
capability details
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR using the bug template
- [x] I have not referenced internal or instance-local issues or links
- [x] My branch name describes the change and contains no internal task
ID
- [x] I have run the focused tests locally and they pass
- [x] I have added tests for the changed behavior
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address reviewer findings before merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 16:17:13 -06:00
DottaandPaperclip 9ca9af544a Drive remaining live run and resource recovery from durable work (#18419)
Reviewed and verified at 6a4c193906.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 16:00:35 -05:00
DottaandPaperclip b6b57079b9 fix: evaluate status cards on committed changes and deadlines (#18401)
Reviewed and verified at 21795e222a.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 11:41:59 -05:00
829f9e7eaf fix(server): settle Muse terminal assignments from committed changes (#17793)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/12/changes/paperclip/12
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its external-agent module owns Muse assignments, operation receipts,
and worker holds.
> - Running assignments and retained terminal history must not cause an
empty maintenance scan every five seconds.
> - Terminal run changes and operation results already have durable
database transactions.
> - This pull request wakes the existing maintenance worker from those
commits and reconstructs work at startup.
> - The worker retries only concrete settlement debt. It retains
qualification deadlines and authority checks.

## Linked Issues or Issue Description

Refs: #17772. This is slice 57. Parent #17772 has merged.
Related: #17000, #17525, #17752, #17758. The publication targets master
and preserves the merged parent.

**What existing behavior does this improve?**

Muse terminal assignment settlement and the existing Muse maintenance
worker.

**Current behavior**

A running assignment can keep maintenance scanning every five seconds.
Terminal reconciliation rewrites unchanged history. Native finalization
also calls that reconciliation during periodic replay. Discovery
excludes a revoked original binding even when its assignment still needs
settlement.

**Proposed behavior**

Committed run changes and operation receipts wake the existing worker
with the original company, run or binding identity. Startup reconstructs
actionable work from existing rows. Maintenance processes bounded
batches and disarms after settlement. Default live and manual callbacks
remain. The periodic callback delegates this one duty.

**Reason and benefit**

Configured but idle instances stop querying for this obligation. Late
receipts, restarts, drain and historical bindings still have recovery
paths. No table or independent timer is added.

**Breaking changes**

No schema, endpoint or provider protocol changes. Explicit manual
reconciliation retains its unbounded default. This does not alter Cloud
sleep admission or claim that all instance work can sleep.

## What Changed

- Add an owner predicate shared by discovery and locked settlement.
Fenced private-worker uncertainty remains held without an automatic
retry loop.
- Preserve original binding generation and company/agent/run identity,
including revoked bindings. Fence only the terminal assignment during
reconciliation.
- Extend existing maintenance with scoped hints, bounded batches,
concrete-debt retry, startup reconstruction and quiet drain behavior.
- Register notifications inside existing terminal, projection,
cancellation, operation-result and send-failure transactions. Keep
qualification and OAuth signals explicitly global when coalesced with
scoped changes.
- Serialize only four domain IDs. Real native-finalizer tests caught and
prevent a circular-object serialization failure at this boundary.
- Use existing restore/worktree suppression and settings reactivation.
Delegate both periodic finalizer branches while retaining resource
callbacks and default live/manual settlement.
- Add the producer audit and regression evidence in
`doc/plans/2026-10-26-muse-terminal-settlement-sleep.md`.

## Verification

Accepted production source: `584015018ed4a9d57d2b0f192b75847503b54e53`.
Publication head: `305014b030d6c5eb891c70bc57036f4192d4fdad`. Normal
adoption of the actual parent squash preserves the complete accepted
child tree. Only audit text, a migration-count clarification, and the
proved Photon test extension follow.

- **3,715 distinct cases in 52 complete server files pass**, plus **20
selected chat cases** (3,735 selected total). The main 49-file batch
passes 3,008; process recovery passes 410; review response passes 273;
the full Photon file passes 24. The other 1,794 chat cases were not run
locally.
- **Quiet producer proof:** moving only the Muse fanout outside its
parent guard produces six candidate SELECTs and fails the regression.
Keeping the guard yields zero and still settles a real terminal run. The
same test hash spans the controlled red/green pair.
- **Photon review evidence:** repeated active callbacks produce zero
settlement hints/candidate SQL. Three concurrent attention recoveries
produce exactly one of each. The existing locked attention guard was
correct; production needed no fix. The parent security check accepted
the resolved finding before merge. This PR keeps the additional
settlement queue in the existing regression.
- **Static checks pass:** database build; server typecheck/build; module
boundaries; migration order; current Copilot identity. Packages equal
the accepted parent byte for byte. No new live provider qualification is
claimed.
- **Fixtures and failures:** 34 fresh PostgreSQL 17 fixtures have 412
applied migration receipts through tag 0413_tired_kronos, plus fresh
per-call helper fixtures. All 52 original test hashes restore exactly.
The additional Photon test hash matches its passing proof. Earlier
remote chat passive-idle failure remains recorded; both passive/missing
cases pass in the new selection. Parent handoff timeout/fresh-file-pass
accounting remains in its audit. No assertion or timeout was weakened.
- [Integration and quiet producer
audit](https://github.com/paperclipai/paperclip/blob/305014b030d6c5eb891c70bc57036f4192d4fdad/doc/plans/2026-10-26-muse-terminal-settlement-sleep.md#current-master-integration-and-quiet-producers-2026-10-29).
Fresh CI, security, and exact-head Greptile 5/5 must pass before merge.

<details><summary>Earlier revision verification and failure
history</summary>

Receiver failure fairness: a controlled 1,100 ms batch reproduced a
30-second delay for healthy due receivers. The worker now supplies its
existing per-item deadline through `nextRunAtOnError`. All 20 receiver
lifecycle tests pass on the owner and combined tip
`31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`; server production
typechecking passes on the owner. The failed receiver retains its own
backoff, and settled receivers return to zero recurring SQL. The
five-second assertion budget is unchanged. See [the owning
audit](https://github.com/paperclipai/paperclip/blob/f7477933469fdc960f555425dce66eff25a94e54/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#healthy-receiver-progress-after-a-slow-failed-batch).
Current-head CI and review remain required.

Approved delegated Test-tab calls now evaluate caller and target
policies in the existing claim transaction. Real HTTP regressions first
showed a missing caller charge and shared-limit self-rejection. All 78
connection-management cases and 92 gateway cases now pass on the owning
branch, along with server production typechecking. All 78
connection-management cases also pass at combined tip
`d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover
shared charges, separate charges, rollback on denial, duplicate
approval, and rejection of a later authorized call. Intermediate
counter/identity fixture failures and a reused-database Slack collision
are recorded in [the owning
audit](https://github.com/paperclipai/paperclip/blob/bbd31dea5ef3a4875950a87b4636af8b6bde858e/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes).
The earlier combined checks below remain evidence for their stated
revisions. Current-head CI and review remain required.

Inherited voice-cleanup verification: the controlled slow-provider
regression and all 13 lifecycle tests passed on its owner and combined
tip `ab22d8ed0c09028315607414b83b375e5cc0a60a`. That repair changed only
the fixture clock advancement and its audit note.

Current head: `f7477933469fdc960f555425dce66eff25a94e54`. This remains a
single-duty PR stacked on #17772.

- Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5`
with normal merges. Every branch passed generated-schema checks and its
target-branch migration-order gate. Existing SQL is unchanged; the 25
stack migrations follow master’s new migration.
- Duty-specific producer coverage, lifecycle tests, and earlier failure
records:
[2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/ff196791db6dada2dc04ea9db0d58c5c3beb4a41/doc/plans/2026-10-24-recurring-server-sleep-stack.md),
[2026-10-26-muse-terminal-settlement-sleep.md](https://github.com/paperclipai/paperclip/blob/ff196791db6dada2dc04ea9db0d58c5c3beb4a41/doc/plans/2026-10-26-muse-terminal-settlement-sleep.md).
- Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355
startup/cancellation/recovery/dispatch/preservation tests**, **75
connection-management tests**, and production server typechecking pass.
These combined checks do not mean every intermediate PR independently
ran the full repository suite.
- Real master-to-stack database upgrade passes: **379 unchanged prior
migration receipts plus 25 new migrations**, with the gateway
responsibility fixture preserved.
- Owning-branch checks: 84 runner installation/settlement cases plus
runner typechecking and generated TypeScript/Rust profile agreement; 253
GitHub review workflow cases. Stop-attribution and held-worker readiness
regressions failed before their fixes.
- Failure accounting: the initial combined run passed 349 cases and
failed six idle assertions in a reused preservation database containing
old cancelled runs. Repeating the combined command with a fresh
preservation database passed. Earlier invalid project selection, merge
import/type errors, and the interrupted full local suite are not counted
as passing checks.
- **Latest-head CI and review remain required.** Prior CI findings
include GitHub idle observation, route teardown, attention cleanup,
touch-picker E2E, workspace health, and voice-cleanup timing; local
passes do not erase those failures. No full local repository-suite,
deployed sleep, fleet cost, or live Copilot qualification claim is made.


</details>

## Risks

- In-process signals do not capture independent SQL or other replicas.
Supported external events still need ingress.
- Pending assignments and operation receipts retain their original
binding identity. A new agent configuration cannot take their ownership.
- Genuine qualification, OAuth, and recovery deadlines remain
obligations. They must not be discarded merely to make the worker quiet.
- The parent coarse-hint guard prevents recurring Muse discovery; real
company/binding authority changes retain their signals.
- No full local repository suite, full chat suite, live Muse or Copilot
qualification, deployed sleep, database suspension, external alarm
coverage, or cost savings is claimed. Existing Cloud blockers remain.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, code execution, real
PostgreSQL tests, and GitHub tools. The exact runtime model identifier
and context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 06:18:40 -05:00
4d0847c4ca fix(server): settle external conversations from committed work (#17772)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/6/changes/paperclip/6
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip manages AI agents and the tasks that they execute.
> - External conversations must become idle after an authorized response
is published and all related work is complete.
> - Native replay and stranded-task recovery still discover this
settlement work through recurring scans.
> - A new deadline worker must preserve response ordering, monitor
handoffs, and unfinished work.
> - This pull request gives settlement one shared readiness predicate
and activates its worker from committed changes.
> - Existing rows provide recovery truth. No new table or per-feature
timer is added.
> - The benefit is a settlement worker that becomes quiet and resumes
when relevant work changes.

## Linked Issues or Issue Description

Refs #17758. This is slice 56. Slice 55 has merged. Related: #17606
(chat milestones), #17663 (bounded run reads), #17665 (monitor proof),
and #13809 (answered Slack conversation lifecycle). The provider
applicability remains Slack and Speko-owned conversations. This does not
extend Telegram settlement from #16559.

**What happened?**

Conversation settlement depended on periodic native replay and generic
task recovery. An exhausted monitor also cleared its schedule before
recovery admission. During that gap, an answered conversation could look
finished. Failed or deferred admission lost the monitor schedule.

**Expected behavior**

Relevant committed changes activate settlement. The worker uses the
owning service's evidence and becomes quiet when no settlement is owed.
Monitor recovery stays pending until admission, and restart does not
admit an already accepted recovery twice.

**Steps to reproduce**

1. Create an active external conversation with a successful run and an
authorized published response.
2. Keep execution, approval, a goal, or a monitor pending. Let the
settlement worker become quiet.
3. Resolve that prerequisite through its owning service. Verify that the
conversation settles without another periodic sweep.
4. Delay or fail exhausted-monitor recovery admission. Verify that the
monitor remains pending and the conversation cannot settle during the
handoff.

**Paperclip version or commit**

Accepted production source: `50f4432ed88537b8d522cf1dce24c33588db0bac`.
Parent #17758 merged as `8ac5a128e66fb672e88106412f76cf278599bfc4`. The
publication adopts that actual squash through a normal merge with an
unchanged complete child tree.

## What Changed

- Preserve quiet after coarse company-only maintenance hints. The shared
inbound-notice and response-availability helpers keep settlement enabled
by default. Only the milestone dispatcher suppresses their duplicate
settlement fanout. Its scoped issue-review signal and the explicit
company-authority signal still wake real work. Four database cases prove
both quiet and real company-wide producer delivery.
- Preserve source-generation checks for chat handoff,
task-before-comment locking, issue-scoped decision invalidation, and
current conversation event columns from master.
- Track initial external conversation settlement through shutdown
without blocking HTTP readiness on endpoint or task locks.

- Reuse the real queue's coalesced recovery receipt after restart,
without admitting the same monitor recovery again.

- Share the existing conversation readiness predicate between bounded
discovery and locked settlement. Preserve current provider policy,
run/publication proof, company boundaries, and endpoint-to-issue lock
order.
- Register a scoped deadline worker through the delivery coordinator.
Preserve startup, restore/worktree suppression, drain/resume, failure
recovery, and joined shutdown.
- Extend existing review, milestone, wake-notice, and finalization
signals. Add notifications at independent issue, comment, goal, session,
and wake-receipt unblocks.
- Keep monitor owner recovery pending until guarded admission. Reuse an
accepted receipt for the same monitor schedule after restart, even when
its limit reason changes; retain compatibility with legacy receipt keys.
Commit board escalation and its monitor clear together.
- Delegate periodic native settlement while preserving resource
callbacks. Keep direct/manual defaults and preserve generic recovery
ordering with a read-only readiness guard.
- Add the producer and lifecycle audit in
`doc/plans/2026-10-26-external-conversation-settlement-sleep.md` and
update the stack ledger.

## Verification

Accepted production source: `50f4432ed88537b8d522cf1dce24c33588db0bac`.
Publication head: `b256bb52647affd9f5279ea8c40f48bbe40f031f`.
The complete accepted child tree stays unchanged when adopting merged
parent #17758. This publication adds only the audit document.

- **3,129 distinct cases in 37 complete server files pass**, plus **18
selected chat-response cases** (3,147 selected cases). The other 1,796
chat cases were not run locally. Full settlement, queue, scheduling,
run-state, goals, interactions, task Retry, decision targets, status
reassessment, workspace, process recovery, and review consumers are
covered.
- **Preserved failure:** the first 35-file command passed 2,445 cases
and timed out one restored-work handoff at its existing 15-second limit.
That aggregate command failed. The unchanged full 89-case actions file
then passed on a fresh fixture; the affected case took 22 ms. Root cause
was not established. No timeout or assertion changed. The 34 other files
passed originally. Distinct counts exclude duplicate actions passes.
Fresh CI remains a separate gate.
- **Indirect wake regression:** the original unconditional baseline
failed two quiet checks. Guards alone left the chat case failing with
three candidate queries. The final controlled indirect baseline fails
one of four cases; the repair passes all four with the same test hash.
Positive controls preserve real company-only producers.
- **Source and fixture controls:** all 38 selected test files restore to
their exact committed hashes after fixture imports. Fresh PostgreSQL 17
fixtures use 413 migrations. All five manual merge resolutions preserve
current authority, locking, and event-column contracts.
- **Static checks pass:** database build; server typecheck and build;
module boundaries; migration order; Copilot profile identity. Runner
packages equal the accepted parent byte for byte. No new Runner/package
qualification is claimed.
- [Current integration and wake
audit](https://github.com/paperclipai/paperclip/blob/b256bb52647affd9f5279ea8c40f48bbe40f031f/doc/plans/2026-10-26-external-conversation-settlement-sleep.md#current-master-integration-and-indirect-wake-audit-2026-10-29).
Latest-head CI and exact-head Greptile 5/5 must pass before merge.

<details><summary>Earlier revision verification and preserved failure
history</summary>

Receiver failure fairness: a controlled 1,100 ms batch reproduced a
30-second delay for healthy due receivers. The worker now supplies its
existing per-item deadline through `nextRunAtOnError`. All 20 receiver
lifecycle tests pass on the owner and combined tip
`31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`; server production
typechecking passes on the owner. The failed receiver retains its own
backoff, and settled receivers return to zero recurring SQL. The
five-second assertion budget is unchanged. See [the owning
audit](https://github.com/paperclipai/paperclip/blob/83204a67880d980cfbea3771cee19a3cda0022b7/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#healthy-receiver-progress-after-a-slow-failed-batch).
Current-head CI and review remain required.

Approved delegated Test-tab calls now evaluate caller and target
policies in the existing claim transaction. Real HTTP regressions first
showed a missing caller charge and shared-limit self-rejection. All 78
connection-management cases and 92 gateway cases now pass on the owning
branch, along with server production typechecking. All 78
connection-management cases also pass at combined tip
`d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover
shared charges, separate charges, rollback on denial, duplicate
approval, and rejection of a later authorized call. Intermediate
counter/identity fixture failures and a reused-database Slack collision
are recorded in [the owning
audit](https://github.com/paperclipai/paperclip/blob/ef9ae5767430d0093efc736ae0b303ba5afaae65/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes).
The earlier combined checks below remain evidence for their stated
revisions. Current-head CI and review remain required.

Inherited voice-cleanup verification: the controlled slow-provider
regression and all 13 lifecycle tests passed on its owner and combined
tip `ab22d8ed0c09028315607414b83b375e5cc0a60a`. That repair changed only
the fixture clock advancement and its audit note.

Current head: `83204a67880d980cfbea3771cee19a3cda0022b7`. This remains a
single-duty PR stacked on #17758.

- Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5`
with normal merges. Every branch passed generated-schema checks and its
target-branch migration-order gate. Existing SQL is unchanged; the 25
stack migrations follow master’s new migration.
- Duty-specific producer coverage, lifecycle tests, and earlier failure
records:
[2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/d6d8e4ffa6b330ebf4647aec997ee2dde6abc321/doc/plans/2026-10-24-recurring-server-sleep-stack.md),
[2026-10-26-external-conversation-settlement-sleep.md](https://github.com/paperclipai/paperclip/blob/d6d8e4ffa6b330ebf4647aec997ee2dde6abc321/doc/plans/2026-10-26-external-conversation-settlement-sleep.md).
- Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355
startup/cancellation/recovery/dispatch/preservation tests**, **75
connection-management tests**, and production server typechecking pass.
These combined checks do not mean every intermediate PR independently
ran the full repository suite.
- Real master-to-stack database upgrade passes: **379 unchanged prior
migration receipts plus 25 new migrations**, with the gateway
responsibility fixture preserved.
- Owning-branch checks: 84 runner installation/settlement cases plus
runner typechecking and generated TypeScript/Rust profile agreement; 253
GitHub review workflow cases. Stop-attribution and held-worker readiness
regressions failed before their fixes.
- Failure accounting: the initial combined run passed 349 cases and
failed six idle assertions in a reused preservation database containing
old cancelled runs. Repeating the combined command with a fresh
preservation database passed. Earlier invalid project selection, merge
import/type errors, and the interrupted full local suite are not counted
as passing checks.
- **Latest-head CI and review remain required.** Prior CI findings
include GitHub idle observation, route teardown, attention cleanup,
touch-picker E2E, workspace health, and voice-cleanup timing; local
passes do not erase those failures. No full local repository-suite,
deployed sleep, fleet cost, or live Copilot qualification claim is made.


</details>

## Risks

- Signals cover supported in-process writers. They do not supply change
capture for independent SQL or other replicas.
- Real company-wide changes can inspect active candidates. Coarse
maintenance hints do not activate settlement. Configured idle and
retained history must remain quiet.
- Monitor receipt binding and admission order must preserve company,
agent, issue, and exact schedule identity. Current parent handoff locks
and source-generation checks remain intact.
- The local restored-work timeout is retained above. A fresh full-file
pass does not establish its cause; latest-head CI must also pass.
- Generic recovery and Muse/resource duties remain separate slices. This
does not make the whole server idle. Existing Cloud sleep blockers
remain. No deployed sleep, database suspension, external alarm coverage,
fleet cost, or live provider qualification claim is made.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, code execution, PostgreSQL
tests, and GitHub tools. The exact runtime model identifier and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 06:02:43 -05:00
8ac5a128e6 fix(server): resolve native finalization notices from committed wakes (#17758)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native finalization can commit before its recovery notice is
resolved.
> - The periodic orphan pass revisits committed runs to resolve those
notices.
> - Retained completed history should not require repeated discovery.
> - This PR moves that one duty to transaction wakes and concrete
failure retries.
> - Existing notice rows reconstruct work after restart. The action
service still owns resolution.
> - Other completion and resource duties keep their current recovery
paths.

## Linked Issues or Issue Description

Refs #17752 (stack parent: committed run projection), #17748
(accepted-result commit), and #17734 (workspace finalization). These are
distinct recovery obligations. This slice incorporates parent #17752 at
`a49a2dac13b0e1c19da5d3d6b435c781703b1c0b`, now merged as
`50fe32684d2d4e1db129e3c1bb455571344a0c09`. Related #17724 retires
obsolete exact-session notices for terminal runs with no result and
preserves their replay hold. This slice resolves notices after a result
committed; it does not change that no-replay policy.

**Current behavior**
Committed native finalization recovery repeatedly inspects active
watchdog notices. A notice can also be updated in place to describe a
different run, so resolving a previously selected ID alone is unsafe.

**Proposed behavior**
Wake a scoped worker from relevant committed writes. Reconstruct active
finalization notices at startup. Resolve only the current notice after
the native owner rechecks its binding under lifecycle locks. Stop
scheduled work when no matching obligation remains.

**Reason and benefit**
Notice recovery becomes quiet after settlement and reacts to real
changes without a polling delay. Reused notices remain attached to their
current incident. This does not claim that every remaining server duty
or the host can sleep yet.

## What Changed

- Track the initial recovery-notice sweep through shutdown without
blocking HTTP readiness on task locks.
- Require issue/run scope for ordinary finalization fanout. Routine
company-only hints leave dormant notices quiet; dedicated owner and
explicit settings signals retain their scopes.

- Extract the native finalization notice predicate and resolution owner.
Keep the existing recovery-action service responsible for writing the
resolution.
- Bind candidates by company, issue and run. Require a committed result
and an active or escalated finalization notice. Exclude held ownership,
pending status effects and unfinished run projection.
- Recheck coordinator, issue, run and matching actions under locks. A
newer incident cannot be resolved from a stale candidate snapshot.
- Use the shared scoped worker, transaction signals, retry backoff and
lifecycle handling. Add no table, per-feature timer or provider branch.
- Signal action insertion, update, supersession and resolution;
finalization and effect changes; run projection and ownership release;
issue changes; settings activation; and the recovery route that reopens
an action.
- Preserve no-op resolution quiet. Serialize only the three supported
notification IDs, even when a caller supplies a larger internal object.
- Delegate only notice resolution from the periodic path. Keep
manual/default finalization and other completion/resource work.
- Add real PostgreSQL producer, concurrency and lifecycle coverage and a
detailed audit.

## Verification

- Current local source `730e00a7b007a36d188ca0a328ef050d3a25d1ee`
incorporates the parent with a normal merge. All 1,209 parent-only,
eight owned-only and 12 automatic shared paths are preserved. The two
manual paths retain both chat import sets and scope the new notice
fanout. The entire package tree matches parent profile 80 byte-for-byte.
- Controlled real-database regression: original notice-only unscoped
fanout produces three candidate SELECTs for routine company hints. The
same zero-query assertion passes after restoring scoped fanout. The real
notice writer then creates and resolves an action for a committed run
and returns to quiet. Both phases restore production bytes exactly; the
22 cases outside this focused pattern are not counted as passing there.
- The complete main command passes **1,873 cases in 21 files**, without
skips, in 309.69 seconds. It includes all 23 notice-worker cases, full
presentation/ownership read coverage, 88 startup cases, and the
projection/result/physical-work consumers. Process recovery passes 410
in 163.54 seconds and external-chat response passes 273 in 53.48
seconds. Total complete-file acceptance: **2,556 cases in 23 complete
files**, with no skips. The additional focused chat selection passes 18
cases in 13.44 seconds; 1,796 unselected chat cases do not count. Total
selected passes: **2,574**. Normal merge
`a0f2b3f4348b078219a53d8cc135c84202ba505e` adopts actual parent ancestry
with this complete tested child tree unchanged.
- The chat selection covers the real reply/notice row-lock regression,
Slack and Telegram response recovery, current-authority denial cases,
exact canonical response binding and internal GitHub attachment
navigation. Other cases in the giant chat file remain unselected; this
is not full chat-suite acceptance.
- Production server typecheck, build, module boundaries, migration order
and source identity checks pass. Twenty dedicated PostgreSQL 17
databases use the unchanged migration 413 schema. Embedded-only
invocations use the existing fresh-per-call recovery helper. All
temporary imports restore exactly and all 24 selected test hashes match
after completion. No assertion or timeout is relaxed.
- Parent package acceptance remains applicable: 130 passing
installation/compatibility/transport cases, 22 installed ACPX/Grok
contract cases and 26 native-provider Rust cases. One existing opt-in
closed Pi snapshot case is skipped because its distribution root is
unset. This does not qualify a live provider.
- Failure history and scope are retained in
`doc/plans/2026-10-26-native-recovery-notice-sleep-audit.md`.
Current-head acceptance: all 57 checks are clean, including protected CI
and security. Greptile gives 5/5 on
`2a8777bbed8b1e10810e3d6d99aae3f25b4854ef`, with no actionable findings.
Prior scores do not substitute. No Ponytail verdict is present under the
authorized missing-review policy. No full local repository suite,
deployed sleep, expanded Cloud wake coverage or measured savings is
claimed.

## Risks

- Resolution takes coordinator, issue, run and action locks in the
native recovery order. Contention must preserve the current incident;
the real concurrent-upsert and competing-resolver cases exercise this
boundary.
- Process-local signals cover instrumented application writers. They do
not add support for independent SQL writers or replicas without
supported ingress. Startup reconstructs the existing durable rows.
- Completed finalization is not proof that external conversation
settlement, Muse reconciliation or physical cleanup finished. Those
obligations retain their owners and recovery paths.
- Discovery uses narrow keyset pages. No fleet benchmark, host
suspension or cost reduction is claimed. Cloud wake coverage remains
unchanged.
- Parent #17752 has merged. This PR targets master at
`50fe32684d2d4e1db129e3c1bb455571344a0c09`. Fresh current-head CI and
review must pass before the authorized protected merge.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The exact serving model ID and context-window size are not exposed in
this session. No subagent was used for this slice.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 05:35:01 -05:00
50fe32684d fix(server): wake committed native run projection from durable changes (#17752)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run has separate result acceptance, task decision, run
status, and resource settlement steps.
> - A task decision can commit before its run status is saved.
> - Recovery currently revisits committed decisions through the periodic
orphan pass.
> - This PR gives the existing run projection repair its own
transaction-driven worker.
> - Existing records reconstruct unfinished work after restart.
Completed history becomes quiet.
> - Other completion and resource obligations keep their existing
recovery paths.

## Linked Issues or Issue Description

Refs #17748 (stack parent: accepted-result commit), #17734 (workspace
finalization), and #17717 (retained session cleanup). Related #17614
preserves terminal notifications. This slice is prepared on current
parent #17748 at `b2efdb2dbbbef5b2227ffa2bd2a9a6cfe3867fe2`; the parent
has merged as `28e38da7e6bdd6af4092dda56bc2be05bed47af5`.

**Current behavior**
The native recovery pass revisits committed decisions to repair the
heartbeat-run status and metadata. A crash between decision and run
writes creates real work, but periodic scanning also revisits retained
history.

**Proposed behavior**
Wake projection repair after committed producer changes. Reconstruct
mismatches on startup and settings activation. Retry concrete failures
through the shared scheduler. Keep the run projection before resource
settlement.

**Reason and benefit**
The repair worker performs no recurring SQL after its work settles. A
real decision, effect settlement, or ownership release reactivates it.
This is one worker migration; it does not establish host or database
suspension.

## What Changed

- Extract the existing projection predicate and write into their
native-runtime owner. The finalizer keeps initial metadata enrichment
and task decision authority.
- Discover accepted, committed, mismatched run projections with company,
issue and run bindings. Exclude pending status effects and held runner
ownership. Read narrow fields instead of whole result payloads.
- Use existing transaction hints, scoped delivery scheduling, failure
backoff and lifecycle handling. Add no tables or per-feature timers.
- Require issue/run scope for ordinary finalization fanout. Routine
company-only hints leave dormant projection quiet; explicit settings and
dedicated producer signals keep their existing contract.
- Connect finalization, pending effects, lifecycle status writes, issue
changes and worktree execution activation. Preserve outer transaction
commit and rollback behavior.
- Lock the current run before deciding which terminal event and
completion time to save. Concurrent live and recovery projection must
not announce the same transition twice.
- Delegate only already-committed projection from the periodic pass. Its
candidate scan uses the same pending predicate to defer remaining
consequences and cleanup until projection finishes.
- Add real PostgreSQL producer, race and lifecycle tests plus an owner
audit.

## Verification

- Current local source `a83b9449233a275176e637c765059f414305d7c1`
incorporates the parent with a normal merge. All 1,208 parent-only,
seven owned-only and ten automatic shared paths are preserved. The only
manual conflict keeps new projection fanout inside the ordinary
issue/run guard. The complete package tree matches parent profile 80
byte-for-byte.
- Controlled real-database regression: the original projection-only
unscoped fanout produces three candidate SELECTs for routine company
hints. The same zero-query assertion passes with scoped fanout. A real
committed result then repairs the run and emits one terminal event
before returning to quiet. Production bytes restore exactly in both
phases. Twenty cases outside that focused pattern are not counted as
passing there.
- The complete main command passes **1,830 cases in 19 files**, without
skips, in 301.14 seconds. It includes all 21 projection cases, 85
startup cases, required quality review and heartbeat run lifecycle, plus
physical-work and result-commit consumers. Separate process recovery
passes 410 in 163.58 seconds and external-chat response passes 273 in
53.26 seconds. Total: **2,513 distinct passes in 21 complete files**,
with no skips. Normal merge `d4b46fd29cb23126d556cc3aeb883e6b41f1a8a5`
adopts the actual parent squash with this entire tested child tree
unchanged.
- Production server typecheck, build, module boundaries, migration order
and source identity checks pass. Seventeen dedicated PostgreSQL 17
databases use the unchanged migration 413 schema. Embedded-only calls
use the existing fresh-per-call recovery helper. All temporary imports
restore byte-exact and all 21 selected test hashes match after
completion. No assertion or timeout is relaxed.
- Parent package acceptance remains applicable: 130 passing
installation/compatibility/transport cases, 22 installed ACPX/Grok
contract cases and 26 native-provider Rust cases. One pre-existing
opt-in closed Pi snapshot case is skipped because its distribution root
is not configured. No live provider qualification is claimed.
- The owner audit at
`doc/plans/2026-10-26-native-run-projection-sleep-audit.md` retains
prior failures and their disposition. Current-head acceptance: all 57
checks are clean, including protected CI and security. Greptile gives
5/5 on `a49a2dac13b0e1c19da5d3d6b435c781703b1c0b`, with no actionable
findings. Old scores do not substitute. No Ponytail verdict is present
under the authorized missing-review policy.
- These checks do not claim a full local repository suite, deployed
suspension, expanded Cloud wake coverage or measured cost savings.

## Risks

- Process-local signals cover instrumented application writers.
Independent SQL writers and replicas without supported ingress are not
newly covered. Durable startup reconstruction remains enabled.
- Projection must precede remaining committed resource settlement. The
real reconciler test proves the periodic path defers cleanup while
projection is pending and resumes it after repair.
- Status decisions, external conversation settlement, watchdog
resolution, Muse reconciliation and resource release keep their own
authority and recovery. A committed result alone is not proof of
completed cleanup.
- Candidate discovery uses narrow bounded pages. There is no large-fleet
performance or deployed sleep claim. Existing Cloud wake coverage and
sleep blockers remain unchanged.
- Parent #17748 has merged. This PR targets master at
`28e38da7e6bdd6af4092dda56bc2be05bed47af5`. Current-head CI and review
must pass before the authorized protected merge.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The exact serving model ID and context-window size are not exposed in
this session. No subagent was used for this slice.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 05:10:16 -05:00
28e38da7e6 fix(server): wake native result commits from durable work (#17748)
## Thinking Path

> - Paperclip manages AI agents and preserves their results through
failures.
> - Native completion must save workspace output before it commits the
accepted result.
> - The orphan recovery pass repeatedly revisits this work even when the
subsystem is idle.
> - Accepted results, workspace receipts, retry dates and leases already
provide durable truth.
> - This PR wakes a scoped worker from those writes and schedules only
real unfinished work.
> - It preserves existing finalization authority and keeps later
projection and resource recovery separate.

## Linked Issues or Issue Description

Refs #17734 and #17717. This is slice 53 of the existing sleep stack,
targeting master after parent #17734 merged at
`a019d00a393b57f531afe8dd8ab0b8a1820d1c4c`.
A current search found no duplicate accepted-result commit PR.

**What problem does this solve?**
Native accepted-result recovery currently depends on periodic orphan
scanning. Quiet instances keep doing work to discover that no result
needs a commit.

**What improvement do you propose?**
Use the shared delivery coordinator with transaction-bound producer
signals. Reconstruct unfinished results at startup. Preserve saved retry
and lease deadlines. Remove only uncommitted-result execution from the
periodic pass.

**Why is this useful?**
The accepted-result worker becomes quiet with no work or retained
completed history. Real results and completed workspace operations
reactivate it without a polling delay. This does not claim that the
whole server or database can sleep yet.

## What Changed

- Track the initial result-commit sweep through shutdown without
blocking HTTP readiness on recovery locks.
- Keep ordinary company-only finalization hints from reconstructing
dormant result work. Existing company settings and dedicated owner
signals remain supported.

- Add a native result-commit worker. Bind candidates by company, issue,
run, accepted result and a successful workspace receipt. Exclude
committed, terminal, held and pending-effect work.
- Signal both accepted-result writers and workspace receipt writes
before their transaction changes. Extend the existing finalization
fanout for retry, lease, effect and completion transitions.
- Add an optional atomic due-date check to the existing finalizer claim.
A lease collision or later retry does not consume a failed attempt.
- Signal native retry cancellation so it removes the saved wake after
commit.
- Register through the existing heartbeat suppression and shared
delivery lifecycle. Preserve outer commit/rollback, drain/resume,
settings activation and shutdown behavior.
- Delegate only uncommitted-result execution from the periodic orphan
pass. Keep committed projection and resource settlement recovery
enabled.
- Add real producer/lifecycle tests and a detailed audit. No new tables,
provider branches or per-feature timers.

## Verification

- Current local source `2af88fa55f961a74340880240019f6cd31e21bf1`
incorporates the parent with a normal merge. All 1,203 parent-only, nine
owned-only and 12 automatic shared paths are preserved. The only manual
conflict keeps the new result fanout inside the existing ordinary
issue/run scope guard. The complete package tree matches parent profile
80 byte-for-byte.
- Controlled real-database regression: the original result-only unscoped
fanout produces three candidate SELECTs for routine company hints. The
same zero-query assertion passes after restoring the scoped fanout. A
real workspace receipt and accepted result then commit exactly one
decision and return to quiet. Red and green restore production bytes
exactly. Twenty cases outside the focused pattern are not counted as
passing there.
- The complete main selection passes **1,746 cases in 16 files**,
without skips, in 277.97 seconds. This includes all 21 result-worker
lifecycle cases and 82 startup cases, including held result recovery,
HTTP readiness and shutdown joining. Separate process recovery passes
410 cases in 162.42 seconds and external-chat response passes 273 in
53.41 seconds. Total: **2,429 distinct passes in 18 complete files**,
with no skips. Normal merge `998239b019defe52e7b5d64f884de4e871ccd736`
adopts the actual parent squash and preserves this complete tested child
tree.
- Production server typecheck, build, module boundaries, migration order
and source identity checks pass. Fifteen dedicated PostgreSQL 17
databases use the unchanged migration 413 schema. Embedded-only calls
use the existing fresh-per-call recovery helper. All temporary imports
restore byte-exact, and every selected test hash matches after
completion. No assertion or timeout is relaxed.
- Runner validation belongs to the unchanged parent package tree: 130
passing installation/compatibility/transport cases, 22 installed
ACPX/Grok contract cases and 26 native-provider Rust cases. One existing
opt-in closed Pi snapshot case is skipped because its distribution root
is not configured. This does not claim live provider qualification.
- Earlier Greptile 4/5 on `656f6722be1aedf3d52b7f853956e15838204278`
correctly identified startup blocking on result recovery. The prior
branch `108cf0dc8296bf451b721ddac55ecbcd035d00f0` already tracks its
ready promise through the existing lifecycle owner. Current full startup
acceptance verifies the fix in the merged context. The later Greptile
service error is not a pass; current-head acceptance now passes: all 57
checks are clean, including protected CI and security. Greptile gives
5/5 on `b2efdb2dbbbef5b2227ffa2bd2a9a6cfe3867fe2`, with no actionable
findings. No Ponytail verdict is present under the authorized
missing-review policy.
- The owner audit at
`doc/plans/2026-10-26-native-result-commit-sleep-audit.md` retains
earlier failures and their disposition. These checks do not claim a full
local repository suite, deployed app/database suspension, expanded Cloud
wake coverage or measured cost savings.

## Risks

- Process-local signals cover instrumented application writes in the
owning server process. They do not cover independent SQL writers or
another replica without supported ingress. Durable startup
reconstruction remains enabled.
- The initial native decision and its run projection are separate
transactions. The periodic reconciler still repairs already-committed
results and releases resources. This PR must not be used to infer that
committed means fully settled.
- Candidate discovery adds bounded, narrow reads at startup and after
signals. No large-fleet query benchmark is claimed.
- Existing Cloud alarm coverage, host sleep safety and resource blockers
are unchanged. No deployed sleep or savings claim is made.
- Parent #17734 has merged. This PR targets master at
`a019d00a393b57f531afe8dd8ab0b8a1820d1c4c`. Fresh current-head CI and
review remain required before merge.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The exact serving model ID and context-window size are not exposed in
this session. No subagent was used for this slice.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 04:53:11 -05:00
a019d00a39 fix(server): wake native workspace finalization from committed work (#17734)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs can accept a result before workspace copy-back finishes.
> - The periodic orphan pass repeatedly tries this recovery, including
cases that require operator action.
> - Copy-back needs exact ownership evidence. A generic timer cannot
grant that authority.
> - This pull request wakes workspace recovery from committed changes
and local owner completion.
> - Saved retries use the existing scheduler. Empty work, completed
barriers, and operator holds become quiet.

## Linked Issues or Issue Description

Refs #17717. This is slice 52 of the recurring-server sleep stack,
targeting master after parent #17717 merged at
`4c56702e06c698cc10d4a86514669aba0f4d9a9b`. Related work: #17709 (status
reassessment), #17687 (unsafe export recovery), and #16478 (workspace
repair holds).

This PR covers native workspace copy-back and validation. Result
commit/projection and resource settlement remain separate recovery
duties. It does not claim full-instance sleep.

## What Changed

- Register a company/run/task-scoped workspace worker. Reconstruct
accepted, unfinished work from existing records. Use saved retry and
lease deadlines.
- Notify from both accepted-result writers (control-plane and PRP
coordinator), workspace receipts, coordinator changes, operator
confirmation, retained workspace repair, and relevant task updates.
Notifications follow the outer transaction.
- Distinguish active local ownership, busy external ownership, pending
Cloud evidence, operator-required evidence, and changed bindings. Only
concrete external ownership/evidence waits receive a scoped retry.
- Preserve advisory locks, signed stop proof, exact binding checks,
physical callback joins, and existing failure policy. Join physical
operations during shutdown.
- Respect the existing heartbeat execution-suppression policy before
discovery and execution; wake suppressed work on settings activation.
- Skip physical workspace recovery in the enabled periodic orphan pass.
Keep direct/startup reconciliation and the remaining result/cleanup
recovery paths.
- Keep routine company-only finalization hints quiet. Ordinary work
requires issue/run scope; explicit settings and owner-release events
keep their company-wide contract.
- Preserve the parent's entire Runner/package tree, profile 80 and all
historical identities. Add the dated obligation, producer, lifecycle and
limitation audit.

## Verification

- Current local source: `4059b24e3ea5c623be54ade18ec9b6ab78eb0289`,
prepared by a normal merge of the current parent. The sole manual
conflict keeps ordinary workspace fanout inside the existing issue/run
guard. All 1,198 parent-only, 15 owned-only and 10 automatic shared
paths are preserved. The complete package tree matches the parent
byte-for-byte.
- Controlled regression: the original workspace-only unscoped fanout
performs three candidate SELECTs for routine company hints. The same
zero-query assertion passes with scoped fanout. A real accepted result
then produces one successful workspace settlement and returns to quiet.
The failed baseline remains in the audit.
- The first complete local group passes **1,693 cases in 14 files**,
including all 19 workspace-worker cases. It covers both accepted-result
producers, physical ownership, export retry/failure, finalization
recovery, arbiter corpus, issue recovery actions, control-plane port,
workspace recovery, heartbeat recovery, startup, retained cleanup and
native execution.
- Separate process-recovery and external-chat response files qualify the
changed recovery/settlement paths without sharing their general test
database with other files. Both pass in full: 410 process-recovery cases
in 162.79 seconds and 273 external-chat cases in 53.79 seconds. The
total is **2,376 distinct passes in 16 complete files**, with no skips.
Normal merge `5c46c52873158529736b8349e4d44658865108ee` adopts the
parent squash with this entire tested child tree unchanged.
- Production server typechecking, build, module boundaries, migration
order and source-profile checks pass. Thirteen dedicated PostgreSQL 17
fixtures use the unchanged migration 413 schema. Embedded-only paths use
the existing recovery helper with fresh per-call fixtures. All temporary
imports are restored byte-exact. No test assertion or timeout is
relaxed.
- Runner validation belongs to the unchanged parent package tree: 130
passing installation/compatibility/transport cases, 22 installed
ACPX/Grok contract cases and 26 native-provider Rust cases. One existing
opt-in closed Pi snapshot case is skipped because its distribution root
is not configured. These are source-scoped checks, with no live provider
qualification claim.
- The owner audit at
`doc/plans/2026-10-26-native-workspace-finalization-sleep-audit.md`
preserves earlier failures and their disposition. Current-head
acceptance: all 57 checks are clean, including protected CI and
security. Greptile gives 5/5 on
`7df5c9c44ea9e171be3c61b4430d3727b10f3337` with no actionable findings.
Earlier scores do not substitute for this result. No Ponytail verdict is
present under the authorized missing-review policy.
- No claim of a full local repository suite, deployed app/database
sleep, expanded Cloud wake coverage or measured cost savings is made.

## Risks

- Supported wake producers are in-process application services. This
adds no cross-replica or independent SQL-writer transport.
- A real unresolved export still needs recovery. Busy owners and pending
Cloud stop evidence retain a 30-second scoped check. No timeout
authorizes copy-back.
- Result projection and resource release still use their existing
recovery. Some existing cleanup failures remain best effort; this PR
does not infer cleanup success from a workspace barrier.
- Cloud alarm coverage and host-sleep safety remain unchanged. No
deployed suspension or cost savings have been measured.
- Parent #17717 is merged. Current base is master at
`4c56702e06c698cc10d4a86514669aba0f4d9a9b`. Fresh current-head CI and
review must pass before this slice merges.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The exact serving model ID and context-window size are not exposed in
this session. No subagent was used for this slice.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 04:40:01 -05:00
4c56702e06 fix(server): schedule retained native-session cleanup (#17717)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs can retain a local session after an interrupted durable
suspend.
> - The orphan reaper repeatedly scans completed history for that
cleanup duty.
> - Idle scans keep the database active. Removing them requires complete
wake and ownership paths.
> - This pull request moves retained-session cleanup to committed
signals, local release events and saved deadlines.
> - Existing physical proof checks still decide whether cleanup is safe.
The shared scheduler does not grant execution authority.
> - The benefit is quiet retained history with recoverable cleanup and
owned shutdown.

## Linked Issues or Issue Description

Refs #17709 and #18135. This is slice 51 of the existing
recurring-server sleep stack. Both referenced dependencies have merged.
The target is `master`.

**What happened?**

The periodic orphan pass scans retained native sessions even after their
cleanup is complete or no automatic action is safe. Some eligible
sessions also depend on a local session, environment lease or original
maintenance callback releasing ownership. An earlier decision
notification can be consumed before the run's later terminal projection
commits.

**Expected behavior**

Reconstruct existing work at startup. Wake from committed writes and
actual owner release. Register the saved claim expiry or a probe for a
specific live process. Stop recurring discovery when no obligation
remains. Preserve original callback ownership and all existing cleanup
proof checks.

**Steps to reproduce**

1. Retain completed native runs with a durable-suspend close failure.
2. Let the periodic orphan pass run with no new work.
3. Observe repeated cleanup discovery. Separately, delay terminal
projection after its decision notification is consumed and observe the
missing cleanup wake.

**Paperclip version or commit**

Parent #17709 at reviewed head
`3b75f4bc1da1370304f069fc287af3d693575716`, landed as
`df6156c2e37f542a69f393dde31e357b82b7cac1`. Current integrated master:
`31b927e3013a8a4b3266854b740cf40a24753788`.

## What Changed

- Move scoped, paged discovery to the retained cleanup owner. Preserve
explicit manual recovery.
- Register the duty with the shared scheduler and remove it from the
periodic orphan pass. Keep concrete lease expiry and live-process probe
obligations.
- Use typed outcomes for settled/refused work and process, session,
environment, maintenance or lease waits. Keep physical proof and
quarantine decisions in the executor.
- Wake from committed finalization and terminal projection writes,
durable process/lease release, and local session or callback settlement.
Routine company-only finalization hints leave dormant cleanup quiet.
- Join duplicate cleanup and serialize inspections on one database lane.
Keep original Runner callbacks owned until settlement and shutdown.
- Preserve all master profile fixtures and the branch's historical
identities. Save the colliding sleep profile 79 separately. Generate
combined Copilot profile 80 from its source and retain master's other
provider profiles.
- Add the owner audit and regression coverage. No new table, migration
or independent maintenance timer is added.

## Verification

- Server acceptance: **2,342 distinct passing cases in 18 complete
files**. The 14-file unaffected group contributes 1,532 passes. Full
startup contributes 76, process recovery 410, environment service 51,
and external-chat response 273. The 29-case cleanup worker suite and
977-case executor suite run in full. No case in these files is skipped.
- Controlled scope regression: removing only the ordinary finalization
issue/run guard wakes dormant retained cleanup; the same assertion
passes after restoring it. A real local owner release still resumes the
durable candidate.
- Preserve the first broad failure: 1,607 passed and one startup
shutdown case failed because the coordinator mock lacked `stop()`. Add
only that mock method. All 76 unchanged startup cases then pass. Do not
treat the first broad command as green.
- Dedicated PostgreSQL 17 databases use the existing migration 413
schema. Embedded-only fixtures use the existing recovery helper with
fresh per-call databases. All temporary imports are restored byte-exact.
No assertion or timeout is relaxed.
- Current-master integration preserves both complete source histories.
The server, UI and database trees are identical to the server-tested
source. All master historical profile fixtures remain byte-exact. The
prepared sleep profile 79 remains independently inspectable.
- Current runtime validation at
`d08f764e3efe75e96929173afe7ee8739121ad9f` passes all 12 commands (22
installed ACPX/Grok contract cases and 26 native-provider Rust cases):
`pnpm install --frozen-lockfile --ignore-scripts`; Copilot/Grok identity
and generated profile checks; Runner and server typechecking; server
build; Copilot/Pi installation, profile compatibility and transport
settlement files (130 passed; one opt-in closed Pi snapshot case skipped
because its distribution root is not configured); installed ACPX
model/reconnect package contract and Grok identity tests; Rust
native-provider backend tests; module boundaries and migration order.
- Earlier 94-case installation/settlement acceptance applies to the
prepared profile 79 source. It is retained as history and does not
substitute for the current profile 80 validation above.
- Failure history and duty-specific evidence remain in
`doc/plans/2026-10-26-retained-native-session-cleanup-sleep-audit.md`.
Exact-current-head acceptance: all 60 checks are clean, including
protected CI and security; Greptile gives 5/5 for
`281d579de0d08e907194b8b790a0597b07d16b7b` with no actionable findings.
No Ponytail review is present under the authorized missing-review
policy.
- These checks do not claim a full local repository suite, live Copilot
or Daytona qualification, deployed host/database suspension, or measured
cost savings.

## Risks

- Missing a supported producer could strand cleanup. Tests exercise real
finalizer, heartbeat, environment and process writers. Process-local
signals cover in-process writers; no multi-replica or external SQL
notification transport is added.
- A timeout can leave an original callback active. Cleanup and shutdown
must keep waiting for it. The existing global maintenance drain is
conservative across roots.
- A retained live process still has a scoped 30-second absence probe. A
saved lease keeps its actual deadline. Operational errors retain retry
work. Immutable proof refusals and operator-required histories do not
gain automatic retry authority.
- Cleanup keeps all existing artifact, identity, claim, no-launch and
quarantine checks. Terminal status alone does not authorize physical
cleanup.
- This PR does not expand Cloud alarm coverage or relax host sleep
eligibility. It proves worker behavior, not deployed app/database
suspension.
- The Copilot source identity has changed; these checks do not claim new
live provider qualification.
- The parent has merged. This PR is merged only after its current-head
protected CI and review gates pass under the existing campaign
authorization.

## Model Used

OpenAI Codex, GPT-6 family, with code execution and repository tools.
The exact serving variant and context-window size are not exposed in
this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 04:24:57 -05:00
df6156c2e3 fix(server): wake native status reassessment from committed changes (#17709)
## Thinking Path

> - Paperclip manages agents, tasks and their durable native results.
> - Changed evidence, task authority or review retirement can require a
new status assessment.
> - Periodic orphan recovery repeatedly reads history to find these
changes.
> - This change gives reassessment an owner driven by committed changes
and saved deadlines.
> - Existing result, assessment, decision and task rows remain durable
truth.
> - The status committer still verifies current sources and authority
before writing.
> - Settled or insufficient evidence becomes quiet until a supported
producer changes it.

## Linked Issues or Issue Description

Refs #17702 (merged), #17679 and #17680. This is slice 50, now based on
master. It owns native status reassessment; workspace settlement,
retained sessions and other orphan duties remain separate.

**What happened?**

Periodic recovery repeatedly inspects completed native results. Removing
that scan without complete writer coverage or current authority checks
could strand new evidence or commit an old result over newer work.

**Expected behavior**

Reassess from existing durable facts after relevant outer transactions
commit. Retain real retry/lease deadlines and restart/drain recovery.
Preserve current authority, quality-review and supersession checks. Make
no recurring SQL when work is settled or dormant.

**Steps to reproduce**

Complete a native result awaiting evidence or review, leave it idle,
then approve a real work product or retire the selected automatic
review. Its saved result should advance once when authorized, or record
the existing nonterminal business gate, without a periodic scan.

## What Changed

- Extract reassessment and its pure status policy from finalization
recovery into their owning modules. Add a scoped worker using the
existing delivery coordinator and scheduler; no queue table or migration
is added.
- Retain the three durable discovery triggers: changed work-product
history, authoritative task status/version, or retirement of the
selected automatic completion review. Preserve selected-source,
ownership, effects, retry and lease fences under the existing committer.
- Notify from actual task/work-product, claim release, approval,
interaction, OAuth, event evidence and workspace-settlement owners
before their transaction writes. Keep provider calls outside short
transactions and nested writes behind outer commit.
- Scope ordinary milestone/finalization/coalesced-input wakes to an
issue/run. Keep genuine company authority restoration explicit. Real
Photon and coalescer regressions cover all three response/status queues.
- Move the current parent's required quality policy into the owner and
apply it to every reassessment that could complete a task. An unmet
review records the existing in-progress decision and next action,
avoiding an operational retry loop. Preserve conversation/session reset
locks and the workspace omission-barrier signal.
- Skip reassessment in periodic recovery while preserving
explicit/startup recovery and the historical supersession guard before
independent workspace work. Document the source contract, producer
inventory, failures and qualifications in the linked audit.

## Verification

- Broad source `9fddb3f0b1157df32b64fb1f22e9d6e9624470e8` passes **2731
distinct cases in 28 complete files**, with no skips: 2458
producer/owner/authority/workspace/quality cases in 394.43 seconds and
all 273 governed-response cases in 53.74 seconds. Nineteen dedicated
fresh PostgreSQL 17 databases migrate through 0413; eighteen
embedded-only invocations preserve fresh-per-call isolation through the
existing recovery helper. All 22 temporary imports restore exactly and
all 28 hashes match.
- Final runtime source `5ea3b0fd016bf7e63ba82a7044825a46a0cd2082` passes
**1211 cases in five complete affected files** in 108.23 seconds after
the narrowly reproduced evidence-trigger fix: all 43 worker, 145 corpus,
32 finalization-recovery, 968 native-session and 23 quality-review
cases. Four fresh databases plus the quality helper's own fresh fixture
use PostgreSQL 17. Its one adapted import restores exactly; all five
hashes match. The other 27 broad-run test files are unchanged. These
qualifications are source-scoped and not cumulatively double-counted.
- Required/optional real regressions reproduce retry loops on
retired-review and evidence-approval paths before repair; both become
SQL quiet afterward with unchanged assertions/time limits. Controlled
removal of only ordinary scope guards reproduces two coarse-hint SQL
failures; exact guard restoration passes. Real producer, outer
transaction, claim, authority, 103-item backlog, retry and drain cases
pass. Photon transport is mocked; callbacks, locks and observed SQL are
real.
- Normal merges preserve 12661 parent/unrelated entries, 35 owned
entries and 21 automatic shared results, with five original conflicts
reviewed explicitly. The parent's actual squash reconciliation preserves
the entire held child tree. The final quality fix changes only its owner
and tests. Fourteen added comments are removed with identical nontrivia
tokens; no directives are removed and the final comment check is empty.
- Database build, server types/build, boundaries and exact-head
migration order pass. The database tree is identical to merged #17702:
no new SQL, queue table or snapshot is generated. Earlier benchmarks,
host limitations, failed assertions and their repairs remain in
`doc/plans/2026-10-26-native-status-reassessment-sleep-audit.md` and
`doc/idle-native-status-reassessment.md`.
- This qualifies targeted external PostgreSQL behavior, not embedded
bootstrap on the semaphore-exhausted host, a full local repository
suite, deployed sleep or measured fleet savings. Fresh exact-head CI and
review remain required.

## Risks

- Notifications retain the process-local supported-writer contract.
Independent SQL writers and uncoordinated replicas are not covered by a
new transport.
- Reassessment competes with current authority changes. Existing
row/source locks and retries remain required; arbitrary missing
identities have no row to lock, so this is not a general predicate-lock
guarantee.
- Discovery pages 100 rows but retains outstanding candidates in memory.
It remains proportional to backlog size, and cancellation occurs between
queries.
- Other orphan/workspace/session duties and Cloud sleep eligibility
remain unchanged. Missing supported producer coverage could strand
dormant work, so the producer audit and real callback/transaction tests
remain part of acceptance.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell
execution and PostgreSQL tests. The runtime does not expose a more
specific serving snapshot or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted scope is stated
above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 03:40:01 -05:00
79ce86630a fix(server): schedule governed review response recovery (#17702)
## Thinking Path

> - Paperclip manages agents, tasks and their external chat responses.
> - A completed chat turn can owe a response while an earlier review
gate remains pending.
> - Periodic recovery revisits that response history even when no work
can advance.
> - This change uses the existing delivery coordinator and saved retry
deadlines.
> - Approval, interaction and authority changes wake the affected issue
or run after commit.
> - The existing presentation owner verifies the durable result and
current authority.
> - Settled and dormant history becomes quiet without losing future
eligible responses.

## Linked Issues or Issue Description

Refs #17696 (merged) and #15625. This is slice 49, now based on master.
It owns governed-review response presentation. Accepted responses and
pending decision effects keep their existing owners.

**What happened?**

Native finalization recovery repeatedly inspects committed review
responses. Removing the scan without covering approval, interaction,
task-policy and authority changes would strand responses that become
eligible later.

**Expected behavior**

Recover an owed response once from existing durable facts. Enforce
authority at publication, wake after relevant outer transactions commit,
retain real retry deadlines and restart/drain recovery, and stop
recurring SQL when no response can advance.

**Steps to reproduce**

Complete an external chat turn while an earlier completion review
remains pending. Block its response behind an approval or interaction,
then resolve that blocker. The response must appear once without a
periodic scan or resolving the original review gate.

## What Changed

- Add a scoped governed-review worker using the existing coordinator,
finalizations, decisions and comment receipts. No queue table or
transaction wrapper is added.
- Keep recovery and publication policy in the native finalizer. Preserve
binding, source/session authority, ownership, deadline and
deleted-comment fences; match coordinator-before-issue lock ordering.
- Notify through actual approval, interaction, OAuth, task-policy and
lifecycle producers before their owned writes. Serialize approval
resolution with task linking so a concurrent link retains its wake.
Nested transactions still settle at outer commit.
- Scope ordinary milestones, finalization and coalesced input to their
issue/run. Use the explicit company authority boundary for genuine
global restoration, including membership changes. Both response workers
share this boundary.
- Register under maintenance and drain gates, disable the two periodic
repair entrances, and retain fresh finalization and explicit recovery.
Add the concurrent 0413 comment-receipt index and migration
registration; retain five snapshots.
- Preserve the producer inventory, source qualifications and earlier
failures in the owner audit. Retain a real Photon regression proving
routine ownership assertions stay quiet and concurrent authority
restoration wakes each worker once.

## Verification

- Scoped repair source `2341714720b5c273f0ce011b48345af0ef39227f` passes
**2053 distinct cases**: 1703 in 15 complete producer/authority/recovery
files, including all 273 governed-response cases and 95 wake-queue
adapter cases, plus 350 selected chat cases. Another 1463 chat cases are
outside that selection. Thirteen fresh PostgreSQL 17 fixtures migrate
through unchanged 0413; the one temporary queue-fixture import restores
exactly. All sixteen captured hashes match.
- Two actual repeated coarse-hint SQL regressions fail before repair and
pass afterward. Actual membership restoration passes before and after.
The same assertions and time limits prove dormant work stays quiet and
real authority restoration still publishes once. Real coalesced input
covers both queues after commit.
- Combined source `586f5674cd9798b33cd22437255e0480b5726d20`, including
independent master #18129, passes **1297 distinct cases**: 1093 in five
complete server files, 185 in five runner files, 14 Node protocol cases
and five UI registry cases. This includes the complete 22-case Photon
file for both response workers. The external Photon transport is mocked;
Paperclip callbacks, transaction fences and observed SQL are real. The
earlier and combined case counts are separate, with no cumulative
double-counting.
- A normal merge retains both complete owned and independent master
binary patches. All twelve new-context hashes and sixteen earlier repair
hashes match. Three dedicated fresh 0413 fixtures plus the live-event
helper's own fresh database use PostgreSQL 17 without source
adaptations. Database build, server types/build, boundaries, full-SHA
migration order and generated TS/Rust profiles pass. The new inherited
Runner vendor inputs rebuild successfully.
- Database bytes remain identical to the previously qualified 0413
source. The recorded populated 0412-to0413 upgrade retains all 411 prior
receipts and 38 companies, appends one receipt and none on replay; the
index is valid and ready. The earlier 2828-case acceptance, CLI journey,
setup failures and their repairs retain their recorded source scope in
`doc/plans/2026-10-26-governed-review-response-sleep-audit.md`.
- Parent #17696 merged with 58 clean checks, Greptile 5/5 and Ponytail
5/5. Its Photon warning was disproved by real callback/SQL evidence and
the security check became successful. This PR still requires its own
fresh exact-head CI and review gates. No deployed sleep, external alarm
or fleet savings claim is made.

## Risks

- Notifications use the existing process-local writer contract.
Independent SQL writers and uncoordinated replicas are outside its
coverage.
- Genuine startup and company authority reconstruction remain
proportional to the owed backlog. Queries page 100 rows, but discovery
accumulates the candidates; this is not a constant-cost global
operation.
- Current publication authorization, durable binding, ownership and lock
ordering remain mandatory. The worker does not resolve the original
review gate.
- The concurrent index uses existing migration replay handling. Other
recovery duties and Cloud sleep eligibility remain separate.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell
execution and PostgreSQL tests. The runtime does not expose a more
specific serving snapshot or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 02:54:46 -05:00
ec3b558277 fix(server): schedule accepted native response recovery (#17696)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs store accepted results and their finalization decisions.
> - A completed run can still owe its public answer after an
interruption.
> - Periodic recovery scans for that answer even when no new work
exists.
> - This pull request wakes recovery from committed changes and saved
retry deadlines.
> - The existing owner still verifies the result, source, and current
publication authority.
> - Settled or dormant response work no longer needs its own periodic
scan.

## Linked Issues or Issue Description

Refs #17690 (merged). This is slice 48 of the sleep migration stack, now
based on master. Related: #17661 handles GitHub terminal publication;
this change handles native accepted-response presentation.
Governed-review presentation remains a separate duty.

**What happened?**

Native finalization recovery repeatedly checks already accepted results
for a missing answer. Removing that check without complete notifications
would lose responses when evidence arrives late, an agent resumes, or
admission settles after execution.

**Expected behavior**

Recover the answer from its saved result when a supported change makes
it eligible. Retain real retry deadlines. Stop querying when the answer
is selected or cannot currently be published. Reconstruct work after
restart.

**Steps to reproduce**

1. Commit a native accepted result with no selected response comment.
2. Run finalization recovery while its source is unavailable or locked.
3. Restore that source or release the lock. The answer must appear once
without repeating provider execution.

## What Changed

- Honor recorded non-create decisions and skip-comment policy in both
discovery and recovery. Apply durable conversation/result/continuation
guards before recovery locks. Preserve genuine unanswered and legacy
recovery.
- Scope routine response wakes to the affected issue/run. Coalesced
input names its active run; pending-effect settlement names its issue.
Sixteen existing global authority/configuration hooks use one explicit
company invalidation boundary. No new transaction wrappers or tables are
added.
- Add an accepted-response worker using the existing delivery
coordinator and scoped deadline helper. Preserve saved retry deadlines
and pending-effect fences under the owner's lock.
- Share the finalization notification boundary. Wake on finalization
settlement, exact accepted-result/final-message evidence, agent
availability, retry admission, and existing chat authority changes.
- Keep lifecycle policy in services. A narrow pre-write status hook lets
lifecycle retain ownership of its transaction and status transitions.
- Register the worker during enabled maintenance startup. Remove its
authority subscription on shutdown. Periodic replay skips this duty on
both committed and fresh-commit paths; ordinary finalization and
explicit recovery retain their behavior.
- Add a partial index and expression statistics for unpresented native
runs. Migration 0412 supports concurrent replay and adds no queue table.
- Add real PostgreSQL and lifecycle coverage. Preserve the
implementation and producer audit in the plan document.

## Verification

- Additional real Photon callback acceptance on the published production
code disproves the security warning: ten active assertions cause zero
response-recovery hints/SELECTs, while three concurrent assertions
during an actual attention-to-active recovery cause exactly one
wake/SELECT. The existing locked status guard already enforces this
boundary. No production change was needed; the resolved finding now has
a successful current-head security check. The regression will be
retained with the next slice.
- Eligibility repair source `892f36e5f66ba1af123034936a7160c64c3ab883`
passes **1489 distinct server cases**: 1139 in six complete
recovery/finalizer/monitor/summary/workspace suites (81.02 seconds),
plus 350 selected chat cases (297.68s). Another 1463 chat cases are
outside the pattern; the complete 1813-case file is not claimed. All
eleven new regressions are included in that count. Five dedicated fresh
PostgreSQL 17 databases migrate through unchanged 0412, with no
fixture-source adaptations. Database build, server types/build,
boundaries and full-SHA migration ordering pass. All seven test source
hashes match.
- The exact published baseline first fails eight behavioral assertions
while the three genuine-answer cases pass. The same corrected test
source passes all eleven after repair: recorded none/reuse remain
suppressed; five ineligible history families take zero recovery row
locks across repeated issue hints; missing/legacy/create-null answers
still publish once; fresh periodic finalization defers presentation to
its worker. Earlier fixture setup and variable/compiler failures remain
explained in the audit, rather than erased by later success.
- Repair source `2fc8b0614aaf9ac7337045de4ced7325f95edc45` passes **1821
server cases**: 1496 in 13 complete producer/lifecycle/authority suites
(110.63 seconds), plus 325 selected chat cases (278.99 seconds).
Coverage includes all native-session cases, queue admission, status
decisions, membership/board access, the complete GitHub workflow,
accepted-response lifecycle and real authority restoration. The other
1477 chat cases are outside this selection; the complete 1802-case file
is not claimed for this repair run.
- Actual PostgreSQL regressions first reproduce a coalesced-input hint
missing its run scope and two routine hint paths producing 12
response-history SELECTs each after three notifications. The same
assertions pass with run-scoped coalescing and zero response-history
SELECTs for unscoped routine hints. Actual membership restoration still
publishes the waiting response once. Assertion time limits are
unchanged; this measures real driver SQL and persisted producer work.
- Eleven dedicated fresh PostgreSQL 17 databases migrate through 0412.
One embedded-only wake-queue fixture import temporarily uses its own
external database and restores byte for byte. All 14 selected source
files match pre-run hashes. This qualifies external-database behavior
rather than reproducing embedded CI locally.
- Master advanced independently with #18126's Pi archive retry fix.
Normal merge `2a535bc61259ef99387369e8b32498f774e5ba2c` preserves both
complete binary patches. All accepted server/database/UI/test/CLI trees
stay identical; the three independent runner installer/manifest files
are retained. All 13 installer tests pass, including stalled header/body
cancellation. Combined server types/build, boundaries and full-SHA
migration order pass. A subsequent normal merge of #18128 at
`e8b78cce5bb557b086bee9ca930eeaab1d09b244` retains its three test-only
file changes with both complete patches preserved. The changed HTTPS
backfill case passes on a fresh PostgreSQL fixture (1 selected case; 178
outside the pattern), and combined types/build, boundaries and migration
order pass again. All 14 repair source hashes remain unchanged. The
independent browser fixture changes are preserved; no local browser run
is claimed.
- The earlier source `9feec170ef8174011fc3c5d3d6d76f0a0f39fea0` passed
the 4171-case full baseline before this scope repair. That scope is
preserved in the audit rather than represented as a current full rerun.
All 0412 database bytes remain unchanged: original concurrent
index/statistics/ANALYZE, prior executable history, journal entries and
retained snapshots. Five snapshots 0408 through 0412 remain. The
populated 0411-to0412 upgrade preserved 410 prior receipts and 38
companies, appended one receipt and none on replay; index/statistic
checks passed.
- The earlier P2 security failure, Greptile's former conditional
dependency on #17702, and the subsequent actual Ponytail eligibility
findings are recorded in
`doc/plans/2026-10-25-native-response-presentation-sleep-audit.md`. This
PR now includes the issue-scoped pending-effect notification fix itself,
so it is independently correct before #17702. Fresh published-head CI,
Greptile 5/5 and a clean security scan remain required. No
deployed-sleep, external-wake or fleet-savings claim is made.


## Risks

- The writer contract is process-local. Independent SQL writers and
other replicas are not covered by these notifications.
- Unpresented backlog reconstruction still has memory and time cost
proportional to candidate count. Pages contain at most 100 rows;
execution uses the shared fair batch. Selected history is excluded.
Health-only signals do not trigger reconstruction.
- Publication policy remains with its current owner. Digest, company/run
binding, source authorization, selected-answer suppression, and
ownership fences remain required.
- The rest of native recovery and governed-review presentation still run
separately. This PR does not relax Cloud sleep eligibility or add
external wake coverage.
- The stack parent has merged and its actual ancestry is reconciled.
This publication still needs fresh current-head CI and review.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell
execution, and PostgreSQL tests. The runtime does not expose a more
specific model identifier or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 02:31:20 -05:00
a8a038e659 fix(server): schedule native pending effect recovery (#17690)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native status decisions store effect targets and a delivery ledger.
> - Historical pending or failed effects still need recovery.
> - The periodic finalizer queries that ledger even when work is
complete.
> - This pull request gives pending effects their own commit-driven
worker and saved deadlines.
> - Completed history becomes quiet while incomplete effects remain
recoverable.

## Linked Issues or Issue Description

Refs #17687 (merged). This is slice 47 of the sleep migration stack, now
based on master.

**What problem does this solve?**

Periodic native finalization repeatedly checks effect delivery. A tenant
with completed history should not perform recurring SQL for this duty.
Historical unresolved effects must still recover without duplicating
their targets.

**Proposed solution**

Reconstruct current pending effects at startup. Wake from committed
coordinator changes. Use the saved retry and ownership deadlines. Keep
the existing dispatcher as the only recovery mutation owner.

**Alternatives considered**

A slower unconditional scan still creates idle traffic. A new queue
table duplicates existing durable state. Replaying materialization can
duplicate effects. This change uses the existing ledger, transaction
signals, and shared scheduler.

## What Changed

- Add a scoped worker with 100-row discovery pages and fair 25-entry
dispatch batches. Delivered, superseded, and terminal-failure history
has no recurring scan.
- Preserve exact company/issue/run/decision binding and validate
materialized targets before settling effects. Recheck phase, saved
result, retry, and lease under the coordinator lock. Missing targets
retain recovery.
- Notify from decision replay, accepted-result recovery, coordinator
claim/retry/release, attention replay, and execution-control recovery.
Only two DB-only producer writes need a new transaction wrapper. Nested
transactions use the existing outer-commit boundary.
- Exclude pending-effect candidates from periodic finalization so it
cannot skip ahead of their owner. Keep explicit and startup recovery
behavior compatible. Respect maintenance disablement, drain, standby,
and shutdown.
- Add concurrent, idempotent migration 0411 with partial indexes for
pending effects and their current coordinator binding. Retain the five
newest schema snapshots.
- Document producers, query evidence, failure history, and scope in
`doc/plans/2026-10-25-native-pending-effects-sleep-audit.md`.

## Verification

- Source `ba38ee465df02b6026286b5d5e15ae49fe07a367` passes all 2284
distinct cases across 25 complete files, without skips. The main
selection passes 2225 in 203.94 seconds; safe-replacement passes 59 in
26.03 seconds. Coverage includes all 968 native-session cases, the
94-case status-authority/pending-effect corpus, actual decision/lease
producers, Stop, queue, interaction, reconstruction and Slack consumers.
The audit gives per-file counts.
- The complete 94-case corpus passes again on the same database in 16.03
seconds. Retained PSC/OUT rows and an unrelated sentinel remain
unchanged. The repeat is not counted again as distinct coverage.
- Twenty-one separate fresh PostgreSQL 17 databases migrate through 0411
before setup. Four embedded-only fixture imports temporarily use their
own external database on this IPC-exhausted host and are restored byte
for byte. Assertions and timing limits stay unchanged. This qualifies
external-database behavior; it does not claim an unmodified embedded CI
run.
- Database build, server typecheck/build, module boundaries and
exact-head migration order pass. Original runtime/schema edits and
unrelated upstream blobs are preserved. Both original concurrent index
statements, all prior executable history, journal entries and retained
snapshot bytes are unchanged; five snapshots 0407 through 0411 remain.
- Populated 0410-to0411 upgrade preserves all 409 prior migration
receipts and 38 companies, appends one receipt and adds none on replay.
Both indexes are valid and ready on fresh and upgraded databases.
- Adopting the latest prepared prerequisite changes ancestry and parent
audits only. All accepted core trees remain identical. The audit retains
earlier failures and qualifications. Actual landed-parent ancestry is
now reconciled; fresh published-head CI and exact-head Greptile 5/5
remain required. No deployed sleep, Cloud-wake or fleet-cost claim is
made.

- Current parent context: integration
`5f7cd7bc589ad0169dfb5f33df611ead56510ca3` changes only the parent's
GitHub observation bound and independent cleanup assertion. All
production code, packages and 25 accepted test files remain unchanged.
Both focused GitHub idle cases pass in 15.26 seconds and the complete
six-case cleanup suite passes in 8.47 seconds on two fresh 0411
PostgreSQL fixtures. This does not claim the full parent workflow was
rerun here.

## Risks

- A missing producer can delay work until restart. The audit covers
current supported writers. Independent SQL and other processes remain
outside the local notification contract.
- Recovery validates existing targets; it cannot recreate a missing
target or silently mark it delivered. Such work retains an error and
retry.
- The indexes add write/storage cost and use the existing restart-safe
concurrent migration runner.
- Other finalization duties still poll until their separate migrations.
This PR does not prove host/database suspension, extend Cloud alarms, or
claim fleet savings.
- The actual prerequisite squash is adopted through a normal merge. Its
complete tree equals the reviewed parent, and the complete accepted
child tree is preserved. Fresh published-head CI and review remain
required.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning and code-execution tools.
The exact serving model ID and context-window size are not exposed in
this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 00:28:00 -05:00
DottaandPaperclip 4e715eb7ea fix(server): schedule historical unsafe export recovery (#17687)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs can retain accepted results after an unsafe workspace
export fails.
> - Historical recovery can retire an old notice or resume the accepted
result without another provider turn.
> - The periodic reaper repeatedly scans failures that cannot currently
recover.
> - Existing rows and committed changes can identify the actual recovery
obligation.
> - This pull request wakes recovery from those changes and retries only
owed work.
> - Dormant history becomes quiet while ownership and explicit operator
decisions remain protected.

## Linked Issues or Issue Description

**What happened?**

The native finalization reaper repeatedly visits historical unsafe
workspace-export failures. Cancelled tasks, newer runs, changed
contracts, and explicit operator decisions can make those rows
permanently ineligible. They still produce periodic database reads.

**What did you expect?**

Retire stale unsafe-export notices and recover eligible accepted
results. Stop reading dormant history until a relevant committed change
occurs. Keep a retry while a real obligation is blocked by ownership or
a transient failure.

**Steps to reproduce**

Retain an unsafe-export terminal failure whose task is cancelled or
superseded. Leave the instance idle after startup. The periodic reaper
continues reading that history.

Prerequisite #17680 is merged into master. Related: #14402 established
safe export finalization; #14329 was an earlier historical recovery
proposal. #17679 owns separate policy-review cleanup. This PR changes
only historical unsafe-export scheduling and its producers.

## What Changed

- Register a scoped worker through the existing coordinator and
scheduler. Remove this duty from periodic reaping. Preserve the
manual/startup recovery entry point.
- Distinguish stale notices, eligible accepted results, dormant history,
and concrete ownership contention. Reuse the existing mutation owner and
lock order. Preserve explicit settlement vetoes and exact
result/contract identity.
- Signal relevant task, run, action, and deletion writes before they
commit. Existing transaction boundaries handle nested commits and
uncertain outcomes. Notifications contain only scope IDs.
- Retain a retry only for owed work. Reconstruct work at startup,
preserve drain/resume, and stop through the shared lifecycle.
- Add concurrent, idempotent partial index migration 0410 for unsafe
terminal failures. Keep the five newest generated schema snapshots.
- Document producer coverage, query plans, supported writer boundaries,
and test failures in the unsafe-export audit.

## Verification

- Source `2a13dd08a6cc52f0adab09e3df9ce31e97b746b9` passes all 2188
cases across 24 complete files, without skips. The main selection passes
2129 cases in 192.58 seconds; safe-replacement passes 59 in 26.04
seconds. Coverage includes all 968 native-session cases, real
export/action producers, explicit continuation, workspace
reconstruction, interaction routing, Stop, queue and Slack consumers.
The audit gives per-file counts.
- Twenty separate fresh PostgreSQL 17 databases migrate through 0410
before setup. Four embedded-only fixture imports temporarily use their
own external database on this IPC-exhausted host, then are restored byte
for byte. Assertions and timing limits stay unchanged. This is qualified
external-database acceptance, not an unmodified embedded CI run.
- Database build, server typecheck/build, module boundaries and
exact-head migration order pass. Normal integration preserves the
complete original runtime/schema patch and unrelated upstream blobs.
Original concurrent index SQL, all prior executable history, journal
entries and retained snapshot bytes are unchanged; five snapshots 0406
through 0410 remain. Added comment removal preserves the full
non-comment TypeScript token streams.
- A populated 0409-to0410 upgrade preserves all 408 prior migration
receipts and 38 companies. It appends exactly one receipt; replay adds
none. The index is valid and ready on fresh and upgraded databases.
- The owning audit retains earlier fixture failures, skipped checks and
notice-retirement regression history. Older combined stack checks retain
their original scope in their owning audits. Actual parent
`73d24aa5acfcf17b3f676b4da7cd05eb70b60b99` is reconciled through a
normal merge. Its tree equals the reviewed ancestor and the complete
accepted child tree is unchanged. Fresh published-head CI and Greptile
5/5 remain required. No deployed sleep, fleet-cost or live provider
qualification claim is made.

- Inherited prefix correction: current integration
`8f43c9f81fac23b50997e7f0e1d0d9cf383a8109` changes only two values in a
previously unselected corpus fixture. All production code, the complete
package tree and all 24 accepted test files remain byte-identical to the
2188-case source above. The additional complete 72-case corpus passes in
14.00 seconds and again on the same database in 13.08 seconds. Retained
PSC/OUT and sentinel rows remain unchanged. The repeat is not
double-counted. The parent audit retains the original collision and
rejected cleanup attempt.

- Latest parent context: integration
`4d51231abe8b68c21ce896d0d2ac16865978c8ab` carries only the parent's
one-argument GitHub observation correction and an independent master
cleanup assertion. All production code, packages, 24 accepted suite
files and the additional corpus are unchanged. The two focused
native-presentation cases pass in 15.14 seconds and the complete
cleanup-removal suite passes all six cases in 8.31 seconds on two fresh
0410 PostgreSQL fixtures. This does not claim the full parent workflow
was rerun here.

Current head `b28399babac72ba9db1c4a4ef882c5e87aded38c` has passing CI
and Greptile 5/5 with no actionable findings. The first CI attempt lost
runners: fourteen jobs explicitly received simultaneous shutdown
signals, two reported lost runner communication, and two aggregate gates
failed downstream. One failed-job retry on unchanged source passes. The
underlying host shutdown cause remains unconfirmed; no code fix or extra
retry is claimed. [CI run and
attempts](https://github.com/paperclipai/paperclip/actions/runs/40819408528).
No Ponytail verdict is present.

## Risks

- A missing future producer can delay dormant-history discovery until
restart. The audit names current producers. Independent SQL writers and
other server processes remain outside the process-local signal contract.
- Physical ownership and uncertain operations retain retries. Time alone
does not authorize taking workspace ownership. No provider turn or
unsafe archive retry is added.
- The recovery owner remains authoritative if state changes after
discovery. Terminal UI status does not authorize resource destruction or
accepted-result replay.
- The partial index adds matching-row write/storage cost. It builds
concurrently through the existing restart-safe migration runner.
- This proves the changed worker becomes quiet. It does not prove host
suspension, external wake coverage, or fleet cost savings. Other sleep
migrations remain. This slice still requires fresh current-head CI and
reviews.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning and code-execution tools.
The exact serving model ID and context-window size are not exposed in
this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-29 00:13:22 -05:00
DottaandPaperclip 73d24aa5ac fix(server): schedule automatic completion review retirement (#17680)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native finalization preserves accepted work and resolves task
status.
> - A historical cleanup removes automatic completion-review cards that
current policy no longer creates.
> - The periodic finalization pass repeats that lookup even after all
matching cards are gone.
> - This PR gives this cleanup its own committed hints and pending-card
retries.
> - The existing owner retains withdrawal receipts and protects genuine
reviews.
> - Idle history stops causing recurring cleanup reads.

## Linked Issues or Issue Description

Refs #17679. This is a separate stacked slice for automatic
completion-review retirement. Obsolete version-change reviews belong to
the parent PR.

**What needs to be improved?**

`reconcileNativeFinalizations` repeatedly discovers old automatic
completion reviews. The discovery helper also converts SQL errors to an
empty result, which would let a new event-driven worker forget failed
reconstruction.

**Why is this improvement important?**

Idle tenants should not query for absent cleanup work. Failed discovery
must preserve the retry, and removing a historical card must preserve
the durable trigger for later evidence reassessment.

**Proposed solution**

Use the existing card, decision, effect, assessment and contract rows.
Reconstruct candidates at startup, signal the existing decision commit,
and retry exact pending cards. Keep the direct cleanup call inside
active finalization. ROADMAP.md and the related native review work were
inspected; this change owns the scheduling migration only.

## What Changed

- Add automatic completion-review registration to the shared scoped
deadline worker. Reconstruct 100-row keyset pages and dispatch fair
batches of twenty.
- Signal company/issue IDs before the native decision transaction
writes. Keep outer-transaction settlement with the existing work-signal
boundary.
- Split raw discovery from the public helper's compatibility error
handling. Worker lookup failures propagate so the shared scheduler
retains dirty scope and retry.
- Reuse the complete historical predicate and existing withdrawal owner.
Recheck the selected card identity, payload and system author fields in
the conditional update.
- Preserve the withdrawal reason, chat publication, activity record,
direct finalizer call, startup/manual recovery, and subsequent evidence
reassessment.
- Skip only this duty in periodic reaping. Retain admission,
drain/standby, retry and joined shutdown.
- Add producer/lifecycle documentation. Add no table, schema migration,
provider branch or independent timer.

## Verification

- Current source `3f4d35ac0154aedc76cc39486d3758c73b1e6ff4`: all 462
cases across seven complete files pass with no skips in 49.31 seconds.
This covers both review workers through the real status-authority
corpus, startup/timing, recovery, finalization, external chat and
governed milestones. Nested corpus fixtures are not double-counted.
- Five separate fresh PostgreSQL fixtures migrated through parent 0409
before setup. Database build, server typecheck/build, module boundaries
and target migration order pass. The whole database source tree matches
the parent; no migration or table is added.
- Normal integration preserves substantive owned edits and unrelated
upstream blobs. Actual landed-parent ancestry and fresh current-head
Greptile 5/5 and clean CI remain required. This is worker and
integration acceptance, not deployed sleep qualification.

- The inherited P2 fix changes only two fixture prefix values. All
production code, other fixtures, assertions and timing limits are
unchanged. The full 72-case corpus also passes again on the same
database in 12.95 seconds. Retained PSC/OUT rows and an unrelated
sentinel remain unchanged after both runs. The repeat is not
double-counted among the 462 distinct cases. The parent audit retains
the original collision and rejected cleanup attempt.

- CI correction: the 40f head fails the passive native-presentation
initial idle observation in chat shard 1. An unchanged local pair
passes. A controlled committed hint plus a 600 ms held run attempt makes
the original one-second observation fail; the identical probe passes
with the established five-second observation bound. This demonstrates
the coalescing boundary, not the unrecorded exact CI token timeline. No
unchanged-head CI retry was used.
- The probe is removed. The only permanent code change is that initial
poll bound. It still requires zero active work and no deadline, followed
by two minutes with no SQL, transaction or provider writes. Production
code, worker/business deadlines, rollback and committed-settlement
assertions are unchanged. All 274 cases in the named GitHub agent-review
workflow pass in 232.06 seconds. Other cases in the large file are
outside that named selection. The earlier 462-case selection retains
identical source. Both owning audits preserve the failure and scope
limits. Fresh updated-head CI and reviews remain required.

- Master integration: independent master
`49fe8b87a49aa5a8985a08dd3d5ce57bdd24756c` adds one company-skill
cleanup assertion. A normal merge preserves both complete binary
patches. All production code and the accepted GitHub integration file
are unchanged. The full six-case cleanup-removal suite passes in 8.16
seconds on fresh PostgreSQL. Database history is unchanged.

## Risks

- A normal native decision can cause one scoped lookup even when it
creates no historical card. It retains no periodic scan when settled.
- Startup reconstruction cost scales with historical data and joins. No
constant-cost or fleet-savings claim is made.
- Process-local signals cover instrumented writers. Independent SQL
writers and concurrent older controllers need explicit integration.
- Retirement and status reassessment are distinct obligations. The
existing durable withdrawal receipt and remaining finalization owner are
preserved.
- Failed withdrawals retain retries while exact pending cards remain. A
database lookup failure must not be interpreted as completion.
- Other finalization and recovery duties remain. Cloud wake coverage and
host-sleep eligibility are unchanged.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL
tests. The runtime does not expose a more precise model ID or
context-window size for this turn.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Remote acceptance: head `be668fb975f119efeb4830405d4bb1ab65791f87` has
53 successful gates, verified with `gh pr checks` (no non-passing
checks). Greptile reviewed that head at 5/5. There are no unresolved
review threads. The PR remains draft and unmerged.

Current published head `181bf800a79449961e09c995f2785dd7f8060bb3` has
passing CI, fresh Greptile 5/5 and Ponytail 5/5 with no actionable
findings. Earlier failures remain recorded in the audits.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 23:40:45 -05:00
DottaandPaperclip dfd5b1ac62 fix(server): wake obsolete policy review cleanup from commits (#17679)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native finalization repairs saved results and related task state.
> - Its periodic pass also searches for obsolete policy-review cards.
> - Those cards have a precise historical signature and existing durable
rows.
> - This PR gives that cleanup its own committed signals and
pending-work retries.
> - The existing owner still protects real reviews and later task
changes.
> - Settled review history no longer causes repeated cleanup queries.

## Linked Issues or Issue Description

Refs #17677. This PR is stacked on cancelled workspace preservation. It
owns only obsolete policy-review cleanup within the native finalization
reconciler.

**What needs to be improved?**

Every periodic native finalization pass queries for old version-change
review cards, even after all such cards are gone. The parent reaper
combines this cleanup with unrelated finalization and recovery duties.

**Why is this improvement important?**

Empty queries prevent idle tenants from becoming quiet. Removing the
query must preserve historical repair, real review requests, task status
authority, and cleanup after a failed write.

**Proposed solution**

Reconstruct matching pending cards at startup. Signal the existing
native decision transaction before its writes. Schedule bounded cleanup
only while an exact card remains pending. Skip this duty in periodic
reaping while keeping startup and manual entry points. ROADMAP.md and
related PR search found no duplicate of this migration.

## What Changed

- Register obsolete-review cleanup with the existing scoped deadline
worker. Reconstruct in 100-row keyset pages and process fair batches of
twenty.
- Use company/issue hints from the existing status-decision commit
boundary. Serialize only the two IDs; preserve outer commit and
rollback.
- Reuse the exact historical signature and existing cleanup owner. Keep
status-version, decision-pointer, separate-review, activity and
chat-publication guards.
- Recheck selected payload and system-card identity in the conditional
withdrawal so a concurrent edit cannot be cancelled from a stale
snapshot.
- Retain a 30-second retry only while a matching pending card survives.
Preserve drain, standby, startup and joined shutdown.
- Skip only this duty in the periodic finalization pass. Automatic
completion reviews, unsafe exports, status effects, evidence
reassessment and workspace finalization remain separate migrations.
- Add the owner audit and validated external-test database support for
the existing status-authority corpus. Add no schema, table, independent
timer or provider branch.

## Verification

- Current source `4cc0d03ae96bd92a4fdad44168c2a8647d42466c`: all 443
cases across seven complete files pass with no skips in 46.46 seconds.
This covers the full status-authority corpus and cleanup worker,
startup/timing, heartbeat recovery, native finalization, external chat
and governed milestones. The audit gives per-file counts without
double-counting fixtures inside the corpus matrix.
- Five distinct fresh PostgreSQL fixtures migrated through parent 0409
before setup. Database build, server typecheck/build, module boundaries
and target migration order pass. The whole database source tree matches
the parent; no migration or table is added.
- Normal integration preserves substantive owned edits and unrelated
upstream blobs. The earlier circular notification-payload failure
remains in the audit. Actual parent
`51c3f8924cb0bd5fe7cbda15ad9b8f06a6a11055` is reconciled through a
normal merge, with its full tree equal to the reviewed ancestor and the
entire accepted child tree unchanged. Fresh current-head Greptile 5/5
and clean CI are still required. No deployed sleep qualification is
claimed.

- Review correction: the old head reproduces a fixed-prefix collision on
a same-database corpus rerun. The initial deletion attempt passed
assertions but failed teardown on non-cascading audit foreign keys; it
was removed. The final two-value change uses each fixture company UUID
in its prefix and changes no production behavior, assertions, timing
limits or teardown.
- The complete 56-case corpus passes a second time on the same database
in 10.15 seconds. Retained PSC/OUT rows and an unrelated sentinel
company remain unchanged after both runs. The repeat is not
double-counted among the 443 distinct cases. The audit retains both
failures and their causes. Fresh CI and Greptile are required for the
updated head.

## Risks

- Startup reconstruction still scans historical candidates and joins;
its cost scales with history. Scoped commits use existing indexes. No
recurring query remains after this worker settles.
- The signal covers instrumented single-process writes. Independent SQL
writers and concurrent older controllers need explicit integration.
- A normal native decision can cause one scoped lookup, even when it
creates no obsolete review. It does not retain a periodic scan.
- Failed withdrawals retain retries while matching cards remain. Actual
cleanup still uses the existing locking, authority and activity owner.
- Other finalization duties still run. This PR does not establish
whole-instance quiet, external alarm coverage, provider suspension or
fleet savings.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL
tests. The runtime does not expose a more precise model ID or
context-window size for this turn.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Current published head `9aa67199da64d700fd6230829d38f2ef522a07fe` has
fresh Greptile 5/5, Ponytail 5/5 with an unconditional ship verdict, and
all current CI checks passing or skipped. The reproduced rerun P2 is
fixed and its thread is resolved.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 22:53:19 -05:00
DottaandPaperclip 51c3f8924c fix(server): schedule cancelled workspace preservation (#17677)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A cancelled native run may still owe preservation of prepared
workspace files.
> - The existing recovery receipt and source lease record that
obligation.
> - The orphan reaper still searches for these cancellations on every
pass.
> - This PR registers committed cancellations with the shared deadline
worker.
> - The existing retention owner keeps file pinning and ownership
checks.
> - Idle and completed history no longer causes this preservation scan.

## Linked Issues or Issue Description

Refs #17674 and #17145. Stopped instruction collection is merged into
master. It owns file-preservation discovery only. The existing
environment cleanup worker continues to own stop-only teardown.

**What needs to be improved?**

`preserveCancelledNativeWorkspaces` runs from every orphan-reaper pass.
It queries even when no cancelled native workspace needs a file-recovery
hold. Current foreground Stop can also defer retention after a lock or
database failure, so startup-only recovery would be insufficient.

**Why is this improvement important?**

Empty queries keep idle tenants active. Removing them must not lose user
files, weaken Stop ownership, replay cancelled work, or turn a retained
allocation into destructive cleanup.

**Proposed solution**

Register the existing durable obligation from committed run-status,
cancellation-acknowledgement and workspace-reference writes. Reconstruct
it at startup. Keep retries while actual preservation remains
unresolved. Disarm after the existing owner records the hold. Related-PR
search and ROADMAP.md review found no duplicate of this scheduling
migration.

## What Changed

- Add a company/run-scoped preservation worker with 100-row keyset
reconstruction and fair batches of five.
- Separate pending file debt from immediate coordinator/lease readiness.
Retain a 30-second retry for known unfinished preservation and shared
failure backoff for errors.
- Signal the existing status, acknowledgement, workspace-reference,
deletion and settings transactions. Add no table or independent timer.
- Keep exact source checks, no-workspace contracts, task/coordinator/run
lock order, retention receipt and no-replay recovery card with their
owners.
- Recheck cancellation acknowledgement, run status and coordinator phase
under the retention locks. Preserve the existing foreground preterminal
API.
- Keep startup/manual recovery; skip only this duty in periodic reaping.
Preserve drain/standby and shutdown behavior.
- Add concurrent, replay-safe partial index migration 0409 and retain
the five newest snapshots.
- Add the producer and lifecycle audit in
`doc/plans/2026-10-25-cancelled-workspace-preservation-sleep-audit.md`.

## Verification

- Current source `21b46a8e4293ca670235117458b984493a181759`: all 1472
cases in 16 complete files pass without skips. The owner passes 84 cases
in 14.93 seconds. Fifteen consumer files pass 1388 cases in 133.46
seconds, including all 968 native-session cases and Stop, dispatch,
restart, workspace-sync and upgrade-readiness coverage.
- Twelve separate fresh PostgreSQL 17 databases migrated through 0409
before setup. The embedded-only lifecycle fixture temporarily used its
own external database on this IPC-exhausted host and was restored byte
for byte. Assertions and timing limits stayed unchanged. This qualifies
external-database behavior; it does not claim an unmodified embedded CI
run.
- Database build, server typecheck/build, module boundaries and
exact-head migration order pass. The complete original runtime/schema
patch and unrelated current-parent blobs are preserved. Original
concurrent index SQL, all earlier executable history and retained
snapshot bytes are unchanged; five snapshots 0405 through 0409 remain.
- A populated 0408-to0409 upgrade preserves all 407 prior receipts and
38 companies. Exactly one receipt is appended; replay adds none. The
preservation index is valid and ready on fresh and upgraded databases.
- Actual parent `82529741418e138d9d623bf13d2fa22c1ff79d36` is merged
through a normal ancestry reconciliation. Its complete tree equals the
reviewed parent. The entire accepted child tree and all source hashes
are preserved. The audit retains earlier failures, qualifications and
per-file counts. Published head
`b34f0589847f33754ecde9413fd36bf007353d2c` has Greptile 5/5 and all 57
checks passing or skipped. No Ponytail review is present. These checks
do not establish deployed host/database suspension or Cloud alarm
delivery.

## Risks

- Signals cover instrumented single-process writers. Independent SQL
writers and additional replicas need integration.
- Status signals may inspect a run without relevant file debt once. They
do not create recurring work. Discovery and memory still scale with
actual candidates.
- Missing ownership or stop evidence remains pending. Active retries are
intentional while a concrete file obligation is unresolved.
- Retry timing is process-local; startup reconstructs durable debt. The
worker does not overwrite another native-finalizer deadline.
- Admitted database operations may finish after drain begins. Shutdown
joins them. Provider teardown remains with the existing cleanup owner.
- The partial index adds storage and write cost. Its concurrent
migration uses the existing interrupted-index repair path.
- Cloud alarm coverage, sleep eligibility and host suspension are
unchanged. This proves worker behavior, not fleet savings.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL
tests. The runtime does not expose a more precise model ID or
context-window size for this turn.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 22:27:11 -05:00
DottaandPaperclip 8252974141 fix(server): schedule stopped instruction collection (#17674)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Runs can stop while their private instruction edits still need
collection.
> - Existing copy rows and stop evidence describe that unfinished work.
> - The orphan reaper still queries for it on every maintenance pass.
> - This PR activates collection from committed copy and run
transitions.
> - It uses saved retry deadlines when stop proof or collection remains
pending.
> - Active copies and completed history no longer cause collection
polling.

## Linked Issues or Issue Description

Refs #17671 and #17670. Both prerequisites are merged into master.
Captured saves and post-save release remain separate reasons with their
own workers.

**What needs to be improved?**

`recoverStopped` queries instruction copies on every orphan-reaper pass,
even when no terminal run owes collection. A run can become terminal
without changing its copy row, so copy-only notifications would miss
real work.

**Why is this improvement important?**

Repeated empty queries prevent idle tenant instances from becoming
quiet. Removing them must preserve edits, stop proof, current ownership,
and recovery after restart.

**Proposed solution**

Use the existing transaction signals, scoped deadline worker, and
scheduler. Reconstruct eligible copies at startup, signal supported copy
and terminal-run writers, and retain saved retry deadlines for
unresolved work. Preserve the existing collection and directory owners.

Related PR search found the warm agent-file ownership work (#14695),
verified stop boundary (#14311), and task-workspace cleanup (#16464).
This PR does not replace those contracts. ROADMAP.md has no duplicate
migration.

## What Changed

- Add scoped stopped-instruction collection with keyset reconstruction
and fair batches of 20.
- Signal before copy preparation, mutation and adoption writes, and
before terminal run writes in their existing transactions.
- Cover heartbeat, native finalization/restart/session recovery,
execution-control, queue and dispatch producers.
- Signal settings changes and company/agent deletion. Preserve drain,
standby, restart and shutdown ownership.
- Recheck terminal-run eligibility in the due-time claim. Keep terminal
status separate from process-stop proof.
- Retain existing stop checks, attempt limits, saved candidate bytes and
remote-resource fencing.
- Skip only this lane in periodic orphan recovery. Preserve
manual/startup recovery.
- Add concurrent, replay-safe partial index 0408 and retain the five
newest snapshots. Add no table.
- Record the owner and producer audit in
`doc/plans/2026-10-25-stopped-instruction-collection-sleep-audit.md`.

## Verification

- Source `cb2818a8f46ef961f744977a258f0e9042e31a42`: all 1469 cases
across 14 complete files pass without skips in 141.69 seconds. This
includes all 968 native-session cases, filesystem saves, startup,
terminal-run producers, dispatch, queue, execution-control, finalization
and lifecycle.
- Eleven separate fresh PostgreSQL 17 databases migrated through 0408.
The embedded-only lifecycle fixture temporarily used its own external
database on this IPC-exhausted host and was restored byte for byte.
Assertions and timing limits stayed unchanged. This qualifies
external-database behavior; it does not claim an unmodified embedded CI
run.
- Database build, server typecheck/build, module boundaries and
exact-current migration order pass. The original concurrent index SQL is
unchanged. All prior executable migration history, journal entries and
retained snapshot bytes are preserved; the five newest snapshots are
0404 through 0408.
- A populated 0407-to0408 upgrade preserves all 406 prior migration
receipts and 38 companies. It appends exactly one receipt. Replay adds
none. The collection index is valid and ready on fresh and upgraded
databases.
- The normal merge of actual parent
`267842fc3c6e6542deed8a1bcc2c78bec8df7862` preserves both complete
patches, all independent sidebar changes from #18117, and the full
accepted server/package/test trees. The complete sidebar test file
passes all 36 cases on the combined tree. React act warnings remain.
- Prior failures and fixture qualifications remain in the audit.
Published head `d176d731f083c7edd2ede688cb901d1e66d8f1d8` has Greptile
5/5, all 57 checks passing or skipped, and no unresolved review threads.
No Ponytail review is present. These local tests do not establish
deployed host/database suspension or Cloud alarm delivery.

## Risks

- Process-local signals cover instrumented writers. Independent SQL
writers and additional replicas still need integration.
- Terminal-run signals can cause one scoped inspection even if no
instruction copy exists. They do not create a recurring scan.
- Reconstruction and memory scale with actual pending candidates; active
candidates are filtered by the run join.
- Admitted file/provider operations can outlive a later drain request.
Shutdown joins them. Uncertain ownership remains pending rather than
being discarded.
- The partial index adds storage and write cost. Its concurrent
migration uses the existing interrupted-index repair path.
- Cloud alarms, sleep eligibility and provider suspension are unchanged.
Worker quiet is not a claim of deployed host/database sleep or measured
savings.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, shell tools, and
PostgreSQL/filesystem tests. The runtime does not expose a more precise
model ID or context-window size for this turn.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 22:13:57 -05:00
scotttongandPaperclip 71f0c0c99e feat: add opt-in Coordinator workspace for chat, tasks, and decisions (#17983)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/147/changes/paperclip/147
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People coordinate work through conversations, tasks, and requests
for input.
> - These surfaces must share task state and preserve the context of
each decision.
> - The existing workspace makes people switch surfaces to follow that
context.
> - This pull request adds an optional Coordinator workspace with linked
conversations, tasks, documents, and decisions.
> - Each person can enable it in Settings > Experimental and return to
the classic workspace.
> - The benefit is a coherent path from a request to an assigned task
and reviewed work.

## Linked Issues or Issue Description

**Subsystem affected**
UI, server, shared contracts, and database.

**Problem or motivation**
People need a consistent way to start tasks from agent conversations,
answer questions, review work, and return to the source context.
Decisions and task state must agree across these surfaces.

**Proposed solution**
Add a personal, default-off workspace experiment. Keep one conversation
per agent. Add task proposals, compact task and decision cards,
contextual document panels, and a Needs You queue with search, filters,
sorting, grouping, and history.

**Alternatives considered**
Replacing the classic workspace for everyone would remove the opt-out
path. Keeping separate decision state in each surface would create
conflicting answers. This change uses the existing canonical task and
interaction records instead.

**Roadmap alignment**
This supports the existing agent chat, task management, and human review
capabilities. It is an experimental presentation choice. The user
requested this isolated implementation after design review. No Discord
discussion is claimed.

## What Changed

- Add per-person workspace preferences, a default-off setting, and an
additive migration.
- Add Coordinator navigation, pinned destinations, task lists, agent
switching, and contextual panels.
- Link task suggestions and creation to the source conversation. Default
new task ownership to the current agent unless the user specifies
another owner.
- Share pending questions, decision receipts, exact review targets, and
history across tasks, chat, and Needs You.
- Preserve task access, assignment authorization, revision binding, and
required output review gates.
- Add compact document links, relative timestamps, composer controls,
and reduced-motion support for task and decision transitions.
- Add component stories, regression coverage, and capability-map
documentation.

## Verification

- Final head: `4b839aae1e314bd7e8635a766f1d205ff1c03ba6`, based on
master `8bc051245f77a47bee0f8ba45896b8bfa5922e0a`. Verified October 29,
2026 at 00:10 UTC: **MERGEABLE, 56 checks passed, four conditional
skips, no pending or failed checks**. [Current CI
run](https://github.com/paperclipai/paperclip/actions/runs/40790639688)
passed without a rerun on this head.
- Fresh Greptile **5/5**, zero new comments and zero open review
threads; Superagent passed with no suspicious changes.
[Checks](https://github.com/paperclipai/paperclip/pull/17983/checks).
- Local workspace typecheck/build passed after the latest rebase. The
final account-read patch passed UI typecheck/build and all 12 related
preference/settings tests, including a delayed real-client request
across account changes. Earlier scoped checks cover review
deduplication/replay, decision deep links, cross-surface state, access
boundaries, and migration replay.
- Local database/browser rechecks after the latest rebase were limited
by exhausted macOS shared-memory IDs: `initdb` failed before database
creation. The local run passed 31 UI/snapshot tests and skipped 24
database tests; those skips and the blocked local browser start are not
claimed as passes. The independent Linux CI database and browser checks
passed. No running dev database or host setting was changed.
- Upstream migration 0403 is preserved. Preferences and durable review
retry aliases are generated as replay-safe migrations
`0404_tense_silver_sable.sql` and `0405_harsh_avengers.sql`; five newest
snapshots are retained. Every detected conflict was rebased and pushed
before waiting for CI/review, as requested.
- The PR is unmerged. The active master ruleset still requires
code-owner approval for the changed Paperclip skill; no approval or rule
change was performed.

<details>
<summary>Earlier verification failures, integration history, and
evidence</summary>

These entries preserve prior checkpoints and failures; the current
acceptance result is above.

- Focused UI and server checks from the isolated build are recorded in
the implementation plan.
- Browser checks covered the opt-in shell, navigation, task/document
links, and decision card states. No new live provider trial is claimed
for this handoff.
- Current acceptance head: `4b839aae1e314bd7e8635a766f1d205ff1c03ba6`,
rebased onto master `8bc051245` immediately after its GitHub
review-repair migration took 0403. Preferences are generated as
idempotent `0404_tense_silver_sable.sql`, preserving upstream
SQL/journal/snapshot history and five newest snapshots. GitHub reports
MERGEABLE after the immediate lease-protected push. Scott's updated
instruction is to rebase and push every conflict resolution first,
without waiting for CI or Greptile; verification follows the push.
- The final reviewed feature tree was consolidated for repeatable
single-commit rebases. The complete previous commit history remains
reachable through local recovery ref
`codex/coordinator-recovery-before-rebase-20261028` (resolved snapshot
`0e1bbdf4e`). The actual rebase completed, and its tree hash exactly
matches that resolved snapshot. The force-with-lease required the known
remote head `0e6ef7c44`, protecting against concurrent branch changes.
No PR merge or permission/runner setting change occurred. Fresh
CI/review are in progress for the current head.
- `2001132fa` received Greptile 5/5 with one nonblocking account-switch
read finding. Current head detaches path-coalesced preference GETs and
disables HTTP caching, retaining React Query's account-scoped sharing. A
delayed real-client GET regression proves account B can load its saved
choice before account A responds, and A's late response only updates A's
cache. Preference/settings tests 12/12 and UI typecheck passed; its
review thread is resolved. Fresh current-head CI/reviews are pending.
The prior CI chat shard 3 ended with “The operation was canceled” and no
assertion failure ([job
123171595631](https://github.com/paperclipai/paperclip/actions/runs/40789688333/job/123171595631),
`/tmp/pr17983-2001132-chat3.log`); no code/timeout change is attributed
to that cancellation.
- `843e823b4` reached fresh Greptile 5/5 with zero comments/open threads
and Superagent pass. All browser tests including the repaired YAML shard
passed, and 54 checks passed. General server shard 9 ended with “The
operation was canceled” and no failed assertion; its test step was
marked cancelled (job
[123164350893](https://github.com/paperclipai/paperclip/actions/runs/40787501761/job/123164350893),
log `/tmp/pr17983-843e-server9.log`). A single unchanged-head rerun was
started after the workflow completed, but master moved while it ran. A
local nested-clone check timed out in its 30-second database-startup
hook and then failed cleanup before the 20 database cases ran; this is
retained as a local verification limit
(`/tmp/pr17983-843e-nested-clone-check.log`), not claimed as a pass or
proof of the CI cancellation cause.
- Following Scott's instruction, the new conflict was rebased and pushed
immediately without waiting for that rerun. Upstream's
`0403_natural_cerise.sql` is preserved; regenerated preferences and
retry-alias migrations are `0404_tense_silver_sable.sql` and
`0405_harsh_avengers.sql`, replay-safe with five newest snapshots
retained. The YAML helper conflict combines upstream's click and header
behavior with the editor-selection guard and injected regression.
Recovery ref `codex/coordinator-recovery-843e823b4` preserves pre-rebase
work. Current head is MERGEABLE. Full workspace typecheck/build passed;
UI and snapshot checks passed 31 tests, but 24 database tests were
skipped by the local support probe, and the local browser server could
not initialize PostgreSQL. A direct disposable `initdb` diagnostic
established the host limitation: `could not create shared memory
segment: No space left on device` (`shmget`, shared-memory IDs
exhausted, not disk space). No dev database or host setting was changed.
Diagnostics: `/tmp/pr17983-2001132-initdb-probe.log`,
`/tmp/pr17983-2001132-db-probe.log`; checks:
`/tmp/pr17983-2001132-tests.log`, `/tmp/pr17983-2001132-browser.log`,
`/tmp/pr17983-2001132-types.log`, `/tmp/pr17983-2001132-build.log`.
These local skipped/blocked checks are not claimed as passes; fresh CI
on independent runners is in progress.
- `d18f683cd` reached fresh Greptile 5/5 with zero comments/open threads
and Superagent pass, but [E2E shard
7](https://github.com/paperclipai/paperclip/actions/runs/40785966243/job/123159719925)
failed parsing an Advanced YAML clipboard read beginning with “Skip to
Main Content.” The log is retained at `/tmp/pr17983-d18f-e2e7.log`.
Current head hardens the test helper to verify editor focus and
selection, clear stale clipboard content, and require the YAML header at
the start of a fresh copy before parsing. Saved-state assertions and
timeouts are unchanged. All six Advanced YAML browser journeys passed on
a disposable local instance (41.0s); the exact lost-response journey
also passed with a one-time injected surrounding-page copy, verifying
rejection/reacquisition (17.6s). The precise browser focus/clipboard
timing in CI remains unproven; the observed invalid-read path is now
guarded and deterministically exercised. Logs:
`/tmp/pr17983-yaml-clipboard-scope-browser.log`,
`/tmp/pr17983-yaml-clipboard-injected-browser.log`. Fresh current-head
CI and reviews are pending.
- `19e6dcb0b` passed Superagent and Greptile 5/5, with one nonblocking
deep-link finding. Current head preserves attention links using
`?decisionId=`: the requested pending decision is rendered even beyond
the first 40 cards, focused and scrolled into view, then only that
parameter is consumed so later refreshes do not refocus. Other URL
parameters remain. Needs You tests 10/10 and UI typecheck passed; its
review thread is resolved. The preceding post-rebase regression run
passed 45/45 tests. Fresh current-head CI/reviews are pending.
- `6d772b7cc` reached Greptile 5/5 with zero comments/open threads and
Superagent pass. While 32 checks had passed, master moved and caused a
documentation conflict in `maps/feature-map/onboarding.md`. Rebased
immediately, preserving both onboarding paragraphs. The complete
four-commit branch was pushed as `19e6dcb0b` before waiting on
CI/review, per Scott's instruction. A transient Git index lock
interrupted the first continue; the command sequence briefly pushed
intermediate `34e298c07`, then the completed rebase was immediately
pushed. All feature and security/retry fixes are present; local recovery
ref `codex/coordinator-recovery-6d772b7cc` preserves the pre-rebase
history. The only tree differences from that ref are upstream's two
smoke-test files and the combined documentation paragraph. GitHub
reports MERGEABLE. Fresh current-head verification is in progress;
prior-head review results are not current acceptance.
- `21efa3120` passed Superagent but Greptile scored 4/5 for a valid
retry-key persistence bug: an acknowledged duplicate key could reopen a
resolved review. Current head adds an additive, replay-safe
`0404_light_swarm.sql` migration for indexed, task-scoped aliases from
acknowledged retry keys to the canonical interaction. Alias registration
and retry lookup occur under the existing task lock; payload/actor
mismatch still conflicts, and each canonical review permits at most 100
aliases. Original and acknowledged keys replay the terminal result
rather than creating a new gate. Five newest migration snapshots are
retained. Service/routes tests 165/165 passed; final quality review plus
migration replay/snapshot checks 25/25 passed; server typecheck passed.
The P1 thread is resolved in code; fresh current-head reviews and CI are
pending.
- `69c2c7543` passed 55 checks and Greptile 5/5 with no open Greptile
findings, but Superagent check
[123140675812](https://github.com/paperclipai/paperclip/pull/17983/checks?check_run_id=123140675812)
correctly flagged unbounded pending quality-card creation. Current head
fixes that under the existing task-row lock: equivalent requests with
fresh idempotency keys reuse the pending card; conflicting requests for
an unresolved deliverable return the existing interaction ID; pending
cards are capped at ten per task across all reviewers. Distinct outputs
remain independent, and already-answered items in a partially pending
batch can receive a fresh review. Added real-database concurrency,
cross-card duplicate, partial-answer, and hard-cap regressions. Quality
review 22/22 tests and final server typecheck passed; earlier
service/routes 163/163 tests passed before the final partial-answer
refinement. One added assertion initially expected a service HTTP error,
but duplicate keys are rejected earlier by the existing shared schema;
the assertion now verifies that validation. Fresh current-head
CI/reviews are pending; the action-required security result is not
treated as passed.
- `c9c575e5f` completed all 56 checks successfully with four conditional
skips, Greptile 5/5, and Superagent pass. Its two P2 review findings are
fixed in the current head: independently bound quality approvals now
each remain visible in attention; changing/clearing URL-backed
assignee/status filters updates the URL and survives task navigation.
Both review threads are resolved. Regression evidence: attention service
34/34 tests passed, CoordinatorTasks 20/20 passed, UI and server
typechecks and the full workspace build passed on `69c2c7543`. The first
new UI test run caught an assertion racing React Router's transition;
waiting for the navigation before asserting passes without weakening the
navigation/remount checks. These older-head green checks are not claimed
for the new fix commit.
- Preceding `0e6ef7c44` completed all 56 checks successfully with four
conditional skips, fresh Greptile 5/5 and no open findings, and passed
Superagent. A fifth migration-metadata collision arrived before the
final Runner job finished. Those green results are preserved as
prior-head evidence, not current acceptance.
- Previous fully verified migration head:
`0e6ef7c44696a377f35e69521b93587576374bd0`, integrating master
`f8226b051` after the fourth migration-number collision. Master added
terminal GitHub-response migration 0401 while the final native check
ran; upstream 0401 SQL/journal/snapshot is preserved unchanged, and
preferences are generated as idempotent `0402_yielding_blink.sql` on
top. Exactly the five newest snapshots remain. GitHub reports MERGEABLE
after the push. All 1,155 relevant
migration/review/interaction/native-execution/remote-assets tests,
workspace typecheck/build, migration guards and whitespace checks pass.
Fresh exact-head CI/Greptile/Superagent are pending.
- Immediately preceding head `36c67cf03` completed all 56 checks
successfully with four conditional skips, fresh Greptile 5/5 and no open
findings, plus passed Superagent security. No retry was needed for that
head's CI. Master advanced and introduced only a migration-metadata
conflict while its final native check ran, so that green result does not
establish acceptance of the current integration. Four consecutive
collisions (0398, 0399, 0400, 0401) during verification are retained as
concrete integration churn; a brief pause of competing migration merges
was requested, without assuming it happened or changing any repo
setting. The PR has not been merged.
- Prior fully verified head: `36c67cf03`, on the clean `30059c0ed`
integration of master `efd65c112`. The last Greptile P2 is fixed: Needs
attention retains the participant-agent scope through authoritative
server membership queries for each pending task, including historical
participation when another agent currently owns the task. It does not
approximate participation with current assignment or enumerate every
company task. The actual view-switch regression and all 31 focused
task/routing/collection tests, UI typecheck/build, token gates and
whitespace checks pass. Greptile completed a fresh 5/5 review on this
exact head (review 19) with zero new comments and no open review
threads. CI and Superagent remain in progress; no current-head CI
failure has been reported.
- Head `30059c0ed` received Greptile 5/5 with that one nonblocking
participant finding, now fixed. Superagent passed that head.
Current-base integration restored the ephemeral RunsOn fleet after a
superseded workflow retained queued aggregate jobs: normal cancellation
was insufficient; the single force-cancel request was interrupted but
readback confirmed that obsolete run cancelled and current CI began on
the trusted fleet. No cancellation retry or runner/security/workflow
changes were made. Its partial CI was superseded by the necessary
participant fix and is not current acceptance.
- Previous compatibility/base-sync head:
`30059c0ededc0eab4e04e8299a884f36db8b292f`, integrating master
`efd65c112` and the Inbox/workspace compatibility fixes. GitHub reports
MERGEABLE after the push. Coordinator keeps Inbox standalone under More
without rewriting the instance's Combined Inbox preference; opting out
restores that preference. Saved merged-Inbox URLs resolve to their
original owner, task status presets remain usable, and canonical
repeated `workspace` parameters retain scoped, deduplicated pagination
(including Needs attention). All 192 broader
routing/Inbox/sidebar/privacy/review tests and the final 16 task-page
tests pass, as do workspace typecheck/build, token gates and whitespace
checks. The two review threads are resolved. Fresh exact-head CI,
Greptile and Superagent results are pending.
- The preceding `d5b9be5bd` remained mergeable but its CI routing gate
intentionally fell back to GitHub-hosted runners: “stacked base merge
commit does not contain the current base branch.” All 45 downstream jobs
stayed queued. Current master was merged cleanly to refresh the pinned
base; no runner/security settings or workflow logic were changed.
Greptile scored that preceding head 4/5 with the two compatibility
findings fixed here. Its automatic Superagent scan was inconclusive at
checkout; the documented retry was still pending when superseded. Those
results do not establish acceptance of this current head.
- Previous approval/migration head:
`d5b9be5bd3b8d6b0437e42bf48138377f34fa67d`, integrating master
`ba5bba86d` and the stale-output approval correction. GitHub reports
MERGEABLE after the push. The fresh P1 finding is fixed: every submitted
item now validates and locks its bound output inside the resolution
transaction, whether or not it carries a required-quality-review
binding. Real database regressions verify that an advanced document
revision or deleted attachment refuses the entire batch without
recording approvals, and current ordinary output reviews still resolve.
All 272 combined migration/review/interaction tests pass; workspace
typecheck/build passed. Fresh exact-head Greptile and CI are pending.
The automatic Superagent scan ended inconclusive at checkout; one
documented on-demand retry is pending. These are explicit verification
limits, not passed security results. The P1 thread is resolved.
- Master consumed the next migration slot three times during this
verification pass: confirmation recovery took 0398, continuation
recovery took 0399, and GitHub acknowledgement recovery took 0400. Each
upstream SQL/journal/snapshot is preserved unchanged. Preferences are
now generated as idempotent `0401_white_lenny_balinger.sql` from
upstream 0400; exactly five snapshots remain, and replay coverage
locates this migration by table content. This is recorded concurrent
integration churn, not a claim that older green results apply to the
latest base.
- Head `ed3b68da9` fixed the two history/URL findings with 26 focused
regression tests, server/UI typechecks, workspace build and token gates
passing. Greptile then scored that head 4/5 with the stale-output P1
corrected in the current head. Its partial CI is superseded, not counted
as current acceptance.
- Previous filter-fix head: `ed3b68da9`, based on the `8458d39d9` merge
of master `063cfd438`. The two fresh Greptile findings are fixed:
history outcome filters now match recorded per-item verdicts in SQL
before pagination (mixed reviews remain in both applicable outcomes),
and same-page URL assignee changes/back/forward navigation update that
filter without clearing independent filters. All 26 focused
database/navigation/collection tests, server/UI typechecks, workspace
build, token gates and whitespace checks pass. The two review threads
are resolved. Fresh exact-head CI, Greptile and Superagent results are
pending.
- The `8458d39d9` migration integration also passed all 328 targeted
migration/interaction/confirmation tests, workspace typecheck/build and
Superagent security scan. Greptile scored that head 4/5 with the two
functional gaps fixed in the current head; those prior-head checks are
not current acceptance.
- Migration integration head:
`8458d39d9721f4cb312c2390070a0609f8cddd1a`, integrating master
`063cfd43813cabf18417605fb9eefb9aacf95c8d`. GitHub reports MERGEABLE
after the push. Master advanced twice during this conflict-resolution
pass: first its confirmation-recovery migration took 0398, then its
continuation-recovery migration took 0399. Both upstream migrations and
journal entries remain unchanged; Coordinator preferences are now
generated as idempotent `0400_sad_guardian.sql` from upstream's latest
snapshot. Exactly the five newest snapshots remain. The replay
regression now locates the table migration by content instead of
hardcoding a renumbered filename.
- All 328 targeted question/review/confirmation/migration tests passed
again after the second merge. Workspace typecheck/build and fresh
exact-head CI/reviews are in progress. The preceding `0c46f46d6` passed
workspace typecheck/build and Superagent security scan, but was
superseded because master consumed migration 0399 while those checks
ran. That prior-head result is not acceptance of the current head. No
production settings or runner/security permissions were changed, and the
PR has not been merged.
- Previous integration head: `0c46f46d6f2b23417ab872ce3cb6ff2673647287`,
integrating master `287b04d72f1b9fca1412e05435c75b8302ba0710`. GitHub
reports MERGEABLE after the push. The renewed conflicts were
`0398_snapshot.json`, `_journal.json`, and the top imports in
`issue-thread-interactions.ts`. Upstream's `0398` confirmation-recovery
index and journal history are preserved unchanged; Coordinator
preferences were regenerated as idempotent `0399_bouncy_miek.sql`, with
replay coverage updated and exactly the five newest snapshots retained.
Both terminal confirmation notifications and Coordinator task
assignment/required review behavior are preserved.
- All 328 targeted interaction/review/confirmation/migration tests pass,
as do workspace typecheck, map validation and whitespace checks. The
workspace build and fresh exact-head CI/Greptile/Superagent checks are
pending. The first migration-generation attempt hit the sandbox's local
IPC restriction before generating files; the unchanged authorized local
generation completed successfully. Prior-head green checks below do not
establish acceptance of this merge.
- Previous acceptance head: `8b7dcf739ed15afc51351fdfa0e0e5fbc75ab788`,
with master `1aa3a21ac` integrated and GitHub reporting MERGEABLE.
Greptile completed a fresh 5/5 review on this exact head with zero new
actionable findings and no open review threads. The [fresh CI
workflow](https://github.com/paperclipai/paperclip/actions/runs/40752864319)
completed SUCCESS on attempt 2 at 19:40:08 UTC, including all eight E2E
shards. Final completed check set: 55 succeeded, four conditional checks
skipped, zero pending/queued/running checks, and one
neutral/inconclusive Superagent scan. The auxiliary commitperclip review
and isolated native Runner compilation/cache-reuse check both passed.
This is not a claim that Superagent produced a passed security verdict.
GitHub still reports MERGEABLE against current master
`60b8089a44b4eaa37e627adbfdf2f130bda13335`.
- Current-head E2E: all eight shards and the aggregate passed, including
the previously failing YAML browser journey. Serialized server shard 9/9
failed: the protected YAML projection test exceeded its original
15-second timeout; its late update request then reached the next test's
reset mock (the captured payload is the prior test's model/label
update). All 88 tests in that exact unchanged file passed locally in
38.03 seconds, with no timeout or assertion changes. The original
slow-request cause remains unproven. CI failure log:
`/tmp/pr17983-8b7-serialized9.log`; local recovery:
`/tmp/pr17983-adapter-yaml-repro.log`. The single bounded unchanged
shard rerun passed, and the workflow completed SUCCESS. Rerun log:
`/tmp/pr17983-8b7-serialized9-rerun.log`. The initial failure is
retained; a passing retry does not establish its original slow-request
cause.
- Greptile scored preceding head `4873704be` 5/5 with one valid P2:
opened task decision records missed company-scoped live invalidation.
The current head uses the existing company decision prefix for
resolution and effect-completion events, and resets private record
payloads on access changes. Its thread is resolved. All 107 focused
record/live-update tests, UI typecheck/build, token checks and
whitespace checks pass.
- CI on `4873704be` failed E2E shard 7/8 in the Advanced YAML
lost-creation-response journey: its clipboard helper selected page text
starting with “Skip to Main Content” instead of YAML. The exact
unchanged journey passed in a disposable local browser instance (1 test,
17.1 seconds). No assertion, timeout or production behavior was
weakened; the original failure cause remains unproven. Failure log:
`/tmp/pr17983-487-e2e7.log`; local check:
`/tmp/pr17983-agent-yaml-repro.log`. The full unchanged Advanced YAML
browser suite also passed locally (6 tests, 37.8 seconds). Fresh
current-head CI subsequently passed this journey and all other E2E
shards.
- Both the automatic and one supported on-demand Superagent scan on
`4873704be` ended neutral/inconclusive: “Pinned PR scan merge-base
checkout failed or timed out.” These do not count as a passed security
verdict. The current head's automatic scan and one documented on-demand
retry also returned the identical neutral checkout failure, with zero
annotations or further diagnostics. This is an external scanner blocker,
not a passed security verdict. No security settings or permissions were
changed.
- Recorded decision links now render and focus the full source-task
record in both Classic and Coordinator, including recorded choice,
submitted inputs, and the existing effect-result resolver. Fresh
terminal records take precedence over older open-decision cache entries.
Context and account checks keep mismatched or inaccessible records out
of the DOM; no decision-history tab was added to tasks or chat. The
shared company live-run feed now drives live-only filtering, including
ended runs and terminal tasks with stale run entries. All 213 focused
tests, UI typecheck/build, token and whitespace checks pass; the final
focus follow-up passed all 188 task-detail tests and UI typecheck. The
first new navigation test failed because its test host omitted company
context; the host was corrected before the passing run.
- Previous integration head: `375c5e89516705050d94bab99336212841b2e6e1`
merges master `1aa3a21ac` after four additional upstream commits. The
sole conflict was competing imports in `server/src/services/issues.ts`;
both quality-review policy and inbound wake-notice notifications are
preserved. The PR-specific review import now sits beside
execution-policy imports instead of the frequently changing first line.
GitHub mergeability is rechecked after the push.
- On the combined master tree, workspace typecheck/build, all 153
affected UI tests, map validation, and whitespace checks pass. Four
server suites passed 77 tests; the fifth suite's module-import hook
timed out while broad verification was running. Its unchanged isolated
run passed all 41 tests in 7.2 seconds with the original timeout. This
records recovery without claiming the original import-timeout cause is
proven. CI on `375c5e895` passed every substantive job, including all
eight E2E shards and Superagent; Greptile found the two functional gaps
fixed in the current head. A short-lived `669d90876` commit added the
main fixes before the focused navigation follow-up.
- Immediately preceding head `9d9737b49` completed all 56 checks
successfully with four conditional skips, Greptile 5/5 and no open
findings, plus a successful Superagent scan. The signoff E2E shard
passed on its single accepted rerun; its initial cancellation/timing
failure is retained below. Master moved again while those checks ran,
requiring this final integration rather than treating the prior head's
green result as current acceptance.
- Previous head `9d9737b496911a6054d0ea22d1b42b28b115e9ab` includes
master `d30c8f42b842327ee1c18c41a2efd6ba623da5cd`; that head was
subsequently conflicted by moving master. The master merge preserved
durable account-bound writes and Coordinator UX. Workspace
typecheck/build, 416 focused UI tests, and 142 affected comment-route
tests passed. A stable full UI run on preceding head `938ad1665` passed
all 9,833 tests across 836 files; its final full workspace build also
passed.
- Greptile found nested proposal project inheritance and Classic
bound-revision links on `91be74952`. Both were fixed with 142
interaction-route tests, 113 focused UI tests, and UI/server typechecks.
Its next review identified a valid draft-preservation bug in the new
Classic loading UI. Head `9d9737b49` keeps loading/error notices outside
editors and annotations while preserving dirty drafts; 115 focused
review UI tests pass, including pending/failed revision loads and
continued draft editing. All related threads are resolved. Greptile
completed a fresh 5/5 review of
`9d9737b496911a6054d0ea22d1b42b28b115e9ab` with no open findings.
- CI on `938ad1665` exposed an integration regression with the newly
merged durable startup saves: the personal preference gate blocked the
task page before its first session lookup. The guarded title editor now
mounts while that lookup is pending, with presentation-specific tools
still gated. All 216 affected component tests pass. The unchanged
five-test `disconnect-write-recovery.spec.ts` browser suite passes
against a newly launched disposable local-trusted PostgreSQL instance on
port 33219, including the exact failed startup-save journey. No live
provider, hosted authentication or production proof is claimed. The
helper-owned test server exited and the port has no listener. UI
typecheck, built-browser assets, token and whitespace checks pass.
- Initial durable-write merge tests found duplicate imports and unscoped
draft fixtures, fixed before 416 focused tests passed. A sandbox
typecheck could not create its IPC socket; the supported rerun passed.
The first revision-link test used a wrong Return to latest button
assertion, corrected before the 113-test pass. The startup E2E
reproduction initially still failed because an outer shell gate also
blocked the page; the subsequent fix passed all five cases without
changing the test or its timeouts. A focused loading-header assertion
initially used an incorrect fixture title; correcting it preceded the
216-test pass. These failures remain recorded.
- Earlier routing gates fell back to GitHub-hosted runners while master
was moving or conflicted. Superseded queued runs were cancelled, then
the `938ad1665` routing gate explicitly selected the ephemeral RunsOn
fleet. Its CI actively executed and completed most tests; only the
diagnosed startup E2E shard and aggregate failed, alongside the valid
Greptile finding. Auxiliary GitHub-hosted workflows also completed
successfully on earlier heads, so queueing alone is not an
administrative blocker. Fresh [CI on
`9d9737b49`](https://github.com/paperclipai/paperclip/actions/runs/40744012678)
is the acceptance target; the previously failing startup E2E shard now
passes. A different signoff E2E shard failed during executor
resubmission when its fixture run was cancelled as
`issue_continuation_waiting_on_review`. The exact unchanged journey
passed locally (1 test, 46 seconds), and the preceding head with
identical server/signoff code passed that CI shard. This supports a
bounded rerun but does not establish the timing failure's root cause.
After the workflow finished, GitHub accepted one rerun of that failed
shard and its dependent aggregate; both passed, completing all checks on
that head. The failure log is retained at `/tmp/pr17983-9d9737-e2e7.log`
and the local reproduction at `/tmp/pr17983-signoff-repro.log`. No
assertion or timeout was weakened.
- On the first merge, full workspace typecheck/build, all 9,622 UI
tests, 267 tests across seven affected server suites, two migration
checks, map validation, token gates, and the PR diff whitespace check
passed. Greptile then found duplicate local task-search filtering and
lost completed item-review detail. Both are fixed, with 42 passing
focused UI tests and passing UI typecheck/build.
- A subsequent Greptile review found attachment confirmations without
working links and mobile history without a visible host. Confirmation
layouts now derive the authenticated immutable-content endpoint from the
attachment ID, and Needs You uses the existing full-screen mobile sheet.
All 119 focused tests pass, including missing/stale display URLs, both
review layouts, and both mobile history entry buttons. UI
typecheck/build and token checks pass. Full workspace typecheck/build
also pass after the second master merge.
- Greptile gave head 22f69c304 a 5/5 score with one non-blocking
desktop-history reopening finding. Head 6dd52b6d3 fixed it, passed all
nine history tests, UI typecheck/build, token gates, and the full
9,679-test UI suite, then received a fresh 5/5 review with zero open
findings. These are prior-head results; review on the master-sync commit
is pending.
- The chat and workspace CI jobs on head 22f69c304 were interrupted by
runner shutdown signals and cancellation before completing their tests.
Their logs do not report failing assertions. Those interrupted jobs are
not claimed as passes. The new head starts fresh runs.
- The first merge CI had a Slack session-sync idle assertion fail after
an unexpected database read. Its exact isolated reproduction passes, and
the chat CI shard passed on head 618c5a2a3. The original failure's cause
remains undetermined. The broader local reproduction, run concurrently
with full verification, instead encountered 21 timeout failures across
its two test chunks; it is not claimed as a passing full local server
run. A concurrent local UI run had eight failures (seven timeouts and
one delayed autosave assertion). The complete UI run on the final head
passed all 9,679 tests across 830 files with two workers; no test
timeout or assertion was weakened. This establishes recovery without
claiming that every earlier timeout's cause has been proved. An earlier
intermediate UI run also overlapped the second master merge and failed
the newly added Advanced-sidebar expectation, so it is not treated as
stable-tree evidence. The newly merged UI dependencies initially
required refreshing the local install from the upstream package
manifest; the generated local lockfile update is not committed, per
repository policy.
- Superagent returned neutral/inconclusive scans on the preceding heads
because its pinned merge-base checkout failed or timed out. The
available retry endpoint returned 404. These are not successful security
verdicts. The fresh scan on `91be74952` passed (PR scan passed; no
suspicious PR changes detected). The `938ad1665` scan again returned
neutral/inconclusive with the same pinned merge-base checkout error; the
check-suite rerun endpoint returned 404. The fresh `9d9737b49` scan
completed successfully at 18:22 UTC with no suspicious changes detected.
Earlier neutral scans are not counted as successful security verdicts.
- Prior head `57902152ac3218096fca669d33ff18400a2a86ae` passed 9,618 UI
tests across 826 files, received Greptile 5/5 with no open findings, and
completed CI with 56 passes and four conditional skips. These are
historical results, not acceptance of the latest merge commit.
- The first full build found generated capability-contract drift. The
generated outputs were refreshed. Initial sandbox runs could not open
local IPC sockets; verification was rerun with local test access.
- Current master was merged. The workspace preference migration was
regenerated after the existing migration history.
- The UI run first found two outdated expectations: early custom-answer
validation and the document access error copy. Both were corrected, and
the full UI suite then passed.
- The local server run found three failures in unchanged tests. The Muse
test passes with Python 3.13 instead of the host's Python 3.9. The two
workspace tests pass when their test process uses the expected master
default branch. These environment settings do not change application
behavior or global machine settings. The unsharded server run was
stopped after those reproductions; CI supplies the full sharded server
result. No successful full local server run is claimed.
- Skill inventory generation needs an external eval checkout, which is
absent locally. The existing generator functions refreshed only the live
skill inventory and its derived files. The checked-in eval inventory was
preserved. The inventory validation gate passes.
- Greptile found a dropped MCP review requirement, missing per-item
output links, generic item-review summaries, and a history-refresh
failure. The fixes have 32 passing database-backed server tests and
focused rendered-card tests. A follow-up partial-review finding was
corrected by forwarding the canonical mutation status before settling
the card. Attachments open through the authenticated immutable-content
route in both card layouts. The final focused UI set has 117 passing
tests, including these regressions.
- Both browser shards that first failed expected a disabled-chat
message. The copy was restored, and those shards passed in CI on the
next commit. The two journeys also passed locally. One local attempt
first timed out before feature disable while waiting for the provider's
initial message; its isolated reproduction passed. That earlier timing
failure remains recorded and is not presented as an explained
infrastructure failure.
- Serialized server shard 9 first failed during pnpm setup with
ECONNRESET, before dependencies or tests ran. Its single rerun passed
setup and all tests. The build and production runtime boot passed on the
first attempt, but that run's public-install step timed out during Pi
setup. The single build-job rerun passed compilation, runtime boot, and
clean public-install verification. Those setup scripts are unchanged by
this PR. The successful retry establishes recovery; the earlier Pi
timeout's underlying cause remains undetermined. The aggregate CI gate
is green: [workflow
run](https://github.com/paperclipai/paperclip/actions/runs/40637789695).

</details>

## Risks

- This touches shared task, interaction, and completion paths.
Regression tests must cover classic and experimental behavior,
authorization, duplicate answers, and stale review targets.
- The additive preference table must migrate before the setting is used.
The setting defaults off.
- The experiment changes navigation and message presentation. Canonical
task and document records remain the source of truth.
- The initial task brief and saved plan can repeat information. This
change fixes chronology; a broader redundancy policy remains a design
decision.

## Model Used

OpenAI GPT-6 Codex with reasoning and tool-assisted code editing, tests,
and browser verification. Earlier iterations used Codex models in this
task; their exact model IDs are not available in the current context.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md
- [x] I have searched GitHub for duplicate or related PRs
- [x] I have described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change
- [x] I have run relevant tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open recommendations or follow-ups
- [x] I will address review comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 17:20:01 -07:00
DottaandPaperclip c74171d620 fix(recovery): retire superseded native workspace retries (#18077)
## Thinking Path

> - Paperclip coordinates agents while retaining their work and
execution authority.
> - A failed native run can schedule an exact-session retry and retain a
sandbox.
> - An explicit task Retry can already have completed successfully on a
successor.
> - Recovery correctly refuses the old run after its task execution lock
changes.
> - It nevertheless leaves that old retry scheduled and its workspace
without a preservation owner.
> - This change retires that proven superseded retry and registers
exact-source preservation together.
> - Existing stop receipts and saved-file verification still control
workspace switching.

## Linked Issues or Issue Description

**What happened?**

A native sandbox run failed before starting a provider turn. An explicit
Retry completed on a linked successor. The original coordinator remained
in `retryable_failure` with an active sandbox lease. Restart recovery
returned `issue_execution_lock_changed`, while the ordinary sweep kept
delaying it. A pending workspace selection then made every new task run
fail during setup.

**Expected behavior**

Stop replaying a failed retry after its linked explicit successor has
completed. Preserve the original files through the existing system
worker. Keep the current task, ownership, pending choice, and unresolved
action outcomes intact.

**Steps to reproduce**

1. Persist a failed, unowned native run with a prepared remote workspace
and `retryable_failure` coordinator.
2. Complete a same-company/task/agent explicit Retry linked to that run.
3. Release the task's old execution lock and schedule a workspace
selection.
4. Run startup or ordinary retry recovery. Observe the old retry remain
scheduled and the selection fail with an unfinished-recovery conflict.

**Paperclip version or commit**

Regression reproduced against `644937c931`. The current repair is based
on refreshed master `60b8089a44`.

**Deployment mode**

Native Runner with remote sandbox workspaces. The reported runtime used
ACPX Claude with per-turn lifecycle. The original provider
session-resume rejection is not independently reproduced or claimed
fixed here; this PR addresses the subsequent stranded recovery.

Related: #17839 records preservation when restart classification becomes
terminal. This change handles the earlier lock-mismatch return and
ordinary retry admission after a completed explicit successor. It uses
the same preservation service and does not require that PR.

## What Changed

- Share a narrow superseded-retry settlement between startup recovery
and the ordinary retry sweep.
- Require a failed source with no coordinator owner or process, its
exact prepared source lease, and a completed linked successor under the
same company, task and agent.
- Atomically retire replay and register preservation. Do not alter task
state or create another provider run.
- Add PostgreSQL lifecycle/safety regressions, a real-file preservation
and workspace-selection regression, and execution/map documentation.

## Verification

- Red: both startup and ordinary admission regressions left the old
coordinator in `retryable_failure` with no preservation obligation.
- Green: the same two cases retire the retry and retain the exact lease
without another run or task mutation.
- The assembled PostgreSQL/filesystem regression passes: workspace
selection remains blocked before preservation, the existing service
saves `saved.txt` and preserves newer local work, then the normal
selection guard applies the pending shared workspace.
- All 90 native retention/resumption cases pass. Three neighboring
startup-recovery cases pass, including the blocked-owner inventory. Full
recursive typecheck and full build pass at runtime head `23ddb213b4`;
final `19d7738ae3` only moves the function comment into execution
documentation. Map, module-boundary and diff checks pass. The duplicate
full local Vitest run was stopped after 20 minutes in its first long
general shard once the complete current-head hosted test gates passed;
no complete local-suite pass is claimed.
- Current-head CI has 53 passing checks and two intended Storybook
skips, including full typecheck, all test shards, build, Runner
verification, browser shards and both aggregate gates. The separate
`commitperclip PR Review` job is still queued on `ubuntu-latest` with no
assigned runner; its outcome remains pending. Greptile reviewed exact
head `19d7738ae3` at 5/5, with zero unresolved threads.
- Live acceptance passes on exact final source `19d7738ae3`: a
disposable Daytona Linux sandbox, the current immutable staging fleet
snapshot, SDK 0.203.0, a Darwin arm64 source host and Node 26.4.0. The
real native workspace preparation/export and provider stop/start paths
preserve an unpredictable report and newer host files. The ordinary
workspace-selection guard stays blocked until preservation commits, then
applies the pending shared workspace.
- The old workload's start count and side-effect marker each remain one.
Recovery creates no new run or native result. The sandbox finishes
stopped and is then deleted by the probe; a final provider read confirms
it is absent.
- A second live acceptance also passes on the original runtime's older
immutable snapshot with the source explicitly archived before recovery.
It verifies cold source reopening, the same report/newer-file checks,
one workload start and side effect, pending workspace application, and
confirmed deletion. Its initial probe attempt used immediate
archive/deletion reads and failed during asynchronous cleanup
observation; bounded provider-state polling fixed the probe without a
product change. The abandoned probe allocation is independently
confirmed absent.
- Scope: database fixtures represent the already-failed run and
completed explicit successor. The old workload is a persistent shell
process with observable file/side-effect markers, not a paid Claude
turn. This qualifies the repaired recovery handoff and real provider
preservation; it does not reproduce the initial ACPX session-resume
rejection.
- The original hosted task has only been inspected read-only. No rollout
or original-task recovery is claimed.

## Risks

- Incorrect successor classification could retire legitimate work.
Matching requires the server-persisted explicit Retry link, same
company/task/agent, later successful committed execution, no current
source execution lock, no recorded process, no coordinator owner, and no
workspace finalization owner.
- Retirement is not a stop receipt. The exact source remains retained,
unknown effects remain unknown, and workspace switching still waits for
the existing worker's stop and file-persistence proof.
- A failed preservation transaction rolls back retirement. No new scan,
retry budget, schema or public API is introduced.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use and code execution.
The host does not expose a more specific model ID or the context-window
size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — focused suites and live
acceptance; full-suite status above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green — separate template-review job
still queued as noted above
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups —
exact head `19d7738ae3`
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 15:15:19 -05:00
Devin FoleyandPaperclip ef0a8db502 perf(native): narrow reconciliation candidate payloads (#17531)
Select the complete workspace-sync record and native status wake identifier for reconciliation candidates. Preserve candidate policy, JSON types, full later evidence, and stored data. The committed replay regression includes the real reconciliation scan. Full build, typecheck, 114 focused tests, complete CI, and exact-head Greptile5/5 pass.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-24 19:01:23 -07:00
Nicky LeachandPaperclip 00331e0254 Connect native Runner admission to plugin-owned workspaces (#16978)
## Thinking Path

> - Paperclip manages work for AI agents.
> - A plugin can own an execution environment and its files.
> - #17029 supplies the task driver and bounded setup verification.
> - Heartbeat still prepares workspace files on Core and expects a shell
transport.
> - This PR gives the selected plugin ownership of workspace preparation
and native task execution.
> - Core keeps admission, tool authority, and recovery checks.

## Linked Issues or Issue Description

**Subsystem affected**

Heartbeat workspace admission and native Runner execution.

**Problem or motivation**

A task-only plugin cannot run through Core filesystem staging or shell
execution. Native admission and recovery must use the same lease and
workspace ownership.

**Proposed solution**

Use the task driver from #17029. Resolve workspace and project
identities without Core file operations. Connect native dispatch and
restart recovery to that driver.

**Alternatives considered**

Keep workspace admission and dispatch in one change. A workspace change
alone would leave dispatch on the previous transport. An execution
change alone would leave Core staging files for the wrong host.

## What Changed

- Enable plugin environment selection for native workspace sessions.
- Resolve workspace metadata and authorized project references without
Core checkout, copy, or provision operations.
- Preserve agent-default and primary workspace policies. Use the pinned
workspace provider for plugin-backed project storage. Reject unsupported
Core service commands and cross-owner reuse.
- Connect native execution to the prepared task driver. Keep assigned
MCP authority on Core.
- Save exact checkpoint identity, transfer selected account refreshes,
and require termination proof before replacement or cleanup.
- Complete managed plugin workspaces through their storage provider. Use
the same check after a restart. Keep branch validation and leave
task-owned files on the plugin.
- Include default instruction content in runtime synchronization when no
instruction directory exists.
- Defer new-agent Copilot verification until the saved-agent lifecycle
can acquire the agent-owned computer.

## Verification

- Rebased onto `origin/master` at `898144f4b3`. Master advanced once
during verification. Both manual conflicts joined import blocks: one in
`native-session-executor.ts`, and one in
`heartbeat/runtime-execution.ts`. The seven patches retain their logic.
The first rebased head passed repository typecheck, build, all 862
focused execution/finalizer cases, and all CI gates. The latest head
`93e5a72892` passes server typecheck and all 880 focused execution,
finalization, and execution-control notification tests. All 54 CI/review
checks pass, with two expected skips. Greptile ran automatically and
scored this head 5/5 with no unresolved findings. No new review or Apex
request was made for these import-only resolutions.
- The successful CI workflow initially left three stale job-check
reports. Targeted policy and summary reruns cleared the reports. No
source change or test fix was needed.
- Repository typecheck and build pass: `pnpm -r typecheck` and `pnpm
build`.
- Focused native execution and restart tests pass: 795 cases. The
account and harness suites pass: 159 cases. Workspace admission passes:
48 cases. The SDK task lifecycle, workspace API, runtime-hook, and
instruction checks also pass.
- The heartbeat integration case uses the merged workspace provider
contract and reaches native dispatch with no project files on Core. The
new-agent case confirms that creation leaves the agent in `preparing`
until lifecycle verification.
- Map validation and the Copilot source-identity check pass. No Runner
implementation or source-identity bump is in this PR.
- The first rebased workspace run found two stale path assertions and a
recovery guard that did not recognize task-owned targets. These are
fixed. The storage integration fixture also needed its persisted source
binding; the production authorization check stays in place.
- Two prior review findings required rejection of Core service startup
and cross-owner workspace reuse. The existing admission fixtures now
check both failures. The installed-account authentication finding was
fixed in merged #17029.
- Tests reuse the current fixtures. No new provider test matrix or live
Runner process is added for these fixes.
- The prior head `2db273ec12` passed CI on its first attempt and reached
standard Greptile 5/5. Apex scored 4/5 and found local-folder checks
during plugin completion and recovery.
- The completion fix reproduces both failures. It passes all 110 cases
in the runtime execution, workspace finalizer, and workspace provider
integration suites. Server typecheck and map checks pass. CI passed on
`62d8335b4e` on its first attempt, and standard Greptile scored 5/5. The
next Apex review scored 4/5. It found that a failed plugin inspection
could abort restart recovery.
- The restart fix records a failed workspace receipt and uses the
current run retry policy. A regression first reproduced the aborted
pass. It now checks retry state, retained results, and completion of
another workspace in the same pass. All 68 finalizer cases and server
typecheck pass. CI passed on `7d4016323b` on its first attempt. Standard
Greptile and Apex both scored 5/5. Apex left one P2 about lost failure
codes.
- The diagnostic fix keeps the known branch and owner mismatch codes in
receipts and run recovery state. It gives repair guidance for those
codes and keeps unknown plugin errors generic. All 96 affected finalizer
cases and server typecheck pass. Before the rebase, required CI passed
on `c2fa61c6c8` on its first attempt. Standard Greptile and Apex both
scored 5/5 on that head, with no unresolved findings. The new head
retains these reviewed patches.
- The earlier Superagent scan on `c2fa61c6c8` was inconclusive after a
pinned merge-base checkout failed or timed out. The current scan on
`93e5a72892` passes.
- The exact rebased head passes all 48 workspace cases and server
typecheck. The first local test launch could not bind PostgreSQL inside
the sandbox. The authorized local-server run passed.
- The prior head reached standard Greptile 5/5. Its first CI run hit an
unchanged Runner cleanup timeout and an external build-runner shutdown.
Both jobs passed on the single retry. The timeout cause is not
established.
- A local regression check found that a retained Core workspace could
also enter a plugin task without file transfer. Core now rejects this
move before provider work. The existing ledger test covers both
directions and confirms that the source files stay intact.
- The first active standard review scored 4/5. It found that a fresh
plugin retry could enter Core Git preparation. This mode now uses the
existing unsupported-storage error before workspace resolution or
acquisition. Both unsupported-mode cases pass, and server typecheck
passes.
- No live Cloud or provider validation was run.

## Risks

- The prerequisite #17029 is merged. This PR targets master.
- Native plugin tasks support per-turn workspace sessions. Core service
commands and fresh retries that import retained files remain
unsupported. A workspace storage plugin can use the modes and strategies
it declares.
- Core-managed credentials support Codex and direct Claude accounts.
Other managed credential routes fail closed. Other harnesses can use an
installed account.
- A provider must preserve its prepared configuration and report process
termination correctly. Uncertain termination keeps the task pending
cleanup.
- Cloud deployment and live provider validation remain separate.

## Model Used

OpenAI Codex, GPT-6. The exact model ID and context window are not
exposed in this session. Tool use includes code editing, local tests,
and the GitHub CLI.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal issue
details
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 08:22:48 -07:00
DottaandPaperclip 30b753f614 perf(budgets): scope accounting and deliver enforcement asynchronously (#17046)
## Thinking Path

> - Paperclip manages AI agents and their paid work.
> - Budgets must prevent new work and stop reported overruns.
> - Cost settlement also ran serial lifecycle reconciliation for every
eligible company agent.
> - That made request latency grow with unrelated idle agents and
delayed results on stop delivery.
> - This change separates atomic admission and accounting from durable
asynchronous enforcement.
> - Scoped live usage detects overruns without writing provisional
ledger charges.
> - The result is constant query scope for ordinary settlement and
recoverable stop delivery.

## Linked Issues or Issue Description

**What happened?**

Settlement and budget checks entered company-wide lifecycle
reconciliation. A company with 1,000 unrelated idle agents performed
1,001 accounting transactions for one ordinary cost evaluation. User
requests could also wait for provider termination. Running usage
checkpoints did not enforce reported monetary exposure.

**Expected behavior**

Keep the final reservation gate authoritative. Commit charges, scope
blocks, and enforcement intent before returning. Deliver stop requests
after commit. Read only applicable policies and execution scopes for
ordinary charges. Stop long-running work when supported reported or
priced usage establishes an overrun.

**Steps to reproduce**

1. Create a disposable company with one company budget and idle agents.
2. Evaluate a charge below the budget with 0, 17, and 1,000 unrelated
idle agents.
3. Compare query counts, company accounting transactions, and duration.
4. Use the controlled partial-usage and delayed-stop fixtures to check
enforcement and request responsiveness.

**Paperclip version or commit**

The measured baseline is `6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837` from
`master`.

Related public work: Refs #16976, #2006, #3539, #13763, #14296, #16853,
#13075. These cover the affected service charge, cancellation, and
accounting paths. This change removes their shared reconciliation
overhead.

## What Changed

- Batch applicable policy exposure under the existing company accounting
lock. Preserve exact fractional money and zero-reservation dispatch
fences.
- Commit a durable final budget dispatch claim before provider I/O.
Retain a separate final issue/run ownership transaction through
synchronous handoff, without holding the company accounting lock or
awaiting provider work. Preserve native recovery lease checks and
duplicate handoff fencing. Browser handoff rechecks current policy; it
reuses a consumed rate slot only for the identical policy generation.
- Separate ledger spend, live observed spend, and effective held
capacity. Use the maximum of reservation and live observation for each
unfinished operation.
- Deliver budget cancellation through the existing coordinator.
Revalidate generations and persist queued/running stop fences before
external termination. Bound stop concurrency to four.
- Apply agent holds only to the affected agent. Project budgets leave
other projects running. Project real company transitions in bounded
lifecycle batches.
- Replace periodic broad reconciliation with durable pending work,
startup repair, receipt replay, and concrete retry/window deadlines.
Retain drain and idle-safety checks.
- Enforce accepted cumulative usage checkpoints without provisional
charges. Final accounting atomically replaces live exposure and settles
reservations.
- Add operator-gated budget spans, regression coverage, map updates, and
a real-PostgreSQL comparison harness. No schema or first-party telemetry
change.

## Verification

- Full workspace `pnpm -r typecheck` passes on the final ownership fix,
rebased onto `bbcf5dfe98`. Production `pnpm build` passes on final head
`4c5f4de8b1`. Map, module-boundary, token, and Storybook catalog checks
pass.
- Focused real-PostgreSQL coverage includes the 292-case rebased
accounting/dispatch run, 63 browser cases, 52 run-control/cancellation
cases, 42 policy-generation cases, 77 lifecycle/commentary cases, and
the exact Muse confirmation regression. Native executor unit coverage
passes all 768 cases. Final ownership coverage passes: 74
adapter/use-case cases, nine public regression cases, and the additional
deleted-project race. These exercise competing ownership, budget changes
and charges between phases, zero/nonzero duplicate claims, rollback,
recovery contention, read-only capacity rechecks, and controlled HTTP
startup after commit. Another 74 pure/startup/tracing cases pass. The
complete CI matrix and both 5/5 reviews passed on `1ac9b58`; its
external security scan identified a stale-owner handoff race. The final
fix restores synchronous issue/run ownership fencing after the
accounting transaction commits. Final head `8e84f55ce8` has fresh
Greptile 5/5 with zero new comments, Paperclip review 5/5, and a passing
security scan. The complete latest-head [CI
matrix](https://github.com/paperclipai/paperclip/actions/runs/39535189808)
passes, including budget, browser, Runner, build, and canary jobs. The
unchanged Runner close fixture passed on its sole same-head job retry.
- Broad local `pnpm test:run` invocations were stopped before required
source updates (rebase and review fixes). They are not claimed as full
passes. Final full-suite coverage comes from the complete latest-head CI
matrix; local focused coverage and the incomplete broad invocations are
reported separately.
- Reproduce performance with `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-budget-enforcement.ts
--baseline-ref=6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837`; repeat with
`--reverse`. The harness uses disposable real PostgreSQL, 20 warm
samples per variant/count, and records raw timings, SQL/lock counts,
agent reads, unchanged agent state, and query plans. Final
production-source measurement revision:
`eff713a4f72c7f4b7a4002a701cb481d4bfccd95`; the extended harness is
committed in `4c5f4de8b1`.

| Unrelated idle agents | Baseline median / p95 | New median / p95 |
Baseline SQL / locks | New SQL / locks | New agent reads |
| --- | ---: | ---: | ---: | ---: | ---: |
| 0 | 1.80 / 2.33 ms | 2.99 / 3.88 ms | 5 / 1 | 7 / 1 | 0 |
| 17 | 15.53 / 19.55 ms | 2.24 / 2.60 ms | 90 / 18 | 7 / 1 | 0 |
| 1,000 | 978.35 / 1,790.43 ms | 2.16 / 2.52 ms | 5,005 / 1,001 | 7 / 1
| 0 |

- These measurements use local PostgreSQL 18.1 and Node 26.4.0 arm64.
They isolate the cost-evaluation core and complete admission path,
without hosted-region or provider latency. Prior-revision reversed-order
controls also preserved the constant query counts; the final
measurements above use the final production source.
- Complete reserve → final ownership gate → immediate offline provider
result is measured separately, with one responsible agent and one active
operation at each unrelated-agent count. Admission p95 baseline/new: 0
agents **11.54 / 29.77 ms**; 17 agents **9.31 / 27.62 ms**; 1,000 agents
**24.51 / 74.05 ms**. Baseline admission was already scoped; the
stronger final fence adds bounded overhead. New counts stay constant at
53 Drizzle statements, three transactions, two company-accounting locks,
and five agent-reading statements bound to the responsible agent. Zero
unrelated-agent reads, exactly one provider callback per sample, and no
leftover runs or holds. All admission scenarios stay below 250 ms. No
external provider calls are included.
- The partial-usage/delayed-termination fixture passed 20/20: receipt
persistence p95 13 ms, receipt return to observed durable Stop fence p95
28 ms, complete receipt-start-to-fence p95 50 ms, unrelated database
request p95 1 ms while termination was blocked. After releasing the
controlled termination fixture, acknowledgement p95 was 14 ms. These are
separately measured phases; fixture acknowledgement is not live-provider
termination latency.
- Models and provider deadlines are unchanged. No deployment, live
inference, or staging acceptance is claimed. A quiet 30-minute follow-up
waits for the merged revision on staging, then repeats classifier and
controlled budget-stop acceptance in staging, recording hosted
bookkeeping, queue delay, and actual provider acknowledgement
separately.
- Development failures retained: finalizer notification regressions
(fixed; 27/27 affected cases passed), invalid browser admission fixture
assumptions (fixed; browser coverage now passes 63/63), and first-head
lifecycle/dispatch CI regressions (reproduced and repaired). A
prior-head Cases E2E test hit its unchanged announcement-dismiss handler
race while opening a filter; the same journey passed on `1ac9b58`. The
external security review then found a stale-owner race between
accounting commit and provider entry; final verification covers a
competing run in that gap while retaining provider I/O outside the
accounting lock. Early PostgreSQL startup attempts hit local semaphore
pressure; serial reruns succeeded. The final-head Runner lane had one
failure among 1,823 tests: the unchanged `timeout-attachment` fixture
failed at its first process close after 6.28 seconds. The sole same-head
job retry passed all 1,819 active tests (four skipped), including that
fixture in 1.85 seconds. Three isolated local probes could not run
correctly because the Rust fake-provider fixture binary was absent;
cleanup masked the original startup failure. CI builds that fixture
through its normal lane. These probes are inconclusive and are not
counted as passes.

## Risks

- This changes accounting, lifecycle, and cancellation ordering. Durable
generations, source/sequence checks, ownership leases, and company
locking remain required for correctness.
- Provider stop acknowledgement can be slow or fail. Stop intent remains
pending until existing termination proof succeeds; cancelled row status
alone does not acknowledge delivery.
- Mid-run limits depend on provider usage reports or supported price
calculations. Providers that report only at completion cannot provide
exact mid-call detection.
- Signals are process-local wake hints for the supported
single-active-writer model. Durable state and startup recovery remain
authoritative.
- There are no migrations, new queue tables, or cached balances. Repeat
the classifier and controlled budget-stop acceptance after normal
rollout.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, code execution,
and parallel coding agents. The host does not expose the exact backend
model revision or context-window size, so those values are not asserted.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 07:23:00 -05:00
36e4c6657f fix(sentry): report a native retry failure only when the retry ends (#16865)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server sends a Sentry `run_failure` event when an agent run
reaches a terminal failure.
> - A native run that waits for a same-run retry is stored as `failed`
with `nativePhase: "retryable_failure"`. That state is not terminal.
> - `reportRunFailure` reported that pending state. Runs that later
recovered, for example after a workspace copy-back retry, still sent a
run failure event.
> - These events make one catch-all Sentry group noisy and hide the real
terminal failures in it.
> - This pull request skips the pending retry state and reports once
when the retry ends as a terminal failure.
> - The benefit is that run failure events match runs that really
failed.

## Linked Issues or Issue Description

**What happened?**

Two staging runs had a workspace copy-back failure during settlement
(`Daytona syncOut tar command failed ... No space left on device`). The
executor stored each run as `failed` with `nativePhase:
"retryable_failure"` and `errorCode: native_session_interrupted`. Sentry
received a `run_failure` event for each run. About two minutes later,
the copy-back retry succeeded and both runs finished as `succeeded` with
`workspaceFinalizeStatus: "succeeded"`.

**Expected behavior**

A run that waits for a same-run retry must not send a terminal run
failure event. If the retry ends as a terminal failure, the server must
send one event at that time.

**Steps to reproduce**

1. Set `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected`.
2. Start a native run that fails retryably. For example, make the
workspace copy-back fail once.
3. See the `run_failure` event in Sentry, and then see the run finish as
`succeeded` after the retry.

## What Changed

- `reportRunFailure` returns without a report for a native run that is
`failed` with `nativePhase: "retryable_failure"`.
- New `isNewTerminalRunFailure(before, after)`: a status change counts
as before. A failed run that leaves the pending retry state but stays
`failed` also counts.
- Three terminal writers can keep the status `failed` → `failed`. They
now use `isNewTerminalRunFailure`, so they still report once:
  - the resume reconciler's ambiguous-state write;
  - the restart recovery `blocked` write;
  - the committed-result projection (`projectCommittedRun`).
- Result-less retries already reset the run to `running` before they
start, so their terminal write still reports through the existing
status-change guard.
- `doc/observability.md` describes the new rule.

## Verification

- `pnpm exec vitest run
src/services/__tests__/run-failure-report.test.ts
src/__tests__/run-failure-sentry.test.ts
src/__tests__/native-session-resumption.test.ts
src/services/native-runtime/native-run-finalizer-telemetry.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/__tests__/heartbeat-status-preserving-telemetry.test.ts
src/services/execution-control-reconciliation.test.ts` (from `server/`):
7 files, 363 passed, 9 skipped.
- New tests:
- `run-failure-report.test.ts`: a pending native retry does not report,
and a `terminal_failure` run does. Unit cases cover
`isNewTerminalRunFailure`.
- `native-session-resumption.test.ts`: a sweep that ends an unreported
pending retry sends one event.
- `native-run-finalizer-telemetry.test.ts`: a committed failure for a
pending-retry row sends one event, and a replay sends none.
- `native-runner-restart-recovery.integration.test.ts`: a restart
recovery `blocked` write for a pending-retry row sends one event.
- I removed each of the three writer changes one at a time. The matching
new test failed each time (`expected [] to have a length of 1` / `to
deeply equal [ [ ObjectContaining{…} ] ]`).
- I did not run the full server `tsc --noEmit` locally. The local
container has a 4 GiB memory limit, and the process was killed (exit
137). CI runs the typecheck.

## Risks

- A run that stays in the pending retry state and never resumes no
longer sends a run failure event. The run stays visible on the board as
failed with a scheduled retry. A later terminal write still reports it.
- The finalizer's `failed` + `retryable_failure` write for a failed
provider result also stops reporting. The terminal write that ends that
retry reports instead: retry exhaustion in `recordRetryableFailure`
reports unconditionally, and commit goes through `finalizeNativeRun` or
`projectCommittedRun`.
- A run that was reported in the pending state before this change can
report once more when it becomes terminal after deploy. This affects
only runs in flight during the upgrade.
- There are no schema, migration, or API changes. Run status, retries,
and recovery do not change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, code search, test runs) and a read-only code
exploration subagent. Context window: the standard window for that
model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 09:29:50 -07:00
Devin FoleyandPaperclip e30550ef39 refactor: remove code comments added by recent PRs (#16681)
CONTRIBUTING.md says PRs add no code comments, but #16485, #16586, #16621 and #16672 added some. This removes the 215 comment-only lines git blame on master attributes to those merge commits, across 12 code files, plus one blank line the removal left doubled. Rewritten (not added) comments in scripts/assert-cloud-image-sentry.mjs and comments in .github workflows are left alone. No code changes.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-17 18:55:39 -07:00
Devin FoleyandPaperclip c7ea5c0216 perf(server): skip settled committed finalizations in unscoped passes (#16485)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - On restart, `/api/health` reports `starting` until startup recovery
finishes, and deploys wait for it
> - Native recovery and the orphan reap each run an unscoped native
finalization reconciliation pass, and readiness waits for both
> - The pass selects every committed finalization with a result,
including runs that ended long ago, and spends dozens of sequential
queries on each one
> - On a busy instance this is about one minute per pass, so about two
minutes of every restart, and the periodic recovery repeats it on every
tick
> - This pull request skips settled committed finalizations in unscoped
passes, and keeps a small rotating upkeep page so their upkeep still
runs
> - The benefit is faster restarts and less steady database load, with
no change for unsettled work

## Linked Issues or Issue Description

Related: #16330 and #16329 (startup and reconciliation timing).

**What existing behavior does this improve?**
Startup recovery time and steady-state recovery load.

**Current behavior**
- The unscoped candidate scan in `reconcileNativeFinalizations` returns
every native finalization with a result and phase other than
`terminal_failure`. Nothing excludes rows that are already settled.
- Each committed row then takes the full per-row path: pending-effect,
assessment, evidence and decision reads, presentation repair (twice),
the workspace barrier read, a `finalizeNativeRun` re-read of the
committed coordinator, and workspace settlement (lease release check and
agent status update). That's about 40–60 sequential round trips per row.
- Startup runs the unscoped pass twice (native recovery, then the orphan
reap), and readiness waits for both. The periodic orphan reap runs it
again on every tick.
- On one instance, all 268 candidates were committed finalizations whose
runs had ended. Native recovery took 50–80 s and the orphan reap 47–80 s
on every restart, with database round trips of about 3.4 ms.

**Proposed behavior**
An unscoped pass skips a committed finalization when:
- its run is `succeeded`, `failed`, `cancelled` or `timed_out`, and
- the run finished more than an hour ago, and
- its decision has no `pending` or `failed` status-decision effect.

Each unscoped pass also includes up to 10 such settled rows in a
rotating order, so the presentation and evidence upkeep still reaches
every row over time. Scoped passes (explicit run IDs) are unchanged.

**Reason and benefit**
These settled rows have nothing left to finish. Recent rows, unfinished
runs and rows with undelivered effects stay in every pass, so
crash-window repairs and effect delivery behave as before. Startup and
periodic passes no longer scale with the full history of committed runs.

**Breaking changes**
None. A late repair on a settled row can be delayed until its turn in
the upkeep rotation instead of the next pass.

## What Changed

-
`server/src/services/native-runtime/native-finalization-reconciler.ts`:
- `settledCommittedFinalization(now)`: the settled predicate above,
including a `NOT EXISTS` on undelivered `status_decision_effects` for
the same company, issue and decision.
- The unscoped candidate scan adds `NOT settled OR run in the upkeep
page`.
- `nextSettledCommittedUpkeep`: up to
`NATIVE_SETTLED_COMMITTED_UPKEEP_PER_PASS` (10) settled run IDs after an
in-process cursor, ordered by run ID, wrapping at the end.
- New constants `NATIVE_SETTLED_COMMITTED_GRACE_MS` (1 hour) and
`NATIVE_SETTLED_COMMITTED_UPKEEP_PER_PASS`, plus
`resetSettledCommittedUpkeepForTests`.
- New
`server/src/__tests__/native-finalization-settled-candidates.test.ts`.
- `doc/execution-semantics.md`: describes the settled rule and the
upkeep rotation.

## Verification

- `pnpm exec vitest run
src/__tests__/native-finalization-settled-candidates.test.ts` (in
`server/`): 4 passed. The test mocks the timing hook to capture the
candidate count and stop the pass after the scan:
- 25 settled rows give 10, 10, 5, then 10 again over four passes
(rotation and wrap).
- A row finished 10 minutes ago, a running row, and a terminal row
without `finished_at` are all candidates.
  - A settled row whose decision has a `failed` effect is a candidate.
  - A scoped pass with an explicit run ID still selects a settled row.
- `pnpm exec vitest run` for `native-finalization-recovery`,
`legacy-finalization-regression`, `native-run-finalizer-telemetry`,
`native-workspace-finalizer`, `native-finalization-reconciler-timing`,
`external-chat-wait.integration` and `native-status-arbiter-corpus`: 316
passed.
- `heartbeat/recovery.test.ts`: the 8 tests that do not need a database
passed. The 20 database tests were skipped locally because embedded
PostgreSQL could not start (the host's shared-memory segments were
exhausted by other processes). CI runs them.
- `pnpm typecheck` in `server/`: passed.

## Risks

- A settled row that needs a late repair (for example a missing chat
presentation) waits for its turn in the upkeep rotation, up to
`ceil(settled rows / 10)` passes, instead of the next pass.
- The upkeep cursor is in memory. After a restart the rotation starts
from the beginning; the work is idempotent.
- A run left `running` with a committed finalization is never considered
settled, so it is still projected before the reaper's active-run loop.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, test runs, database and log analysis).
A separate Claude agent mapped the recovery call tree.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-15 16:42:15 -07:00
Devin FoleyandPaperclip 307c27e3d8 fix(native): reclaim task execution before same-run recovery
Restore the original task execution binding atomically during native same-run recovery. Preserve current owners, stops, pauses, assignments, and dependency waits.

Isolate candidate failures, bound deferral writes, prioritize undated retries, and allow recovery ahead of queued waiters without premature queue promotion. Add PostgreSQL attachment, ownership, contention, fairness, and queue regressions.

Verification: 99 focused tests, workspace typecheck, build, full current-head CI, and Apex 5/5.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-15 01:53:26 -07:00
Devin FoleyandPaperclip cda77edc60 perf(server): stop readiness waiting on startup housekeeping (#16329)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - On restart, `/api/health` reports `starting` until startup recovery
finishes, and deploy tooling waits for `ok`
> - After heartbeat recovery, readiness also waits for housekeeping that
every scheduler tick repeats and that serving requests does not need
> - #16330 added OpenTelemetry spans for the main startup phases; native
recovery and orphan reaping are the slow ones, and both run the native
finalization reconciliation pass, which has no inner breakdown
> - This pull request stops readiness waiting on that housekeeping, and
records the phases of slow reconciliation passes as spans
> - The benefit is faster readiness on every restart, and traces that
show where a slow reconciliation pass spends its time

## Linked Issues or Issue Description

Builds on #16330, which records `controller.startup.<phase>` spans.
Related to #16328.

**What existing behavior does this improve?**
Server readiness time after a restart, and diagnosis of slow native
recovery.

**Current behavior**
- After heartbeat recovery, readiness waits for the custom-image setup
cleanup, tool connection health (remote MCP checks), decision expiry,
the adapter and setup-token login reapers, environment lease cleanup
(provider calls) and the retention sweep. Each also runs on every
scheduler tick. On one instance they added 11–28 s to every measured
restart. A throw in any of them fails startup.
- On the same instance, native recovery took from 33 s to 11.5 min and
orphan reaping from 30 s to 4 min across restarts in one hour. Both run
the native finalization reconciliation pass. The #16330 spans time each
step as a whole, but not the phases inside that pass.

**Proposed behavior**
- The housekeeping steps run after readiness. The scheduler interval
still starts after them.
- With an OTLP endpoint, a reconciliation pass of at least one second
records a span with one child span per phase.

**Reason and benefit**
Readiness depends only on what serving requests and run recovery need.
Slow reconciliation passes become attributable to a phase in the
existing traces, without adding trace volume for the periodic pass.

**Breaking changes**
None. A failed housekeeping step no longer fails startup; it is logged
and the next step runs.

## What Changed

- `server/src/index.ts`: moves the housekeeping steps into
`runStartupHousekeeping`, which runs after readiness under
`trackHeartbeatSchedulerWork` (idle sleep and shutdown wait for it).
Each step catches and logs its own failure (`startup housekeeping step
failed`). The heartbeat scheduler interval starts when housekeeping
finishes, unless the scheduler was stopped, so the reapers still finish
before the first timer tick.
- New
`server/src/services/native-runtime/native-reconciliation-timing.ts`:
records the phase boundaries of one pass. When the pass took at least
one second and an OTLP endpoint is configured, it emits a
`controller.native_reconciliation` root span
(`paperclip.native-reconciliation`) with child spans
`legacy_workspace_exports`, `obsolete_policy_reviews`,
`automatic_completion_reviews`, `candidate_scan` and `candidates`, using
the recorded start and end times. It records `scoped` and the candidate
count only. A failed pass marks the pass span and the running phase.
Tracing failures are swallowed.
- `native-finalization-reconciler.ts`: the exported function wraps the
pass in that timing and finishes it in `finally`, so failed passes are
recorded; the original error is rethrown.
- `doc/execution-semantics.md`: what readiness waits for, and the
housekeeping order. `doc/observability.md`: the reconciliation spans.

## Verification

- `pnpm exec vitest run
src/__tests__/server-startup-feedback-export.test.ts
src/services/native-runtime/native-reconciliation-timing.test.ts
src/services/native-runtime/native-finalization-reconciler-timing.test.ts
src/startup-recovery-timing.test.ts` (in `server/`): 52 passed. New
tests:
- Readiness is `ready` while a housekeeping step hangs, and the
heartbeat interval starts only after the step settles.
- A failed housekeeping step is logged, and the scheduler still starts
after the remaining steps.
- Span timing: nothing for a fast pass or without an endpoint; a slow
pass emits the pass span and one child per phase with the recorded
times; a failed pass marks the running phase and the pass; a failing
tracer does not throw.
- The reconciler records its phases and a failed outcome when a pass
throws, and rethrows.
- Three existing startup tests settle housekeeping before they read the
scheduler interval or the setup cleanup.
- `pnpm exec vitest run src/services/native-runtime`: 4,399 passed.
- `pnpm typecheck` in `server/`: passed.

## Risks

- Housekeeping now runs concurrently with the first requests and with
`waitForExternalAdapters`. None of these steps is needed to serve
requests, and each one also runs on every scheduler tick.
- Readiness is reported earlier; the heartbeat timer starts at the same
point relative to housekeeping as before.
- The new spans exist only with an OTLP endpoint and only for passes of
at least one second.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-14 15:55:13 -07:00
DottaandPaperclip a6306ba606 feat(runner): consolidate Cursor production integration (#15075)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native Runner keeps provider sessions under company authority,
approvals, budgets and durable recovery.
> - Cursor work was spread across candidate branches. The published
branch lacked later plan, permission and cleanup fixes.
> - Production also needs public installation and matching runtime
assets for local and Daytona execution.
> - This pull request consolidates Cursor onto current mainline recovery
behavior and completes that installation path.
> - The installed v11 release passed focused local and Daytona
qualification after the generic mode and lifecycle cleanup. The later
model-selection correction and current mainline merge produce v14
artifacts that need matching release qualification.
> - Cursor admission is enabled in source; publish only an artifact
combination with matching qualification. Native AskQuestion and complete
per-run dollar accounting remain excluded.

## Linked Issues or Issue Description

Refs: #14435, #14631, #14669, #14699, #14724.

This completes the Cursor implementation by @cryppadotta from combined
source `22c78242a4e0c2369fecf0c2dc4e7600fbad6706`. It preserves newer
mainline recovery, completion and warm-directory behavior. Pi and
Copilot remain gated.

## What Changed

- Generate named Rust and TypeScript ACPX release profiles from one
manifest. Share runtime pins with packaging and server verification.
Preserve vendor runtime versions; bind the updated ACPX patch to Cursor
profile v14 and reject stale generated declarations at build/typecheck.
- Remove ACPX model allowlists, including the former Codex and Pi
restrictions and the duplicate developer test-drive gate. Send any
explicit model ID unchanged to its provider and verify the effective
selection before prompting. The bundled ACPX package forwards unlisted
IDs, rejects mismatched acknowledgements, and restores the exact
selection after session load. It does not expand Cursor model aliases.
Provider rejection, mismatch, or missing model controls fails without a
fallback. Model examples live in evaluation fixtures, outside runtime
declarations.

- Add pinned Cursor execution, contained instructions, exact model
verification and Agent/Plan/Ask modes.
- Carry an opaque generic `mode` identifier in shared native execution,
sidecar, Rust and recovery contracts. The provider adapter owns
supported modes, defaults, native translation and acknowledgement.
- Keep native RPC recognition, accepted-plan interpretation and
permission evidence behind provider adapters. Shared settlement and
recovery verify normalized facts and their committed evidence.
- Replace the Cursor-only warm-attachment branch with a runner-owned
capability. Only Cursor opts into it. Move profile compatibility and
optional usage parsing into provider metadata and adapters.
- Write generic plan-wait receipts. Read exact historical Cursor
receipts through a separate compatibility decoder. Reject mixed formats
and preserve existing authority checks.
- Carry native plans, semantic questions, todos, child activity,
permission identities and partial usage diagnostics through the Runner.
- Preserve durable response delivery, cancellation, warm ownership and
process retirement.
- Finish accepted planning runs successfully. Keep their tasks open for
explicit direction. Acceptance does not start implementation.
- Ship `paperclipai runtime setup cursor` and its provisioner through
the public package. npm installation does not download Cursor. Setup
uses the OS account's closure-keyed cache so system-wide npm packages
can remain read-only. Run it as the Paperclip service account.
- Include Cursor in normal provider packs and Daytona images for macOS
ARM64/x64 and Linux x64.
- Reject stale release packs by source revision and current ACPX/Cursor
pins before assembly writes files. Verify current Cursor
version/profile/closure again at runtime.
- Ship all three daemon targets and the expected Linux image-pack
identity. A macOS controller uses its packaged Linux daemon for Daytona.
Image mismatches fail before provider launch.
- Use the vendored Runner boundary for installed readiness probes.
Verify the actual installed Cursor probe.
- Verify compiled public Daytona plugins and their release versions in
installed smokes.
- Record exact artifacts, the acceptance matrix, retained failures,
supported capabilities and rollback behavior in the [readiness
report](https://github.com/paperclipai/paperclip/blob/codex/cursor-production-readiness/doc/plans/2026-10-03-cursor-production-readiness.md).

## Verification

- Current head `f7ec5cc1f0e30c62a829c016ff2013a2a9d79037` merges
mainline `faa8e452c73bae5e044dd6379179a00106abb131`. It keeps Cursor
plan and cancellation guards alongside mainline historical-question
filtering. The evaluation catalog includes both Cursor and expanded
adapter accounting cases (683 total). Recursive typecheck, full build,
696 lifecycle/recovery tests, 45 fixture tests and fixture typecheck
passed. Current-head CI passed: 56 successful checks, one neutral and
four skipped. [Complete
CI](https://github.com/paperclipai/paperclip/actions/runs/37557996535).
The fresh Base Greptile review is 5/5 on this exact head, with 304 files
reviewed, zero new comments and zero unresolved threads. The user
authorized overriding the CODEOWNER review gate after checks passed; no
failing checks are overridden. Prior results below retain their own head
identities.
- Corrective head `3d2b168366258036f6b6a6fccb382c49138cc601` fixes the
post-merge Apex finding. Automatic-review and new-evidence
reconciliation preserve pending child results and recheck delivery under
the status lock before completing. Account repair now excludes unrelated
secret consumers and requires the failed agent's identity. Regression
coverage includes the commit race, delivery statuses,
current-run/current-intent exclusions, repeated reconciliation, both
database reconciliation paths, and credential consumer boundaries. All
184 affected tests, server typecheck and server build passed.
Current-head Base Greptile review is 5/5, with 304 files reviewed, zero
new comments and zero unresolved threads. Current-head CI passed: 56
successful checks, one neutral and four skipped. [Complete
CI](https://github.com/paperclipai/paperclip/actions/runs/37535994724).
This Base review is distinct from the earlier Apex review.
- Merge head `5957c257a` reconciles mainline `b508a05c4`. It preserves
both accepted-plan waits and pending-child-completion checks, current
provider selectors, task-creation response identities, and mainline ACPX
missing-file handling. The combined patch is bound to Cursor profile
v14; historical records keep their original identities.
- Merge head `5957c257a` passed recursive typecheck, full build, 43
installed ACPX/package contracts, 107 provider UI and plan/recovery
tests, 593 database-backed lifecycle tests, 49 profile/native contract
tests, 45 Product E2E fixture tests, fixture typecheck, token gates,
three provider-free browser task-creation cases, and Runner
conformance/replay checks. Its complete CI passed (55 successful checks,
one neutral and four skipped), while Apex returned 2/5 with a
child-delivery finding addressed below.
- The local full-suite attempt again failed the unchanged Git streaming
test (360-second timeout) and was stopped. The concurrent local Rust
attempt failed four unchanged Codex process/deadline tests; all four
passed serially without code changes in 7.29 seconds after removing the
competing test load. These failed commands are retained and are not
reported as full-suite passes; the fresh Linux CI runs are tracked
separately.
- The previous head `907bdb2a2778c7ffeb4a662a91460c9d1ddfc9c5` earned
Apex 5/5 with zero comments after fixing all three findings: per-user
install cache, stale release-pack rejection, and public Linux smoke
account/home handling. Its real built installer passed from read-only
public packages on macOS ARM64 and Linux x64. All 137 release-registry
checks and 64 ACPX package contracts passed. That review does not cover
this mainline reconciliation.
- Prior `beadd3654` passed the full CI matrix; its one unchanged chat
test failure and successful single retry remain in the [CI
history](https://github.com/paperclipai/paperclip/actions/runs/37521449327).
Historical results below remain attributed to their original builds.

- Fixture follow-up `dd59d7e82b103a88b7cbd7d2c38b612c0fbbff7a` removes
provider-specific model choices from generic offline ACPX tests. The
fake sidecar preserves the model and session identity selected at open
through suspension. Affected verification passed: 106 Rust tests and 73
TypeScript tests. This commit changes test code only; the
production-code checks below retain their recorded identities. Its CI
and Greptile review later passed; those results belong to that
historical head.
- Model-selection cleanup `9a070808b48960a41fdfd369ae0636b95af82459`:
252 focused Runner tests passed (six platform skips), covering all six
ACPX agents, native model acknowledgement, rejected selections,
installation integrity and recovery identity. The merged branch passed
recursive typecheck, full build, token gates, server admission (19
tests), and the Product E2E catalog (45 tests). The acceptance catalog
passed all four tests. The full Rust suite passed: 643 tests, 2 ignored.
It verifies sidecar acknowledgement of unlisted models and rejection of
model mismatches. The final commits only update Rust tests; production
sources match the verified build at
`65ec3279ac50185e3cda109b5cfd9b4f56105de0`. No new paid provider calls
were made.
- The merge preserves both Cursor and the new mainline public-MCP
fixture cases. Auto-merge remains disabled; the latest follow-up status
is recorded above. The local `pnpm test:run` attempt hit the unchanged
Git streaming test's 300-second timeout and was interrupted before
merging mainline. The broad Runner attempt found obsolete single-model
assertions plus three macOS fixture-path failures caused by a
`/private/tmp` override. The assertions are corrected; affected
TypeScript checks passed with the standard macOS temporary directory,
and the complete Rust suite passed. Neither interrupted command is a
full-suite pass.
- Earlier declaration-cleanup head `6f4a5e9e2` passed recursive
typecheck, build, Rust and focused tests. Its CI later exposed a test
expecting duplicated Grok digest literals. The current source fixes that
assertion to compare launcher bytes with the shared manifest. Historical
successes and failed attempts are retained; no new live provider
qualification is claimed.
- Previous head `e75fde6098b0ddd8cec765bfb6ecaeecb88a26a6` passed
complete CI (56 successful checks, one neutral, four skipped) and
Greptile 5/5. [Historical complete
CI](https://github.com/paperclipai/paperclip/actions/runs/37489112305).
Those results are not claimed for the cleanup head.
- Frozen live application: `d7b696f9b8f79095233e9e3d56d23e6a6018dd48`.
Public package version: `0.0.0-cursor-verify.3d0c9b7761c6`. The
declaration cleanup preserves release pins and does not relabel that
tested artifact as a build of the new source. Mainline through
`e34abee670` was reconciled while preserving accepted-plan waits,
provider-capacity handling, and both Cursor and public-MCP fixtures.
- Clean normal installation, explicit Cursor setup and daemon resolution
passed on macOS ARM64, macOS x64 under Rosetta, and Linux x64. npm
lifecycle hooks ran without silently downloading Cursor.
- Historical v11 live matrix: **18/18 passed with cleanup** (nine local,
nine Daytona) after the generic mode and lifecycle cleanup. The campaign
has 23 attempts; all five failures and their diagnoses remain recorded.
Exact case identities, hashes and limits are in the readiness report.
All provider calls are real, use the explicit Luna model and
company-bound credentials, and run without qualification or
runtime-asset overrides.
- The immutable Daytona image is
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:d6259b6bba094702c13fc2283bd85550849c1c53145b656fb2746778f9fa1747`.
The public Daytona plugin is installed independently and its version is
checked.
- Recursive typecheck, full build, token gates and Runner
contract/conformance/replay checks passed on the frozen application. Its
complete Linux CI suite passed. The duplicate local full-suite command
was incomplete after timing failures; affected repeats passed, but that
command is not reported as a clean pass.
- Qualification fixtures passed typecheck, 1,675 Vitest tests (one
skip), 128 Node checks, three provider-free browser tests, and 150
focused lifecycle tests after the final diagnostic correction. The
affected legacy Cursor command file also passed all five tests after
removing its shorter 10-second override; it now inherits the suite’s
standard 15-second timeout. Greptile is 5/5 on `e75fde609` with no
unresolved review threads. CI results above are recorded separately from
historical build results.

## Risks

- Cursor v14 includes the updated ACPX dependency patch and release
identity. The v11 live matrix and image below remain historical
evidence. They do not certify new v14 package/image artifacts.

- ACPX accepts models beyond the qualification fixtures. Availability
and entitlement depend on the provider. Successful configuration is not
a claim of live qualification for every model.
- Shared mode is an opaque identifier. Provider adapters own its
meaning. Incompatible historical sessions remain fenced; exact committed
plan waits and task history remain inspectable.
- Native AskQuestion is excluded. Paperclip semantic questions are
supported. Authoritative per-run dollar accounting is unavailable;
partial counters remain diagnostics and unknown cost is not zero.
- Image input, detailed native diffs, deeper child transcripts and
native plan-file export remain follow-ups.
- macOS x64 has clean-install and daemon-startup proof under Rosetta,
not a separate live campaign on Intel hardware.
- Release only the tested package/image combination. Merging this PR
does not publish npm packages or deploy that image. Later builds need
their own release verification. Rollback disables new Cursor admission
while preserving records and recovery inspection.
- A model can fail an exact instruction: one cancelled-plan attempt
returned the wrong summary marker despite correct cancellation. The
unchanged repeat passed; both results remain in the report.

> ROADMAP.md was checked. This completes existing native Runner/Cursor
work; it does not add an independent core feature proposal.

## Model Used

OpenAI Codex, GPT-6. The exact serving variant and context window are
not exposed in this session. The agent used reasoning, repository
inspection, code execution, protocol tests and browser-backed Product
E2E tools. Cursor acceptance uses the explicit
`gpt-5.6-luna[context=272k,reasoning=medium,fast=false]` model. That is
the evaluated provider model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — affected suites passed;
full CI and the retained local failed attempts are recorded separately
above.
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green — 56 successful checks, one
neutral and four skipped on `f7ec5cc1f0e30c62a829c016ff2013a2a9d79037`
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups —
fresh Base review passed on `f7ec5cc1f0e30c62a829c016ff2013a2a9d79037`;
zero new comments and no unresolved threads. The earlier Apex finding
remains fixed.
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-06 20:48:15 -05:00
DottaandPaperclip a65ca09508 fix(runner): settle accepted results after shutdown failures (#15217)
## Thinking Path

> - Paperclip manages AI agents and the tasks they perform.
> - The native runner saves tool results and completion reports before
it releases a session.
> - Large project discovery responses can exceed the durable command
limit.
> - A shutdown failure can leave a saved answer waiting for workspace
repair.
> - Recovery reused the old assessment for a different status decision,
which violated a database constraint.
> - This pull request bounds discovery responses and lets recovery
commit the saved result after workspace repair.
> - The benefit is a task that reaches its correct final status without
another provider turn.

## Linked Issues or Issue Description

**What happened?**

A native run can save its final answer, fail during shutdown, and leave
the task In Progress after workspace repair succeeds. Reconciliation
tries to reuse the failed-workspace assessment for a new decision. The
one-decision-per-assessment constraint rejects the write. Replaying the
old decision can also retain a fresh coordinator lease. Separately,
retained session cleanup only recognizes the old `adapter_failed` label.

The project-list tool returns full project records, including large
descriptions and workspace configuration. A large response exceeds the
runner's durable command limit. The settlement diagnostic previously
recorded only a failure flag.

**Expected behavior**

Project discovery stays within the command limit. Recovery finishes the
saved result after workspace repair, releases its lease, and preserves
the original error for inspection. It does not repeat provider work or
relax session ownership checks.

**Steps to reproduce**

1. Return large project records from `list_projects` and observe an
oversized semantic result.
2. Persist an accepted native completion result, then record a shutdown
failure.
3. Finalize with a failed workspace, repeat that attempt, then record
successful workspace repair.
4. Reconcile the run. Before this fix, the issue stays In Progress.

**Paperclip version or commit**

Reproduced on `a386a599983519eb1d399f8b770bfccdb2a74762`.

**Deployment mode**

Self-hosted server with Paperclip Runner.

Related transport work: #12208 drains queued events; #12241 resumes
interrupted semantic calls. This change addresses bounded project
discovery and accepted-result finalization.

## What Changed

- Read bounded project summary projections from the database and return
at most 50 authorized summaries with a continuation cursor and explicit
description truncation. Agent and run trust boundaries narrow the
database candidates; project-specific policies still receive full
authorization. Only visible projects determine continuations. The
default project-list API remains unchanged.
- Record bounded, content-free settlement failure causes for command
limits, storage errors, and rejected dispatch.
- Include workspace state in assessment identity. Preserve the initial
assessment for interrupted finalization, and commit replacement
assessment and decision references together.
- Release the coordinator lease when an existing decision is replayed,
without repeating its effects.
- Clear stale errors when recovery succeeds and retain them in
`recoveredExecutionFailure`.
- Accept both current and legacy close-failure labels in the existing
exact-state cleanup path.
- Add regression tests and update the tool contracts and recovery
documentation.

## Verification

- Red: the new project paging, settlement diagnostic, current cleanup
label, and repaired-workspace regressions failed on the original
implementation.
- Green: protocol/catalog/tool checks (117 tests), the full project-tool
and finalizer suites (47 tests), cleanup ownership cases (70 tests), and
cleanup sweep cases (4 tests) pass. Database-backed pagination covers
large descriptions/configuration, complete enumeration, uppercase
cursors, agent/run restrictions, project-policy scope contributions, and
identical results/cursors when hidden projects are added.
- The initial CI failures in interrupted Board waits, contended endpoint
proof, and semantic schema validation were reproduced and fixed; all
affected cases pass locally.
- `pnpm -r typecheck` — passed.
- `pnpm build` — passed.
- `pnpm test:run` — started before the review corrections; it spanned
several source revisions and was stopped after reporting old-behavior
and timing failures. It is not claimed green. Fresh project and recovery
suites pass; the recovery suite also passes all 25 cases with the broad
runner’s isolated home/config. The Slack timing case passed in
isolation. Latest-head CI is the authoritative complete test matrix.
- Existing authorization suite — 66 tests passed.
- Latest-head CI on `8e5763d915aa6f375bdab6601996899ea01496fc` — 55
checks passed, 4 intentionally skipped, no pending or failing checks.
- Greptile — 5/5, zero unresolved threads on the same commit.
- `git diff --check` — passed.

## Risks

- `list_projects` now returns summaries. Callers must follow
`nextCursor` and use the authorized project API for full records.
Candidate narrowing is only an optimization: project policy and
responsible-user authorization remain authoritative.
- Assessment identity changes for workspace finalization. Existing
evidence remains intact; no migration is needed.
- Cleanup still requires matching identities, settled tool evidence, and
verified process ownership. Unknown tool outcomes remain blocked from
session reuse.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, code execution, and GitHub
tools. The exact serving model ID and context window are not exposed in
this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-05 10:31:11 -05:00
DottaandPaperclip 14795136f5 fix(runtime): finalize and recover sandbox workspace exports safely (#14402)
Serialize native workspace finalization, validate streamed archives within bounded limits, and quietly recover unsafe exports from saved results. Preserve exact allocations for exhausted transient failures and provide export-only retry without rerunning the provider.

Consolidates #14314, #14315, #14329, and #14334 while preserving the already-merged finalization label changes.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-28 10:57:32 -05:00
Nicky LeachandPaperclip c276d3fdc3 feat(observability): report terminal run failures to Sentry (#13446)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server records the state of each agent run.
> - Terminal run failures need clear error tracking for operators.
> - The server did not report terminal `failed` or `timed_out`
transitions to Sentry.
> - This pull request reports each genuine terminal failure transition
with safe diagnostic data.
> - The benefit is faster diagnosis without changing run control flow or
exposing credentials.

## Linked Issues or Issue Description

**What happened?**

The server wrote terminal run failures but did not report them to
Sentry. Operators could not see these failures in error tracking.

**Expected behavior**

The server should report each genuine transition to `failed` or
`timed_out` to Sentry.

**Steps to reproduce**

1. Run an agent task that reaches a terminal failure state.
2. Inspect the Sentry events for the server.
3. Observe that the terminal run failure has no matching Sentry event.

**Paperclip version or commit**

The change targets the current `master` branch.

No public GitHub issue or pull request covers this change.

## What Changed

- Add `captureRunFailure()` as a fail-open Sentry entry point.
- Add `reportRunFailure()` to filter status, resolve the adapter, redact
text, and report the failure.
- Call `reportRunFailure()` beside each of the eight terminal status
writers.
- Report six diagnostic values: the instance host, task identifier, run
identifier, error message, error code, and agent adapter.
- Group events by error code and agent adapter while keeping the
redacted message in the event.
- Report only genuine transitions and avoid duplicate finalization
events.
- Keep Sentry failures outside run control flow.

## Verification

- `pnpm vitest run
server/src/services/__tests__/run-failure-report.test.ts
server/src/__tests__/run-failure-sentry.test.ts
server/src/__tests__/native-session-resumption.test.ts`
- `pnpm vitest run
server/src/services/execution-control-reconciliation.test.ts`
- `pnpm --filter @paperclipai/server typecheck`
- The full continuous-integration suite must run on this pull request.

## Risks

- The report path can add diagnostic events when Sentry is configured.
- The report path returns without action when Sentry is not configured.
- Redaction runs before length limits and before the event leaves the
process.
- The change has no migration and no schema change.

## Model Used

OpenAI Codex, GPT-5, tool use and code review support. The exact context
window and reasoning configuration are not exposed by the runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-14 23:32:15 -07:00
DottaandPaperclip 422287eecd fix: preserve runner recovery, warm sessions, and task outcomes (#13338)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runner connects task messages, provider execution, and
task outcomes.
> - First-time user tests exposed gaps in recovery, completion
permissions, message delivery, and Stop behavior.
> - These gaps left usable output hidden, completed work waiting for
bookkeeping, or safe work unable to continue.
> - This pull request fixes the shared lifecycle and receipt paths while
preserving process ownership and action checks.
> - Users can continue work with accurate task state and durable
messages.

## Linked Issues or Issue Description

**What happened?**

A stopped local Codex execution could remain blocked even after its
processes had stopped and its complete transcript proved that no
external action needed replay. Claude under Conservative permissions
could fail to call task completion tools. Recovery could reuse an
assistant item ID and overwrite prior output. A delivered comment could
remain marked uncertain after navigation. Stop could look like Pause or
a new recovery incident. Workspace contention could look like
cancellation. A direct reply reopening Done could enter a clarification
loop.

**Expected behavior**

Recover automatically only with verified termination and complete action
receipts. Preserve answers and messages. Keep task completion available
under Conservative permissions without broad tool access. Show crashes
as Blocked, actual human decisions as In Review, and ordinary workspace
contention as waiting. Stop the current response and allow a new
direction.

**Steps to reproduce**

1. Create ordinary response tasks with local Codex and Claude Code, then
send follow-up messages through the task composer.
2. Interrupt a disposable local Codex runner during text-only work.
Verify automatic continuation and retained output.
3. Stop a response, send a new request, answer a clarification, and
reopen completed work with another message.
4. Navigate or reload while a comment submission is pending. Confirm the
exact persisted request receipt settles it without removing newer draft
text.
5. Run two tasks in a shared Daytona workspace. Confirm waiting does not
appear as failure.

**Paperclip version or commit**

Initial acceptance baseline: `c9021c6721f91e2c74bd9fee9d3fd41c999d17b7`.
Current integration base: `6cef9743c`. Both operator-interruption and
workspace-waiting guards are preserved; native restart and legacy
permission rules remain documented.

**Deployment mode**

An isolated source-built test-drive instance, with real local Codex and
Claude Code providers and disposable Daytona environments.

Related work: #13314, #13316, #13327, #13344, #13239, #13254, #13163.
This PR addresses additional failures from ordinary task journeys,
including controller restart handoff and repeated warm sandbox setup.
Historical task status reconciliation is excluded.

## What Changed

- Persist runner ownership immediately at spawn and resume an explicitly
adopted runner even when the controller crashed before the first driver
checkpoint. Detach the controller safely across graceful restarts,
including session startup. Prevent an old finalizer from suspending or
signaling an adopted runner. Checkpoint idle warm sessions before
shutdown. Preserve the same run and queued follow-up messages.
- Scope saved legacy queue successor checks to the queue owner while
preserving ordinary task locks, operator identity, assignment gates, and
exactly-once delivery.
- Preserve managed Codex credential files when an old session is
detached for restart; normal owned cleanup still copies refreshed auth
back and removes the scoped copy.
- Reuse the bound warm shared sandbox and fully verify an existing
staged provider pack before using it. This avoids repeated uploads when
the pack is already valid.
- Add a narrow local Codex replacement path with stopped-process proof,
a closed transcript inventory, exact completion receipts, and
fresh-session lineage. Preserve no-replay holds when evidence is
incomplete. Recovery may clear only the same run's recorded Blocked
status version; manual re-blocking and dependency changes invalidate
that receipt, while queued comments do not. Later blocks stop scheduled,
queued, and final dispatch; queued/final checks re-read dependencies
even when the task status stays In Progress.
- Permit only task delivery and human-input tools through the isolated
Claude runner's exact task bridge.
- Scope assistant item identity to the provider turn and ignore only
authority-free Codex skill-change notifications during startup.
- Reconcile composer submissions by client request ID across response
loss, navigation, and reload. Retain text typed during delivery.
- Keep acknowledged run-only Stop neutral and show workspace contention
as waiting. Project exhausted native failures as Blocked.
- Restore the guarded task-page retry action for failed legacy runs,
including the server-supported explicit new-attempt path for stopped
conversation adapters. Preserve native/process recovery holds and avoid
promising Retry while a decision or execution gate hides it.
- Refresh delivered artifacts and handle direct user replies that reopen
completed work without a clarification loop.
- Check the embedded PostgreSQL PID, data directory, and actual port
before connecting or migrating.
- Document accepted behavior and add focused regressions at lifecycle,
route, transcript, and UI boundaries.

## Verification

- Final head `fece606ac2` passes the complete GitHub CI matrix: **34
green checks, two expected Storybook skips, no failures or pending
checks**, including `ci / verify`, `ci / e2e`, full runner verification,
typecheck, build, every server/workspace shard, and all browser shards.
[CI
run](https://github.com/paperclipai/paperclip/actions/runs/34727183287).
Greptile is **5/5 with no open findings**. The final two commits only
refine test fixtures; both affected suites pass 24/24 locally and in CI,
with server typecheck green.
- Complete local Vitest coverage uses the canonical groups/shards: all
635 general server suites, all 145 serialized suites, and all workspace
packages. The aggregate began on `0a8001c18` while the final queue fix
arrived: 23,903 passed, five failed, 87 skipped. The five
port/socket/timing failures passed unchanged in follow-ups (60 tests in
the exposure/file suites and 412 tests covering the serialized failures
and unrun tails). The final queue/operator-identity suites separately
passed 52/52. This is aggregate coverage plus explicit reruns, not a
pristine single-command final-head run.
- After integration with current master,
queue/operator-identity/continuation suites passed 162/162 and affected
UI suites passed 140/140. ACP Stop/continuation and legacy
task/Inbox/message browser suites passed 9/9, including both task
recovery Retry and thread Try again, automatic saved-message delivery,
exactly one new run, Done, and retained output after reload. The default
process Stop/Pause/Resume browser case passed (the native-provider case
is opt-in and skipped by default). The complete Board attachment/receipt
browser suite passed 11/11 on a disposable instance, covering both
composers, exact receipts after lost responses, no replay, bound
attachments, and newer drafts after reload.
- Blocking-intent regressions cover pre-existing Blocked, a mismatched
run/cause, an explicit manual re-block, changed dependencies, a queued
comment after failure, and a block arriving between scheduling and
provider dispatch. The negative cases reproduced before the fix. All 478
affected executor/recovery/dispatch tests passed; both database suites
ran separately after availability-probe skips in the first combined
command. The final late-dependency check passed all 143 affected
recovery/dispatch tests (zero skips) after two new negative cases
reproduced the bug.
- Focused runtime regressions cover awaited runner ownership
publication, authenticated adoption before the first checkpoint,
old-finalizer detachment, idle and busy warm-session shutdown, rejected
checkpoint propagation, provider-pack verification, and managed-Codex
credential preservation. Four managed credential detachment cases
reproduced the bug before the fix; normal owned cleanup still succeeds
exactly once.
- Live local Claude: SIGKILL 2.6 seconds into startup recovered the same
run automatically in 53 seconds, then a normal follow-up completed in 24
seconds. SIGTERM 2.5 seconds into startup preserved the same run (54
seconds) and its queued follow-up (21 seconds). Answers remained visible
and the task reached Done.
- Live Claude Daytona: a warm follow-up retained its sandbox and fell
from 121 seconds to 44 seconds. A separate cold turn took 127 seconds;
after controller shutdown and checkpointing, its follow-up completed in
33 seconds with the same sandbox, workspace, native session, and runner.
Both answers remained visible and the task was Done.
- Other live journeys covered task completion and follow-up with local
and Daytona Codex, local Codex crash recovery, Stop then new direction,
clarification response, live artifact refresh, and shared-workspace
waiting.
- Validation limits: the opt-in native composer Stop/Pause→subtree
Resume fixture exposes terminal/result ordering and subtree-cancellation
attribution bugs that can leave a child task blocked; that new finding
is assigned to a separate follow-up and is not claimed fixed here.
Default CI skips this optional native-provider fixture. Managed-Codex
credential handoff and the queue-agent integration use automated
regression evidence. Cold custom provider-pack uploads still add startup
latency.

## Risks

- Automatic replacement remains deliberately narrow: local Codex,
verified stopped identities, unchanged retained state, and a complete
text/completion-only turn. Unknown actions, partial history, or changed
ownership remain blocked.
- Claude completion permission handling changes an upstream package
patch. The exact isolated task bridge must remain pinned; unrelated
tools keep their existing permissions.
- New task failure projection changes user-visible status. No historical
status backfill or database migration is included.
- This is a broad lifecycle fix across server and UI. Live proof covers
graceful local Claude restart during startup and idle Claude Daytona
session recovery across controller shutdown. Live abrupt SIGKILL during
local Claude startup also recovered the same run. Unknown ownership or
missing action evidence still blocks reuse. Cold custom provider-pack
uploads still add startup latency; this change avoids unnecessary repeat
uploads.

## Model Used

OpenAI GPT-6 (Codex), with reasoning, code execution, browser
automation, and tool use. The exact hosted model ID and context window
are not exposed in this task.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-12 19:41:15 -05:00
DottaandPaperclip c9021c6721 fix: require explicit native completion reviews (#13314)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs report their outcome through paperclip_finish.
> - The server previously turned incomplete reports into human approval
requests.
> - Those requests could block a later successful run, even when no
person had requested review.
> - This pull request creates review cards only for explicit attention
requests and withdraws proven old fallback cards.
> - Agents receive useful completion feedback, while explicit approval
gates and task state protections remain in force.

## Linked Issues or Issue Description

Related: #13266 removed reviews caused by policy upgrades. This change
removes the separate completion fallback.

**What happened?**

An agent reported needs_review while waiting for checks without
requesting a human decision. Paperclip created a generic Native
completion review. A later successful report could not complete the task
because that old card remained pending.

**Expected behavior**

Ordinary low-risk work completes after a valid done report, a successful
run, and workspace finalization. Incomplete work stays with the agent.
Explicit approval requests remain visible and must be resolved.

**Steps to reproduce**

1. Complete a native run with needs_review and no attention requests.
2. Continue the task and submit a successful done report.
3. Observe that the old implementation leaves the task in review behind
a generic confirmation card.

## What Changed

- Require explicit attention requests to create native review cards.
Route each request independently and preserve pending or declined
decisions.
- Withdraw only pending system cards with matching old decision,
assessment, effect, contract, and prompt provenance. Preserve history
and explicit or answered requests.
- Reassess an affected current result without overwriting later task
edits, runs, contracts, or workspace failures.
- Return pending approval links and required actions through the
completion tool. Reject contradictory done reports and empty review
requests before accepting a result.
- Allow one corrective continuation for incomplete results, then expose
a recovery action.
- Update status fixtures, database regressions, runner tests, and the
completion contract documentation.

## Verification

- `pnpm -r typecheck` passed after merging current master.
- `pnpm build` passed after merging current master.
- The combined branch passed 89 completion and Agent Chat tests. Other
targeted tests passed: 170 external-chat and reconciliation tests; 50
runner-resume and control-plane tests; 13 arbiter tests; 7 chat delivery
tests; 21 runner completion and runtime-context tests.
- The full local test attempt exposed old review fixtures and a missing
fake-provider binary. The fixtures are fixed and the helper is built.
All affected suites pass in fresh reruns. The timing-sensitive Discord
test also passed on rerun.
- All latest-head CI checks passed, including build, typecheck, general
and serialized tests, runner verification, browser tests, and canary dry
run. Greptile is 5/5 with zero unresolved comments.

## Risks

- Cleanup changes existing pending cards. It requires exact system
provenance and only applies to low-risk agent-claim contracts. It does
not delete history or dismiss explicit requests.
- Status still commits after the turn and workspace finalization.
Completion feedback reports current constraints and does not claim an
early status commit.
- Incomplete reports now request a bounded corrective run instead of an
automatic approval. Repeated failures expose recovery.

## Model Used

OpenAI Codex, GPT-6, with repository inspection, code execution, and
test tools. The exact deployment identifier and context window size are
not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-12 13:00:04 -05:00
DottaandPaperclip a12bbd1824 fix: stop completion reviews caused by policy upgrades (#13266)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runtime records completion assessments and task
decisions.
> - An application update can change the assessment policy version.
> - The previous code treated that version change as a reason for human
review.
> - An upgrade alone does not give the user a new decision to make.
> - This change keeps existing decisions and withdraws obsolete upgrade
review cards.

## Linked Issues or Issue Description

**What happened?**

A rules version change moved unfinished tasks into review and created a
card that said, "Review the superseding native policy assessment." The
task did not need new work or a human decision.

**Expected behavior**

New runs use the current rules. An upgrade leaves existing task
decisions alone. The saved policy version remains available in the audit
history.

**Steps to reproduce**

1. Save a native run assessment and a task decision.
2. Change the native status policy version.
3. Run finalization reconciliation without new task evidence.
4. The previous code created a review card. The corrected code keeps the
saved assessment and decision.

Related context: #13038 changed the native policy version. Searches
found no duplicate fix for upgrade-only completion reviews.

## What Changed

- Remove policy-version mismatch as a reconciliation trigger.
- Withdraw pending cards only when their source decision, effect ledger,
creator, key, and prompt match the old upgrade-only review.
- Restore the previous status only while that decision and status
version remain current and no other review gate is pending.
- Preserve answered cards, real review requests, later task changes, and
historical assessments and decisions.
- Record cleanup activity and retire any corresponding chat review
actions.
- Isolate cleanup failures so one old card cannot block other cleanup or
normal finalization.
- Update the status conformance fixture, regression tests, and
architecture documentation.

## Verification

- Passed: `pnpm exec vitest run
server/src/__tests__/native-status-arbiter-corpus.test.ts` (23 tests,
including the 53-fixture status corpus).
- Passed: `pnpm -r typecheck`.
- Passed: `git diff --check`.
- Passed: `pnpm build`.
- Passed: fresh Greptile review at 5/5 on `d24e5ecaf`, with no open
review threads.
- Passed: the 23 focused tests in the isolated full-suite environment.
- Local `pnpm test:run`: the server group finished with 10,638 passed
and one missing-fixture failure. Built the required
`fake-codex-app-server` fixture and reran the entire affected native
session-resume suite: 37/37 passed. The initial full command exited on
that server-group failure, so remaining groups are covered by CI.
- Passed: the workspace-runtime-exposure suite (25 tests, 3 platform
skips).
- Passed: all CI gates, including every test shard, browser tests,
typecheck, and build. The server shard passed on one retry after an
unrelated host-port conflict in the unchanged workspace-exposure tests.
- Cleanup tests cover repeat runs, real reviews, answered cards, later
statuses, newer decision identity, status changes back to review, other
pending requests, run scope, cleanup failures, retry, and live
publication failures.

## Risks

- Cleanup changes stored task state. It checks the exact obsolete
decision and status version under database locks before restoring
status.
- Pending approvals, interactions, and execution stages prevent
restoration out of review.
- The cleanup handles at most 100 matching cards per reconciliation
pass. It does not rewrite old decisions or accept an agent's completion
claim.
- New evidence and explicit task changes still use the existing
reconciliation paths. This change does not reevaluate old work merely
because Paperclip was updated.

## Model Used

OpenAI Codex, GPT-6. The exact deployment identifier and context window
are not exposed in this session. Used reasoning, repository inspection,
code editing, shell execution, and automated tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-11 16:55:46 -05:00
DottaandPaperclip 889947c238 feat: add experimental native chat connectors (#13038)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People also ask agents for work in their existing chat tools.
> - Each external conversation needs one task and a current authorized
source.
> - Retries, Stop, and provider failures must not duplicate work or
expose private data.
> - The first chat PR establishes the opt-in provider and data
contracts.
> - This PR adds experimental channel integration and its durable
control plane.
> - Users can request work from connected channels and inspect delivery
in Paperclip.

## Linked Issues or Issue Description

Refs #13100 and #13092. This is the second of exactly two chat PRs.
Foundation #13100 is merged and changed 143 files. Runner prerequisite
#13092 is also merged. This PR changes 400 files against master, below
the 500-file review limit. It contains no wireframe images or HTML
galleries.

## What Changed

- Add native Slack, GitHub, Microsoft Teams, Telegram, and Discord chat
connections. Keep chat disabled unless the operator enables experimental
chat connectors. Preserve the production GitHub tool connection and its
normal setup path.
- Bind each provider bot identity to one immutable Paperclip agent. Bind
each admitted external conversation to one task. Paperclip owns tasks,
runs, permissions, and audit records.
- Add durable admission, per-conversation queues, questions, task
controls, progress, final replies, images, files, and delivery receipts.
Board comments remain internal unless explicitly sent to the channel.
- Check current identity, provider reach, resource access, credentials,
runtime generation, and exact source before provider effects. Keep
private responses private. Never send raw reasoning, private logs,
credentials, or tool arguments.
- Hold uncertain sends for explicit audited resolution. Make Board
Send-to-channel atomic and idempotent. Keep reconnect and setup
credentials in Paperclip secret storage.
- Preserve current native-runner authority across retries, lost
acknowledgements, and recovery. Keep immutable input and completion
contracts separate from newer user input. Receipt reconciliation cannot
launch a provider.
- Reconcile chat close/new ordering and provider-effect lock order.
Audit resource access changes in the same transaction. Submit only the
selected resource from each UI toggle so stale pages cannot undo
unrelated access changes.
- Drain Codex stdout before certifying process exit. Bound the drain
with the existing shutdown grace. Preserve observed terminal authority
without treating an undrained process as successful or reusable.
- Incorporate master `018ca5da` with its ACP Stop, mobile task layout,
runner packaging, and official lock changes. Preserve dedicated
chat-answer continuations in both directions when ordinary queued
comments are adopted after Stop.
- Fence late adapter readiness behind an earlier Stop for the same run.
Preserve verified cleanup for registered adapters. Handle single Stop,
agent pause, duplicate Stops, and failure release without creating a
false cancellation receipt.
- Incorporate master's `6dd48cad4` wake-queue extraction. Preserve exact
failed-chat retry authorization and lineage, retired question-source
suppression, and the block on generic recovery that would discard the
admitted source. Fresh deferred input retains its separate promotion
path.
- Incorporate master `2a05b5ed3` and its queue-admission extraction,
simplified transaction ports, and separate runner CI job. Preserve exact
durable receipts, actor separation, and dedicated-answer isolation
through the new module. A failed receipt insert rolls back the
accompanying deferred-wake merge.

## Verification

Current head: `afe19299d06253cb628eb398e91d1200ea9f412a`, incorporating
master `2a05b5ed3457ea33efd6895520447d1d97fe98d8`. The conflicts are
resolved. This successor fixes two test-harness boundaries exposed by
CI: per-case route-module preparation and actual durable-save completion
before intentional runner termination. Production code and all existing
test/turn deadlines are unchanged. [Exact-head Greptile
review](https://github.com/paperclipai/paperclip/pull/13038#issuecomment-5587250594)
is **5/5**, completed September 10 at 13:20:55 UTC, with no actionable
findings or open review threads. [Fresh exact-head
CI](https://github.com/paperclipai/paperclip/actions/runs/34481724341)
passes **all 24 jobs**, including Build and both required aggregates.
Normal exact-head guarded merge was attempted and rejected by the
remaining branch approval policy: CODEOWNER review is required and no
human approval is present. Normal **squash auto-merge is enabled** as of
September 10 at 13:36:26 UTC. Requested CODEOWNERS have been notified;
no approval bypass or self-approval was used. Earlier-head results below
remain historical evidence, not qualification of this successor.

- Final exact-head Linux evidence: 995/995 chat integration cases; 36/36
agent-skills routes; 35/35 runner live-session cases, including real
process kill/resume; 1948 runner Vitest cases with three existing
benchmark/platform guards; 870/870 API-authority cases; and 104 browser
cases with four existing optional skips. Rust, conformance/replay, full
repository build, typecheck, canary, all server/workspace shards, and
both required aggregates pass with normal CI concurrency. Earlier failed
attempts remain recorded below.

- Latest test-only qualification: 141/141
route/permissions/authentication cases pass in separate cold forks, with
plain server types and independent review clear. The real-runner suite
passes 35/35, with plain runner types and independent review clear. A
controlled premature-save acknowledgement fails as expected; matching
ownership/effect/process evidence, rejected saves, real turn outcome,
test abort, and pre-kill liveness are covered. No local reproduction of
the original CI scheduling failure is claimed. The preceding [CI
run](https://github.com/paperclipai/paperclip/actions/runs/34479680858)
passes 21/24 jobs, including all 995 Linux chat cases and browser
aggregate (104 passed, four existing optional skips); only Build, the
skills serialized shard, and the required verification aggregate fail.
Its exact-head Greptile review was 5/5. Both failed job logs are
retained.

- Final fixture qualification: all eight focused Discord cases and all
995 chat integration cases pass. The exact modal statement/PID is
observed before taking the real connection lock; the test then proves
its actual blocking relationship before mutation. Original SQL
execution, provider behavior, negative assertions, and 1s/15s timeouts
remain unchanged. Independent review is clear and test/production hashes
remain frozen. The preceding [CI
attempt](https://github.com/paperclipai/paperclip/actions/runs/34477184777)
passed 22 jobs, including Build/runner, typecheck, canary, all other
test shards, and browser aggregate (104 passed, four existing optional
skips); the two fixture failures and failed verification aggregate
remain recorded, not relabeled as a pass.

- Current queue-module composition: 308/308 recovery/batching/queue/Stop
tests; 995/995 full chat integration; 89/89 module tests, including real
PostgreSQL receipt-insert rollback; 24/24 workflow/module-boundary
tests; plain server and UI types. All four actual local process/ACP
browser paths pass in 1.4 minutes. Fresh databases, no skips or retries,
stable reviewed source hashes. The initial boundary failure is retained;
its no-op service wrapper was removed without changing recovery context
or weakening the check. An exploratory standalone test-directory
typecheck fails because its new upstream transformation config is not a
standalone typechecking project; standard CI/build does not invoke it,
and no configuration was weakened to suppress those diagnostics.

- The preceding head `e02a63d462ce5d47433b0aeb632bb6fd20aab1ba` passed
[all 24 CI
jobs](https://github.com/paperclipai/paperclip/actions/runs/34436462958)
and exact-head Greptile review at 5/5. Required CODEOWNER review
prevented its normal merge before master advanced again.

- Final extracted-module composition: 307/307 recovery, batching, queue
and Stop-control tests; 995/995 full chat integration; 49/49 module
tests including eight PostgreSQL adapter cases; and 19/19 issue-update
tests. Plain server types pass. All four actual local process/ACP
browser paths pass in 1.3 minutes. Fresh databases, no skips or retries
in these cohorts, frozen source hashes, and independent review clear.

- The preceding head `3e4e1c1c` passes [all PR CI
jobs](https://github.com/paperclipai/paperclip/actions/runs/34415826820),
including Build and required `ci / verify` and `ci / e2e`. Both the
original Rust failure and the previously load-sensitive lineage fixture
pass with unchanged Linux concurrency. Master advanced afterward and
required this reconciliation.
- Final master composition: 448/448 focused UI tests, 186/186 adapter
tests, 24/24 queue/control tests, and 11/11 packaging tests. Plain UI,
server, shared, and adapter types pass. Token gates and diff checks
pass. Independent server and UI reviews are clear.
- Stop-registration regression: both real-service cases fail against
exact `a95` source and pass with the fix. The full corrected
recovery/control suite passes 265/265. Duplicate-owner and failed-Stop
controls also pass. Plain server types pass. The readiness barrier
prevents provider startup without adding an acknowledgment to an already
terminal run.
- Final qualification strengthens terminal-field equality and repeats
both affected cases successfully on a fresh database. All four actual
local process/ACP browser paths pass again in 1.3 minutes, without skips
or retries. The final screenshot shows Cancelled, a paused subtree,
retained input, and no error toast.
- Two new actual-service regressions fail before the merge fix. They
prove that queued-comment adoption could consume a dedicated chat answer
or add unrelated input to that answer. The fixed four-case cohort
passes, including ordinary upstream continuation and adapter Stop
controls. Full recovery passes 257/257. All four actual local
process/ACP Stop browser flows pass in 1.4 minutes, without skips or
retries, on a fresh database.
- The unchanged runner artifact was qualified with 171/171 transport
tests, 870/870 API-authority tests, conformance 1/1, and replay 11/11.
Six controlled reader tests prove the exit/drain repair. Its local
serial Rust workspace passed 546 top-level cases plus two invoked
helpers; the later passing Linux CI supplies default-concurrency
evidence.
- Prior exact-source full chat integration passes 995/995. Settings
regressions cover concurrent stale pages, 501 destinations, pending
state, rejected updates, and explicit retry. These deterministic tests
do not prove live provider behavior.
- Retained failed attempts and their causes are in the [qualification
log](https://github.com/paperclipai/paperclip/blob/afe19299d06253cb628eb398e91d1200ea9f412a/doc/plans/chat-adapters/2026-09-08-chat-queue-and-webhook-repair.md).
The first merge adapter run timed out while macOS slept for 290 seconds.
Its unchanged repeat passed with a temporary sleep guard. No assertion,
deadline, or CI gate was weakened.

Review commands include `pnpm --filter @paperclipai/server exec vitest
run src/__tests__/heartbeat-process-recovery.test.ts
src/__tests__/issue-queued-comments-routes.test.ts` and `pnpm exec
playwright test --config tests/e2e/playwright.config.ts
tests/e2e/acp-stop-continuation.spec.ts`. Database suites require fresh
disposable databases. See the [browser
runbook](https://github.com/paperclipai/paperclip/blob/afe19299d06253cb628eb398e91d1200ea9f412a/doc/plans/chat-adapters/2026-09-04-chat-adapters-browser-e2e-runbook.md)
for provider setup and separate live acceptance steps.

## Risks

- This remains experimental. Deterministic tests and bounded live
evidence do not establish every provider feature, tenant, permission
layout, or media shape. Teams work-tenant qualification is still open.
- Failed and uncertain provider effects remain visible and can require
operator action. A transport receipt does not prove recipient
visibility.
- Native controller and runner artifacts must remain compatible.
Preserve lease ownership, terminal authority, source binding, and
quarantine during future changes.
- Access and audit rows commit together, but activity notifications
remain best-effort. This is not a new durable event outbox.
- The PR operation does not deploy a live server, replace its runner, or
change provider permissions. Remaining live qualification is documented
in the [temporary
handoff](https://github.com/paperclipai/paperclip/blob/afe19299d06253cb628eb398e91d1200ea9f412a/doc/plans/chat-adapters/2026-09-08-open-qualification-followups.md).

## Model Used

OpenAI Codex assisted with implementation, tool execution, testing, and
review. The work records `gpt-6-astra` assistance. The environment does
not report a context-window size. No private reasoning traces are
included.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-10 10:06:45 -05:00
Dotta 1dceee9a4e fix(runner): persist warm Daytona workspaces (#12901)
## Thinking Path

> - Paperclip manages AI agent work and the execution state for each
task.
> - Remote agents run in sandbox environments such as Daytona.
> - Daytona keeps files while a sandbox is stopped, but deletion removes
those files.
> - Runner Codex did not copy successful remote workspace changes back
to the host workspace.
> - A warm sandbox could therefore hide data loss until Daytona replaced
or deleted the sandbox.
> - This pull request makes the host workspace durable after every
successful turn and keeps verified reusable sandboxes warm.
> - The benefit is reliable multi-turn work across warm reuse, restart,
stop, and sandbox replacement.

## Linked Issues or Issue Description

**What happened?**

A successful native Codex turn in Daytona could leave workspace changes
only in the remote sandbox. A later warm turn appeared to work because
it reused that filesystem. A replacement sandbox could start from stale
host data and lose the successful changes.

**Expected behavior**

Paperclip must merge each successful remote turn into the authoritative
host workspace before it completes the run. A verified warm lease may
reuse its remote files. A replacement lease must reconstruct the exact
durable workspace seed.

**Steps to reproduce**

1. Run Codex in a reusable Daytona environment.
2. Write a file during one successful turn.
3. Replace the Daytona sandbox before the next turn.
4. Observe that the next turn can start without the prior file on the
unpatched code.

Related remote workspace foundation: #10070.

## What Changed

- Added explicit `host_current`, `durable_seed`, and `adopt_remote`
workspace preparation modes.
- Added atomic, versioned native workspace descriptors and seed archives
under `PAPERCLIP_HOME`.
- Added real native sandbox export and three-way host merge before
terminal result completion.
- Added workspace-only recovery after a proposed result. Recovery does
not submit another provider turn or consume the provider retry budget.
- Added fail-closed handling when a sandbox with unexported changes is
gone.
- Kept healthy reusable Daytona sandboxes started for legacy Codex and
Runner Codex.
- Kept the Runner Codex process and provider session across verified
warm turns.
- Added the paid `daytona-warm-continuity` browser suite. It contains
exactly the legacy Codex and Runner Codex cells. Each cell performs
three measured turns.
- Documented `pnpm test:e2e:runner -- --suite daytona-warm-continuity`.
No package script was added.
- Added no database migration. The metadata format is backward
compatible and idempotent.

## Verification

- `pnpm typecheck`
- `pnpm test:e2e:runner:unit` — 114 passed
- Native workspace, finalizer, session, and environment tests — 232
passed
- Daytona provider tests — 150 passed
- Workspace staging and merge tests — 98 passed
- Runner transport tests — 63 passed
- Legacy Codex restore tests — 5 passed
- Rust format and compile checks pass through root typecheck
- The paid Daytona suite was not run locally because the required
Daytona, OpenAI, and immutable image credentials are not present.

## Risks

- The main risk is an incorrect workspace identity or merge after a
crash. Durable descriptors bind the run, workspace, lease, provider
lease, local root, remote root, and baseline digest. Ambiguous evidence
fails closed.
- The host merge may conflict with concurrent host edits. The existing
three-way merge and exclusion rules handle this case and surface
failures.
- A deleted sandbox cannot recover unexported bytes. Paperclip now
blocks with `workspace_sync_out_unrecoverable` instead of reporting
success or rerunning the provider.
- There is no database migration. Descriptor writes and recovery are
atomic and idempotent.

## Model Used

OpenAI Codex with GPT-5. The run used agentic reasoning, repository
inspection, code execution, test execution, Git, and GitHub CLI tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-09-05 13:00:57 -05:00
Nicky LeachandPaperclip 184b014c25 feat(telemetry): add the agent.task_run event and emit it at every terminal run transition (#12809)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip records agent run outcomes through telemetry and run
lifecycle services
> - Terminal run transitions need one consistent event for outcome
analysis
> - The current paths do not report every terminal transition through
one event
> - This pull request adds the agent.task_run event and emits it at each
terminal transition
> - The benefit is complete run outcome data without exposing raw task
identifiers

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Paperclip telemetry reports agent activity, but it does not report every
terminal task run through one event.

**Subsystem affected**

Cross-cutting (multiple of the above): packages/shared telemetry and
server run lifecycle services.

**Current behavior**

Several run paths write a terminal status without a matching
agent.task_run telemetry event.

**Proposed behavior**

Each terminal run transition emits one agent.task_run event. The event
records the terminal state and uses the existing pseudonym helper for
the optional task identifier.

**Reason and benefit**

Complete terminal-run data helps operators measure agent outcomes. The
pseudonym helper prevents the raw task identifier from leaving the
installation.

**Breaking changes**

None. The change adds an event and keeps existing event behavior
compatible.

## What Changed

- Add the agent.task_run telemetry contract and client helper.
- Reuse the existing pseudonym helper for the task identifier. The
helper hashes the identifier with a per-installation salt and returns 16
hexadecimal characters. The raw identifier never leaves the
installation. Existing identifiers do not move.
- Emit one event from each legacy, native, recovery, and issue terminal
transition.
- Keep emissions outside database transactions and make delivery
best-effort.
- Add regression tests for event shape, hashing, terminal transitions,
and emission failures.
- Document the event and its privacy rule in the telemetry data
contract.

## Verification

- `npx tsc --noEmit` in `server/` passes at the submitted commit.
- The pull-request CI suite must pass. CI is the authority because local
Vitest has a known dependency artifact.
- The added regression tests cover event output shape, per-installation
hash divergence, raw identifier handoff, omitted identifiers, and
non-throwing emits.

## Risks

- A missed terminal path could reduce event coverage.
- Telemetry delivery remains best-effort and cannot change run
finalization.
- The pseudonym helper uses installation-specific state, so identifiers
differ between installations.

## Model Used

OpenAI Codex, GPT-5, tool use and code execution. Context window details
were not provided.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I have addressed all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-04 08:21:32 -07:00