Commit Graph
14 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 340157d1d6 perf(recovery): scope run lifecycle recovery hints to the run's task graph (#21552)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs issue recovery queues in the background. Two of
them, `stranded-assignments` and `assigned-work-invariant`, find
assigned tasks that have no live run and restart them.
> - Every run dispatch and every run state change sent a recovery hint
for the whole company. On each hint these queues reload every eligible
open task of the company and check each one again, at most every 30
seconds per queue.
> - With live runs the hint comes all the time, so the queues never
stop. Their cost is (run activity) x (open-task backlog). On a hosted
instance with about 1,400 open tasks and 5 live runs, the server main
thread stayed at 87-92% of a core.
> - A run event can only change the tasks of its own task graph. This
pull request makes run and task producers name that graph, and keeps one
slow full pass as a backstop.
> - The benefit is that a run event costs a few checks, whatever the
size of the backlog, and a missed hint still delays recovery by at most
five minutes.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The issue recovery queues in `server/src/services/recovery/work.ts`
(`stranded-assignments`, `assigned-work-invariant`,
`resolved-dependencies`, `stale-issue-locks`) and the producers that
wake them on run dispatch, run start, run state change, run finalization
and accounting settlement.

**Subsystem affected**

Server background recovery (`server/src/services/recovery/`,
`server/src/services/scoped-deadline-work.ts`, the notification helpers
in `server/src/services/*-notifications.ts`).

**Current behavior**

`notifyRunDispatch`, `notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` sent `notifyIssueRecovery(tx, {
companyId })`. Most run lifecycle paths call one of them, often two or
three in the same transaction (for example `notifyIssueReviewState`).
Some other run paths sent the company hint directly: the wake status
update, the execution-control deadline writes, legacy run
terminalization, scheduled retries, accounting settlement, task checkout
and task document writes. Each company hint makes each task queue reload
all eligible open tasks of the company, 100 per page, and check each
task again. A company hint is reconstructed at most every 30 seconds per
queue, so one run event per 30 seconds is enough to keep a full pass
running all the time. `issue-recovery-deadlines.test.ts` pinned this: a
company hint re-checks every eligible task.

**Proposed behavior**

Design. For each task queue, a run event of run R on task T can change:

| Queue | What the check reads | What one run event can change |
| --- | --- | --- |
| `stranded-assignments` | T's own runs, wakes, handoffs, questions,
recovery actions, goal, assignee eligibility | T (its runs). The parent
of T and the tasks T blocks when the event also writes T's status. |
| `assigned-work-invariant` | the same plus open blockers and open
children | T, the parent of T (open children), the tasks T blocks (open
blockers) |
| `resolved-dependencies` | blocked tasks whose blockers are all done |
the tasks T blocks |
| `stale-issue-locks` | tasks whose checkout or execution run is R | T |

The assignee's agent status (`running`, `idle`, `error`) does not change
whether the agent is invokable
(`packages/shared/src/agent-eligibility.ts`), and a freed concurrency
slot starts the agent's next queued run through the
`queued-run-dispatch` queue, which keeps its company hint. So the
agent's other tasks are not in the set.

- `notifyIssueGraphRecovery(tx, { companyId, runId?, issueId? })` sends
`{ companyId, graphRunId?, graphIssueId? }` to the four task queues. It
runs no query in the producer transaction.
- When a task queue loads that hint, it resolves the graph with one
bounded, indexed query (`resolveIssueGraph`): the task of the run
(`issue_id`, `native_issue_id`, or the saved `issueId`/`taskId`,
validated and cast so the primary key is used), its parent, and the
tasks it blocks. The four queues share the result for 2 seconds. With no
task or more than 100 tasks, the queue turns the hint into the plain
company hint, so task-less runs coalesce under its 30-second gate
instead of each starting its own full pass. The lookup runs after
commit, so it sees the written state. Graph hints do not wait for the
30-second reconstruction gate (they load at most 100 tasks); hints for
the same run coalesce while dirty, and the lookup is shared across the
four queues, so it runs about once per pass instead of once per producer
call. An earlier round ran the lookup in every producer transaction: 34
lookups/s at 1,500 open tasks and 70/s at 5,000.
- The three company-level queues (`orphan-blockers`,
`active-recovery-actions`, `task-watchdogs`) keep the company hint. They
hold one entry per company or per watchdog, not per open task.
- `notifyRunDispatch`, `notifyScheduledRetryDispatch`,
`notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` use it when the scope names a run
or a task. A scope with only a company keeps the company hint.
- Run paths that knew the run or the task but sent only the company now
pass it. This includes three `writeExecutionControl` deadline writes,
legacy run terminalization (`notifyExecutionRunWork`) and native safe
replacement. Those helpers also forward the scope to
`notifyLiveRunRecovery` and `notifyLiveResourceRecovery`, so their hints
to the live-run queues now carry the run id.
- Recovery action changes (`issue-recovery-actions.ts`: create,
supersede, update, exhaust, resolve) name their source task. A recovery
action is a fact of its source task for these queues. The company-level
`active-recovery-actions` queue still gets the company hint. In the
fixture, failing runs create and resolve recovery actions every few
seconds, and that alone kept a full pass running.
- `writeExecutionControl` and `notifyExecutionRunWork` send the
attention retention hint for a task through
`notifyIssueAttentionRetentionWork` (the task, its parents and the tasks
it blocks), a superset of the single-task hint they sent before. A
recovery action of a blocker can change the blocker attention of its
dependents, so the narrower recovery action scope needs this.
- `promoteDeferredWorkIntents` scopes `lock_released` with a task to
that task, and `accounting_settled` to the run when the company has no
active hard-stop budget policy. Only a reached hard stop can refuse work
because of accounting (`evaluateAdmissionInTransaction`), so without
such a policy the settlement of one run cannot admit another task. The
other hold releases keep the company hint.

Backstop. Each scoped hint also sends `{ companyId, backstop: true }` to
the same four queues. Nothing else arms it. The queue runs it as a full
company pass, but not before 5 minutes after the last full pass of that
queue for that company or of the startup pass. It is a scheduler
deadline (`nextWakeAt` reports it), but it holds no idle intent and does
not mark the queue unreconciled. So:

- Startup still reconstructs every company.
- A failed scoped load retries with the existing back-off (30 seconds to
5 minutes). The backstop is pending in any case.
- A graph with no task or more than 100 tasks loads the whole company at
once.
- A task that a scoped hint missed is checked at most
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes) after the event. While runs
continue, each queue runs at most one full pass per company per 5
minutes. With no activity, nothing is scheduled.

Why arming on activity only cannot leave a stranded task undetected when
the instance then goes quiet. A task becomes stranded only through a
write or through a run that stops without a write:

1. The write names the task's graph (run and task producers). The scoped
pass checks the task at once.
2. The write sends a company hint (pause, budget, agent and company
configuration, tree holds, the producers listed below). The company pass
checks it within the 30-second gate, as before.
3. The write is a run event whose graph does not contain the task. That
same event armed the backstop. The backstop deadline stays scheduled
while the instance is quiet and checks the task at most 5 minutes later.
4. A run stops without any write (process lost, controller gone). The
existing deadline paths find it: `stale-issue-locks` waits for the
stored controller lease, and `active-run-silence` and
`orphan-active-leases` act on their own deadlines. Each of them
terminalizes the run, which is a run write and falls under 1 or 3. The
new test `restarts a task whose run vanished without a final event once
its lease lapses` covers this with the production backstop.
5. The process restarts or a quiet instance is woken by a restart. The
startup pass reconstructs every company.
6. A check that found a task waiting (grace, back-off, monitor) returns
its own deadline. That entry keeps its deadline whatever hints arrive.

Five minutes is the existing upper bound of recovery back-off in this
area: the maximum retry delay of a failed scoped-deadline check
(`scoped-deadline-work.ts`), and `DEFERRED_WAKE_PENDING_BOUND_MS`, the
time `assigned-work-sweep.ts` already trusts a deferred wake before it
calls a task stranded.

Producers that keep the company hint, and why. They can change
eligibility of tasks outside one task graph, or I could not show that
their effect stays in the graph. None of them fires once per run event
in the fixture.

- Eligibility for many tasks: company status (`companies.ts`), budgets
(`budgets.ts`), agent lifecycle and configuration (`agent-lifecycle.ts`,
`agent-configuration.ts`), instance settings (`instance-settings.ts`,
`*`), tree holds (`issue-tree-control.ts`), company deletion and import
(`company-deletion.ts`, `company-portability.ts`), and the hold releases
`pause_lifted`, `tree_hold_released`, `scheduling_resumed` and
`reconciler_backstop`. `accounting_settled` also keeps it when the
company has an active hard-stop budget policy.
- Task writes that can move a task between graphs or change eligibility:
task creation and a task update that changes the parent in `issues.ts`
(other task updates and checkout releases now name the task), document
delete and restore paths in `documents.ts`, task watchdogs
(`task-watchdogs.ts`), system and workspace recovery
(`system-recovery.ts`, `task-retry.ts`,
`workspace-recovery-capture-policy.ts`,
`workspace-recovery-stop-proof.ts`).
- Conversation and chat paths: `agent-conversations.ts`, the daily-cap
release in `heartbeat/queue.ts`, the Telegram attachment failure,
`chat-attachment-reuse.ts`, `native-runner-file-handoff.ts`,
`notifyChatRunMilestones` without a task or run (company authority
change), and `notifyNativeStatusReassessment` from `access.ts`.
- `writeExecutionControl` and `notifyExecutionRunWork` callers that pass
only a company.

**Reason and benefit**

Background work must not grow with the backlog of open tasks. The checks
a run event causes now depend on its task graph, not on the company.

## What Changed

- `server/src/services/issue-recovery-notifications.ts`:
`notifyIssueGraphRecovery`, `resolveIssueGraph`, the scope matcher and
decoder, `ISSUE_RECOVERY_MAX_ISSUE_IDS` (100) and
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes).
- `server/src/services/recovery/work.ts`: the task queues resolve a
graph hint when they load it (shared for 2 seconds), remember the last
full pass per company, and gate the backstop hint on it.
`registerIssueRecoveryWork` takes an optional `backstopMs`.
- `server/src/services/recovery/assigned-work-sweep.ts`: `listSuspects`
accepts `issueIds`.
- `server/src/services/scoped-deadline-work.ts`: an optional
`backstop(scope)` that returns the earliest time of a backstop hint. A
pending backstop hint keeps its deadline but does not count as pending
work or as unreconciled.
- Producers: `issue-recovery-actions.ts`,
`execution-control-notifications.ts`, `run-dispatch-notifications.ts`,
`native-status-reassessment-notifications.ts`,
`external-conversation-settlement-notifications.ts`,
`work-intent-outbox.ts`, `run-cost-accounting.ts`,
`heartbeat/run-lifecycle.ts`, `heartbeat/runtime-execution.ts`,
`heartbeat/retries.ts`, `heartbeat.ts`, `legacy-execution-recovery.ts`,
`issues.ts` (checkout), `documents.ts` (task document writes),
`modules/wake-queue/adapters/postgres.ts` (recovery runs for a task),
`heartbeat/queue.ts` (wake enqueue), `recovery/service.ts`,
`native-runtime/native-provider-capacity-retry.ts`,
`native-runtime/native-safe-replacement.ts`,
`modules/run-dispatch/adapters/workspace-busy-wait.ts` (scheduled
retries).
- `bounds subsidiary scans for100 assignment candidates`: a resolved
recovery action now adds one scoped load per task queue (bound 8 to 11),
and the coalescing check sends one company hint before the burst of 100,
because the first full pass after the resolve is now a backstop pass
with its own key.
- `checks only blocked tasks whose blockers are done`: uses the shared
recording harness and runs out the backstop before its quiet check,
because the blocker's status change is now a graph hint (its 300
dependents overflow to a company load).
- Tests. `issue-recovery.test-fixture.ts` and
`live-recovery-work.test.ts` register the queues with `backstopMs: 0`,
so their quiet checks keep the earlier timing. The new tests use the
production value.

## Verification

New tests. I saw each red test fail before the change.

- `issue-recovery-deadlines.test.ts` `re-checks only the tasks a run
lifecycle event can affect, whatever the backlog` (red before): a run of
the agent on a task with a parent and a dependent, 1,000 backlog tasks
of another agent, and a second open task of the same agent. Seven real
producers (`notifyNativeStatusReassessment`, `notifyExecutionRunWork`,
`notifyRunDispatch`, `promoteDeferredWorkIntents` lock release and
accounting settlement, recovery action create and resolve) check exactly
the 3 tasks of the graph. The recovery action steps were red before the
third commit. With 4,000 more backlog tasks the event issues the same
number of statements. Before the change it checked 1,004 tasks.
- `resolves a run's task graph through index lookups, not the backlog`
(red before the second commit, from a Greptile finding): with 2,000
backlog tasks the graph lookup read 2,005 issue rows
(`pg_stat_xact_user_tables`) because it compared `id::text`. It now
validates and casts the saved id, reads fewer than 50 rows, and returns
exactly the run's graph.
- `finds a stranded task outside the affected set within the backstop
bound` (red before): a task inserted without a hint is not checked by
the run event, and is checked at most 5 minutes after the last full
pass. After 20 quiet minutes, a run event checks a new unrelated task
within 5 seconds, because the last full pass is older than 5 minutes.
Then the queues are quiet.
- `keeps the company pass for hold releases that can admit any task`
(guard, passes before and after): accounting settlement with a hard-stop
policy and a lifted pause still check every task.
- `falls back to the company pass when a run affects more tasks than a
hint names` (guard): a run task that blocks 100 more tasks checks the
whole company.
- `limits an ordinary comment to its issue among one thousand open
assignments`: the old final assertion (a company hint checks all 1,000
tasks) stays, and the test now also runs out the backstop and ends
quiet.
- `issue-recovery-assignments.test.ts`, real heartbeat, real producers,
production backstop (guards; they also pass with the old company hint):
a task whose run failed, a task whose run the system cancelled, a run
that vanished without a final event and is found when its controller
lease lapses, and an agent's queued task that starts when its previous
run ends. Each one recovers within 15 seconds, well before the backstop.
- `scoped-deadline-work.test.ts` `keeps a backstop hint's deadline
without holding the worker busy` (red before).

Suites, each file alone, in `server/` (issue recovery, live recovery,
recovery action and attention suites against dedicated local `_test`
databases): `issue-recovery-deadlines` (13/13),
`issue-recovery-assignments` (7/7), `issue-recovery-work-lifecycle`
(14/14), `scoped-deadline-work` (24/24), `live-recovery-work` (51/51),
`assigned-work-invariant` (32/32), `assigned-work-sweep-round-trips`
(2/2), `saved-work-recovery-overdue` (4/4),
`issue-recovery-dependencies` (5/5), `issue-recovery-conversations`
(3/3), `issue-recovery-actions` (112/112), `attention-retention-work`
(55/55). `recovery-deadline-acceptance` skips without its external
database. `pnpm check:module-boundaries` passes.

Local benchmark. The real server ran against PostgreSQL with
`log_statement=all` and instrumented counters (not committed). Fixture:
1 company, 60 agents, open tasks at 150 / 1,500 / 5,000 (52% in review,
39% blocked behind open blockers, 3% in progress, 2% to do), with
process agents that fail at once so recovery re-dispatches runs all the
time. The tables are analyzed after loading. Each window is 180 s,
starting 150 s after the server is ready, so each "after" window
contains one backstop pass. Base is `master` at 54c3616ad9; after is
this branch at dcd170e29f. The last commit (graph hints without the
30-second gate, task updates and checkout releases scoped) is covered by
the tests below but not re-measured. The machine was shared (load
average 9-16, one base window at 45). Run churn differs between runs, so
the table gives counts per run event (runs created in the window).

| Open tasks | Runs created (/s) base → after | Stranded checks per run
base → after | Assigned-work checks per run base → after |
`agent_task_sessions` reads (/s, 4 shapes, stranded check) | Statements
per run event base → after |
| --- | --- | --- | --- | --- | --- |
| 150 | 0.08 → 0.14 | 45.1 → 6.2 | 2.00 → 0.96 | 3.75 → 1.27 | 2,260 →
1,257 |
| 1,500 | 1.09 → 1.00 | 22.0 → 3.9 | 1.24 → 1.53 | 26.5 → 7.2 | 1,313 →
1,060 |
| 5,000 | 1.12 → 1.99 | 20.4 → 1.6 | 4.10 → 0.91 | 29.3 → 8.1 | 1,400 →
829 |

New statements: the task graph lookup (`with seed(id) ...`) at 0.8 / 9.3
/ 18.5 per second, and scoped loads of the task queues (`select "id",
"company_id" from "issues" ...` grew from 0.7 / 6.6 / 7.8 to 2.9 / 20.6
/ 34.4 per second, each a lookup of at most 100 ids). Stale-lock checks
grew from 0 / 50 / 52 to 22 / 190 / 445 per window: each run event now
checks the locks of its own task at once instead of waiting for the next
full pass. In an earlier round the graph lookup ran in every producer
transaction: 34/s at 1,500 and 70/s at 5,000.

With no activity the server issues no statements before and after this
change (quiet checks in `issue-recovery-deadlines`,
`issue-recovery-assignments`, `live-recovery-work`).

## Risks

- A task outside the graph of the run whose state the event changes is
now found by the backstop, at most 5 minutes later, instead of within 30
seconds. One known case: from #21427, `assigned-work-invariant` counts a
held recovery wake as pending while its agent has queued recovery runs.
When the last one starts, the continuation queue delivers the held wake;
the invariant check of that task waits for the backstop.
- A producer that names a task but changes something wider would now
miss the wider effect. Producers that change eligibility for many tasks
(pause, budget, agent and company configuration, tree holds) still send
the company hint.
- The graph is resolved after commit, from the written state. A parent
change is not a run event and keeps its company hint.
- The live-recovery and issue-recovery suites opt out of the backstop
deadline with `backstopMs: 0`.
- The execution-control deadline writes and legacy run terminalization
now pass the run id. `writeExecutionControl` also forwards that scope to
the live-run queues (`active-run-silence`, `native-process-sessions` and
others), so those hints now name the run instead of the company. Those
queues already handle run-scoped hints from native finalization.
- Test-only option: `registerIssueRecoveryWork(..., { backstopMs })`.
The shared recovery fixtures pass `0` so their quiet checks keep the
earlier timing; the new tests use the production value.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 01:29:22 -06:00
DottaandClaude Fable 5.2 18fb6ee571 fix(recovery): continue a native terminal failure whose provider is never proven stopped (#21364)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs that end in a terminal provider failure go through
automatic recovery: an exact replacement first, then a fresh
continuation from saved work
> - The exact replacement waits for proof that the previous provider
stopped. Without that proof it writes neither a successor nor a denial
> - That wait has no end. If the old sandbox or Runner can never be
proven stopped (for example after a server swap), the recovery resolver
keeps waiting for the replacement, and the task sits with no run and no
reason a person can see
> - This pull request bounds the wait. After 5 minutes on the database
clock from the run's terminal time, the replacement pass records a
denial, and the existing fallback continues the task in a fresh run in
an isolated workspace
> - The benefit is that these tasks continue by themselves instead of
stalling forever, and exact replacement still wins when proof arrives in
time

## Linked Issues or Issue Description

**What happened?**
A native run that failed with a terminal provider failure stayed stuck
when its previous provider could not be proven stopped. The replacement
pass skipped the run on every pass. It recorded no successor and no
denial, and it set no deadline. The recovery resolver waits for one of
those two records, so it never continued the task. The replacement queue
also did not wake again on its own, because a completed scan reports
nothing pending.

**Expected behavior**
Exact replacement gets a bounded first chance. After the bound, the task
continues in a fresh run that cannot use the old writer's workspace.

**Steps to reproduce**
1. Create a native run in `terminal_failure` with code
`native_provider_terminal_failed` and a finished time more than 5
minutes ago.
2. Record a process on the run that this server cannot prove stopped
(for example a PID recorded by another server).
3. Run the replacement pass and then the recovery resolver.
4. Before this change: nothing happens, on every pass. After this
change: the pass records a denial and the resolver records one fresh
continuation with isolation `fresh_isolated_workspace`.

**Paperclip version or commit**
`master` at 2b69cdfcd6.

## What Changed

- `native-safe-replacement.ts`: an unverified-ownership decision now
waits only until 5 minutes after the run's terminal time (`finished_at`,
else the finalization's last update), compared with the database
`now()`. After that, the pass records `replacementDenied` and the
existing watchdog hold, with a plain-language next action. The expiry is
checked again inside the locked transaction, so two servers cannot both
act, and a successor that a concurrent pass committed first still wins.
- The deadline applies only to `native_provider_terminal_failed` runs
whose finalization has no lease owner. Other failure codes and
person-owned causes are unchanged.
- The bound reuses `RECOVERY_CONTINUATION_MAX_HOLD_MS` (5 minutes) from
`recovery-continuation-pacing.ts`, as `NATIVE_OWNERSHIP_PROOF_BOUND_MS`.
- The wake has no time window. The wake query returns the deadline of
every marked run that is still undecided (no successor, no denial, no
lease owner), with the same eligibility as the denial. When that
deadline is past, the queue asks for a pass at once. That pass always
removes the run from the query:
  - It denies the run, which ends the wait.
  - Or it starts the exact replacement, when ownership was proven.
- Or it finds the run legitimately undecidable and clears the marker.
This happens when the history has unresolved interactions, or when the
task is not eligible for a successor (for example in backlog). The
existing task and interaction notifications wake the queue again later.
- More than 25 overdue runs: the scan cursor requests the next pass at
once.
- A lost lock or any other error: the pass throws, and the coordinator
retries after its existing `retryMs`
(`EXECUTION_RECONCILIATION_INTERVAL_MS`, 15 seconds). That is the same
backoff as every replacement-decision error today.
- An environment lease that is never released no longer blocks the
decision after the deadline. Before the deadline, the run waits as
before and records the marker. After it, an unreleased lease counts as
"provider not proven stopped", so the pass denies the run and never
starts an exact replacement next to a live lease. The fallback then
treats the unreleased lease as pending cleanup and picks
`fresh_isolated_workspace`. Admission (`heartbeat/queue.ts`, the
wake-queue adapter) does not block a run that a fresh continuation
superseded, so the new run does not wait for or touch the old lease. The
environment orphan-settlement modules are not changed.
- When the pass first sees unverified ownership before the deadline, it
writes `ownershipUnverifiedAt` once into the failure detail. It does not
change `updated_at`. This marker only limits which runs schedule a wake.
The deadline still counts from the terminal time.
- `execution-control-work.ts`: when no replacement is pending, the
replacement queue schedules its next pass at the earliest future
deadline of a marked run. This is a single timer, not a polling loop.
Rows already past their deadline are handled by the pass that runs at
startup, with or without the marker.
- No changes to `execution-recovery-resolution.ts` or
`fresh-continuation-fallback.ts`. The fallback already selects
`fresh_isolated_workspace` when its own owner proof cannot show that the
old local writer ended, `remote_epoch` for a remote workspace, and waits
(bounded) only for a process proven alive on this host.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/native-safe-replacement.test.ts`: new cases:
- Past the deadline (6 minutes, and 2 days for a row stuck before this
release): denial recorded on the first pass, no exact successor, then
exactly one fresh continuation with `fresh_isolated_workspace` after two
resolver passes.
- Before the deadline: no denial, no hold, no continuation, and the
replacement schedule wakes at terminal time + 5 minutes.
- Ownership proven before the deadline: the exact replacement is
scheduled and no denial is written.
- Red before the fix: 3 of the 4 new cases failed (no denial written; no
wake scheduled). The exact-replacement case passed, as expected, because
it guards existing behavior. Green after: 93 of 93 passed in
`native-safe-replacement.test.ts` and `execution-control-work.test.ts`.
- `tsc --noEmit` for `server`: clean.
- Review fixes, red before and green after:
- A deadline that passed 10 minutes before the schedule was asked: the
queue requests an immediate pass, which denies the run and continues it
isolated.
- An overdue run with an unreleased environment lease: denied, no exact
successor, continued with `fresh_isolated_workspace`.
- An overdue run the pass cannot decide (task in backlog): one wake,
then no second wake.
- 26 overdue runs: 25 decided on the first pass, all 26 on the second,
then no wake. This one also passed before, because the existing scan
cursor covers it. It stays as a guard.
  - On the earlier head, the first three failed.
- Green after all commits: 97 of 97 passed in
`native-safe-replacement.test.ts` and `execution-control-work.test.ts`.
- `pnpm verify` selected 2624 of 3444 test files, because
`execution-control-work.ts` is widely imported. I stopped it on a
heavily loaded machine (load average about 20). Its partial output
showed only unrelated attention-test timeouts near 30 seconds. Required
CI supplies the broad coverage.

## Risks

- After 5 minutes, a run whose old provider is truly still alive on
another server is continued in a new isolated workspace. The old writer
cannot write to it, and its files are not applied. A process proven
alive on this server still makes the fallback wait for its own bound
first.
- Deploy burst: rows stuck before this release are denied on the first
pass, at most 25 per pass (existing scan limit). Continuations then go
through the existing pacing: 6 per task per hour, 5 per company per
pass, one per agent per pass, and the agent's concurrency limit.
- No schema or API change. No change before `turn.start` for healthy
runs.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 06:03:53 -06:00
Devin FoleyandPaperclip 53c9be8d41 Reuse identical captured workspace archives (#20476)
Reuse the current committed snapshot when a fresh capture matches its manifest and plaintext archive hash and size. Preserve the original encryption binding and verify each new run receipt for recovery, concurrent replay and storage request statistics.

Verified with 338 focused save and recovery tests, server typecheck and build, all final-head CI checks, and Greptile 5/5.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-09 23:27:25 -08:00
DottaandClaude Opus 5.5 177f642da7 fix(recovery): continue native runs that failed before their first claim (#19911)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs go through a coordinator claim before any provider work
starts
> - A run can fail before that claim, for example while it restores the
previous harness backup
> - The failure path then records a no-replay hold, as if the run had
acted
> - Workspace recovery saves the files, but the task stays blocked and
asks a person to send a message
> - Recoveries must continue without a human step when no side effect is
possible
> - This pull request lets the existing safe-replacement worker continue
such runs once, after their files are saved
> - The benefit is that these tasks resume automatically, and runs that
may have acted keep today's reconciliation

## Linked Issues or Issue Description

**What happened?**
Native runs that failed before their first coordinator claim were never
continued automatically. They had no claim, no provider process and no
provider or tool events. Their result recorded `executionRecovery: {
kind: "bootstrap", providerWorkStarted: false }`. The failure path still
moved the coordinator to `terminal_failure` with
`native_continuation_requires_reconciliation` and recorded a resolved
`preserve_without_replay_v1` hold (replay blocked, board owner).
Workspace recovery then saved the files, but the next action stayed
"Saved work recovered. Send a message to continue."
`nativeReplacementPending` accepted only a few failure codes, and the
interrupted-preparation path covers only legacy runs.

**Expected behavior**
When a native run provably failed before its first claim, Paperclip
continues it once, automatically, after its files are saved (or when it
had no workspace to save). Runs with a claim, a process, a provider or
tool event, or recorded provider work keep the existing reconciliation.

**Steps to reproduce**
1. Start a native run with a remote workspace.
2. Make it fail before `native.coordinator.claim` (for example, abort
the harness-backup restore).
3. Let workspace recovery save the files.
4. See that the task stays blocked and asks for a message.

**Paperclip version or commit**
Observed on e2d490c61. Reproduced on master before this change.

Refs #19906 (fixes the restore deadline that caused the observed
failures).

## What Changed

- `native-safe-replacement.ts`: add one class-level candidate, "failed
before its first claim". It requires the
`native_continuation_requires_reconciliation` coordinator at attempt 0
and controller generation 0, with no lease, assessment, decision or
result.
- The candidate also requires no process identity, no `sessionIdAfter`,
no provider/turn/item/tool/`run.result` events, no tool invocations,
`providerWorkStarted` not `true`, no Stop or cancellation, no explicit
user continuation, and no later run for the task.
- A no-replay hold must have completed file recovery (the existing
`completedWorkspaceRecoveryCondition`). A blocked task must carry this
failure's block receipt at the current status version.
- Discovery checks saved-file evidence (snapshot, stopped sandbox) or a
workspace-less run before it selects the candidate.
- The reconciler skips the stopped-session probe for this class. It
feeds the decision with known failure meaning and a complete effect
inventory, since no provider ran. It rechecks the candidate under the
task and coordinator locks before it inserts the successor.
- The successor is the existing fresh-session `native_safe_replacement`.
Its context carries `nativePreClaimContinuationOf`, so a failed
successor is not continued again. The existing transaction resolves the
hold (`replay: verified_safe_replacement`) and restores the task block.
- `doc/execution-semantics.md`: describe the rule.

Previous-release story: holds recorded by earlier releases (resolved,
`replay: blocked`) carry forward. The test seeds that exact shape. No
schema or persisted identity changes.

## Verification

- `cd server && npx vitest run
src/services/native-preclaim-continuation.test.ts`
- Before the fix: "continues automatically exactly once after its saved
work is recovered" fails (no successor).
  - After the fix: all 9 tests pass.
- The tests cover: the staging shape with an old-release hold continues
exactly once under concurrent and repeated sweeps; it waits for saved
files; it does not chain from its own successor; and it keeps
reconciliation when the coordinator claimed, a provider event exists, a
tool call started, provider work is recorded, a process started, or a
later run owns the task.
- `npx vitest run
src/services/native-runtime/native-safe-replacement.test.ts` passes (62
tests).
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- The successor uses a fresh provider session, like the other safe
replacements. History comes from `retryOfRunId`.
- The new predicate adds one indexed `not exists` on `heartbeat_runs
(company_id, context_snapshot->>'issueId', created_at)` per candidate.
It runs only for the narrow pre-claim failure code.
- A run whose prepared workspace uses the older `nativeWorkspaceSync`
descriptor without remote-workspace evidence stays held, as today.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 15:12:36 -06:00
DottaandPaperclip e3fb4bc73a fix(native): recover saved listener startup failures (#19479)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native recovery must preserve saved work and respect current task
authority.
> - A closed local API listener can stop startup before a provider
starts.
> - #19462 gives new failures a bounded bootstrap retry.
> - Older saved failures still have a generic terminal code and a
recovery hold.
> - This pull request verifies the old producer outcome and uses the
existing safe replacement path.
> - Eligible tasks continue once. Unknown provider outcomes and user
holds remain blocked.

## Linked Issues or Issue Description

Refs: #19462

**What happened?**

A restart can leave an old listener startup failure in
`terminal_failure` with `native_session_interrupted`. Startup recovery
does not claim it after #19462. A completed file recovery can also
retain its no-replay hold.

**Expected behavior**

Continue automatically when durable records prove that no provider
started, the source stopped, and the current files were saved. Keep the
existing attempt budget and normal admission checks.

**Steps to reproduce**

1. Close the actual API listener after a fresh native coordinator claim.
2. Persist the old executor's generic failure. Save the workspace and
confirm source termination.
3. Restart recovery. The baseline schedules no successor. The repair
schedules one successor after all receipts pass.

**Paperclip version or commit**

Baseline: `0252421d5c`. Old failure producer: `2f35eb1080`.

**Deployment mode**

Built from source. Local macOS arm64, Node 26.4.0, real embedded
PostgreSQL, independent filesystem execution. No external AI service is
used by the controlled acceptance.

## What Changed

- Recognize the exact saved local-listener guard through one proof
predicate shared by discovery and the locked transaction.
- Require a fresh first attempt, matching execution bindings, no prior
provider lineage, no provider events or tool calls, no checkpoint or
admission receipt, and no cancellation.
- Reuse current snapshot, exact stop, source block, attempt budget, and
successor idempotency checks.
- Wait for completed source file-recovery receipts. Retire only that
same source's verified preservation hold in the successor transaction.
- Bound discovery to 100 candidates per pass, keep completed history
quiet, and wake through existing transactional producer notifications.
- Retain retries after transient database and final ownership-proof
errors; advance the bounded cursor and force a rescan so a failing page
cannot hide later work.
- Let only the replacement duty discharge a proved-empty notification
during drain through its existing read-only readiness check, once per
notification generation. Keep real work and concurrent changes
outstanding without running replacement during drain.
- Wake replacement at canonical local-stop proof writes and the existing
dispatch eligibility boundary; these hints grant no new execution
authority.
- Add database regressions, scheduler quietness checks, real controller
handoff coverage, and an opt-in controlled Runner continuation.

## Verification

Final production head: `0df41b2fef2644195d1224d36984af3827da3e3b`.

Local environment: macOS arm64, Node 26.4.0, TypeScript 7.0.2, Vitest
5.0.3, frozen-lockfile dependencies. Tests execute freshly;
`--fsModuleCache` reuses only content-validated module transforms.

- Native safe replacement: 62/62 PASS on isolated PostgreSQL 17 (63.18
seconds), including clean per-suite database disposal. All 62 assertions
also passed on embedded PostgreSQL 18 beta, but its Darwin cleanup hook
timed out; that invocation is not counted as a pass. Docker's disk was
full, so the disposable PostgreSQL 17 data used a size-limited RAM
mount, with no host mounts. The owned container was removed afterward.
- Actual stopped-process producer suite: 31/31 PASS. A real exited child
produces the stop receipt, and its replacement notification settles only
after the outer transaction commits.
- Workspace and separate-process drain regressions: 33/33 PASS (75.90
seconds).
- Shared coordinator and execution-control lifecycle checks: 39/39 PASS
(16.43 seconds). Pending work, concurrent changes, cold startup,
shutdown, and passive drain boundaries remain covered.
- Actual suppressed Cloud routine/drain workflow: PASS (34.64 seconds),
using the unchanged test and timeouts. Its matching local RED reproduced
CI's 462-query, zero-pending-query, one-idle-hold timeout after an
admitted write committed during drain (69.46 seconds).
- Actual historical controller handoff: PASS (48.40 seconds). A separate
controller loads the original executor from `2f35eb1080`, writes its
generic listener failure, and exits. Recovery at the final head admits
exactly one successor. The real Rust Runner and controlled Codex process
complete governed `paperclip_finish`; attempt two commits and the saved
file remains intact. Only the child loads the old artifact, and both
isolated artifacts are restored afterward. Rust source subtree
`77961b70f177ec3c38d11abcaec92fd3d0e96ec0` matches the built binary. No
external AI service is used.
- Seven corrected existing fixtures pass: transactional stop-proof calls
(5 cases) and notification inventory (2 cases). Their earlier CI
failures exposed fixture calls outside the documented transaction and an
outdated expected queue set.
- Server typecheck: PASS.

Red/green proofs also cover the original saved failure on baseline
`0252421d5c` (zero successors instead of one), held-history quietness,
bounded forward scanning through 100 parked rows and concurrent wakes,
real snapshot/cleanup producer notifications, actual issue-row lock
timeout, and task/history eligibility notifications. An obsolete
prior-run question can be skipped to request another proof check; actual
skip while the task is paused or the source owns provider tool activity
schedules no successor.

The failed-page regression starts with 25 persistent verifier errors and
a healthy candidate 26. Baseline RED repeats the first page. Repair
GREEN advances the healthy candidate, revisits the original failures
after their fault clears, creates all 26 successors exactly once, and
becomes quiet.

Greptile is SUCCESS, 5/5 on the exact final head (2026-11-06 17:09:21
UTC). Final-head CI is GREEN: 56 successful checks, three intentional
skips, one neutral result, and no pending or failing checks. [Attempt
two and
results](https://github.com/paperclipai/paperclip/actions/runs/42258352876/attempts/2)
include passing build, typecheck, Rust, native server, package, browser
E2E, serialized server, general server/chat tests, and final
verification. Five jobs in the first attempt were interrupted together
at 17:13:12 UTC; the native-server log records an explicit runner
shutdown signal. The single infrastructure retry reran only those five
jobs and dependent verification, preserving the 49 successful checks on
the unchanged head. No timeout was weakened, and setup failures or
interrupted invocations are not counted as passes.

Focused commands from the repository root:

```sh
./server/node_modules/.bin/vitest run --project @paperclipai/server server/src/services/remote-workspaces.integration.test.ts server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts -t 'native safe-replacement decision|reconciles the drained remote backend' --fsModuleCache --reporter=dot
./server/node_modules/.bin/vitest run --project @paperclipai/server server/src/__tests__/execution-control-work.test.ts server/src/__tests__/delivery-work-coordinator.test.ts -t 'execution-control lifecycle|work coordinator' --fsModuleCache --reporter=dot
node node_modules/typescript/bin/tsc -p server/tsconfig.json --noEmit --pretty false
```

The historical acceptance sets `PAPERCLIP_TEST_HISTORICAL_LISTENER=1`
and uses an original-executor child artifact with a `try/finally`
restore. Normal regressions require no Rust binary.

## Risks

- Compatibility is deliberately limited to positively proved fresh
remote startup failures. Other old ambiguous failures remain held.
- Existing provider work, a changed task block, incomplete preservation,
or revoked authority must prevent execution. Normal retry admission
remains in control.
- This PR does not deploy the change or mutate a hosted task. Hosted
recovery still requires the repaired revision and all current proof
gates.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use, and code
execution. The exact backend model identifier and context window are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 11:49:28 -06:00
DottaandPaperclip 5e4cc01073 feat(workspaces): keep remote files off the webserver disk (#18403)
Keep native sandbox workspace files independent of the webserver disk. Use encrypted saved versions and Postgres ownership so a replacement server can recover files and continue work. Enable the path by default for new eligible workspaces when durable storage is configured.

Preserve explicit opt-outs and existing host workspaces. Include safe snapshot repair, Files access, import, migration and retention. Bound saved-Files resources, preserve imported Git recovery, reject unfinished Git operations, publish immutable concurrent captures, and verify permanent object-version deletion.

Validated 471 targeted local controls, build/typechecks, all 40 CI partitions, fresh Greptile 5/5 and security review. The PR records original failed attempts and scoped retries.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 14:54:01 -05:00
DottaandPaperclip baf651c8d0 perf(heartbeat): drive retry promotion from durable deadlines (#18387)
Reviewed and verified at f1c48c924d.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 09:07:52 -05:00
df6156c2e3 fix(server): wake native status reassessment from committed changes (#17709)
## Thinking Path

> - Paperclip manages agents, tasks and their durable native results.
> - Changed evidence, task authority or review retirement can require a
new status assessment.
> - Periodic orphan recovery repeatedly reads history to find these
changes.
> - This change gives reassessment an owner driven by committed changes
and saved deadlines.
> - Existing result, assessment, decision and task rows remain durable
truth.
> - The status committer still verifies current sources and authority
before writing.
> - Settled or insufficient evidence becomes quiet until a supported
producer changes it.

## Linked Issues or Issue Description

Refs #17702 (merged), #17679 and #17680. This is slice 50, now based on
master. It owns native status reassessment; workspace settlement,
retained sessions and other orphan duties remain separate.

**What happened?**

Periodic recovery repeatedly inspects completed native results. Removing
that scan without complete writer coverage or current authority checks
could strand new evidence or commit an old result over newer work.

**Expected behavior**

Reassess from existing durable facts after relevant outer transactions
commit. Retain real retry/lease deadlines and restart/drain recovery.
Preserve current authority, quality-review and supersession checks. Make
no recurring SQL when work is settled or dormant.

**Steps to reproduce**

Complete a native result awaiting evidence or review, leave it idle,
then approve a real work product or retire the selected automatic
review. Its saved result should advance once when authorized, or record
the existing nonterminal business gate, without a periodic scan.

## What Changed

- Extract reassessment and its pure status policy from finalization
recovery into their owning modules. Add a scoped worker using the
existing delivery coordinator and scheduler; no queue table or migration
is added.
- Retain the three durable discovery triggers: changed work-product
history, authoritative task status/version, or retirement of the
selected automatic completion review. Preserve selected-source,
ownership, effects, retry and lease fences under the existing committer.
- Notify from actual task/work-product, claim release, approval,
interaction, OAuth, event evidence and workspace-settlement owners
before their transaction writes. Keep provider calls outside short
transactions and nested writes behind outer commit.
- Scope ordinary milestone/finalization/coalesced-input wakes to an
issue/run. Keep genuine company authority restoration explicit. Real
Photon and coalescer regressions cover all three response/status queues.
- Move the current parent's required quality policy into the owner and
apply it to every reassessment that could complete a task. An unmet
review records the existing in-progress decision and next action,
avoiding an operational retry loop. Preserve conversation/session reset
locks and the workspace omission-barrier signal.
- Skip reassessment in periodic recovery while preserving
explicit/startup recovery and the historical supersession guard before
independent workspace work. Document the source contract, producer
inventory, failures and qualifications in the linked audit.

## Verification

- Broad source `9fddb3f0b1157df32b64fb1f22e9d6e9624470e8` passes **2731
distinct cases in 28 complete files**, with no skips: 2458
producer/owner/authority/workspace/quality cases in 394.43 seconds and
all 273 governed-response cases in 53.74 seconds. Nineteen dedicated
fresh PostgreSQL 17 databases migrate through 0413; eighteen
embedded-only invocations preserve fresh-per-call isolation through the
existing recovery helper. All 22 temporary imports restore exactly and
all 28 hashes match.
- Final runtime source `5ea3b0fd016bf7e63ba82a7044825a46a0cd2082` passes
**1211 cases in five complete affected files** in 108.23 seconds after
the narrowly reproduced evidence-trigger fix: all 43 worker, 145 corpus,
32 finalization-recovery, 968 native-session and 23 quality-review
cases. Four fresh databases plus the quality helper's own fresh fixture
use PostgreSQL 17. Its one adapted import restores exactly; all five
hashes match. The other 27 broad-run test files are unchanged. These
qualifications are source-scoped and not cumulatively double-counted.
- Required/optional real regressions reproduce retry loops on
retired-review and evidence-approval paths before repair; both become
SQL quiet afterward with unchanged assertions/time limits. Controlled
removal of only ordinary scope guards reproduces two coarse-hint SQL
failures; exact guard restoration passes. Real producer, outer
transaction, claim, authority, 103-item backlog, retry and drain cases
pass. Photon transport is mocked; callbacks, locks and observed SQL are
real.
- Normal merges preserve 12661 parent/unrelated entries, 35 owned
entries and 21 automatic shared results, with five original conflicts
reviewed explicitly. The parent's actual squash reconciliation preserves
the entire held child tree. The final quality fix changes only its owner
and tests. Fourteen added comments are removed with identical nontrivia
tokens; no directives are removed and the final comment check is empty.
- Database build, server types/build, boundaries and exact-head
migration order pass. The database tree is identical to merged #17702:
no new SQL, queue table or snapshot is generated. Earlier benchmarks,
host limitations, failed assertions and their repairs remain in
`doc/plans/2026-10-26-native-status-reassessment-sleep-audit.md` and
`doc/idle-native-status-reassessment.md`.
- This qualifies targeted external PostgreSQL behavior, not embedded
bootstrap on the semaphore-exhausted host, a full local repository
suite, deployed sleep or measured fleet savings. Fresh exact-head CI and
review remain required.

## Risks

- Notifications retain the process-local supported-writer contract.
Independent SQL writers and uncoordinated replicas are not covered by a
new transport.
- Reassessment competes with current authority changes. Existing
row/source locks and retries remain required; arbitrary missing
identities have no row to lock, so this is not a general predicate-lock
guarantee.
- Discovery pages 100 rows but retains outstanding candidates in memory.
It remains proportional to backlog size, and cancellation occurs between
queries.
- Other orphan/workspace/session duties and Cloud sleep eligibility
remain unchanged. Missing supported producer coverage could strand
dormant work, so the producer audit and real callback/transaction tests
remain part of acceptance.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell
execution and PostgreSQL tests. The runtime does not expose a more
specific serving snapshot or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted scope is stated
above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 03:40:01 -05:00
DottaandPaperclip 4e715eb7ea fix(server): schedule historical unsafe export recovery (#17687)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs can retain accepted results after an unsafe workspace
export fails.
> - Historical recovery can retire an old notice or resume the accepted
result without another provider turn.
> - The periodic reaper repeatedly scans failures that cannot currently
recover.
> - Existing rows and committed changes can identify the actual recovery
obligation.
> - This pull request wakes recovery from those changes and retries only
owed work.
> - Dormant history becomes quiet while ownership and explicit operator
decisions remain protected.

## Linked Issues or Issue Description

**What happened?**

The native finalization reaper repeatedly visits historical unsafe
workspace-export failures. Cancelled tasks, newer runs, changed
contracts, and explicit operator decisions can make those rows
permanently ineligible. They still produce periodic database reads.

**What did you expect?**

Retire stale unsafe-export notices and recover eligible accepted
results. Stop reading dormant history until a relevant committed change
occurs. Keep a retry while a real obligation is blocked by ownership or
a transient failure.

**Steps to reproduce**

Retain an unsafe-export terminal failure whose task is cancelled or
superseded. Leave the instance idle after startup. The periodic reaper
continues reading that history.

Prerequisite #17680 is merged into master. Related: #14402 established
safe export finalization; #14329 was an earlier historical recovery
proposal. #17679 owns separate policy-review cleanup. This PR changes
only historical unsafe-export scheduling and its producers.

## What Changed

- Register a scoped worker through the existing coordinator and
scheduler. Remove this duty from periodic reaping. Preserve the
manual/startup recovery entry point.
- Distinguish stale notices, eligible accepted results, dormant history,
and concrete ownership contention. Reuse the existing mutation owner and
lock order. Preserve explicit settlement vetoes and exact
result/contract identity.
- Signal relevant task, run, action, and deletion writes before they
commit. Existing transaction boundaries handle nested commits and
uncertain outcomes. Notifications contain only scope IDs.
- Retain a retry only for owed work. Reconstruct work at startup,
preserve drain/resume, and stop through the shared lifecycle.
- Add concurrent, idempotent partial index migration 0410 for unsafe
terminal failures. Keep the five newest generated schema snapshots.
- Document producer coverage, query plans, supported writer boundaries,
and test failures in the unsafe-export audit.

## Verification

- Source `2a13dd08a6cc52f0adab09e3df9ce31e97b746b9` passes all 2188
cases across 24 complete files, without skips. The main selection passes
2129 cases in 192.58 seconds; safe-replacement passes 59 in 26.04
seconds. Coverage includes all 968 native-session cases, real
export/action producers, explicit continuation, workspace
reconstruction, interaction routing, Stop, queue and Slack consumers.
The audit gives per-file counts.
- Twenty separate fresh PostgreSQL 17 databases migrate through 0410
before setup. Four embedded-only fixture imports temporarily use their
own external database on this IPC-exhausted host, then are restored byte
for byte. Assertions and timing limits stay unchanged. This is qualified
external-database acceptance, not an unmodified embedded CI run.
- Database build, server typecheck/build, module boundaries and
exact-head migration order pass. Normal integration preserves the
complete original runtime/schema patch and unrelated upstream blobs.
Original concurrent index SQL, all prior executable history, journal
entries and retained snapshot bytes are unchanged; five snapshots 0406
through 0410 remain. Added comment removal preserves the full
non-comment TypeScript token streams.
- A populated 0409-to0410 upgrade preserves all 408 prior migration
receipts and 38 companies. It appends exactly one receipt; replay adds
none. The index is valid and ready on fresh and upgraded databases.
- The owning audit retains earlier fixture failures, skipped checks and
notice-retirement regression history. Older combined stack checks retain
their original scope in their owning audits. Actual parent
`73d24aa5acfcf17b3f676b4da7cd05eb70b60b99` is reconciled through a
normal merge. Its tree equals the reviewed ancestor and the complete
accepted child tree is unchanged. Fresh published-head CI and Greptile
5/5 remain required. No deployed sleep, fleet-cost or live provider
qualification claim is made.

- Inherited prefix correction: current integration
`8f43c9f81fac23b50997e7f0e1d0d9cf383a8109` changes only two values in a
previously unselected corpus fixture. All production code, the complete
package tree and all 24 accepted test files remain byte-identical to the
2188-case source above. The additional complete 72-case corpus passes in
14.00 seconds and again on the same database in 13.08 seconds. Retained
PSC/OUT and sentinel rows remain unchanged. The repeat is not
double-counted. The parent audit retains the original collision and
rejected cleanup attempt.

- Latest parent context: integration
`4d51231abe8b68c21ce896d0d2ac16865978c8ab` carries only the parent's
one-argument GitHub observation correction and an independent master
cleanup assertion. All production code, packages, 24 accepted suite
files and the additional corpus are unchanged. The two focused
native-presentation cases pass in 15.14 seconds and the complete
cleanup-removal suite passes all six cases in 8.31 seconds on two fresh
0410 PostgreSQL fixtures. This does not claim the full parent workflow
was rerun here.

Current head `b28399babac72ba9db1c4a4ef882c5e87aded38c` has passing CI
and Greptile 5/5 with no actionable findings. The first CI attempt lost
runners: fourteen jobs explicitly received simultaneous shutdown
signals, two reported lost runner communication, and two aggregate gates
failed downstream. One failed-job retry on unchanged source passes. The
underlying host shutdown cause remains unconfirmed; no code fix or extra
retry is claimed. [CI run and
attempts](https://github.com/paperclipai/paperclip/actions/runs/40819408528).
No Ponytail verdict is present.

## Risks

- A missing future producer can delay dormant-history discovery until
restart. The audit names current producers. Independent SQL writers and
other server processes remain outside the process-local signal contract.
- Physical ownership and uncertain operations retain retries. Time alone
does not authorize taking workspace ownership. No provider turn or
unsafe archive retry is added.
- The recovery owner remains authoritative if state changes after
discovery. Terminal UI status does not authorize resource destruction or
accepted-result replay.
- The partial index adds matching-row write/storage cost. It builds
concurrently through the existing restart-safe migration runner.
- This proves the changed worker becomes quiet. It does not prove host
suspension, external wake coverage, or fleet cost savings. Other sleep
migrations remain. This slice still requires fresh current-head CI and
reviews.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning and code-execution tools.
The exact serving model ID and context-window size are not exposed in
this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-29 00:13:22 -05:00
DottaandPaperclip 898144f4b3 fix: stop idle execution-control recovery polling (#17094)
Replace idle execution-control polling with workers driven by committed writes and persisted deadlines. Preserve startup recovery, bounded retries, and shutdown safety. Stabilize the Slack setup browser regression by waiting for the restored setup screen before simulating its webhook.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-21 09:54:28 -05:00
DottaandPaperclip 992f720262 fix: make runner task context ownership explicit (#13753)
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100). -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task descriptions, comments, continuation data, skills, and
execution rules enter several agent adapters.
> - The same source can be rendered by more than one automatic input
carrier.
> - Failed resumes can also rebuild input from stale or compact context.
> - This pull request gives each Paperclip-owned source one delivery
owner and preserves the required transport boundaries.
> - It adds deterministic adapter, interaction, runner, and browser
tests for these boundaries.
> - The benefit is more predictable context delivery with explicit
evidence for later live qualification.

## Linked Issues or Issue Description

Related: #13144 removes a duplicate environment payload and bounds wake
lists. Related: #11360 addresses Hermes resume behavior. This pull
request preserves compatible active-session formats while repairing
context ownership and stale question creation.

**What happened?**

Task descriptions and comments could enter more than one automatic
context block. Native transports could wrap a complete model input in a
second task envelope. Some legacy and gateway adapters could omit the
owned assignment on ordinary tasks or rebuild a failed resume with stale
compact context. A continuation could also request a question after
newer human comments had arrived.

**Expected behavior**

Each task or comment source has one automatic model-facing owner.
Distinct comment IDs and repeated wording remain distinct. Fresh
fallback attempts rebuild the required full context. A question request
is rejected when newer queued human direction makes it stale. Harness
access policy remains owned by execution configuration.

**Steps to reproduce**

1. Build a task with a description and current comments.
2. Capture the actual adapter or runner input.
3. Compare source ownership and task-envelope nesting.
4. Queue a human comment before a continuation requests a question.
5. Trigger a failed resume and inspect the fresh retry input.
6. Run the focused adapter, interaction, runner, and browser checks.

## What Changed

- Add shared prompt-section selection at the provider-attempt boundary.
- Deliver owned assignment context through native, legacy CLI, ACP,
gateway, cloud, Pi, Kimi, Grok, Gemini, OpenCode, Cursor, OpenClaw, and
Hermes paths.
- Rebuild full or compact context after resume recovery changes the
attempt. Add native and Claude ACP tests of actual recovery requests.
- Preserve custom templates, loaded instruction files, execution
policies, and older active-session formats.
- Record continuation source metadata and reject stale question creation
under the issue-row lock.
- Add explicit Product E2E context-integrity profiles, prerequisite
gates, credential-isolation checks, and report fixtures.
- Bypass service-worker forwarding for same-origin Vite development
modules. A real Chromium test fails with resource exhaustion before the
repair and passes after it. Production asset caching keeps its existing
policy.
- Add browser diagnostics and service-worker module-loading regressions.
- Add an explicit zero-retry eval option. The default retry behavior
remains unchanged. Each campaign records its effective policy.
- Remove the model-facing working-directory sentence from four prompt
builders. Existing workspace, sandbox, permission, and custom-template
configuration remains unchanged.
- Align the everyday workflow assertion with the current 47-entry
catalog.

Compared with current upstream master, the branch carries the
context-ownership implementation and its tests, the explicit
context-integrity catalog and evidence harness, and the focused browser
regression checks.

## Verification

**Merge assessment:** focused regression evidence supports merge. This
is not full completion of the original broad qualification matrix. The
maintainer has authorized merge after fresh verification of the master
integration.

- Current head: `bbd52f82114eabf09bc7b1a7e97d54a5b43bbc00`. This
integrates current master `2f585ef26a1814fa209715242d1ca791b63e4c4e`.
All 14 conflicts are resolved. Cancellation checks, workspace
finalization, native Grok support, and both sets of tests are retained.
- Current-head Greptile: **5/5**, with no blocking findings. The review
names this exact commit. All **59 reported checks are terminal: 55
successful, 4 skipped, zero pending or failing**. This includes the full
root general and serialized suites, separate runner checks, typecheck,
build, canary, browser E2E, Docker, and security checks. The successful
legacy security status is included in that total.
- After integration: workspace typecheck and full build passed. Separate
runner checks passed: **2,160 TypeScript tests (10 skipped), 582 Rust
tests, and 39 preparation checks**. Other passing checks include 621
Product E2E harness units, 376 focused shared/adapter tests, 160
real-database/API tests, 86 Hermes tests, 18 browser-support checks, and
Product E2E typechecking. The complete root suite passed in CI. The
duplicate local monolithic root run was stopped after that CI result; it
is not counted as a completed local pass.
- New native recovery coverage retains full assignment, completion
contract, and explicit skill selection after safe replacement, for old
and prepared input formats. Full native session test file: **136/136
passed**.
- New Claude ACP coverage captures actual fresh, resumed, and
missing-session fallback requests. It verifies one assignment copy,
comment order, identical text under distinct comment IDs, and full
fallback context. Full file: **33/33 passed**. Both affected TypeScript
checks passed.
- Existing deterministic tests cover source revisions, approval and
trust boundaries, completion validation, custom templates, compatible
sessions, standalone driver wrapping, and maintained adapter transport
requests.
- Provider-free browser support: **17/17 passed** after the master
merge. Service-worker unit tests: **33/33 passed**. The module-overload
regression failed before the repair and passed after it in real
Chromium.

### Fresh live comparisons

The new batch ran exactly four Product E2E attempts. **All four passed
on the first attempt; no retries.** Each has six terminal matchers plus
the existing browser lifecycle and invariant checks.

| Exact case ID | Control | Candidate |
|---|---|---|
| `core-compatibility.runner-codex.local.plan-revise-accept` | Passed |
Passed |
|
`local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`
| Passed | Passed |

The plan case checks a revised canonical plan and revision-bound
approval before completion. The question case restarts the server before
submitting the answer, then verifies the continuation completes.

Control source is `dfa4e1bda8d50a1a01746603251a9128dbe9d0d6`. Candidate
source is `79fcdb5dece501d28064ea9da306603881b46f0c`. They use identical
frozen definitions and provider versions: Codex `0.156.0` with
`gpt-5.6-sol`; ACPX `0.13.1` / Claude ACP `0.73.0` with
`claude-sonnet-5`. The September 24 head added master browser recovery
and test-only changes. The September 28 head also integrates newer
master changes, including cancellation, workspace finalization, and
native Grok. These are frozen-source live results, not exact-head live
runs.

The candidate received one description copy where the control initially
received three. The submitted initial plan envelopes were 7,969 versus
19,097 characters. Question envelopes were 7,592 versus 18,919. These
are structural measurements, not whole-provider token or dollar savings.

### Earlier evidence and failed attempts

- The preceding fresh batch has four effective passing pairs: OpenCode
comment continuation and assigned skill, native Codex comment
continuation, and native Claude comment continuation. It retains **11
attempts: eight passed and three failed**.
- Original failures remain recorded: missing local PostgreSQL library
links before task creation; host-sleep cleanup after task/page checks
passed; and a Claude **control** session-open rejection before a model
turn. Setup was repaired identically on both worktrees. The permitted
unchanged infrastructure retries passed. The underlying Claude provider
startup error was not retained and remains unknown.
- Older R2 retains **17 passes and one failure** across 18 attempts,
including eight both-pass native/legacy Codex/Claude pairs. Its OpenCode
blank-page failure led to the service-worker repair. R2 is historical
evidence: master changed the native fixed prompt and removed duplicate
wake environment data afterward.
- The September 24 CI run initially failed one unrelated preview
readiness test (`ECONNREFUSED` on its local fixture). Its test and
production code match master. Isolated local verification passed **28
tests, 3 skipped**. One unchanged CI retry passed the full shard: **831
passed, 1 skipped**, including all **31 preview-exposure tests**. The
aggregate CI gate passed afterward. The precise startup cause remains
unknown; a port race is a hypothesis, not a proved cause.

### Limits

The original wider profile/workflow matrix, repeated trials, and remote
Daytona qualification are incomplete. These results support a focused
merge recommendation, not statistical equivalence or universal harness
qualification. Some usage receipts are missing in both variants, so no
token or dollar savings are claimed. The $500 ceiling was preserved
using conservative allowances; failed attempts and unknown charges
remain in the ledger.

Reproduce the focused additions with `pnpm exec vitest run
packages/adapters/claude-local/src/server/acp.test.ts` and `pnpm
--filter @paperclipai/paperclip-runner exec vitest run
src/native-session-runtime.test.ts`. Full checks use `pnpm -r
typecheck`, `pnpm test:run`, `pnpm build`, and the separate runner
checks. Paid evals require the frozen definitions, profiles, and
credentials; do not use `--all` as a substitute for the selected cases.

## Risks

- Context placement changes can affect model behavior. Deterministic
checks cover the selected paths, but live qualification remains
incomplete.
- The stale-question guard can reject a request when queued human
comments arrived during the run. This is intended.
- New stored inputs and model envelopes retain compatibility readers for
older active sessions.
- Custom templates may intentionally repeat content.
- Removing a model-facing working-directory sentence does not change
filesystem, command, sandbox, or permission configuration.
- The worker bypass applies only to same-origin development module
paths. Cache-policy tests preserve private-response handling and
production asset caching. Mounted HTTP fixture changes remain test-only.
- This PR does not claim measured token savings or statistical
equivalence across every harness.

## Model Used

OpenAI Codex, exact model gpt-6-astra, with repository tools and code
execution. Bounded supporting work used gpt-5.6-luna and gpt-6-luna. The
serving context-window size is not exposed in this task.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR using the required issue fields
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id
- [x] I have run the focused local checks and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect these changes
- [x] I have considered and documented risks above
- [x] All current-head Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
for the current head
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-28 14:49:14 -05:00
DottaandPaperclip 422287eecd fix: preserve runner recovery, warm sessions, and task outcomes (#13338)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runner connects task messages, provider execution, and
task outcomes.
> - First-time user tests exposed gaps in recovery, completion
permissions, message delivery, and Stop behavior.
> - These gaps left usable output hidden, completed work waiting for
bookkeeping, or safe work unable to continue.
> - This pull request fixes the shared lifecycle and receipt paths while
preserving process ownership and action checks.
> - Users can continue work with accurate task state and durable
messages.

## Linked Issues or Issue Description

**What happened?**

A stopped local Codex execution could remain blocked even after its
processes had stopped and its complete transcript proved that no
external action needed replay. Claude under Conservative permissions
could fail to call task completion tools. Recovery could reuse an
assistant item ID and overwrite prior output. A delivered comment could
remain marked uncertain after navigation. Stop could look like Pause or
a new recovery incident. Workspace contention could look like
cancellation. A direct reply reopening Done could enter a clarification
loop.

**Expected behavior**

Recover automatically only with verified termination and complete action
receipts. Preserve answers and messages. Keep task completion available
under Conservative permissions without broad tool access. Show crashes
as Blocked, actual human decisions as In Review, and ordinary workspace
contention as waiting. Stop the current response and allow a new
direction.

**Steps to reproduce**

1. Create ordinary response tasks with local Codex and Claude Code, then
send follow-up messages through the task composer.
2. Interrupt a disposable local Codex runner during text-only work.
Verify automatic continuation and retained output.
3. Stop a response, send a new request, answer a clarification, and
reopen completed work with another message.
4. Navigate or reload while a comment submission is pending. Confirm the
exact persisted request receipt settles it without removing newer draft
text.
5. Run two tasks in a shared Daytona workspace. Confirm waiting does not
appear as failure.

**Paperclip version or commit**

Initial acceptance baseline: `c9021c6721f91e2c74bd9fee9d3fd41c999d17b7`.
Current integration base: `6cef9743c`. Both operator-interruption and
workspace-waiting guards are preserved; native restart and legacy
permission rules remain documented.

**Deployment mode**

An isolated source-built test-drive instance, with real local Codex and
Claude Code providers and disposable Daytona environments.

Related work: #13314, #13316, #13327, #13344, #13239, #13254, #13163.
This PR addresses additional failures from ordinary task journeys,
including controller restart handoff and repeated warm sandbox setup.
Historical task status reconciliation is excluded.

## What Changed

- Persist runner ownership immediately at spawn and resume an explicitly
adopted runner even when the controller crashed before the first driver
checkpoint. Detach the controller safely across graceful restarts,
including session startup. Prevent an old finalizer from suspending or
signaling an adopted runner. Checkpoint idle warm sessions before
shutdown. Preserve the same run and queued follow-up messages.
- Scope saved legacy queue successor checks to the queue owner while
preserving ordinary task locks, operator identity, assignment gates, and
exactly-once delivery.
- Preserve managed Codex credential files when an old session is
detached for restart; normal owned cleanup still copies refreshed auth
back and removes the scoped copy.
- Reuse the bound warm shared sandbox and fully verify an existing
staged provider pack before using it. This avoids repeated uploads when
the pack is already valid.
- Add a narrow local Codex replacement path with stopped-process proof,
a closed transcript inventory, exact completion receipts, and
fresh-session lineage. Preserve no-replay holds when evidence is
incomplete. Recovery may clear only the same run's recorded Blocked
status version; manual re-blocking and dependency changes invalidate
that receipt, while queued comments do not. Later blocks stop scheduled,
queued, and final dispatch; queued/final checks re-read dependencies
even when the task status stays In Progress.
- Permit only task delivery and human-input tools through the isolated
Claude runner's exact task bridge.
- Scope assistant item identity to the provider turn and ignore only
authority-free Codex skill-change notifications during startup.
- Reconcile composer submissions by client request ID across response
loss, navigation, and reload. Retain text typed during delivery.
- Keep acknowledged run-only Stop neutral and show workspace contention
as waiting. Project exhausted native failures as Blocked.
- Restore the guarded task-page retry action for failed legacy runs,
including the server-supported explicit new-attempt path for stopped
conversation adapters. Preserve native/process recovery holds and avoid
promising Retry while a decision or execution gate hides it.
- Refresh delivered artifacts and handle direct user replies that reopen
completed work without a clarification loop.
- Check the embedded PostgreSQL PID, data directory, and actual port
before connecting or migrating.
- Document accepted behavior and add focused regressions at lifecycle,
route, transcript, and UI boundaries.

## Verification

- Final head `fece606ac2` passes the complete GitHub CI matrix: **34
green checks, two expected Storybook skips, no failures or pending
checks**, including `ci / verify`, `ci / e2e`, full runner verification,
typecheck, build, every server/workspace shard, and all browser shards.
[CI
run](https://github.com/paperclipai/paperclip/actions/runs/34727183287).
Greptile is **5/5 with no open findings**. The final two commits only
refine test fixtures; both affected suites pass 24/24 locally and in CI,
with server typecheck green.
- Complete local Vitest coverage uses the canonical groups/shards: all
635 general server suites, all 145 serialized suites, and all workspace
packages. The aggregate began on `0a8001c18` while the final queue fix
arrived: 23,903 passed, five failed, 87 skipped. The five
port/socket/timing failures passed unchanged in follow-ups (60 tests in
the exposure/file suites and 412 tests covering the serialized failures
and unrun tails). The final queue/operator-identity suites separately
passed 52/52. This is aggregate coverage plus explicit reruns, not a
pristine single-command final-head run.
- After integration with current master,
queue/operator-identity/continuation suites passed 162/162 and affected
UI suites passed 140/140. ACP Stop/continuation and legacy
task/Inbox/message browser suites passed 9/9, including both task
recovery Retry and thread Try again, automatic saved-message delivery,
exactly one new run, Done, and retained output after reload. The default
process Stop/Pause/Resume browser case passed (the native-provider case
is opt-in and skipped by default). The complete Board attachment/receipt
browser suite passed 11/11 on a disposable instance, covering both
composers, exact receipts after lost responses, no replay, bound
attachments, and newer drafts after reload.
- Blocking-intent regressions cover pre-existing Blocked, a mismatched
run/cause, an explicit manual re-block, changed dependencies, a queued
comment after failure, and a block arriving between scheduling and
provider dispatch. The negative cases reproduced before the fix. All 478
affected executor/recovery/dispatch tests passed; both database suites
ran separately after availability-probe skips in the first combined
command. The final late-dependency check passed all 143 affected
recovery/dispatch tests (zero skips) after two new negative cases
reproduced the bug.
- Focused runtime regressions cover awaited runner ownership
publication, authenticated adoption before the first checkpoint,
old-finalizer detachment, idle and busy warm-session shutdown, rejected
checkpoint propagation, provider-pack verification, and managed-Codex
credential preservation. Four managed credential detachment cases
reproduced the bug before the fix; normal owned cleanup still succeeds
exactly once.
- Live local Claude: SIGKILL 2.6 seconds into startup recovered the same
run automatically in 53 seconds, then a normal follow-up completed in 24
seconds. SIGTERM 2.5 seconds into startup preserved the same run (54
seconds) and its queued follow-up (21 seconds). Answers remained visible
and the task reached Done.
- Live Claude Daytona: a warm follow-up retained its sandbox and fell
from 121 seconds to 44 seconds. A separate cold turn took 127 seconds;
after controller shutdown and checkpointing, its follow-up completed in
33 seconds with the same sandbox, workspace, native session, and runner.
Both answers remained visible and the task was Done.
- Other live journeys covered task completion and follow-up with local
and Daytona Codex, local Codex crash recovery, Stop then new direction,
clarification response, live artifact refresh, and shared-workspace
waiting.
- Validation limits: the opt-in native composer Stop/Pause→subtree
Resume fixture exposes terminal/result ordering and subtree-cancellation
attribution bugs that can leave a child task blocked; that new finding
is assigned to a separate follow-up and is not claimed fixed here.
Default CI skips this optional native-provider fixture. Managed-Codex
credential handoff and the queue-agent integration use automated
regression evidence. Cold custom provider-pack uploads still add startup
latency.

## Risks

- Automatic replacement remains deliberately narrow: local Codex,
verified stopped identities, unchanged retained state, and a complete
text/completion-only turn. Unknown actions, partial history, or changed
ownership remain blocked.
- Claude completion permission handling changes an upstream package
patch. The exact isolated task bridge must remain pinned; unrelated
tools keep their existing permissions.
- New task failure projection changes user-visible status. No historical
status backfill or database migration is included.
- This is a broad lifecycle fix across server and UI. Live proof covers
graceful local Claude restart during startup and idle Claude Daytona
session recovery across controller shutdown. Live abrupt SIGKILL during
local Claude startup also recovered the same run. Unknown ownership or
missing action evidence still blocks reuse. Cold custom provider-pack
uploads still add startup latency; this change avoids unnecessary repeat
uploads.

## Model Used

OpenAI GPT-6 (Codex), with reasoning, code execution, browser
automation, and tool use. The exact hosted model ID and context window
are not exposed in this task.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-12 19:41:15 -05:00
DottaandPaperclip 5545f6d166 feat: let user messages continue stopped native tasks (#13239)
## Thinking Path

> - Paperclip manages AI agents and their tasks.
> - A failed native run can leave a durable execution hold.
> - The hold prevents automatic replay of actions with unknown outcomes.
> - It can also prevent the agent from answering a new user message.
> - A new user message should authorize a fresh turn after the prior
execution stops.
> - This pull request adds that admission path and preserves the
existing execution gates.
> - Users can continue the conversation without certifying every past
action.

## Linked Issues or Issue Description

**Subsystem affected**

Server task wake admission and native execution recovery.

**Problem or motivation**

A native task can remain blocked after automatic recovery stops. A new
user message is saved, but its run is cancelled before the agent can
answer.

**Proposed solution**

Use a new authenticated user comment to authorize a fresh turn. Check
stopped predecessor ownership and available history. Retain uncertain
action outcomes. Commit the new run and hold retirement together.

**Roadmap alignment**

This is a focused improvement to the existing self-healing runs and
recovery behavior.

Builds on merged #13237, which covers legacy conversation continuation.
This PR adds native admission and preserves native automatic-recovery
eligibility and budgets.

## What Changed

- Admit a fresh native turn for a new user comment after every held
predecessor has stopped.
- Validate the comment author, task, timing, process ownership,
controller, and cleanup leases.
- Preserve failed runs, unknown action outcomes, and the failed
incident's attempt count.
- Record the new comment and run in the existing recovery audit history.
- Validate the saved continuation source and discard consumed user-wake
authority from later automatic replacements.
- Keep pause, approval, budget, ownership, and dependency interaction
rules.
- Add database and actual wake-path regressions. Update the execution
contract.

## Verification

- [Full CI run
34626750213](https://github.com/paperclipai/paperclip/actions/runs/34626750213)
passed on `c58e6c087e0df7530c747d80b27d491da925a9c4`: all 31 reported
checks passed, including all server/workspace suites, browser shards,
native runner verification, build, typecheck, release dry run, and
aggregate gates. The two conditional Storybook checks were skipped.
- Greptile reviewed this exact head at 5/5. All review threads are
resolved, and security checks passed.
- All 218 local targeted tests passed across explicit native
continuation, continuation history, safe replacement, durable chat
wakeups, wake queue, issue liveness, native session resume, and run
dispatch. The native implementation is unchanged by the final rebase
onto master.
- Full workspace `pnpm -r typecheck` and `pnpm build` passed on the
final head. Complete test coverage is supplied by the green CI suites;
local tests used the targeted suites above.
- Regressions cover scoped authorization, concurrent delivery, live
ownership, later admission gates, retained message receipts, and
automatic replacement after terminal-task or reviewer changes.

## Risks

- A fresh model turn can choose to repeat an action. Paperclip preserves
prior history and does not replay recorded calls.
- Missing process identity and remote ownership without a target-aware
stop proof retain the hold. A terminal database row alone does not prove
that execution stopped.
- No schema or dependency changes. Existing historical tasks are not
awakened by deployment.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, code
execution, and test execution. This session does not expose a more
specific model build ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-11 12:33:57 -05:00
DottaandPaperclip 35fdc0c66b fix: make task recovery durable and preserve current requests (#13075)
Make task recovery durable and preserve the latest user request across native and legacy continuations. Keep routine recovery quiet and prevent replay when action outcomes are uncertain.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-09 09:14:25 -05:00