Commit Graph
8 Commits
Author SHA1 Message Date
DottaandPaperclip f6ba9afab0 perf(server): narrow explicit continuation run reads (#17896)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Recovery must prove that old execution stopped before it admits new
user input.
> - Explicit continuation reads the source run before and after it takes
the run lock.
> - Both reads transfer the saved prompt and the full Runner profile.
> - The preflight needs only identity, comment, adapter, and workspace
receipt fields.
> - This pull request narrows those reads and keeps the complete locked
profile for session proofs.
> - The benefit is less database transfer with the same admission and
recovery rules.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Explicit native continuation admission after a failed or stopped run.

**Current behavior**

The preflight and locked reread select complete heartbeat run rows.
Large saved contexts and execution profiles cross the database
connection twice.

**Proposed behavior**

Read only required scalar fields and typed context keys. Read only
historical adapter identity and workspace sync receipts before the lock.
Keep the complete Runner profile in the locked reread. Keep lock order,
company scope, text lookup of historical run IDs, stop proofs, queued
input, and replay rules.

**Reason and benefit**

A synthetic PostgreSQL 17 fixture removes 74.9% of the two queries'
DataRow bytes. This result is local query evidence. It is not a claim
about deployed latency or monthly transfer.

**Breaking changes**

None intended. No schema, API, migration, workflow, or dispatch contract
changes.

**Additional context**

Related: #17491 narrows the separate execution blocker read path. This
PR covers explicit continuation admission. The PR has no dependency on
the other byte reduction PRs.

## What Changed

- Select required continuation fields and typed context keys instead of
complete run rows.
- Read a small historical adapter/workspace profile during preflight.
Preserve the full locked profile for exact suspended-session and
quarantined-cleanup proofs.
- Narrow four helper input types to their actual field requirements.
Their runtime behavior is unchanged.
- Add real PostgreSQL admission comparison coverage and a reproducible
baseline/treatment benchmark.
- Document the query boundary and benchmark command.

## Verification

- Baseline: `663fe667d66d3863d0919a8724f1d99500feef0b`.
- Disposable PostgreSQL 17.11. One source run has a 192 KiB
incompressible context and two 96 KiB profile payloads.
- Command: `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-continuation-run-reads.ts
--baseline-ref=663fe667d66d3863d0919a8724f1d99500feef0b`. Set
`PAPERCLIP_CONTINUATION_BENCHMARK_DATABASE_URL` to a disposable database
ending in `_test` to use an external server.
- The harness captures SQL generated by both real service versions.
Baseline imports use current helpers; only their TypeScript input types
changed in this PR. Their runtime behavior stays fixed.
- Preflight DataRow bytes: 394,067 to 466 (99.88% less). Locked reread:
394,067 to 197,168 (49.97% less). Combined: 788,134 to 197,634 (74.92%
less).
- Byte counts cover exact PostgreSQL text-format DataRow frames. They
exclude RowDescription, command metadata, TLS, TCP, and other queries in
the service.
- Seven measured samples per query follow two warmups. The order
alternates. Final-head median preflight time: 6.53 to 2.28 ms (ranges
5.34–7.41 and 1.84–4.46 ms). Median locked read: 8.06 to 5.26 ms (ranges
5.94–13.84 and 4.61–6.81 ms). These are warm local SQL measurements
under shared machine load. The deterministic byte counts are the
acceptance criterion; these noisy timings do not establish a deployed
latency gain.
- The harness compares the admitted continuation, blocked reasons, every
retained scalar column, kept JSON, and the full locked profile. It fails
unless the fixture removes more than half of the two reads' DataRow
bytes. It records SQL, parameters, revision identity, samples, and
EXPLAIN plans.
- All 291 continuation tests pass together on the final head with no
skips in a clean isolated PostgreSQL 17 database. This includes all
seven new projection comparisons and six malformed/missing JSON cases.
- Edited server TypeScript check passes: `node
node_modules/typescript/bin/tsc --noEmit -p server/tsconfig.json`.
- `pnpm -r typecheck` and `pnpm build` pass on the final head. The
normal server build verifies the complete pinned Copilot closure. Local
full `pnpm test:run` is not claimed: host embedded PostgreSQL startup
limits prevent that gate. The isolated PostgreSQL regressions pass; the
full exact-head test matrix runs in CI. The targeted exact-head test
retry passes all 4,086 tests in 98 files. All CI verification checks and
the dependent aggregate are complete and successful on head
`47c1b5f7d17a58310f7a3ebd494fdcaa814777c2`. Greptile completed cleanly
on head `47c1b5f7d17a58310f7a3ebd494fdcaa814777c2`: 5/5, no findings,
and no unresolved review threads.
- Local setup evidence: embedded initdb could not load its ICU data
library. Tests therefore use a dedicated PostgreSQL 17 container. A run
against reused test fixtures produced stale sweep failures; a new clean
database passed all 285 cases. The first comparison mock replaced an
unrelated aliased query; it was narrowed to the two admission reads
before the clean pass.

- The first [server test shard
6/12](https://github.com/paperclipai/paperclip/actions/runs/40529210562/job/122277218565)
failed one existing workspace-recovery case: a third export stayed
`scheduled` at its 20-second poll deadline; 4,085 other tests passed.
The exact case passes on the pinned baseline and final head with the
same PostgreSQL 17 fixture. Temporary spies recorded zero calls to both
changed admission functions in either replay. Other touched helpers have
type-only changes. This supports one timing-flake retry; the exact
scheduler delay remains unproven. The single [targeted shard
retry](https://github.com/paperclipai/paperclip/actions/runs/40529210562/job/122285744135)
passes all 4,086 tests in 98 files. The exact case completes in 17.25
seconds. The dependent aggregate is complete and successful. No timeout
or unrelated runtime code was changed.

- The separate Paperclip Review check ended `NEUTRAL` without a final
report. The completed source review is the exact-head Greptile 5/5
result above.

## Risks

- A future admission check may need an additional saved context key.
Keep the projection and helper contracts in sync with that consumer.
- Exact suspended-session and quarantined-cleanup proofs still require
the complete Runner profile. This PR retains it under the lock rather
than weakening those checks.
- JSON values retain their types and missing keys stay absent.
Malformed-context comparisons preserve the existing controller hold.
- These measurements use synthetic local data. Deployed transfer savings
require a fresh capture after rollout. Nothing was deployed.

## Model Used

OpenAI GPT-6 through Codex. The exact runtime model ID and context
window are not exposed by this session. The work used repository tools,
code execution, and real PostgreSQL validation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 12:05:51 -05:00
Devin FoleyandPaperclip 19ec787e5b perf(recovery): release native process identity without saved payload reads (#17622)
Keep locked process and terminal-state evidence plus JavaScript-compatible checkpoint presence. Preserve every host, provider, lease, coordinator and stop-receipt check without transferring saved prompts or checkpoint bodies.

53 focused tests, server typecheck, build, all 55 CI checks, and exact-head Apex 5/5 pass.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-25 10:06:27 -07:00
Devin FoleyandPaperclip b7dc594bfa perf(recovery): narrow saved-message execution-wait reads (#17588)
Read current ownership, startup, cancellation, task-binding, and adapter evidence without transferring unused saved profiles, launch context, or output. Preserve normal continuation admission, recovery holds, and locked mutation reads.

Verified with full build and typecheck, 537 affected tests across 12 suites, a counterfactual real PostgreSQL byte regression, complete exact-head CI, and Greptile 5/5 Medium with no unresolved comments.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-25 03:54:59 -07:00
Devin FoleyandPaperclip b5bf86ea09 Clear process identity when settled native runs stop (#17374)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each heartbeat run stores the process identity of its runner in
`heartbeat_runs.process_pid`, `process_group_id` and
`process_started_at`
> - Only cancellations (`release_run_resources`), restart recovery and
heartbeat recovery cleared these fields
> - Failed native runs take the `schedule_retry` path, so they kept a
PID after the run finished, for weeks
> - Liveness checks for old runs then depend on PID-reuse defenses, and
code that reads a set PID as "running" is misled
> - This pull request releases the process identity of settled native
runs when the process is confirmed gone, and backfills old sandbox rows
> - The benefit is that a finished run no longer looks like it owns a
live process, while live, warm and retrying processes keep their
identity

## Linked Issues or Issue Description

Refs #17358. Bug report described in-PR (bug report template).

**What happened?**
On one instance, 16 of 23 old native workspace sync folders belonged to
failed or cancelled runs that finished more than a week earlier and
still had `process_pid` set. The PID is written when the run's process
starts (for remote targets it is the sandbox PID). The status arbiter
clears it only through `release_run_resources`, which only cancellations
emit. Failed native runs take `schedule_retry`, and other terminal paths
(finalization reconciler, restart recovery terminal failures, watchdog
timeouts) do not clear it either.

**Expected behavior**
When a run is terminal and its process is confirmed gone (or no longer
owned), the process identity is cleared. A process that may still be
alive keeps its identity.

**Steps to reproduce**
1. Run a native task whose provider turn fails, so the arbiter decides
`run_failed_partial_evidence_preserved` with a `schedule_retry` effect.
2. Wait for the run to finish and the runner to exit.
3. Read the run row: `status = 'failed'` and `process_pid` /
`process_group_id` / `process_started_at` are still set.

**Paperclip version or commit:** `master` at 75fcb0b3fc.
**Deployment mode:** any.
**Agent adapter(s) involved:** native runtime (`paperclip_runner`);
local and sandbox targets.

## What Changed

- `releaseSettledNativeRunProcess` (in `native-local-process-stop.ts`)
clears `process_pid`, `process_group_id` and `process_started_at` of a
native run under a row lock, only when all of these are true:
- the status is `failed`, `cancelled`, `timed_out` or `interrupted`, and
`finished_at` is set;
- the finalization coordinator is absent, `committed` or
`terminal_failure` (a `retryable_failure` same-run retry keeps the
existing heartbeat recovery path; unfinished finalization keeps its
identity);
- the run has a session checkpoint or provider evidence, so native
session resume never mistakes it for an unused bootstrap;
- local target: every lease is released without failed cleanup, the
latest `native.process_identity_recorded` event names this host and the
same PID/group, and the PID and group probe absent. The release writes
the existing `native.local_process_stopped` receipt in the same
transaction, so `hasNativeLocalProcessStop`,
`readNativeLocalProcessStop`, explicit continuation and acknowledged
Stop keep their proof;
- sandbox target: every lease has a remote termination receipt (sandbox
PIDs are never probed on this host).
- Succeeded runs keep their identity: a warm runner can serve the next
turn, and chat retry matches warm predecessors by PID.
- The heartbeat executor calls the release after lease and scratch
cleanup at a terminal boundary (not when a same-run resume, workspace
finalization or held ownership is pending).
- Background work stays idle (see
`rules/sleep-aware-background-work.mdc`):
- Trigger: the executor settlement calls `settleNativeRunProcess`. If
the release is not ready yet (process still alive, local lease not
released, remote termination receipt missing), the run goes into an
in-memory pending set.
- Startup: the first `reapOrphanedRuns` pass after boot runs
`discoverSettledNativeRunProcesses` once, in pages of 50, over the
durable rows. This reconstructs work after a restart and covers terminal
transitions outside an executor (reconciler, restart recovery, watchdog,
timeouts).
- Retry: later passes call `retryPendingSettledNativeRunProcesses` only
for pending runs. It makes no database call when the set is empty. A run
leaves the set when it is released, becomes ineligible, or reaches 20
attempts.
  - Runs that this process still executes are skipped.
- Stop receipts now also record `processStartedAt`.
`withReleasedNativeRunProcess` restores a released identity from the
latest stop receipt. Retained session cleanup
(`reconcileRetainedNativeSessionCleanup`) and warm transition restart
checks use it. Quiescent runnerd state recovery reads the stop receipt
when the prior run's identity was released. Without a receipt, all three
still treat the identity as unproven.
- Migration `0377_clear_settled_native_run_process_identity` backfills
sandbox runs (without bumping `updated_at`) with the same rules: native,
settled status, `finished_at` older than 1 day, settled coordinator
phase, checkpoint or provider evidence, and at least one lease where
every lease has a valid remote termination receipt. The snapshot is
carried forward as `0377_snapshot.json`.
- Not changed: legacy (non-native) runs. Case handoff release, scratch
cleanup and conversation continuation read the terminal row's local
PID/group as process-tree absence evidence, so clearing them needs
separate work. `release_run_resources` cancellation behavior and
`native_run_finalizations.controller_*` fields (they track the server
controller, not the run process) are unchanged.

### Migration margin

The 1-day margin is defense in depth. The real proof is the remote
termination receipt: the provider confirmed that the sandbox stopped or
was destroyed. One day is far longer than lease release, sandbox stop,
restart recovery takeover and the finalization reconciler sweep, so no
in-flight teardown or same-run recovery can still read these rows. Local
rows are not backfilled in SQL: only the host that recorded the process
can verify that it stopped and write the stop receipt. The runtime sweep
releases them on that host.

## Verification

- `pnpm exec vitest run
src/__tests__/native-settled-process-release.test.ts` (new, 20 tests):
release with a stop receipt and restore of the identity from it;
`cancelled`/`timed_out`/`interrupted`; live process kept, then released
after exit; kept for succeeded, `retryable_failure`,
`workspace_finalizing`, unreleased local lease, another host, no
evidence, unconfirmed sandbox, mixed leases; sandbox release with
receipts; startup discovery paging and skip; pending retry; retry cap;
no database access while nothing is pending; `reapOrphanedRuns`
discovery, then retry after the process exits; executor release for a
failed native run and no release for a succeeded run.
- `native-provider-failure.integration.test.ts`: a real `schedule_retry`
decision (`run_failed_partial_evidence_preserved`) leaves the failed run
with its PID after finalization, and the release then clears it and the
stop receipt reads back.
- `native-session-executor.test.ts`: quiescent recovery and retained
session cleanup accept a released identity with a stop receipt and
reject one without a receipt.
- `packages/db` `client.test.ts` migration rule: the backfill does not
set `updated_at`.
- `native-finalization-migration.test.ts`: migration 0377 clears only
old settled native sandbox rows with confirmed termination (15 fixtures,
including recent, succeeded, running, legacy, unsettled phases, no
evidence, unconfirmed or local leases, no lease).
- Related suites, all passing: `pnpm exec vitest run` on
heartbeat-process-recovery, native-workspace-sync-retention(-config),
native-finalization-{migration,settled-candidates,recovery},
chat-failed-run-retry-boundary, acknowledged-native-stop,
explicit-native-continuation, conversation-continuation,
native-session-resume, native-run-finalizer-telemetry,
native-session-executor, native-safe-replacement,
native-finalization-reconciler-timing, native-restart-recovery,
native-finalization-recovery-state, run-lifecycle,
agent-instruction/directory working copies, heartbeat-case-handoff (1920
tests across 23 files).
- `pnpm run typecheck` in `server/`: passes. `pnpm run check:migrations`
and the snapshot drift test in `packages/db`: pass.
- Mutation checks (each reverted alone; a test failed every time):
executor call, startup discovery, pending retry in `reapOrphanedRuns`,
retry cap, pending-set cleanup on retry, start time in the stop receipt,
retained cleanup receipt fallback, coordinator phase guard, succeeded
exclusion, liveness probe, host check, remote receipt check,
provider-evidence guard, local lease check, quiescent recovery receipt
fallback, migration margin, migration NULL-safe receipt check, migration
phase guard. The NULL-safe check came from a real bug the migration test
found. The warm transition restart call site has no direct test: that
path has no existing fixture. It uses the same tested helper.
- Not run: the full server suite and e2e tests.

## Risks

- Behavior change: more native runs now have a null `process_pid` after
they settle. Every native reader was checked: explicit continuation,
acknowledged Stop and agent stop use the stop receipt for null
identities; cancelled-startup detection also requires no process events,
so a released run is not mistaken for a cancelled startup; native
session resume selection is protected by the evidence guard; chat retry
reads its warm-predecessor PIDs from succeeded runs, which are not
released.
- A recycled local PID keeps a row unreleased (conservative), and
sandbox runs from providers without termination receipts keep their
identity.
- Background cost: one paged discovery at startup. After that, only
pending runs are checked, at most 20 times each. With nothing pending
there is no recurring query.
- A transition outside an executor that happens after startup, with no
executor settlement, is released at the next startup discovery.
- Migration: a data-only `UPDATE` on `heartbeat_runs` with a selective
`WHERE`; it passes the migration safety check. It is safe to re-run.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-23 15:09:43 -07:00
DottaandPaperclip d33b40a5c3 fix: recover saved work and provide reliable task Retry (#16965)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The control plane must retain work when an execution or workspace
export fails.
> - Retained-workspace holds can leave saved input waiting with no
usable Retry action.
> - Users cannot verify process ownership or repair provider
allocations.
> - This pull request gives system recovery a durable budget and gives
each deliberate Retry a durable intent.
> - Safe recovery can preserve files, settle an accepted result, or
prepare an independent workspace.
> - One task state keeps infrastructure diagnostics out of the ordinary
conversation while preserving real permission and approval gates.

## Linked Issues or Issue Description

**What happened?**

A task with a retained workspace can show recovery, failed-run and
saved-message warnings at the same time. Historical runs without native
workspace descriptors cannot use the native restore path. Old recovery
metadata can prevent a new attempt even when an independent workspace
would be safe.

**Expected behavior**

Paperclip owns bounded infrastructure retries and file preservation. A
user can request another attempt with ordinary Retry. Thinking appears
immediately. The service preserves completed effects and current user
controls. It does not claim that starting fresh recovered the old files.

**Steps to reproduce**

1. Retain a failed run and its workspace after export or process-stop
failure.
2. Save a follow-up while the recovery hold is active.
3. Open the task and try to continue without infrastructure credentials
or operator certification.

Refs #16906, #16835, #16883. This extends their recovery and receipt
contracts. Related work: #16962 and #16963.

## What Changed

- Add task-level Retry with company-scoped authorization, all-status
idempotency and durable admission before physical work. Keep exact-run
Retry compatible.
- Add typed recovery phases, finite persisted backoff, historical
adoption and durable local failure diagnostics.
- Preserve native workspaces or verified historical snapshots. Admit an
independent fresh workspace only after execution containment. Keep
old-source preservation and cleanup obligations separate.
- Carry completed publication and API receipts forward. Fence uncertain
mutation retries across replacement runs. Preserve connected-chat
identity and authorization.
- Re-enter normal dispatch for saved input. Reuse accepted results
through the existing finalizer.
- Use one task interruption projection across the thread, composer and
task list. Show optimistic Thinking, ordinary Retry and genuine user
gates.
- Add PostgreSQL, filesystem, API, browser-fixture and controlled
provider coverage. Document the behavior and update the feature/access
maps.

## Verification

- All GitHub Actions CI jobs pass on
`6d8c6d27d50586d8548b31d03b47f6d73a0ee11d`: typecheck, all
server/chat/workspace and serialized suites, Runner checks, eight
browser shards, canary and build. The required merge gate passes.
- Greptile reviewed the final commit at 5/5, with no actionable findings
or unresolved review threads. Its two earlier P1 findings are fixed.
- Full recursive typecheck and build pass locally on the final commit.
All 16 task Retry cases pass against isolated real PostgreSQL. The
unrelated-success case and both stale Retry cases fail on the previous
code and pass after the fix.
- The corrected gateway fixture passes its uncertain-mutation and
independent-read case. Four real HTTP API cases pass for definite
403/422 rejection, completed-effect reuse, and uncertain-effect fences
across deliberate and automatic retries. These focused gateway/API
checks used a temporary database lifecycle harness on an existing
healthy local PostgreSQL process. The standard repository helper is
unchanged.
- Earlier qualification covers 291 task UI cases, saved-file copyback
and ancestry, current Agent Chat ownership races, nine chat Retry API
browser cases, and six assembled browser recovery fixtures. A real
API/PostgreSQL/native-runner journey showed Thinking in 25 ms despite a
delayed response, admitted one successor in a separate workspace,
retained original files, and survived reload.
- The controlled Daytona test interrupts export, joins the controller,
restarts it in a separate process, and verifies one agent start, one
external sentinel operation, retained file contents and no premature
cleanup. It does not establish abrupt-kill or live LLM/GitHub
acceptance.
- The full local `pnpm test:run` was attempted and was not green: an
OAuth row-lock timing assertion failed, and a later suite exhausted
macOS PostgreSQL semaphores. The ten OAuth timing cases passed on rerun
against a separate database. Later embedded database initialization
failures remain recorded; the final CI matrix passes.
- Security scanning passed on production-code commit
`63953edf069b111c6becc3144dffa6deb52f68b5`. Its only successor changes
one test-fixture identity line. The final scan is neutral/inconclusive
because its pinned merge-base checkout failed or timed out. Both rerun
API requests returned 404, so no fresh successful scan is claimed.
Socket and Snyk checks pass on the final head.
- The optional Paperclip Review remains queued behind an earlier run in
workspace finalization. Its prior completed assessment was incomplete
because a comparison tool was blocked by the platform filter. It is not
counted as a completed code review. GitHub does not require this check
for merge.
- Hosted acceptance remains pending deployment. Local and CI results do
not establish recovery of a hosted task or its retained files.

## Risks

- Recovery changes span admission, physical ownership and dispatch.
Incorrect evidence could permit overlapping work. Existing physical
locks and authoritative stop requirements remain in place.
- The migration adds concurrent indexes. It requires the existing
top-level migration runner and retries incomplete index creation safely.
- A fresh attempt can complete while old-source preservation remains
unresolved. The service retains that obligation and does not report
those files as recovered.
- Unknown tool mutations remain blocked until their authoritative
receipt owner resolves them. Generic upstream tools have no universal
result-query protocol.
- Recovery and diagnostics stay inside the stack. No Cloud incident
endpoint, notification worker or new outbound reporting channel is
included.

## Model Used

OpenAI Codex, GPT-6 family. The runtime does not expose a more specific
model version or context-window size. The model used reasoning, shell
execution, source editing and browser tools. No model subagents were
used for this implementation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused suites; the full
local run limitation is documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 19:58:53 -05:00
DottaandPaperclip 4cd7b40255 fix: recover new messages after historical native runs stop (#13405)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native recovery must distinguish a fresh user request from replay of
failed work.
> - Older runs can lose their process fields before a local stop receipt
exists.
> - A suspended durable session can still prove that the exact runner
and provider session are idle.
> - The message admission path ignored that evidence and kept new user
messages blocked.
> - This pull request uses the existing exact-state verifier for those
historical runs.
> - The user can start one fresh turn while the old history and unknown
outcomes remain intact.

## Linked Issues or Issue Description

**What happened?**
A user sent a new message after a native run exhausted recovery.
Paperclip saved the message but said the previous run had no verified
stop record. The old runner was suspended, with no active provider turn
or pending output. Its process fields had been cleared before stop
receipts were added.

**Expected behavior**
A new user message starts a fresh turn when the exact retained session
proves it is suspended and the other execution gates pass.

**Steps to reproduce**
1. Retain a failed native run with a terminal controller, cleared
process fields, and no process receipt events.
2. Retain its exact suspended runner state and idle provider state. Keep
its recovery hold.
3. Send a new user comment. Before this fix, admission returns no
successor.

**Paperclip version or commit**
Reproduced on master at d351e08de.

**Deployment mode**
Self-hosted server. The regression uses an embedded PostgreSQL test
database and real durable state files.

Related work: Refs #13270, Refs #13338. This adds compatibility for
older stopped runs. Refs #13332 concerns separate recovery-hold scope
rules.

## What Changed

- Add exact suspended-state evidence to explicit native message
admission for runs that predate process receipts.
- Reuse the existing failed-retry verifier for run, runner, workspace,
provider identity, and pending-work checks.
- Reject this fallback if any server-authored process receipt or launch
event exists.
- Add a red/green regression with real message admission, duplicate
delivery, dry-run behavior, and blocked-state cases.
- Document the new-message recovery rule.

## Verification

- Red: the historical suspended-state regression failed on master
because admission returned null. Eight rejection cases passed.
- Green: 469 tests passed across explicit native continuation and native
session execution.
- Full repository `pnpm -r typecheck` and `pnpm build` passed locally.
- The complete Vitest CI matrix and all browser test shards passed. The
duplicate local `pnpm test:run` was stopped after these CI results; it
did not complete locally.
- The first runner verification worker received an infrastructure
shutdown signal during compilation. The single retry passed.
- Greptile reviewed commit `6836d7310` at 5/5 with no findings.
- Inspected the affected server's database and durable state read-only.
It has the historical missing-PID shape and an exact suspended runner
with no active provider turn, pending tools, or undelivered output.

## Risks

- Incorrect idle evidence could allow overlapping work. The fallback
requires an exact suspended root and rejects active or pending work,
mismatched identities, missing files, and newer process evidence.
- Normal task, controller, lease cleanup, decision, and active-run gates
remain in force.
- This does not resume old provider actions or reset recovery attempts.
Unknown outcomes remain unknown.
- No schema change or deployment is included.

## Model Used

OpenAI Codex, GPT-6. The session does not expose the exact model
snapshot or context-window size. Used reasoning, repository inspection,
code editing, shell execution, and test tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-14 09:45:34 -05:00
DottaandPaperclip 422287eecd fix: preserve runner recovery, warm sessions, and task outcomes (#13338)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runner connects task messages, provider execution, and
task outcomes.
> - First-time user tests exposed gaps in recovery, completion
permissions, message delivery, and Stop behavior.
> - These gaps left usable output hidden, completed work waiting for
bookkeeping, or safe work unable to continue.
> - This pull request fixes the shared lifecycle and receipt paths while
preserving process ownership and action checks.
> - Users can continue work with accurate task state and durable
messages.

## Linked Issues or Issue Description

**What happened?**

A stopped local Codex execution could remain blocked even after its
processes had stopped and its complete transcript proved that no
external action needed replay. Claude under Conservative permissions
could fail to call task completion tools. Recovery could reuse an
assistant item ID and overwrite prior output. A delivered comment could
remain marked uncertain after navigation. Stop could look like Pause or
a new recovery incident. Workspace contention could look like
cancellation. A direct reply reopening Done could enter a clarification
loop.

**Expected behavior**

Recover automatically only with verified termination and complete action
receipts. Preserve answers and messages. Keep task completion available
under Conservative permissions without broad tool access. Show crashes
as Blocked, actual human decisions as In Review, and ordinary workspace
contention as waiting. Stop the current response and allow a new
direction.

**Steps to reproduce**

1. Create ordinary response tasks with local Codex and Claude Code, then
send follow-up messages through the task composer.
2. Interrupt a disposable local Codex runner during text-only work.
Verify automatic continuation and retained output.
3. Stop a response, send a new request, answer a clarification, and
reopen completed work with another message.
4. Navigate or reload while a comment submission is pending. Confirm the
exact persisted request receipt settles it without removing newer draft
text.
5. Run two tasks in a shared Daytona workspace. Confirm waiting does not
appear as failure.

**Paperclip version or commit**

Initial acceptance baseline: `c9021c6721f91e2c74bd9fee9d3fd41c999d17b7`.
Current integration base: `6cef9743c`. Both operator-interruption and
workspace-waiting guards are preserved; native restart and legacy
permission rules remain documented.

**Deployment mode**

An isolated source-built test-drive instance, with real local Codex and
Claude Code providers and disposable Daytona environments.

Related work: #13314, #13316, #13327, #13344, #13239, #13254, #13163.
This PR addresses additional failures from ordinary task journeys,
including controller restart handoff and repeated warm sandbox setup.
Historical task status reconciliation is excluded.

## What Changed

- Persist runner ownership immediately at spawn and resume an explicitly
adopted runner even when the controller crashed before the first driver
checkpoint. Detach the controller safely across graceful restarts,
including session startup. Prevent an old finalizer from suspending or
signaling an adopted runner. Checkpoint idle warm sessions before
shutdown. Preserve the same run and queued follow-up messages.
- Scope saved legacy queue successor checks to the queue owner while
preserving ordinary task locks, operator identity, assignment gates, and
exactly-once delivery.
- Preserve managed Codex credential files when an old session is
detached for restart; normal owned cleanup still copies refreshed auth
back and removes the scoped copy.
- Reuse the bound warm shared sandbox and fully verify an existing
staged provider pack before using it. This avoids repeated uploads when
the pack is already valid.
- Add a narrow local Codex replacement path with stopped-process proof,
a closed transcript inventory, exact completion receipts, and
fresh-session lineage. Preserve no-replay holds when evidence is
incomplete. Recovery may clear only the same run's recorded Blocked
status version; manual re-blocking and dependency changes invalidate
that receipt, while queued comments do not. Later blocks stop scheduled,
queued, and final dispatch; queued/final checks re-read dependencies
even when the task status stays In Progress.
- Permit only task delivery and human-input tools through the isolated
Claude runner's exact task bridge.
- Scope assistant item identity to the provider turn and ignore only
authority-free Codex skill-change notifications during startup.
- Reconcile composer submissions by client request ID across response
loss, navigation, and reload. Retain text typed during delivery.
- Keep acknowledged run-only Stop neutral and show workspace contention
as waiting. Project exhausted native failures as Blocked.
- Restore the guarded task-page retry action for failed legacy runs,
including the server-supported explicit new-attempt path for stopped
conversation adapters. Preserve native/process recovery holds and avoid
promising Retry while a decision or execution gate hides it.
- Refresh delivered artifacts and handle direct user replies that reopen
completed work without a clarification loop.
- Check the embedded PostgreSQL PID, data directory, and actual port
before connecting or migrating.
- Document accepted behavior and add focused regressions at lifecycle,
route, transcript, and UI boundaries.

## Verification

- Final head `fece606ac2` passes the complete GitHub CI matrix: **34
green checks, two expected Storybook skips, no failures or pending
checks**, including `ci / verify`, `ci / e2e`, full runner verification,
typecheck, build, every server/workspace shard, and all browser shards.
[CI
run](https://github.com/paperclipai/paperclip/actions/runs/34727183287).
Greptile is **5/5 with no open findings**. The final two commits only
refine test fixtures; both affected suites pass 24/24 locally and in CI,
with server typecheck green.
- Complete local Vitest coverage uses the canonical groups/shards: all
635 general server suites, all 145 serialized suites, and all workspace
packages. The aggregate began on `0a8001c18` while the final queue fix
arrived: 23,903 passed, five failed, 87 skipped. The five
port/socket/timing failures passed unchanged in follow-ups (60 tests in
the exposure/file suites and 412 tests covering the serialized failures
and unrun tails). The final queue/operator-identity suites separately
passed 52/52. This is aggregate coverage plus explicit reruns, not a
pristine single-command final-head run.
- After integration with current master,
queue/operator-identity/continuation suites passed 162/162 and affected
UI suites passed 140/140. ACP Stop/continuation and legacy
task/Inbox/message browser suites passed 9/9, including both task
recovery Retry and thread Try again, automatic saved-message delivery,
exactly one new run, Done, and retained output after reload. The default
process Stop/Pause/Resume browser case passed (the native-provider case
is opt-in and skipped by default). The complete Board attachment/receipt
browser suite passed 11/11 on a disposable instance, covering both
composers, exact receipts after lost responses, no replay, bound
attachments, and newer drafts after reload.
- Blocking-intent regressions cover pre-existing Blocked, a mismatched
run/cause, an explicit manual re-block, changed dependencies, a queued
comment after failure, and a block arriving between scheduling and
provider dispatch. The negative cases reproduced before the fix. All 478
affected executor/recovery/dispatch tests passed; both database suites
ran separately after availability-probe skips in the first combined
command. The final late-dependency check passed all 143 affected
recovery/dispatch tests (zero skips) after two new negative cases
reproduced the bug.
- Focused runtime regressions cover awaited runner ownership
publication, authenticated adoption before the first checkpoint,
old-finalizer detachment, idle and busy warm-session shutdown, rejected
checkpoint propagation, provider-pack verification, and managed-Codex
credential preservation. Four managed credential detachment cases
reproduced the bug before the fix; normal owned cleanup still succeeds
exactly once.
- Live local Claude: SIGKILL 2.6 seconds into startup recovered the same
run automatically in 53 seconds, then a normal follow-up completed in 24
seconds. SIGTERM 2.5 seconds into startup preserved the same run (54
seconds) and its queued follow-up (21 seconds). Answers remained visible
and the task reached Done.
- Live Claude Daytona: a warm follow-up retained its sandbox and fell
from 121 seconds to 44 seconds. A separate cold turn took 127 seconds;
after controller shutdown and checkpointing, its follow-up completed in
33 seconds with the same sandbox, workspace, native session, and runner.
Both answers remained visible and the task was Done.
- Other live journeys covered task completion and follow-up with local
and Daytona Codex, local Codex crash recovery, Stop then new direction,
clarification response, live artifact refresh, and shared-workspace
waiting.
- Validation limits: the opt-in native composer Stop/Pause→subtree
Resume fixture exposes terminal/result ordering and subtree-cancellation
attribution bugs that can leave a child task blocked; that new finding
is assigned to a separate follow-up and is not claimed fixed here.
Default CI skips this optional native-provider fixture. Managed-Codex
credential handoff and the queue-agent integration use automated
regression evidence. Cold custom provider-pack uploads still add startup
latency.

## Risks

- Automatic replacement remains deliberately narrow: local Codex,
verified stopped identities, unchanged retained state, and a complete
text/completion-only turn. Unknown actions, partial history, or changed
ownership remain blocked.
- Claude completion permission handling changes an upstream package
patch. The exact isolated task bridge must remain pinned; unrelated
tools keep their existing permissions.
- New task failure projection changes user-visible status. No historical
status backfill or database migration is included.
- This is a broad lifecycle fix across server and UI. Live proof covers
graceful local Claude restart during startup and idle Claude Daytona
session recovery across controller shutdown. Live abrupt SIGKILL during
local Claude startup also recovered the same run. Unknown ownership or
missing action evidence still blocks reuse. Cold custom provider-pack
uploads still add startup latency; this change avoids unnecessary repeat
uploads.

## Model Used

OpenAI GPT-6 (Codex), with reasoning, code execution, browser
automation, and tool use. The exact hosted model ID and context window
are not exposed in this task.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-12 19:41:15 -05:00
DottaandPaperclip 51b0e01ead fix: resume saved user messages after execution recovery (#13270)
Preserve verified native process-stop evidence and retry saved user messages through normal continuation admission after recovery cleanup. Show the current wait reason and serialize delivery so a saved message starts one fresh turn.

Validated with 410 focused tests, typecheck, build, token gates, all PR CI checks, and Greptile 5/5.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-11 17:12:55 -05:00