Commit Graph
8 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 f71aa135a0 fix(server): release sandbox leases for runs stopped before runner attach (#21788)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents in a `sandbox` environment run each task in a paid remote
sandbox that an environment lease tracks
> - When a run stops, the control plane must release that lease, and
terminating an agent waits until every lease of the agent is released
> - In a load test, a run stopped after its sandbox was ready but before
its runner started never released its lease, so the sandbox kept running
and the agent stayed in "terminating" forever
> - In the same test, terminating an agent whose failed run had a
scheduled same-run retry waited about 18 minutes, until the stale retry
time
> - This pull request makes lease release finish for every stop window,
records any release that must wait with a retry time, adds a hint-driven
safety net, and makes termination cancel a pending native retry at once
> - The benefit is that a stopped run never leaves a paid sandbox
running, and agent termination finishes in seconds without manual repair

## Linked Issues or Issue Description

**What happened?**
A load test with ten OpenCode agents in a `sandbox` environment stopped
its agents mid-run (pause, then terminate). Two defects appeared.
1. A run cancelled while it selected its provider pack (after the
sandbox was acquired and the workspace was prepared, before the runner
started) kept its environment lease `active` for more than 40 minutes.
The agent stayed in lifecycle `terminating`. The lifecycle worker
retried every pass and wrote `updatedAt` each time with no recorded
reason. Delete returned 409 "Complete termination before deleting the
agent".
2. A failed native run with a scheduled bootstrap retry kept its lease
until the stale retry time. Agent termination took about 18 minutes. The
run still showed "retry_scheduled" after the agent was terminated.

**Expected behavior**
A stopped run releases its sandbox lease within seconds. If release must
wait, the wait has a recorded reason and retry time, and a background
pass finishes it. Terminating an agent cancels a pending retry, releases
its lease, and reaches `terminated` promptly.

**Steps to reproduce**
1. Start a native run in a `sandbox` environment.
2. Stop it after the sandbox is acquired and the workspace is prepared,
before the runner starts.
3. Pause and then terminate the agent.
4. See that the lease stays `active` and the agent stays `terminating`.

**Paperclip version or commit**
Reproduced on `master` at 62033b9994 (load test build 2026.1106.0+2564).

**Relevant logs or output**
Root causes found in the code:
- `releaseEnvironmentLeasesForRun` returned with no record when
workspace retention refused a cancelled run. The refusal was permanent
when the run's native finalization owner had expired but was not
cleared. The owner is cleared only after `agentExecutionsHaveStopped`
sees every lease released, so the two waited on each other.
- `releaseRunLeases` silently skipped a lease whose prepared native
workspace was still "pending export", even for a run that never started
a runner and so never changed those files.
- The destroy path deferred on any busy warm session in the same
environment, also with no record.
- `stopInvocationsForAgents` returned on the first run that was still
live, so it did not release the leases of the other runs of the agent.
- The agent stop cancelled only `queued`, `running` and
`scheduled_retry` runs. A failed native run with a pending same-run
retry kept its lease until that retry fired.
- The lifecycle worker deferred a pending host step by 2 seconds every
pass, with no reason and no backoff.

## What Changed

- `releaseEnvironmentLeasesForRun` (`run-control.ts`):
- A native run with no runner launch evidence (no
`native.process_start_requested`, no runner event, no process id) skips
workspace retention and warm-session closing. No runner process exists
to fence, and no work ran against the prepared files.
- When retention is refused only because the run's native finalization
owner lease has expired, the release clears that expired owner
(compare-and-set on the exact expired owner of a cancelled run with no
result) and retries retention once. A recorded recovery then stops and
retains the sandbox, so the files stay recoverable.
- Every release that cannot finish now records
`metadata.releaseDeferral` (`reason`, `attempts`, `retryAt`,
disposition) on the active lease and publishes an `orphan-active-leases`
hint. The retry time backs off from 30 seconds to 15 minutes, then
retries hourly without end. The record keeps `failingSince` and the last
reason. A new Stop or terminate operation, or a server restart, triggers
an immediate attempt.
- `nativeWorkspaceExportPendingForLease` treats a cancelled run with no
runner launch evidence as owing no export.
- Safety net: `sweepOrphanedActiveLeases` (the existing hint- and
deadline-driven orphan-lease work) now retries the run's own release for
a due deferral or for a cancelled native run, after the native owner
lease has expired. The orphan-lease loader uses the stored `retryAt` as
the due time. No new timer or poller.
- Agent stop:
- It cancels a failed native run that has a pending same-run retry (or
an interrupted cancellation of one), through the existing
retry-cancellation path. A pause stamps `pauseStop` on those runs too,
so a budget resume can continue them.
- It releases the leases of every terminal run, and it skips a lease
whose deferral is not yet due.
- `agentInvocationStopBlocker` and `agentExecutionStopBlocker` return
why the stop must still wait.
- Lifecycle worker: a pending host or plugin step records
`pendingReason` and `pendingPasses` on the operation, and its retry
backs off from 2 seconds to a 60 second cap. When a release leaves a run
with no unsettled lease, the stopping agent's saved retry time is
cleared and the lifecycle queue is notified, so Pause or Terminate
finishes on the next pass.
- Cancelled-workspace preservation: a stopped run that never started a
runner has its prepared reference moved to
`nativeWorkspaceSyncReleasedUnchanged` when its lease is released, so
the preservation worker no longer selects it. The worker records the
same settlement itself for older runs, and waits on a live owner's exact
lease expiry.

## Verification

- Red on `master`, green on this branch (embedded PostgreSQL, fake
plugin sandbox provider):
-
`server/src/__tests__/heartbeat-lease-release-before-runner-attach.test.ts`:
agent pause then terminate, and board Stop, for each window between
sandbox acquire and runner attach (artifact discovery, pack select, pack
upload, runner launch, transport connect, session open). Each test
checks that the provider gets a stop or destroy call, the lease is
released, the run is cancelled, and the agent reaches `terminated`
within one extra worker pass. It also checks the orphan-lease safety net
for a leaked lease and a recorded deferral that the background pass
finishes when due.
-
`server/src/__tests__/heartbeat-terminate-pending-native-retry.test.ts`:
terminating an agent whose failed run has a bootstrap retry 17 minutes
away cancels the retry, releases the lease and reaches `terminated` at
once. The test covers an open task and a cancelled task.
-
`server/src/modules/agent-lifecycle/__tests__/pending-backoff.test.ts`:
a host step that stays pending records its reason, and its retry delay
grows to a 60 second cap.
- Related suites pass: run-control, recovery, orphaned active lease
sweep, environment runtime, agent lifecycle, native workspace
preservation and recovery, held comment wake.
- `server` typecheck passes.

## Risks

- A cancelled native run with no launch evidence now gets its prepared
sandbox destroyed instead of kept. The proof is the same launch evidence
that restart recovery uses.
- A launched cancelled run whose retention owes nothing is stopped and
retained, as before. A retention that throws (lock or write error, or a
live owner) now keeps the lease active with a recorded reason and retry
time, instead of a silent skip.
- Release now clears an expired native finalization owner of a cancelled
run. An old controller whose lease already expired then fails its next
owner check and stops, which is the normal lease rule.
- On the first start after deploy, the orphan-lease work releases leases
already leaked by cancelled native runs.
- Task state after pause then terminate: the run-scoped stop keeps the
task status (an open task stays open, a cancelled task stays cancelled),
and a run that held prepared files gets the existing workspace recovery
record on its stopped sandbox. No new state is added.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (code search, shell, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 12:54:23 -06:00
DottaandClaude Opus 5.5 0477f622d6 fix(recovery): keep the sandbox that holds abandoned unsaved files out of replacement and cleanup (#21304)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents run in remote sandboxes, and the workspace files live in
those sandboxes until a save
> - When a run cannot resume exactly, the fresh continuation fallback
(#21135) abandons the old workspace epoch and keeps its unsaved files
"recorded and unapplied" on the old sandbox
> - The abandonment clears the dirty flag, the writer and the location,
so the workspace looks clean. No code reads the abandonment receipt
> - A later failed resume can then replace the sandbox, and cleanup can
destroy it. That sandbox holds the only copy of the unsaved files
> - This pull request adds one shared hold. The task continues on a new
sandbox, and the old sandbox is parked until its files are saved,
discarded, or kept for 7 days
> - The benefit is that the fallback keeps its promise. Abandoned files
are not lost

## Linked Issues or Issue Description

Refs #21135
Refs #21297

**What happened?**
After the fresh continuation fallback abandoned a workspace epoch, the
receipt in `remoteWorkspaceAbandonedEpochs` was write-only.
`reusableSandboxReplacementIsSafe` saw a clean workspace with a
committed save and allowed replacement.
`retireReplacedReusableSandboxLease` then moved the old lease to
`pending_cleanup`, and the cleanup sweep destroyed the sandbox with the
unsaved files. `nativeWorkspaceExportPendingForLease` (the check that
the sweep, the retained orphan settlement and the scoped destroy paths
use) also answered "nothing pending".

**What did you expect?**
A sandbox that holds abandoned, unsaved files stays until the files are
saved, a person discards them, or a stated retention period ends.

**Steps to reproduce**
1. Abandon a remote workspace epoch whose location names sandbox L.
2. Let the next run fail to resume L.
3. The replacement decision says safe, L goes to `pending_cleanup`, and
the sweep destroys it.

## What Changed

- New `server/src/services/remote-workspace-abandoned-files-hold.ts`
holds the single predicate for "this lease holds abandoned files". It is
one SQL fragment (`abandonedWorkspaceFilesHeldSql`) plus a TypeScript
wrapper (`abandonedWorkspaceFilesHeldOnLease`). A lease is held while a
workspace in its company has an abandoned-epoch receipt whose
`location.providerLeaseId` equals the lease's provider lease id, and
that receipt has no `released` object. A receipt written by the current
release (no new fields) counts as unreleased.
- Replace and park. A failed resume of a held sandbox still lets the run
replace it, so the task continues at once on a new sandbox restored from
the last committed save. `retireReplacedReusableSandboxLease` parks the
old lease as `retained` with the `abandonedWorkspaceFilesParked` marker
(failure reason `abandoned_workspace_files_held`) instead of
`pending_cleanup`. `reusableLeaseCanBeResumed` and
`reusableLeaseCanBeCleanedUp` skip a parked lease, so later runs never
select it again. The check is a metadata read with no query, so the run
path does no extra work.
- Every path that can destroy a sandbox now honours the hold:
- `nativeWorkspaceExportPendingForLease` answers pending for a held
lease. This covers the pending cleanup sweep (defer), the orphaned
active lease sweep, the retained orphan settlement `heldBy`,
`destroyClaimedSandboxLease` (terminal workspace reaper
`destroyRemoteWorkspaceLease`, `destroyReusableSandboxLeases` and the
environment delete path, and run lease release with `destroy`), the
`destroyRunLease` dispatcher, and the plugin sandbox teardown retry.
- The run lease settlement (`environment_run_lease_active`) is the one
caller that ignores the hold (`{ destructive: false }`). It never
destroys a sandbox. It stops the sandbox and keeps it, or it moves a
lease without a sandbox to `pending_cleanup`, where the sweep parks a
held lease. If it waited for the hold, a dead run's lease would stay
`active` and block the retry for the whole retention period.
- `destroyReusableSandboxLease` parks the lease and returns without a
provider call. This covers obsolete reusable lease cleanup, the destroy
after an expired or mismatched resume, release with `expired`, and the
sandbox driver's `destroyRunLease`.
- The `retryPendingSandboxTeardown` dispatcher refuses a destructive
teardown of a held lease. Stop-only cleanup still runs, because a stop
keeps the filesystem.
- The retained orphan listing (`alwaysHeldSql`) uses the same SQL
fragment, so held orphans are not polled.
- Release. `releaseAbandonedWorkspaceFiles` marks a receipt `released`
with a reason (`recovered`, `discarded`, `retention_expired`) and an
actor. It moves the parked leases for that sandbox to `pending_cleanup`
when no other unreleased receipt holds them, and it writes a
`workspace.abandoned_files_released` activity entry on the task in plain
language. No export path for abandoned epochs exists today. The
`recovered` reason is the hook for one; this pull request adds no new
export mechanism.
- Retention expiry. New receipts carry `retainUntil` (7 days). An old
receipt uses `abandonedAt` plus 7 days. `nextEnvironmentCleanupAt`
includes the earliest retention end of any unreleased receipt. The
shared cleanup worker therefore wakes at that real deadline, and a
restart rebuilds it from the database. At the deadline the sweep runs
`expireAbandonedWorkspaceFileHolds`. This releases each due receipt as
an attributed system decision (actor `workspace-retention`) and logs the
held receipts and parked leases for each company. Each expiry always
writes a task entry: "Unsaved files from an interrupted run were kept
for N days, and their retention period has ended." The entry adds "They
are scheduled for removal." only when a parked lease was queued for
cleanup. A separate entry, "... kept for N days and have now been
removed.", is written only after the provider confirms the delete. Each
receipt is released in its own transaction. A failed receipt is logged
and skipped, and it is retried after a backoff (5 minutes, doubling, at
most 6 hours). The other receipts are still processed. The entries
reference a run or a task only if it still exists. If the task was
deleted, the decision goes to the server log only.
- Upgrade. A `pending_cleanup` lease (not in flight, not stop-only) that
an unreleased receipt holds goes back to `retained` and is parked by the
sweep. A sandbox that a release before this change already destroyed
cannot be recovered. Its receipt ends at its retention time like any
other receipt.
- Receipt trim (review P1). `abandonUnverifiedRemoteWorkspaceEpoch`
trims only released receipts and keeps the last 20 of them. It never
drops an unreleased receipt. A new abandonment of the same sandbox and
root replaces the unreleased receipt for that root. So a workspace holds
at most one unreleased receipt for each distinct sandbox and root
abandoned in the last 7 days.

## Verification

- `server/src/__tests__/abandoned-workspace-files-hold.test.ts` (19
tests). It follows the lease through the real acquisition path with a
failed resume, the real `abandonUnverifiedRemoteWorkspaceEpoch`, the
real `sweepPendingCleanupLeases`, and `settleRetainedOrphanSandbox`:
- `gives the next run a new sandbox at once and parks the held one,
which cleanup and later runs leave alone`
- `pulls a held lease already queued for cleanup back to retained, and
leaves it on the provider`
- `treats a receipt written before release fields existed as unreleased`
  - `records a retention expiry on the receipt`
- `hands the parked lease to normal cleanup once the files are
recovered` / `... discarded`
- `removes the files only after the retention period, and says removed
only after the provider confirms it`
  - `does not report removal when the provider delete fails`
- `records the end of retention without scheduling removal when another
receipt still holds the sandbox`
- `records the end of retention when it expires before the lease was
ever parked`
- `wakes the cleanup worker at the receipt's retention end, also after a
restart, and then goes quiet`
- `releases a receipt whose run was deleted and records it without the
run reference`
- `releases a receipt whose task was deleted without writing a task
entry`
- `keeps expiring later receipts when one fails, and retries the failed
one after a backoff`
- `keeps a held sandbox when the retained orphan settlement reviews it`
- `lists the sandbox for the retained orphan settlement again once the
receipt is released`
- `keeps holding the first sandbox after many later abandonments on
another sandbox`
- `replaces and cleans the sandbox as before when the receipt names a
different sandbox`
- `replaces and cleans the sandbox as before when no epoch was
abandoned`
- Red on the base revision (62d673cc3f): the park, upgrade, old-receipt,
expiry, many-abandonments and scheduling tests fail. The controls pass.
- Green: 19 of 19. Also
`environment-runtime-reusable-resume-replacement.test.ts` 17/17,
`environment-retained-orphan-sandbox.test.ts` 32/32,
`heartbeat-pending-cleanup-sweep.test.ts` 51/51,
`fresh-continuation-fallback.test.ts` 58/58.
- `tsc --noEmit -p server/tsconfig.json` and a tsc pass over the new
test file: clean. `environment-cleanup-work.test.ts` +
`heartbeat-pending-cleanup-sweep.test.ts`: 80/80.

- No existing test expectation was changed. Three CI failures on bb43f52
came from this change. Two came from the hold blocking the run lease
settlement: `ownership-reconciliation-report.test.ts > releases the
lease and frees the workspace for the retry once a preparation-only
epoch ends` and `remote-workspace-save-concurrency.test.ts > lets a
retry prepare the workspace again after its first writer was killed
during preparation`. The third, `native-session-executor.test.ts >
restores a warm backup into a created lease only when a settled restore
hold retained the stopped source`, came from a mocked database that has
no `execute`; the predicate now runs through `select`. All three pass
locally now. So do `heartbeat-orphaned-active-lease-sweep`,
`environment-run-lease-ownership`, `native-workspace-warm-reuse` and
`remote-workspaces.integration`.

## Risks

- Cost of a parked sandbox. Parking makes no provider call and changes
only the lease row. A sandbox that could not be resumed is not running.
A reusable sandbox that was released is already stopped and kept. In
both cases the cost is stopped storage for up to 7 days. One case is
different: if a destructive path finds a held warm sandbox that is still
running (`retained` after a keep-running release), the sandbox stays
running until the receipt is released. The expiry log line counts parked
leases for each company, so this cost can be seen.
- Run lease release for an `ephemeral` lease calls the provider's
release. Some providers delete the sandbox on that call. A held sandbox
normally has a reusable lease, so this pull request does not gate that
call.
- No schema change. The receipt is JSON in
`execution_workspaces.metadata`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edit) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:00:20 -06:00
7d1e9b3477 fix(recovery): retain exact warm native export sources (#17916)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native task recovery must save unfinished workspace exports before
continuation.
> - A prepared export can still refer to a sandbox with a warm reuse
policy.
> - The strict recovery inspector rejects that policy and exhausts
source verification.
> - Ordinary reuse must not give the same unfinished allocation to
another run.
> - An existing user Retry can authorize bounded file maintenance after
exact ownership checks.
> - This change retains that exact source and preserves the original
failure record.
> - Recovery completes only when files, stop evidence and policy
settlement agree.

## Linked Issues or Issue Description

**What happened?**

A failed native run has a prepared `keep_running` workspace export. Its
exact sandbox lease still has `reuse_by_environment` policy. Native
export cleanup can preserve that policy while it stops the sandbox.
Generic workspace recovery then rejects the source with
`lease_policy_not_retained`. A saved user Retry remains blocked before
file export.

**Expected behavior**

Keep unfinished exports out of ordinary warm reuse. With current user
authority and an unchanged saved Retry, retain the exact source for one
bounded maintenance generation. Preserve files and return continuation
to normal admission. Keep all ambiguous sources held.

**Steps to reproduce**

1. Prepare a native workspace export with a warm sandbox and
`keep_running` disposition.
2. Save a normal board Retry for the failed source run.
3. Persist its first exhausted source-verification failure before file
export.
4. Run the normal maintenance worker. Before this change, source policy
rejection leaves the saved files unrestored.

Refs #17773 for source diagnostics. Related #17839 records a missing
preservation obligation during blocked restart. This change handles an
existing prepared warm export whose verification already failed.

## What Changed

- Exclude unfinished physical allocations from reuse and obsolete-lease
cleanup across every historical lease row. Reuse a clean candidate or
acquire a fresh sandbox through normal selection. Preserve exact
same-run reattachment, workspace-less execution and ordinary task
admission checks.
- Preserve the newer prepared remote-workspace writer proof for native
and legacy runs. Share its binding check between single-source recovery
and one batched allocation read.
- Keep persisted recovery-action IDs when constructing continuation
source checks. Synthetic sources cannot grant snapshot receipt
authority.
- Permit one audited maintenance adoption for the first exhausted
source-verification attempt. Require an unadmitted canonical saved board
Retry, current user authority, unchanged assignment and latest source
run.
- Keep all company, source, allocation, workspace writer, controller and
physical finalization ownership checks. The public read-only inspector
keeps its strict policy check.
- Pin the exact source for file maintenance. Preserve the previous
generation and reuse the existing bounded recovery worker. Create no
provider turn or new user request.
- Restore the warm policy in the same transaction as verified native
copyback completion. A failed write or changed settlement keeps a
recovery obligation.
- When local merge metadata is missing, settle the exact verified
snapshot while keeping its unfinished allocation retained. Deliver the
saved Retry through normal fresh isolation and attach the verified
files. Keep the accepted result as history.
- Require native copyback proof before reopening an accepted result's
original workspace finalizer.
- Validate snapshot receipts through one shared boundary for warm,
retained and ephemeral sources. Bind the receipt to the current
generation's operation IDs, company, issue, run, recovery action, lease
and provider allocation.
- Retire fixture input before removing recovery proof, and use the same
configured heartbeat for HTTP and worker dispatch. Keep the exact
catalog refusal and no-provider-work assertions.
- Document the recovery contract in technical documentation. Leave
capability maps unchanged for this bug fix.

## Verification

- Candidate-selection RED: all six clean-candidate and fresh-allocation
controls fail on unchanged `f54e7baf538` because a pending export aborts
the whole acquisition. GREEN: the focused suite passes **15/15 cases**
after candidate filtering. Both provider paths preserve the protected
lease and never resume or destroy its allocation. Obsolete fingerprints
cannot send that allocation to cleanup. Historical-row fencing, modern
native/legacy writers, exact same-run reuse, workspace-less execution
and the single-read allocation-history check remain covered.
- The existing reusable-runtime selection cases passed **40/40**.
Driver/capability contract suites passed **5/5**. Reusable service
controls passed **2/2**. After merging current master, the combined
lease/provider suites pass 61 cases and the recovery integration
selection passes 36 cases, including the upstream inherited-session and
resolved-interruption assertions. This gives 97 passing targeted cases
on the final source. A duplicate test-app declaration from the automatic
merge was removed while retaining the upstream assertions.
- The source-history repair has meaningful red/green evidence: three
valid retired-handoff cases could not complete under the old blanket
history-row rejection. Settlement now uses the same full source
inspector as adoption. It accepts exact retired handoffs and still
rejects a predecessor that becomes active after export. The 26 focused
history, rollback, snapshot and ambiguity cases passed before this last
merge.
- Server typecheck passes on the final merged source. Module boundaries
and `git diff --check` pass. Full local repository tests and a
successful current-head full build are not claimed. Current-head CI is
complete on `0ef974d76483e11fb0b0304bcb05bbd3b56e2ef9`: both required
gates pass, with 54 successful checks, three expected skips, one neutral
Paperclip review check and a successful security status. [Full CI
evidence](https://github.com/paperclipai/paperclip/actions/runs/41904339370).
Greptile reviewed that exact head at **5/5** with no actionable
findings. Its check completed successfully. [Ponytail also reviewed the
exact head at
5/5](https://github.com/paperclipai/paperclip/pull/17916#issuecomment-6640083408).
No review threads remain unresolved, and the branch is mergeable.
- Earlier local broad runs exposed a dropped recovery-action ID in the
merged source projection and deferred test input being revisited after
its records were removed. Both were repaired. Later local disk
exhaustion caused manifest and saved-file reserve failures; generated
caches were removed only from these checkouts. The affected saved-file
control passed unchanged after cleanup. The original logs are retained.
No disk reserve or timeout was lowered.
- On the prior head, the first [full CI
attempt](https://github.com/paperclipai/paperclip/actions/runs/41899038530)
received simultaneous runner shutdowns. Four jobs were cancelled, and
two late jobs have lost-runner annotations with unavailable logs. The
Runner job had five earlier ACPX failures whose full assertion report
was lost on interruption. Their cause remains unproven. Identical Runner
source passed the [#19004
control](https://github.com/paperclipai/paperclip/actions/runs/41898305521/job/126814429725),
including all seven enabled tests in that file.
- That attempt's [browser
shard](https://github.com/paperclipai/paperclip/actions/runs/41899038530/job/126817404595)
returned to the exact saved task URL, but its conversation was still
loading at the unchanged heading deadline. The
[artifact](https://github.com/paperclipai/paperclip/actions/runs/41899038530/artifacts/13358840931)
is retained. All five Cloud-auth cases passed locally on the unchanged
head and in the [same-source CI
control](https://github.com/paperclipai/paperclip/actions/runs/41898305521/job/126814429966).
The delay remains unexplained. One rerun of failed/interrupted jobs was
submitted after the workflow completed. This new source revision
requires fresh CI.
- The earlier [original-head
workflow](https://github.com/paperclipai/paperclip/actions/runs/40552228175)
passed all 48 jobs, and prior head `f54e7baf538` received Greptile 5/5.
These results do not establish readiness for the new head. The earlier
Superagent scan was neutral after a pinned-base checkout failure; its
external analysis remains unverified.
- These tests use real PostgreSQL and controlled provider boundaries.
Hosted rollout and original-task acceptance remain unverified.


## Risks

- Maintenance can resume and stop the exact sandbox for file export. It
cannot admit a provider turn, choose another allocation or bypass normal
continuation gates.
- A saved Retry alone is insufficient. A changed source, newer run,
active writer, competing lease, lost user authority or previous adoption
keeps the hold.
- Policy restoration can fail. Completion and restoration now commit
together, so normal recovery remains responsible for a retry.
- Snapshot preservation cannot repair missing local merge metadata. Its
old allocation remains explicitly retained. A snapshot receipt grants no
permission to reuse or destroy that allocation, or to replay the
accepted result.
- The allocation check uses one projected join per candidate over the
existing provider lease index. It adds no polling timer or recurring
scan. A fresh fallback can create another sandbox while the unfinished
source remains retained; it grants no authority to execute a provider
turn.
- This change does not establish the cause of a prior provider failure
or prove that any specific hosted sandbox still contains its original
files.

## Model Used

OpenAI Codex, GPT-6 family, for the original change. OpenAI GPT-6.1 Sol
(`gpt-6.1-sol`), xhigh reasoning, for the merge and integration repairs.
Both used repository tools and PostgreSQL verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Devin Foley <devin@paperclip.ing>
2026-11-04 14:47:42 -08:00
DottaandPaperclip 669538534c fix(recovery): preserve failed remote reattach workspaces (#18936)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native runs can edit files in a remote execution workspace.
> - A failed reattach can leave an observed coordinator and unpublished
files.
> - The failure handler recognizes the older workspace-sync descriptor
but misses the modern workspace reference.
> - The released task lock then prevents ordinary session recovery from
claiming that old run.
> - This pull request retains the files through the existing Stop,
cleanup and workspace recovery owners.
> - Exact ownership and Stop evidence permit file preservation while
unknown provider effects remain under manual reconciliation.

## Linked Issues or Issue Description

Related: #18877 protects native manual admission holds. #18879 handles
scheduled native retry lock reclaim. #18918 fixes remote workspace
row-lock order. Those changes do not record the missing modern
failure-retention obligation.

**What happened?**

A native reattach failure can leave a failed run, a dirty modern remote
workspace, an expired observed coordinator and a free task execution
lock. The run has no accepted result. The failure handler skips file
retention because it checks only the older descriptor. Recovery cannot
export while the old coordinator remains owned. A mismatched workspace
generation can also make orphan cleanup miss the existing file claim.

**Expected behavior**

Retain the exact original allocation and files. Stop the allocation
through the existing lifecycle owner before retiring the stale
coordinator. Keep checkpoints and unknown external effects available for
manual reconciliation. Unsafe or changed source authority must prevent
physical cleanup.

**Steps to reproduce**

1. Prepare a modern remote workspace for a native run.
2. Record a provider checkpoint and an active interruption incident.
3. Fail reattach preparation and release the task execution lock.
4. Let the coordinator lease expire.
5. Run the existing orphan, environment cleanup and workspace recovery
workers.
6. Verify the original files from a cold archive read and verify that no
new native session starts.

## What Changed

- Include modern prepared references in the existing heartbeat
failure-retention producer.
- Add a file-only historical intake for an exact failed observed run in
the normal expired reattach state.
- Require an untouched active ephemeral lease, a free blocked task,
matching workspace writer and generation, no accepted result and
complete single-owner allocation history.
- Persist the existing StopAndRetain intent before provider I/O. Use
nonblocking row locks and classify busy ownership only after rollback.
- Use the existing native environment settlement owner after exact Stop
and retention receipts arrive. Keep its single scoped discovery query.
- Preserve source and coordinator fingerprints across the handoff. Keep
original checkpoints, errors and manual unknown-effect incidents.
- Keep stale or malformed modern workspace claims held when preservation
intake refuses them.
- Add actual PostgreSQL, heartbeat, allocator, worker restart, archive
and safety controls. Document the narrow recovery contract.

## Verification

Current published head `b1f49691187a398415cf9b16b723e3ab473f0f91`: all
required CI checks pass (54 success, 3 skipped, 1 neutral); Greptile is
5/5 with no open threads. The separate Paperclip assessment is queued
and its current response says work is paused. That assessment is
unavailable; no additional review nudge or hosted retry was issued.

All local commands use Node 24.22.2 on macOS arm64.

- Exact published head: `b1f49691187a398415cf9b16b723e3ab473f0f91`.
Required CI passes (54 success, 3 skipped, 1 neutral); Greptile is 5/5
with zero open threads. The optional Paperclip assessment remains
queued/paused without a completed score.
- Final recursive `pnpm -r typecheck`: passed.
- Final `pnpm build`: passed.
- Final three native producer, containment and scheduling files: **67/67
passed**. These include the real coordinator and worker restart, the
production heartbeat recovery callback, exact Stop receipts, cold
archive restore, and zero native backend, new allocation or destructive
release calls.
- The plugin allocator cases use the actual selector with `reuseLease:
true`. Active and released ephemeral leases receive a fresh allocation.
- The assembled reattach producer checks that the task and coordinator
claim commit before physical resume. A historical pin refuses that
current owner. A later claim behind a committed pin remains held.
- The safety controls cover reusable allocations, manual and unknown
states, cleanup in flight, accepted but unbound results, late allocation
owners, wrong or malformed workspace authority, changed
generation/checkpoint, lost Stop acknowledgement and a real
cleanup-producer lock conflict.
- Untouched base `e7549ab6131555f655736e6b544892a3b1e323cb`: four
expected behavioral failures and one passing legacy-descriptor control.
The failures cover the modern producer, original provider history,
historical retention and wrong-epoch cleanup. Fixed production files
were restored byte-for-byte after this comparison.
- The first affected six-file run completed **562 passed / 1 failed**.
The failure was an existing HTTP case with `socket hang up`. That exact
case passes separately on both untouched base and this branch. The
original connection failure remains unexplained. At `0e92de8c`, the
complete workspace family passes **188/188** with unchanged assertions.
The environment cleanup owner family also passes all 23 cases. The later
guard changes are qualified by the complete 67-case native owner group.
A full repository test run was not repeated locally; the requested
full-ci workflow runs the complete repository suites.
- Published `0e92de8c` fails both new explicit-null Stop/export marker
controls because it creates a new retention pin. The follow-up uses the
shared presence-aware Stop guard. Both controls now preserve the marker
and record no new retention pin, Stop or destructive release.
- Paperclip's review found that malformed workspace authority could
throw and abort an orphan page. Published `5f6cd4f5` fails all three new
controls: intake, the actual page and after-Stop settlement. The same
strict schema now uses a safe parse, and the existing OBSERVED hold runs
before the prepared-source lookup. The malformed source stays held while
the next valid source receives its retention pin.
- Two existing scheduling tests caught the draft's second global
discovery query. The final single-query repair passes all 36 unchanged
scheduling tests.
- An earlier restarted archive assertion stopped after 20 seconds. Its
recorded Stop was valid, and the coordinator had settled. The existing
evidence recheck deadline was still 13 seconds ahead. The final test
observes the unchanged 24–36 second policy within its existing 60 second
case limit.

The external fake provider Stop and file transport are controlled test
boundaries. Database ownership, lifecycle workers, archive bytes,
encrypted local storage and cold restore are real. This does not claim a
live cloud sandbox, S3 or model run. No hosted task was changed for this
PR.

## Risks

- Historical intake is deliberately narrow. Reusable, already released,
mismatched or competing allocations remain held.
- File preservation does not settle unknown provider effects or
authorize a repeated provider turn. The task can still require manual
reconciliation.
- A busy or changed owner delays preservation. The original files and
claims remain retained.
- This does not repair the underlying sandbox readiness failure or
establish the cause of a separate archive-capture failure.
- Deployment and recovery of original hosted tasks remain unverified.

## Model Used

OpenAI Codex, GPT-6 family, with repository inspection, code execution
and GitHub tools. The runtime does not expose the exact serving model ID
or context window size. A separate Codex agent and the coordinator
performed read-only source review.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 07:19:03 -06:00
DottaandPaperclip 367f29478d fix(recovery): preserve the bound remote writer before continuation (#18803)
## Thinking Path

> - Paperclip manages agent work and preserves task files.
> - A remote workspace records the exact run that owns its unsaved
files.
> - A later setup failure can become the latest task run without owning
those files.
> - Retry only inspected that latest run, so the original writer had no
file-recovery owner.
> - A released reusable source also failed the legacy retention rule.
> - This change registers the bound writer, saves its files, and then
admits the saved user request once.
> - Existing stop, ownership, approval and execution-evidence checks
remain effective.

## Linked Issues or Issue Description

**What happened?**

A native run prepared a remote workspace and failed before finalization.
Its sandbox stopped with a dedicated stop receipt. The workspace stayed
dirty and retained that run as its writer. A later setup failure became
the latest task run. Retry inspected the newer run and left the original
file obligation unregistered. New attempts failed at the writer fence.

Registration alone was insufficient. After file recovery, continuation
still required the Retry source to be one of the older recovery sources.
A newer pre-provider failure did not satisfy that check. Workspace
selection could also choose a fresh directory instead of the recovered
files.

**Expected behavior**

Find the writer through the task's current workspace binding. Register
only its exact prepared source. Preserve the released lease and its stop
receipts. Save the files before continuation. Independently prove that a
newer setup failure never dispatched work. Select the verified saved
workspace and admit the Retry once.

**Steps to reproduce**

1. Fail a native run with a prepared remote workspace and a released
reusable sandbox.
2. Keep matching dedicated stop receipts and the workspace's dirty
writer authority.
3. Record a newer pre-provider setup failure for the same task.
4. Request Retry. Before the fix, no file-maintenance record owns the
actual writer.

**Paperclip version or commit**

Reproduced before production edits on master `f7ad1dc7ca`. The branch
was rebased on master `adb49a44346849e751548b2fb640693dbfad4b6f`.

**Deployment mode**

Authenticated hosted execution with Paperclip Runner and remote
workspace storage.

Companion: #18798 (now merged upstream) fixes a pre-existing inbox test
synchronization race. Its test-only commit
`110aa88489cad926a451c972ad731ede469347c1` is cherry-picked here with
attribution so this PR can run the complete CI matrix.

Related: #17800 registers missing preservation for the selected source
run. #18678 holds unfinished remote exports. #18718 repairs a prior
warm-controller projection failure. #17916 addresses a separate legacy
warm-copyback policy case.

## What Changed

- Discover the failed writer through the company-scoped task/workspace
binding during Retry and authenticated message admission.
- Reuse the exact prepared-source validator for released remote
workspaces. Registration changes neither the lease policy nor its stop
receipts. Physical recovery retains its full allocation checks.
- Check a newer failed preparation separately before allowing an older,
repaired source to release continuation. Provider events, leases,
results and execution ownership remain disqualifying evidence.
- Select the bound saved workspace through the continuation's verified
repair receipt.
- Add PostgreSQL safety cases and an HTTP Retry workflow with real file
capture and saved-copy reads. Document the execution contract.
- Reuse #18798's completion barrier for the existing inbox cancellation
test; preserve its eight-slot limit, rejection and coalescing
assertions.

## Verification

- RED before production edits: the exact stopped writer has no
`workspaceRestoreRecovery` after Retry. Six unsafe-source controls pass
on the baseline.
- GREEN: the expanded Retry/message/preview selection passes 17 cases.
All 627 tests in the five affected recovery suites pass on recovery
commit `b572304f772d236b8a8bdf095e13e651d808f41c`, including Stop,
ownership, saved messages and duplicate Retry.
- The HTTP workflow registers maintenance, keeps execution held before
save, reads back the saved file bytes, and admits exactly one
continuation with `resume_preserved_workspace`. A registration-only
repair first failed after save because it did not reconcile the newer
Retry source. The incident-shaped pair passes with an independent active
stranded-task action present.
- Greptile identified the path where the newer setup failure already
owns a reconciliation hold. Its new regression failed before the
correction (saved files, zero queued continuations); the
contradictory-dispatch control passed. All four HTTP variants pass after
the correction, covering both hold shapes and both dispatch-evidence
controls.
- Repository typecheck and build passed. Server typecheck and build also
passed after the recovery changes. Module-boundary and whitespace checks
pass.
- The full local Vitest command failed in the GitHub fast-response
acknowledgement test (`delivery_unknown`: one response write instead of
zero). That isolated case passes on this branch and unchanged master;
the original full-run cause remains unestablished. A sibling run passed
199 test cases but timed out in workspace-suite database cleanup.
Unchanged master also passed all 126 workspace cases and timed out in
the same cleanup hook; all five acknowledgement outcomes passed in that
baseline comparison. These local failures remain recorded rather than
being reported as suite passes.
- The first hosted matrix failed the pre-existing inbox cancellation
assertion (expected nine computes, saw eight). The same assertion failed
in [master
CI](https://github.com/paperclipai/paperclip/actions/runs/41664339150/job/126015526840)
and reproduced locally before applying #18798's existing repair. All six
inbox HTTP/database cases pass after that repair. Final head
`7f2fafcbace74cb1ad8ea68c8375a69b42ed30d9` adds only that attributed
test change to the verified recovery code.
- [Final hosted
CI](https://github.com/paperclipai/paperclip/actions/runs/41677310377)
passes on head `7f2fafcbace74cb1ad8ea68c8375a69b42ed30d9`, including `ci
/ verify`, `ci / e2e`, typecheck, build, every server/chat/workspace
group, Runner checks and all eight browser shards. There are 54
successful check results, two informational neutral results and three
skipped results, with no pending or failing results.
- Greptile and the Paperclip reviewer both report 5/5 for that final
head; no review threads remain open. The Paperclip review check is
informational (`neutral`) and records its local-testing limitations.
- The file workflow uses real PostgreSQL, subprocess file capture,
encrypted saved objects and production recovery code. Provider
resume/stop calls are simulated at the environment driver boundary. This
does not prove a live Daytona or Codex recovery. No deployment or
original-task retry is claimed.

## Risks

- File-maintenance discovery must not authorize another executor.
Existing writer, coordinator and physical allocation checks still own
that decision.
- A false bootstrap summary cannot prove that a newer run never
executed. The new path also requires its immutable native-adapter claim,
unresolved preparation state, absent
coordinator/process/session/lease/result evidence, and no dispatch
events.
- Additional database reads run only during explicit task admission or
Retry strategy selection. They are company/task scoped and add no
polling loop. The saved-writer query returns one matching snapshot and
omits prompt bodies.
- No schema, credential, provider profile or deployment setting changes.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection, code
execution and test tools. The runtime does not expose the exact serving
model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 11:25:24 -06:00
DottaandPaperclip 5e4cc01073 feat(workspaces): keep remote files off the webserver disk (#18403)
Keep native sandbox workspace files independent of the webserver disk. Use encrypted saved versions and Postgres ownership so a replacement server can recover files and continue work. Enable the path by default for new eligible workspaces when durable storage is configured.

Preserve explicit opt-outs and existing host workspaces. Include safe snapshot repair, Files access, import, migration and retention. Bound saved-Files resources, preserve imported Git recovery, reject unfinished Git operations, publish immutable concurrent captures, and verify permanent object-version deletion.

Validated 471 targeted local controls, build/typechecks, all 40 CI partitions, fresh Greptile 5/5 and security review. The PR records original failed attempts and scoped retries.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 14:54:01 -05:00
DottaandPaperclip f5534717b0 fix(runtime): preserve unexported native work during lease cleanup (#16583)
Preserve the exact native workspace source until its export is finalized. Apply the same guard to cleanup sweeps and direct environment teardown. Keep stop-only recovery and healthy warm retention available.

Verified with 317 targeted tests, server typecheck, module boundary check, all current-head CI gates, and Greptile 5/5.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-16 12:56:09 -05:00
Devin FoleyandPaperclip 7a58591721 fix(recovery): preserve accepted workspace finalization during reaping (#15718)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run saves its accepted result before it copies workspace
changes back.
> - A restart can leave that copyback under an unverified physical
owner.
> - The generic orphan reaper can mistake the exited provider for lost
work and release the source lease.
> - Recovery must preserve the accepted result and let the existing
finalizer settle its remaining work.
> - This pull request checks that authority at the conditional
process-loss write and preserves the matching recovery hold.

## Linked Issues or Issue Description

Related: #15475 added bounded process-loss evidence. The earlier #9687
addresses dead-run finalize barriers with synthetic receipts. This
change preserves an existing accepted native result and requires real
workspace settlement.

**What happened?**

After a server restart, the native finalizer required operator
verification of unfinished copyback. The generic orphan reaper then
marked the run `process_lost` and released its source lease. A new
execution path also made the recovery reconciler clear the copyback hold
before physical ownership was verified.

**Expected behavior**

An accepted result bound to the same company, issue, and run keeps its
unfinished workspace suffix under native finalization. The generic
reaper must not discard its source. The existing finalizer must still
complete the work, apply its bounded failure policy, or require operator
verification.

**Steps to reproduce**

1. Accept a native result and leave its coordinator in
`workspace_finalizing`.
2. Stop the controller before copyback completes, leaving its physical
owner receipt.
3. Start recovery, then run the generic orphan and lease cleanup sweeps.
4. Before this change, the run can fail with `process_lost` despite its
accepted result and explicit copyback hold.

**Paperclip version or commit**

Reproduced against `011e5669`; the same recovery code remains at base
`1602512820`.

**Deployment mode**

A hosted server with PostgreSQL and native runner execution. The tests
use only isolated local PostgreSQL and synthetic fixtures.

## What Changed

- Check the exact accepted result and `workspace_finalizing` coordinator
before generic process-loss cleanup.
- Repeat the check in the conditional status write. Compare the
heartbeat tuple version to reject a result publisher that commits while
the update waits on its row lock.
- Preserve nonterminal copyback holds when another execution path
appears. Resolve only the matching run's hold after verified
finalization commits.
- Add fourteen real PostgreSQL service regressions and document the
recovery boundary.

## Verification

- `pnpm -r typecheck` passed.
- `pnpm exec vitest run
server/src/__tests__/heartbeat-process-recovery.test.ts -t 'native
finalization'`: 14 passed. The test filter excluded 383 existing tests.
- Tests cover startup, repeated reaper and cleanup sweeps, malformed and
foreign owners, an expired old controller, exact database bindings,
legacy and unsupported states, verified settlement, and unchanged
terminal-task dispositions.
- The concurrent publisher test uses the real accepted-result
transaction and proves the reaper update is blocked by its row lock
before the publisher commits.
- The local `pnpm test:run` aggregate was stopped after setup timeouts
in the chat and tool-access files and a damaged-epoch Runner test
failure. It is not a full local pass.
- The chat file passed all 1,145 tests when run alone. The tool-access
file then passed 419 of 420; one case returned `socket hang up`. That
single case passed on its isolated rerun. The cause of the socket
failure is not established here.
- The damaged-epoch Runner case still fails locally at its existing
completion assertion. The same assertion failed on untouched baseline
`556daf9a` before this patch; its test and Runner sources are unchanged.
Exact-head CI passes this coverage. No workaround for that local fixture
is included.
- `pnpm build` passed.
- Independent review approved this exact commit. All 14 new PostgreSQL
regressions and 181 existing recovery/finalizer cases passed. One
existing suite needed an isolated rerun after a setup timeout.
- Independent negative controls reproduced the original bug. Removing
only the tuple-version guard reproduced the blocked-row race. Removing
only the hold exclusion cleared the safety hold too early. Restoring
this patch passed all three checks.
- GitHub CI passed all 53 applicable checks. The two Storybook jobs were
skipped as expected. Greptile scored this commit 5/5 with no unresolved
review threads.
- Local runtime: Node 25.6.1. GitHub CI supplies the repository's Node
24 checks.
- `git diff --check`, Gitleaks, and a local diff scan for secrets and
private data passed.

## Risks

- PostgreSQL tuple versions are used only as an optimistic fence for
`process_lost`. Both accepted-result publishers lock and update that
heartbeat in their result transaction.
- The guard requires the bound accepted result and the durable
coordinator state. A heartbeat phase or arbitrary owner JSON alone
cannot suppress ordinary process loss.
- Existing finalizer retries, ownership verification, explicit
cancellation, and terminal-task dispositions remain unchanged. A
physical owner can still require operator verification.
- This change does not replay provider work or restore a source already
destroyed by an older controller. No migration or runtime operation is
included.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, code
execution, and independent agent review. The exact service model ID and
context-window size are not exposed in this worker session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-09 12:28:16 -07:00