Commit Graph
11 Commits
Author SHA1 Message Date
28e38da7e6 fix(server): wake native result commits from durable work (#17748)
## Thinking Path

> - Paperclip manages AI agents and preserves their results through
failures.
> - Native completion must save workspace output before it commits the
accepted result.
> - The orphan recovery pass repeatedly revisits this work even when the
subsystem is idle.
> - Accepted results, workspace receipts, retry dates and leases already
provide durable truth.
> - This PR wakes a scoped worker from those writes and schedules only
real unfinished work.
> - It preserves existing finalization authority and keeps later
projection and resource recovery separate.

## Linked Issues or Issue Description

Refs #17734 and #17717. This is slice 53 of the existing sleep stack,
targeting master after parent #17734 merged at
`a019d00a393b57f531afe8dd8ab0b8a1820d1c4c`.
A current search found no duplicate accepted-result commit PR.

**What problem does this solve?**
Native accepted-result recovery currently depends on periodic orphan
scanning. Quiet instances keep doing work to discover that no result
needs a commit.

**What improvement do you propose?**
Use the shared delivery coordinator with transaction-bound producer
signals. Reconstruct unfinished results at startup. Preserve saved retry
and lease deadlines. Remove only uncommitted-result execution from the
periodic pass.

**Why is this useful?**
The accepted-result worker becomes quiet with no work or retained
completed history. Real results and completed workspace operations
reactivate it without a polling delay. This does not claim that the
whole server or database can sleep yet.

## What Changed

- Track the initial result-commit sweep through shutdown without
blocking HTTP readiness on recovery locks.
- Keep ordinary company-only finalization hints from reconstructing
dormant result work. Existing company settings and dedicated owner
signals remain supported.

- Add a native result-commit worker. Bind candidates by company, issue,
run, accepted result and a successful workspace receipt. Exclude
committed, terminal, held and pending-effect work.
- Signal both accepted-result writers and workspace receipt writes
before their transaction changes. Extend the existing finalization
fanout for retry, lease, effect and completion transitions.
- Add an optional atomic due-date check to the existing finalizer claim.
A lease collision or later retry does not consume a failed attempt.
- Signal native retry cancellation so it removes the saved wake after
commit.
- Register through the existing heartbeat suppression and shared
delivery lifecycle. Preserve outer commit/rollback, drain/resume,
settings activation and shutdown behavior.
- Delegate only uncommitted-result execution from the periodic orphan
pass. Keep committed projection and resource settlement recovery
enabled.
- Add real producer/lifecycle tests and a detailed audit. No new tables,
provider branches or per-feature timers.

## Verification

- Current local source `2af88fa55f961a74340880240019f6cd31e21bf1`
incorporates the parent with a normal merge. All 1,203 parent-only, nine
owned-only and 12 automatic shared paths are preserved. The only manual
conflict keeps the new result fanout inside the existing ordinary
issue/run scope guard. The complete package tree matches parent profile
80 byte-for-byte.
- Controlled real-database regression: the original result-only unscoped
fanout produces three candidate SELECTs for routine company hints. The
same zero-query assertion passes after restoring the scoped fanout. A
real workspace receipt and accepted result then commit exactly one
decision and return to quiet. Red and green restore production bytes
exactly. Twenty cases outside the focused pattern are not counted as
passing there.
- The complete main selection passes **1,746 cases in 16 files**,
without skips, in 277.97 seconds. This includes all 21 result-worker
lifecycle cases and 82 startup cases, including held result recovery,
HTTP readiness and shutdown joining. Separate process recovery passes
410 cases in 162.42 seconds and external-chat response passes 273 in
53.41 seconds. Total: **2,429 distinct passes in 18 complete files**,
with no skips. Normal merge `998239b019defe52e7b5d64f884de4e871ccd736`
adopts the actual parent squash and preserves this complete tested child
tree.
- Production server typecheck, build, module boundaries, migration order
and source identity checks pass. Fifteen dedicated PostgreSQL 17
databases use the unchanged migration 413 schema. Embedded-only calls
use the existing fresh-per-call recovery helper. All temporary imports
restore byte-exact, and every selected test hash matches after
completion. No assertion or timeout is relaxed.
- Runner validation belongs to the unchanged parent package tree: 130
passing installation/compatibility/transport cases, 22 installed
ACPX/Grok contract cases and 26 native-provider Rust cases. One existing
opt-in closed Pi snapshot case is skipped because its distribution root
is not configured. This does not claim live provider qualification.
- Earlier Greptile 4/5 on `656f6722be1aedf3d52b7f853956e15838204278`
correctly identified startup blocking on result recovery. The prior
branch `108cf0dc8296bf451b721ddac55ecbcd035d00f0` already tracks its
ready promise through the existing lifecycle owner. Current full startup
acceptance verifies the fix in the merged context. The later Greptile
service error is not a pass; current-head acceptance now passes: all 57
checks are clean, including protected CI and security. Greptile gives
5/5 on `b2efdb2dbbbef5b2227ffa2bd2a9a6cfe3867fe2`, with no actionable
findings. No Ponytail verdict is present under the authorized
missing-review policy.
- The owner audit at
`doc/plans/2026-10-26-native-result-commit-sleep-audit.md` retains
earlier failures and their disposition. These checks do not claim a full
local repository suite, deployed app/database suspension, expanded Cloud
wake coverage or measured cost savings.

## Risks

- Process-local signals cover instrumented application writes in the
owning server process. They do not cover independent SQL writers or
another replica without supported ingress. Durable startup
reconstruction remains enabled.
- The initial native decision and its run projection are separate
transactions. The periodic reconciler still repairs already-committed
results and releases resources. This PR must not be used to infer that
committed means fully settled.
- Candidate discovery adds bounded, narrow reads at startup and after
signals. No large-fleet query benchmark is claimed.
- Existing Cloud alarm coverage, host sleep safety and resource blockers
are unchanged. No deployed sleep or savings claim is made.
- Parent #17734 has merged. This PR targets master at
`a019d00a393b57f531afe8dd8ab0b8a1820d1c4c`. Fresh current-head CI and
review remain required before merge.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The exact serving model ID and context-window size are not exposed in
this session. No subagent was used for this slice.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 04:53:11 -05:00
a019d00a39 fix(server): wake native workspace finalization from committed work (#17734)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs can accept a result before workspace copy-back finishes.
> - The periodic orphan pass repeatedly tries this recovery, including
cases that require operator action.
> - Copy-back needs exact ownership evidence. A generic timer cannot
grant that authority.
> - This pull request wakes workspace recovery from committed changes
and local owner completion.
> - Saved retries use the existing scheduler. Empty work, completed
barriers, and operator holds become quiet.

## Linked Issues or Issue Description

Refs #17717. This is slice 52 of the recurring-server sleep stack,
targeting master after parent #17717 merged at
`4c56702e06c698cc10d4a86514669aba0f4d9a9b`. Related work: #17709 (status
reassessment), #17687 (unsafe export recovery), and #16478 (workspace
repair holds).

This PR covers native workspace copy-back and validation. Result
commit/projection and resource settlement remain separate recovery
duties. It does not claim full-instance sleep.

## What Changed

- Register a company/run/task-scoped workspace worker. Reconstruct
accepted, unfinished work from existing records. Use saved retry and
lease deadlines.
- Notify from both accepted-result writers (control-plane and PRP
coordinator), workspace receipts, coordinator changes, operator
confirmation, retained workspace repair, and relevant task updates.
Notifications follow the outer transaction.
- Distinguish active local ownership, busy external ownership, pending
Cloud evidence, operator-required evidence, and changed bindings. Only
concrete external ownership/evidence waits receive a scoped retry.
- Preserve advisory locks, signed stop proof, exact binding checks,
physical callback joins, and existing failure policy. Join physical
operations during shutdown.
- Respect the existing heartbeat execution-suppression policy before
discovery and execution; wake suppressed work on settings activation.
- Skip physical workspace recovery in the enabled periodic orphan pass.
Keep direct/startup reconciliation and the remaining result/cleanup
recovery paths.
- Keep routine company-only finalization hints quiet. Ordinary work
requires issue/run scope; explicit settings and owner-release events
keep their company-wide contract.
- Preserve the parent's entire Runner/package tree, profile 80 and all
historical identities. Add the dated obligation, producer, lifecycle and
limitation audit.

## Verification

- Current local source: `4059b24e3ea5c623be54ade18ec9b6ab78eb0289`,
prepared by a normal merge of the current parent. The sole manual
conflict keeps ordinary workspace fanout inside the existing issue/run
guard. All 1,198 parent-only, 15 owned-only and 10 automatic shared
paths are preserved. The complete package tree matches the parent
byte-for-byte.
- Controlled regression: the original workspace-only unscoped fanout
performs three candidate SELECTs for routine company hints. The same
zero-query assertion passes with scoped fanout. A real accepted result
then produces one successful workspace settlement and returns to quiet.
The failed baseline remains in the audit.
- The first complete local group passes **1,693 cases in 14 files**,
including all 19 workspace-worker cases. It covers both accepted-result
producers, physical ownership, export retry/failure, finalization
recovery, arbiter corpus, issue recovery actions, control-plane port,
workspace recovery, heartbeat recovery, startup, retained cleanup and
native execution.
- Separate process-recovery and external-chat response files qualify the
changed recovery/settlement paths without sharing their general test
database with other files. Both pass in full: 410 process-recovery cases
in 162.79 seconds and 273 external-chat cases in 53.79 seconds. The
total is **2,376 distinct passes in 16 complete files**, with no skips.
Normal merge `5c46c52873158529736b8349e4d44658865108ee` adopts the
parent squash with this entire tested child tree unchanged.
- Production server typechecking, build, module boundaries, migration
order and source-profile checks pass. Thirteen dedicated PostgreSQL 17
fixtures use the unchanged migration 413 schema. Embedded-only paths use
the existing recovery helper with fresh per-call fixtures. All temporary
imports are restored byte-exact. No test assertion or timeout is
relaxed.
- Runner validation belongs to the unchanged parent package tree: 130
passing installation/compatibility/transport cases, 22 installed
ACPX/Grok contract cases and 26 native-provider Rust cases. One existing
opt-in closed Pi snapshot case is skipped because its distribution root
is not configured. These are source-scoped checks, with no live provider
qualification claim.
- The owner audit at
`doc/plans/2026-10-26-native-workspace-finalization-sleep-audit.md`
preserves earlier failures and their disposition. Current-head
acceptance: all 57 checks are clean, including protected CI and
security. Greptile gives 5/5 on
`7df5c9c44ea9e171be3c61b4430d3727b10f3337` with no actionable findings.
Earlier scores do not substitute for this result. No Ponytail verdict is
present under the authorized missing-review policy.
- No claim of a full local repository suite, deployed app/database
sleep, expanded Cloud wake coverage or measured cost savings is made.

## Risks

- Supported wake producers are in-process application services. This
adds no cross-replica or independent SQL-writer transport.
- A real unresolved export still needs recovery. Busy owners and pending
Cloud stop evidence retain a 30-second scoped check. No timeout
authorizes copy-back.
- Result projection and resource release still use their existing
recovery. Some existing cleanup failures remain best effort; this PR
does not infer cleanup success from a workspace barrier.
- Cloud alarm coverage and host-sleep safety remain unchanged. No
deployed suspension or cost savings have been measured.
- Parent #17717 is merged. Current base is master at
`4c56702e06c698cc10d4a86514669aba0f4d9a9b`. Fresh current-head CI and
review must pass before this slice merges.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The exact serving model ID and context-window size are not exposed in
this session. No subagent was used for this slice.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 04:40:01 -05:00
df6156c2e3 fix(server): wake native status reassessment from committed changes (#17709)
## Thinking Path

> - Paperclip manages agents, tasks and their durable native results.
> - Changed evidence, task authority or review retirement can require a
new status assessment.
> - Periodic orphan recovery repeatedly reads history to find these
changes.
> - This change gives reassessment an owner driven by committed changes
and saved deadlines.
> - Existing result, assessment, decision and task rows remain durable
truth.
> - The status committer still verifies current sources and authority
before writing.
> - Settled or insufficient evidence becomes quiet until a supported
producer changes it.

## Linked Issues or Issue Description

Refs #17702 (merged), #17679 and #17680. This is slice 50, now based on
master. It owns native status reassessment; workspace settlement,
retained sessions and other orphan duties remain separate.

**What happened?**

Periodic recovery repeatedly inspects completed native results. Removing
that scan without complete writer coverage or current authority checks
could strand new evidence or commit an old result over newer work.

**Expected behavior**

Reassess from existing durable facts after relevant outer transactions
commit. Retain real retry/lease deadlines and restart/drain recovery.
Preserve current authority, quality-review and supersession checks. Make
no recurring SQL when work is settled or dormant.

**Steps to reproduce**

Complete a native result awaiting evidence or review, leave it idle,
then approve a real work product or retire the selected automatic
review. Its saved result should advance once when authorized, or record
the existing nonterminal business gate, without a periodic scan.

## What Changed

- Extract reassessment and its pure status policy from finalization
recovery into their owning modules. Add a scoped worker using the
existing delivery coordinator and scheduler; no queue table or migration
is added.
- Retain the three durable discovery triggers: changed work-product
history, authoritative task status/version, or retirement of the
selected automatic completion review. Preserve selected-source,
ownership, effects, retry and lease fences under the existing committer.
- Notify from actual task/work-product, claim release, approval,
interaction, OAuth, event evidence and workspace-settlement owners
before their transaction writes. Keep provider calls outside short
transactions and nested writes behind outer commit.
- Scope ordinary milestone/finalization/coalesced-input wakes to an
issue/run. Keep genuine company authority restoration explicit. Real
Photon and coalescer regressions cover all three response/status queues.
- Move the current parent's required quality policy into the owner and
apply it to every reassessment that could complete a task. An unmet
review records the existing in-progress decision and next action,
avoiding an operational retry loop. Preserve conversation/session reset
locks and the workspace omission-barrier signal.
- Skip reassessment in periodic recovery while preserving
explicit/startup recovery and the historical supersession guard before
independent workspace work. Document the source contract, producer
inventory, failures and qualifications in the linked audit.

## Verification

- Broad source `9fddb3f0b1157df32b64fb1f22e9d6e9624470e8` passes **2731
distinct cases in 28 complete files**, with no skips: 2458
producer/owner/authority/workspace/quality cases in 394.43 seconds and
all 273 governed-response cases in 53.74 seconds. Nineteen dedicated
fresh PostgreSQL 17 databases migrate through 0413; eighteen
embedded-only invocations preserve fresh-per-call isolation through the
existing recovery helper. All 22 temporary imports restore exactly and
all 28 hashes match.
- Final runtime source `5ea3b0fd016bf7e63ba82a7044825a46a0cd2082` passes
**1211 cases in five complete affected files** in 108.23 seconds after
the narrowly reproduced evidence-trigger fix: all 43 worker, 145 corpus,
32 finalization-recovery, 968 native-session and 23 quality-review
cases. Four fresh databases plus the quality helper's own fresh fixture
use PostgreSQL 17. Its one adapted import restores exactly; all five
hashes match. The other 27 broad-run test files are unchanged. These
qualifications are source-scoped and not cumulatively double-counted.
- Required/optional real regressions reproduce retry loops on
retired-review and evidence-approval paths before repair; both become
SQL quiet afterward with unchanged assertions/time limits. Controlled
removal of only ordinary scope guards reproduces two coarse-hint SQL
failures; exact guard restoration passes. Real producer, outer
transaction, claim, authority, 103-item backlog, retry and drain cases
pass. Photon transport is mocked; callbacks, locks and observed SQL are
real.
- Normal merges preserve 12661 parent/unrelated entries, 35 owned
entries and 21 automatic shared results, with five original conflicts
reviewed explicitly. The parent's actual squash reconciliation preserves
the entire held child tree. The final quality fix changes only its owner
and tests. Fourteen added comments are removed with identical nontrivia
tokens; no directives are removed and the final comment check is empty.
- Database build, server types/build, boundaries and exact-head
migration order pass. The database tree is identical to merged #17702:
no new SQL, queue table or snapshot is generated. Earlier benchmarks,
host limitations, failed assertions and their repairs remain in
`doc/plans/2026-10-26-native-status-reassessment-sleep-audit.md` and
`doc/idle-native-status-reassessment.md`.
- This qualifies targeted external PostgreSQL behavior, not embedded
bootstrap on the semaphore-exhausted host, a full local repository
suite, deployed sleep or measured fleet savings. Fresh exact-head CI and
review remain required.

## Risks

- Notifications retain the process-local supported-writer contract.
Independent SQL writers and uncoordinated replicas are not covered by a
new transport.
- Reassessment competes with current authority changes. Existing
row/source locks and retries remain required; arbitrary missing
identities have no row to lock, so this is not a general predicate-lock
guarantee.
- Discovery pages 100 rows but retains outstanding candidates in memory.
It remains proportional to backlog size, and cancellation occurs between
queries.
- Other orphan/workspace/session duties and Cloud sleep eligibility
remain unchanged. Missing supported producer coverage could strand
dormant work, so the producer audit and real callback/transaction tests
remain part of acceptance.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell
execution and PostgreSQL tests. The runtime does not expose a more
specific serving snapshot or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted scope is stated
above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 03:40:01 -05:00
Devin FoleyandPaperclip dfe7df0015 fix(native): expire settled native workspace sync folders (#17358)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs sync their workspace with a remote sandbox through a
per-run folder under `<instance>/native-workspace-sync/<run>/`
> - The folder holds a descriptor, a manifest, and workspace and git
seed archives, and a seed can be more than a gigabyte
> - Only a few finalization commit paths delete the folder, so many runs
leave it on disk forever
> - Instance disks fill up with folders that no code will read again
> - This pull request adds a bounded, deadline-driven retention worker
that removes a folder only when no reader can still need it
> - The benefit is that instance disks stop growing with stale sync
folders, and recovery and export paths keep their files

## Linked Issues or Issue Description

**What happened?**

`cleanupNativeWorkspaceSync(runId)` is the only code that deletes
`native-workspace-sync/<run>/`. It runs only when finalization commits
through specific paths. Runs that end in `terminal_failure`, retained
recovery, reconstruction, superseded runs, commits through other paths,
and server crashes leave the folder forever. On one instance, 22 folders
older than 7 days held 6.8 GB. Single seed archives were up to 1.4 GB.

**Expected behavior**

A run's sync folder is removed after a retention window, once the run
and its finalization are settled and no recovery or export path can
still read it. The shared `generations` seed cache stays.

**Steps to reproduce**

1. Run a native agent with a remote sandbox and let the run end in
`terminal_failure` (or crash the server during finalization).
2. Wait more than 7 days.
3. Look in `<instance>/native-workspace-sync/`. The run's folder and its
seed archives are still there.

**Paperclip version or commit**

`master` at 837add0caa

**Deployment mode**

Any deployment that runs native agents with remote sandboxes.

**Agent adapter(s) involved**

Native runtime with remote sandbox workspace sync (for example Codex on
Daytona).

## What Changed

- Added
`server/src/services/native-runtime/native-workspace-sync-retention.ts`.
It removes `native-workspace-sync/<name>/` when `<name>` is a UUID, the
entry is a real directory, its newest file is older than the window, and
one of these holds:
  - no `heartbeat_runs` row has that id (orphan);
- the run has no valid `nativeWorkspaceSync` reference and is settled:
terminal status, `finished_at` before the window, no live recorded
process (see below), no live controller lease, no
`nativeWorkspaceFinalizationOwner` receipt;
- the run is settled as above, its `native_run_finalizations` row (if
any) has no `lease_owner` and no `next_attempt_at` and is `committed`
(any recovery state) or `terminal_failure` with no or `blocked` recovery
state, no active/escalated export-cause or owner-unverified recovery
action names it, no lease carries `nativeWorkspaceExportResume` or
`pendingCleanupInFlight`, retained recovery cannot still use it
(referenced lease destroyed, or no scheduled/running/awaiting_evidence
system recovery and no queued/recovering workspace recovery), and
historical recovery adoption (`adoptHistoricalWorkspaceRecoveries`) has
no pending candidate for it.
- Each folder is decided in one transaction that holds
`pg_try_advisory_xact_lock` on the run's finalization ownership key. The
rule is evaluated while the lock is held. A held lock skips the folder.
The folder is renamed to `.expired-<uuid>` in that transaction and
deleted after commit. The next sweep removes leftover `.expired-*`
folders. `generations`, non-UUID names, symlinks and files are never
touched. Each tick tries at most 50 leftovers, then checks at most 200
folders and removes at most 50. Leftovers and folders each continue from
their own cursor, so undeletable leftovers do not block run folders. A
leftover that cannot be deleted schedules the 15-minute retry. Deadline
discovery reads run state 200 IDs at a time and stops when asked to. It
supports dry run and `shouldContinue`.
- A recorded `process_pid` / `process_group_id` blocks removal only
while it can still be the run's own process. This reuses the rule from
`conversation-continuation.ts`: PIDs are ignored when every lease of the
run is remote (they are sandbox PIDs), and a local PID or process group
blocks only while it exists on this host and its start time matches
`process_started_at`. An unreadable start time keeps the folder. A
recycled PID does not.
- Separate finding, not changed here: native runs do not clear
`process_pid` when they finish. It is cleared only by the status
arbiter's `release_run_resources` effect (cancellations and commits) and
by the recovery paths. A failed run gets a `schedule_retry` effect
instead, so its PID stays set. For remote targets the stored value is
the sandbox PID. Other readers already treat these PIDs as liveness
hints only, so this PR does not change that write path.
- Exported `nativeWorkspaceFinalizationLockKey` from
`native-workspace-finalization-ownership.ts` and used it in
`withNativeWorkspaceFinalizationOwnership` and the retention worker, so
both use the same key derivation.
- Exported `nativeWorkspaceSyncStateRoot()` from
`native-workspace-sync.ts`; `descriptorDirectory` now uses it.
- Registered the worker on the shared delivery-work coordinator
(`DELIVERY_QUEUES.nativeWorkspaceSyncRetention`) in
`server/src/index.ts`, next to the git restore point worker. It follows
`rules/sleep-aware-background-work.mdc`: no interval. It runs at startup
and at real deadlines: the earliest settled run's finish (or an orphan's
newest file) plus the window; a recheck one window later for folders it
kept; and a 15-minute retry after a failure or a held lock. With no
folders, `nextRunAt` is `null`. The run folder is now created through
one helper in `native-workspace-sync.ts` that notifies process-local
listeners; the worker wakes on that signal only while it has no
deadline, because any scheduled deadline is never later than a new
folder's own deadline. `index.ts` unsubscribes the listener on shutdown.
It pauses passive maintenance during idle drain.
- Added `PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_ENABLED` (default on
unless `false`), `PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DRY_RUN`
(default off) and `PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DAYS`
(default `7`, minimum `1`) in `server/src/config.ts`. Documented them in
`doc/DEVELOPING.md` and added a "Native workspace sync folders" section
to `doc/recoverable-disk-cleanup.md`.

## Verification

- `pnpm exec vitest run
src/__tests__/native-workspace-sync-retention.test.ts
src/__tests__/native-workspace-sync-retention-config.test.ts` in
`server/`: 62 tests pass. They cover:
- deletes a settled committed run (also with a left-over
`resuming_session` recovery state), a blocked `terminal_failure` with no
recovery owner, an orphan, and an unreferenced settled run;
- keeps runs whose local process or process group is still alive, or
whose live process start time cannot be read; keeps a running run with a
PID; deletes terminal runs whose recorded local process exited, whose
PID was recycled, or whose PID came from a destroyed sandbox;
- keeps recent files, recently finished runs, unsettled runs, a live
controller lease, an owner receipt, `observed` and `retryable_failure`
phases, a live `lease_owner`, a scheduled `next_attempt_at`, and each
non-blocked recovery state on `terminal_failure`;
- keeps runs with an open export-cause or owner-unverified recovery
action, `nativeWorkspaceExportResume` or `pendingCleanupInFlight` lease
metadata, retained recovery in flight on a lease that was not destroyed,
or pending historical recovery adoption; deletes when the retained lease
was destroyed;
- never touches `generations`, non-UUID names, symlinks or files; dry
run deletes nothing; leftover `.expired-*` folders are removed; an
advisory lock held by another transaction makes it skip; ticks are
bounded and resume from a cursor;
- `nextDeadlineAt` returns `null` with no folders, now with leftovers,
the settled finish plus the window, the orphan mtime plus the window,
and one window later for kept folders; the registered worker goes quiet
with no folders, retries after a held lock or an undeletable leftover,
and the real delivery-work coordinator wakes again when the real
producer (`writeDescriptor`) creates a run folder; leftover removal is
bounded and undeletable leftovers do not block run folders.
- Mutation checks: I removed each guard one at a time (advisory lock,
owner receipt, process pid, finished_at window, lease owner, next
attempt, phase, recovery state, export recovery, owner-unverified,
export resume, retained recovery, backfill pending,
symlink/real-directory check, dry run, and both mtime checks together,
plus the folder-creation wake, the process liveness rule (restoring the
old `process_pid IS NULL` rule, the sandbox exemption, the process group
check, the recycled-PID check, the unreadable start time), the leftover
batch limit, the leftover cursor and the leftover retry delay). Each
mutation made at least one test fail. I restored the code after each
one. Removing only the first mtime check does not fail a test, because
the second check inside the transaction still keeps the folder.
- `pnpm exec vitest run` on related suites in `server/`:
`native-workspace-sync.test.ts`,
`native-workspace-sync-history.test.ts`,
`native-workspace-finalizer.test.ts`,
`native-workspace-finalizer-failure.test.ts`,
`native-harness-backup-expiration*.test.ts`,
`git-restore-point-ledger*.test.ts`,
`delivery-work-coordinator.test.ts`, `chat-delivery-work.test.ts`. All
pass.
- `pnpm run typecheck` in `server/` passes.
- Greptile review: earlier rounds; all findings fixed with tests; final
score 5/5 with no open threads.
- Not run: the full server test suite and an end-to-end run against a
real instance. I did not measure disk savings on a live instance.

## Risks

- Deleting a folder that a reader still needs would make that path fall
back or fail. Same-run resume and export run under the finalization
ownership lock or before the run settles, and the worker checks the rule
while it holds the same lock. Retained workspace recovery already
handles a missing folder (ENOENT) by snapshotting the sandbox. The rule
keeps the folder for every in-flight or reopenable recovery state listed
above.
- A committed finalization row with a stale recovery state and no lease
owner is treated as settled. Nothing reads the recovery state of
committed rows.
- The worker is on by default. Operators can set
`PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DRY_RUN=true` first to see
candidates in the log, or turn it off with
`PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_ENABLED=false`.
- The first sweep on a large instance does more filesystem work. It is
bounded to 200 folders checked and 50 removed per tick.
- The folder-creation wake is process-local. A folder created by another
server process sharing the same instance directory is picked up at the
next deadline or restart.
- A stale `process_pid` no longer blocks removal. If a run's local
process is still alive with its recorded start time, the folder stays.
Live work is also still guarded by the run status, controller lease,
finalization lease and ownership lock checks.
- No schema migration.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-23 14:05:55 -07:00
DottaandPaperclip d33b40a5c3 fix: recover saved work and provide reliable task Retry (#16965)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The control plane must retain work when an execution or workspace
export fails.
> - Retained-workspace holds can leave saved input waiting with no
usable Retry action.
> - Users cannot verify process ownership or repair provider
allocations.
> - This pull request gives system recovery a durable budget and gives
each deliberate Retry a durable intent.
> - Safe recovery can preserve files, settle an accepted result, or
prepare an independent workspace.
> - One task state keeps infrastructure diagnostics out of the ordinary
conversation while preserving real permission and approval gates.

## Linked Issues or Issue Description

**What happened?**

A task with a retained workspace can show recovery, failed-run and
saved-message warnings at the same time. Historical runs without native
workspace descriptors cannot use the native restore path. Old recovery
metadata can prevent a new attempt even when an independent workspace
would be safe.

**Expected behavior**

Paperclip owns bounded infrastructure retries and file preservation. A
user can request another attempt with ordinary Retry. Thinking appears
immediately. The service preserves completed effects and current user
controls. It does not claim that starting fresh recovered the old files.

**Steps to reproduce**

1. Retain a failed run and its workspace after export or process-stop
failure.
2. Save a follow-up while the recovery hold is active.
3. Open the task and try to continue without infrastructure credentials
or operator certification.

Refs #16906, #16835, #16883. This extends their recovery and receipt
contracts. Related work: #16962 and #16963.

## What Changed

- Add task-level Retry with company-scoped authorization, all-status
idempotency and durable admission before physical work. Keep exact-run
Retry compatible.
- Add typed recovery phases, finite persisted backoff, historical
adoption and durable local failure diagnostics.
- Preserve native workspaces or verified historical snapshots. Admit an
independent fresh workspace only after execution containment. Keep
old-source preservation and cleanup obligations separate.
- Carry completed publication and API receipts forward. Fence uncertain
mutation retries across replacement runs. Preserve connected-chat
identity and authorization.
- Re-enter normal dispatch for saved input. Reuse accepted results
through the existing finalizer.
- Use one task interruption projection across the thread, composer and
task list. Show optimistic Thinking, ordinary Retry and genuine user
gates.
- Add PostgreSQL, filesystem, API, browser-fixture and controlled
provider coverage. Document the behavior and update the feature/access
maps.

## Verification

- All GitHub Actions CI jobs pass on
`6d8c6d27d50586d8548b31d03b47f6d73a0ee11d`: typecheck, all
server/chat/workspace and serialized suites, Runner checks, eight
browser shards, canary and build. The required merge gate passes.
- Greptile reviewed the final commit at 5/5, with no actionable findings
or unresolved review threads. Its two earlier P1 findings are fixed.
- Full recursive typecheck and build pass locally on the final commit.
All 16 task Retry cases pass against isolated real PostgreSQL. The
unrelated-success case and both stale Retry cases fail on the previous
code and pass after the fix.
- The corrected gateway fixture passes its uncertain-mutation and
independent-read case. Four real HTTP API cases pass for definite
403/422 rejection, completed-effect reuse, and uncertain-effect fences
across deliberate and automatic retries. These focused gateway/API
checks used a temporary database lifecycle harness on an existing
healthy local PostgreSQL process. The standard repository helper is
unchanged.
- Earlier qualification covers 291 task UI cases, saved-file copyback
and ancestry, current Agent Chat ownership races, nine chat Retry API
browser cases, and six assembled browser recovery fixtures. A real
API/PostgreSQL/native-runner journey showed Thinking in 25 ms despite a
delayed response, admitted one successor in a separate workspace,
retained original files, and survived reload.
- The controlled Daytona test interrupts export, joins the controller,
restarts it in a separate process, and verifies one agent start, one
external sentinel operation, retained file contents and no premature
cleanup. It does not establish abrupt-kill or live LLM/GitHub
acceptance.
- The full local `pnpm test:run` was attempted and was not green: an
OAuth row-lock timing assertion failed, and a later suite exhausted
macOS PostgreSQL semaphores. The ten OAuth timing cases passed on rerun
against a separate database. Later embedded database initialization
failures remain recorded; the final CI matrix passes.
- Security scanning passed on production-code commit
`63953edf069b111c6becc3144dffa6deb52f68b5`. Its only successor changes
one test-fixture identity line. The final scan is neutral/inconclusive
because its pinned merge-base checkout failed or timed out. Both rerun
API requests returned 404, so no fresh successful scan is claimed.
Socket and Snyk checks pass on the final head.
- The optional Paperclip Review remains queued behind an earlier run in
workspace finalization. Its prior completed assessment was incomplete
because a comparison tool was blocked by the platform filter. It is not
counted as a completed code review. GitHub does not require this check
for merge.
- Hosted acceptance remains pending deployment. Local and CI results do
not establish recovery of a hosted task or its retained files.

## Risks

- Recovery changes span admission, physical ownership and dispatch.
Incorrect evidence could permit overlapping work. Existing physical
locks and authoritative stop requirements remain in place.
- The migration adds concurrent indexes. It requires the existing
top-level migration runner and retries incomplete index creation safely.
- A fresh attempt can complete while old-source preservation remains
unresolved. The service retains that obligation and does not report
those files as recovered.
- Unknown tool mutations remain blocked until their authoritative
receipt owner resolves them. Generic upstream tools have no universal
result-query protocol.
- Recovery and diagnostics stay inside the stack. No Cloud incident
endpoint, notification worker or new outbound reporting channel is
included.

## Model Used

OpenAI Codex, GPT-6 family. The runtime does not expose a more specific
model version or context-window size. The model used reasoning, shell
execution, source editing and browser tools. No model subagents were
used for this implementation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused suites; the full
local run limitation is documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 19:58:53 -05:00
DottaandPaperclip bc091f0a21 fix(recovery): recover retained workspaces before saved messages (#16906)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A stopped run can leave useful files in its original sandbox.
> - The server keeps that sandbox when it cannot confirm file copyback.
> - The task can then save new user input but offers no safe way to
recover the files.
> - A retry of the old agent turn can repeat work that already finished.
> - This pull request recovers files through the existing workspace
owner and delivers only saved human input.
> - The user can recover a task without certifying infrastructure facts
or losing completed work.

## Linked Issues or Issue Description

Refs #16883 and #16835 for publication receipts and verified process
termination.
Related: #16478 clears stale holds after copyback already succeeded.
This change performs the missing export and records its proof. #16901
manages the main GitHub response comment and does not perform workspace
repair.

**What happened?**

A failed run retained its workspace. A later user message stayed saved
behind a file-repair hold. The task asked the user to verify process
termination and file recovery but offered no repair control. Sending a
message could show an optimistic Working state and move the task to Todo
before admission succeeded.

**Expected behavior**

The harness should recover the original files when it can prove
ownership and termination. It should preserve newer edits and completed
operations. Only waiting user input may start another turn. Missing
evidence must keep the source held with a specific explanation.

**Steps to reproduce**

1. Interrupt a sandbox run before workspace finalization succeeds.
2. Retain its exact allocation and save a user follow-up.
3. Observe the file-repair hold and missing end-user repair path.

## What Changed

- Add source-bound background recovery for native and legacy runs. An
accepted native result is not required.
- Share the physical copyback lock and durable owner. Resume only the
exact stopped allocation, export files, verify persistence and obtain a
fresh stop receipt.
- Persist retry limits and idempotent requests on the existing recovery
record. Limit each server to two background exports. Add an indexed
candidate scan and a separate cleanup obligation.
- Add a task-editor recovery endpoint. Deny agents and viewers. Reject
caller-supplied stop or repair claims.
- Deliver waiting human input once through ordinary admission. Preserve
pause, approval, budget and assignment gates. Require historical
conversational-adapter evidence for legacy-to-Runner continuation.
- Include exact GitHub publication receipts in continuation context.
Preserve unknown effects and the original failed run.
- Show observed recovery progress, a transient-failure retry action and
specific service blockers. Keep saved messages visible. Suppress false
startup feedback and implicit Todo promotion behind a file hold. Hide
unavailable workspace links.
- Update execution semantics, capability maps and production-component
stories.

## Verification

- Red: the original read model failed the new recovery-progress
assertion; the other 23 checks in that file passed.
- Passed: recovery regressions with 28 cases with real PostgreSQL,
files, tar and Git operations. Lifecycle provider calls are controlled
test boundaries. Coverage includes native/legacy export, concurrent
owners, rollback after export, interrupted archive export with a new
database connection, lost replies, historical discovery, durable retry
limits, separate cleanup and saved-input delivery. The actual candidate
query uses its partial index with 3,000 unrelated history rows. A first
automatic repair can bind verified old-controller stop proof without an
accepted native result. Review also reproduced and fixed export-only
continuation and cleanup lost after a committed repair. Cleanup remains
discoverable without waiting input and after task reconciliation.
- Passed: two GitHub integration cases verify that continuation context
carries the existing publication receipt and does not write another
response. Company and task boundaries are checked.
- Passed: 293 continuation, workspace-sync, platform-receipt and OpenAPI
checks. After file repair, actual dispatch retains pause, cancellation,
approval, real budget-policy and assignment gates.
- Passed: 174 route checks, including editor admission, agent/viewer
denial and rejection of fabricated repair claims.
- Passed: 351 task detail, workspace properties and comment-reopen
checks after updating the new read boundaries and asynchronous
workspace-link expectations. Two new route cases keep held tasks Blocked
when a comment is saved.
- Passed: 265 task-thread and recovery-notice checks. The broader
continuation suite also passed; its new legacy migration case first
exposed a gap that the focused recovery suite now covers.
- Passed after rebase: `pnpm -r typecheck` and `pnpm build`. The later
master merge adds an unrelated permission migration; recovery migration
generation and safety checks passed after the merge.
- Passed: token gates, module boundaries, map structure and migration
checks. The recovery index is migration 0360 after current master.
- Browser: inspected the production recovery component and its
retry-to-queued transition with fixture data. The first dev dependency
load failed; a reload after optimization resolved it. This is UI
evidence, not a live provider recovery claim.
- Every test shard passed on final commit `bbbc09efb4`. The older local
`pnpm test:run` was stopped after the complete CI test matrix passed.
Local full typecheck and production build passed. The first CI build
compiled and booted, then lost its runner during the clean public
npm-install check. An isolated retry passed the build, production boot
and clean public npm-install verification.
- Greptile reviewed final commit `bbbc09efb4` at 5/5. All review threads
are resolved.
- The change is not deployed. Hosted recovery remains unverified until
the served revision includes this change.

## Risks

- Recovery crosses process, provider and persistence boundaries. Missing
or mismatched ownership, stop, descriptor or file evidence keeps the
hold. It never selects a replacement allocation or silently starts over.
- File recovery does not certify external action outcomes or authorize
replay. New input still uses normal execution admission.
- Historical sources without sufficient durable evidence can remain
blocked. Destructive reconstruction stays separate.
- The index migration is idempotent. It changes no task data. Snapshot
pruning follows repository policy.

## Model Used

OpenAI Codex, `gpt-6.1-sol`, with code execution and browser tools.
`gpt-6-astra` assisted earlier investigation. The runtime does not
expose the context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-19 21:51:46 -05:00
DottaandPaperclip cccde7122b Trace execution preparation and durable response persistence (#16419)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server prepares an execution environment before it starts an
agent.
> - Operators need to distinguish preparation, provider wait, and
response persistence.
> - Existing preparation phases start separate traces, and the native
session trace ends before the final response is saved.
> - This pull request adds an execution-attempt trace and detailed spans
at the existing work boundaries.
> - The benefit is a measurable startup and settlement path without
changing execution policy.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Operator-configured OpenTelemetry tracing for agent preparation,
startup, and durable response storage.

**Subsystem affected**

Server orchestration, native Runner startup, and the Honeycomb run link.

**Current behavior**

Preparation phases start independent roots. Workspace lock waits,
authorization groups, runtime asset work, provider-pack fallback, and
the gap after provider completion are hard to attribute.

**Proposed behavior**

Use a server-owned async context for each execution attempt. Parent
nested work to that attempt. Link the separately sampled native session
because its ingress and queue history can begin earlier. Record
transaction entry, lock waits, grouped authorization, asset preparation,
artifact fallback, copyback, and response commit.

**Reason and benefit**

Operators can identify expensive work from recorded boundaries instead
of inferring the cause from elapsed time. Nested and concurrent
durations remain visible and must not be added together.

**Breaking changes**

No API or execution-policy change. New Observability span names and
closed count attributes are documented. The existing local
run-performance schema also records the additional provider-pack and
harness measurements.

Refs: #16178, #16205, #16330.

## What Changed

- Add `task.execution_attempt` with same-run async ownership,
independent root sampling, and native-session links.
- Measure workspace transaction entry and settlement, advisory and row
waits, source scans, reader groups, notifications, and writes.
- Measure runtime selection and each concurrent instructions, skills,
and MCP branch. Separate manifest, cache, materialization, and
verification work.
- Measure provider-pack discovery, verified reuse or upload, final
verification, and remote harness state checks.
- Measure workspace settlement, accounting, liveness, response
authorization, comment commit, and publication enqueue.
- Preserve the silent endpoint-off path, including when another library
has registered a tracer. Bound repeated detail spans to 16 per name per
attempt.
- Extend the Honeycomb run link and document the actual coverage and
remaining gaps.

## Verification

- After resolving the conflict with retained-reader batching on
`b98cdaea531c8447637cd4dcc19c61b3f5f41844`, all 61 targeted query,
authorization, workspace-preparation and trace-boundary tests passed.
The server typecheck passed. Reader tests still bound query count, avoid
loading retained run payloads and enforce fresh project grants on every
publication attempt.
- Before that base integration, the tracing/exporter group passed 855
tests with 2 optional-SDK skips across 10 files and no inherited OTLP
endpoint. It covers real PostgreSQL waits/rollback, durable response
commit, assembled heartbeat preparation, concurrent parentage, sampling
and tracer failure isolation. Native workspace/history tests passed 23
cases. These results remain supporting evidence for the unchanged
tracing paths.
- Full workspace typecheck and build passed during implementation. Full
sharded CI, release validation, E2E and Apex 5/5 passed on prior head
`8cebad80f3856dbcff6f3a6d5d34880ff77a20f2`. Fresh CI, security and Apex
review are required on the current head before merge.
- `pnpm check:token-gates` passed. No styles changed.
- Earlier CI runs exposed exact-source-expression assertions changed by
wrappers, missing run hashes on transaction-boundary spans, and sampling
fixtures without an enabled endpoint. The fixes preserve the calls, add
the declared hash and span cap, and enable tracing explicitly in the
sampling fixtures. Focused repairs passed; the full prior-head CI passed
after those repairs. Earlier failed run:
https://github.com/paperclipai/paperclip/actions/runs/38729131366.
- The full local `pnpm test:run` was stopped after the base advanced
during that process. Its mixed-version results are not final-head
evidence. A full local-suite pass is not claimed.
- The old-head Superagent scan stalled at 35%. The current head has a
fresh scan; no security pass is inferred from the old pending result.
- No deployed Honeycomb acceptance is claimed by this code change.

## Risks

- Enabled tracing creates more spans. Repeated detail spans are bounded,
and inventories use group spans.
- The attempt and native session remain linked traces. Historical queue
time can precede the attempt, and nested intervals overlap.
- A committed comment proves durable server storage. Publication enqueue
and realtime dispatch do not prove external delivery or browser paint.
- New custom attributes use only the existing run hash and validated
file/byte counts. No SQL, paths, permissions, content, or exception text
is added.

## Model Used

OpenAI GPT-6 through Codex. The serving model ID and context window size
are not exposed in this session. The agent used reasoning, tools, code
editing, and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-15 08:13:53 -05:00
DottaandPaperclip aa04c5c2a7 fix(runtime): keep the copyback lock transaction active (#16340)
Keep the dedicated workspace-copyback transaction active with bounded SELECT 1 probes while filesystem work runs. Preserve existing connection, owner-token, takeover and publication fences.

Verified head 7fb0ad45bf: full CI, E2E, Superagent, legacy statuses and Apex 5/5; no unresolved review findings. Real PostgreSQL idle-timeout reproduction and 71 affected cases pass. Strengthened disabled-timeout controls fail when their guard is removed.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-14 17:52:26 -05:00
0b0e53029f fix: recover saved workspace results automatically (#15824)
## Thinking Path

- Paperclip preserves accepted agent results before workspace copyback
and result commitment.
- A replaced controller can leave a physical copyback owner receipt
behind. A lost database lock or expired lease cannot prove that its
tar/Git children stopped.
- Infrastructure recovery should belong to the runtime and deployment
harness; task users should not investigate processes or attest shutdown.
- New receipts record the exact deployment. The harness independently
verifies that deployment fully stopped and signs a short-lived proof
bound to the stack, run, owner and deployment.
- The existing guarded reconciler then finishes the same accepted result
without a new agent turn. Legacy receipts retain the documented
administrator API after independent platform verification.

## Linked Issues or Issue Description

**What happened?**

An accepted native result can remain at `workspace_finalizing` with an
abandoned physical copyback owner. Repeated recovery sweeps could
replace specific instructions with advice to start new agent work. The
first version of this PR exposed a shutdown-evidence form in the task;
that asks end users to resolve infrastructure details.

**Expected behavior**

Automatically verify and recover the exact stopped deployment, preserve
the original accepted result and existing approval/monitor waits, and
show only a quiet task status. Unknown or live workers must remain
fenced. Older receipts without platform identity need independently
verified operator assistance through the administrator API.

**Steps to reproduce**

1. Persist an accepted native result and a copyback receipt from a prior
controller.
2. Replace that deployment before workspace finalization commits.
3. Let the normal finalization reconciler revisit the saved result.
4. With a platform-bound receipt and a fully stopped deployment,
recovery should finish automatically. With missing, live, unverifiable
or changed ownership, it must not write files or create another provider
turn.

Related public work: #14402 introduced copyback ownership; #9687 handles
dead-run finalize barriers; #15795 handles terminal stale-run cleanup.
This change addresses an accepted result whose physical copyback owner
prevents commitment.

## What Changed

- Record Railway project, environment, service and deployment identity
on new physical copyback owners.
- Request read-only platform verification from the managed cloud
harness. Require an Ed25519 stop receipt with a disjoint purpose,
configured signing key, exact stack/run/owner/deployment binding and
valid expiry.
- Keep live, unknown, malformed, unavailable and legacy owners fenced.
Compare the whole prior receipt during ownership takeover, under the
existing dedicated advisory lock.
- Persist platform proof and audit automatic takeover. Resume normal
finalization of the same saved result, preserving task ownership,
approved boundaries, monitors and pending confirmations.
- Remove process metadata, shutdown evidence, confirmation checkboxes
and infrastructure-repair forms from task views. Suppress the duplicate
technical owner-recovery card. Show “Restoring saved work…” only from
server-confirmed recovery or queued operator recovery; platform identity
alone shows the operator-assistance fallback. Remove the status when
recovery resolves.
- Share current-task, current-running-execution, accepted-result,
authoritative-reviewer and newer-execution checks between automatic and
operator recovery. Validate after the provider lookup, atomically with
the owner claim, and again before file publication; closed or superseded
tasks cannot start copyback.
- Keep the separately authorized exact-owner API for operator recovery
of legacy receipts, with active-lock, live-controller, stale-result and
newer-execution rejection. Generic dismissal cannot bypass the copyback
fence.
- Replace obsolete form stories with production-backed automatic,
queued, legacy, completed and mobile examples; retain stable suite IDs.
Update the operator runbook and scoped Storybook catalog.

## Verification

- Red/green: new platform-proof tests failed before implementation; the
database takeover regression failed against the original ownership
guard; reduced-UI assertions failed against the original form. All pass
after the changes.
- Passed: 59 physical-owner/finalization database tests using isolated
PostgreSQL, including real reconciler commitment from all three
saved-result phases through both automatic and operator recovery. They
preserve result identity, exact saved file bytes, pending approval and
overdue monitor waits, and create no new run or wake request. Unknown
evidence and changed ownership remain fenced.
- Passed: 162 focused proof, cloud identity, recovery route,
recovery-card and reduced-UI tests.
- Passed: full workspace typecheck and build, final server/UI
typechecks, token gates, and Storybook build.
- Browser inspection: desktop task composition shows only the quiet
status; completed example removes it and retains the approval-wait
message. Mobile shows the existing bottom navigation with no horizontal
overflow. These are fixture UI observations, not live provider
acceptance.
- A companion harness change implements the actual provider check,
signed receipts, server-token authentication and bounded/coalesced
provider reads. Its full typecheck and test suite passed: 3,240 passed,
92 skipped, zero failures. This application alone cannot enable platform
verification.
- The updated full application `pnpm test:run` was stopped after
repeated local embedded-Postgres startup failures; it is not claimed
green. The same affected recovery suites pass against isolated
PostgreSQL. CI passed on final head `9ad5c3529e`: 54 successful checks,
two intentional skips, and successful legacy Snyk status. Greptile is
5/5 with no unresolved findings.
- Global Storybook catalog validation is green: 225 files, 2,224
stories. Its 17 catalog/index tests pass. Master was merged and the
inherited index metadata was reconciled after source review.
- Review correction red/green: two UI and two database assertions
reproduced misleading progress / missing server recovery state; all pass
after server-confirmed state was added. All 153 focused server tests and
68 UI tests pass on the final recovery code, including repeated-sweep
idempotence. Full workspace typecheck/build, token gates and Storybook
build also pass.
- Safety regression red/green: stale-task copyback and publication tests
failed against the previous guard. The final suite covers changes during
proof lookup, closed/cancelled tasks, replaced execution/checkout, newer
and queued legacy runs, reassignment and rejected/unbound results, plus
the real reconciler selecting closed or superseded tasks. Files remain
untouched and no workspace operation starts.
- No staging mutations, deployment, live provider probes or live
production recovery were performed. The three historical receipts lack
deployment identity and still require independent operator proof.

## Risks

Automatic takeover requires the companion harness release and configured
runtime-identity keys. Deploying only this application leaves
unavailable evidence fenced. Legacy receipts are not retrospectively
assigned a deployment identity. An authorized operator can still make an
incorrect attestation through the fallback API; its administrator gate
and exact-owner safeguards remain. Provider history that is missing or
deeper than the bounded lookup defers automatic recovery. Copyback can
still fail under its existing retained-workspace policy.

No migration, telemetry change or cloud provider credential delivery is
required. The companion harness PR also has passing CI and a 5/5 review;
neither branch is deployed.

## Model Used

OpenAI Codex, GPT-6, with repository inspection, browser verification,
tool use and code execution. The exact serving model ID and
context-window size were not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run the relevant tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Paperclip <paperclip@paperclip.ing>
Co-authored-by: Devin Foley <devin@paperclip.ing>
2026-10-13 10:14:45 -05:00
83e13db8fa fix(inbox): keep automatic recovery cleanup out of personal participation (#16063)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The personal inbox uses issue activity to identify a person's
participation.
> - Issue reads can cancel stale recovery actions and record that
cleanup as the reader's action.
> - This adds unrelated tasks to the inbox and refreshes their activity
timestamps.
> - A workspace owner hold can also alternate between creation and
cancellation after a task finishes.
> - This pull request corrects cleanup attribution, excludes historical
read cleanup, and retains one exact owner hold.
> - Agents can read authorized issue activity to diagnose the same
problem themselves.

## Linked Issues or Issue Description

Refs #15742 excludes logs by the execution-recovery actor from activity
timestamps. This PR covers reader-attributed stale recovery cleanup,
personal inbox participation, exact workspace owner holds, and sandbox
issue activity access. I searched open issues and PRs for recovery inbox
and activity bridge fixes. I found no duplicate for this defect.

**What happened?**

Loading issue lists cancels stale recovery actions under the reader's
identity. The personal inbox treats those rows as user participation.
One completed issue had hundreds of such rows and no edits or comments
by the affected reader. A retained native workspace owner repeatedly
creates a verification action that read projections then cancel. Sandbox
agents cannot inspect the issue activity because the reverse bridge
rejects the route.

**Expected behavior**

Automatic cleanup belongs to the system and does not add an issue to a
reader's inbox. Historical cleanup rows remain auditable but do not
count as participation or recent activity. An unresolved physical
workspace owner keeps one stable verification action. Agents can inspect
activity only for issues their identity may read.

**Steps to reproduce**

1. Finish a task whose native run retains an unverifiable workspace
owner receipt.
2. Repeat workspace recovery checks and GET requests for issue detail,
compact lists, and recovery actions.
3. Observe new recovery action IDs and reader-attributed cancellations
in the activity history.
4. Query the personal inbox for that reader and observe the unrelated
task.
5. From a sandbox agent, request GET /api/issues/:id/activity and
observe Route not allowed.

**Paperclip version or commit**

Observed at f5346c05b9. Regression tests
use current master behavior.

**Deployment mode**

Authenticated hosted instance with native and legacy sandbox agents. The
inbox defect is independent of the adapter.

## What Changed

- Attribute automatic stale recovery cleanup to the recovery system
actor.
- Exclude exact historical read cleanup rows from personal inbox
membership, unread counts, and both activity timestamp queries. Keep all
audit rows and explicit user recovery decisions.
- Retain owner verification holds only while the exact company, run,
issue, and owner fingerprint still match. Apply the same rule to reads
and background recovery.
- Make repeated checks for the same owner hold leave its identity,
attempts, and timestamps unchanged. Keep physical takeover blocked.
- Allow GET issue activity through the sandbox bridge. Keep the existing
company and issue-read authorization and reject broader audit routes.
- Add regressions and update the runtime API and recovery documentation.
Refresh generated source references for the documentation edits. Update
the paired activity, inbox, and recovery maps.

## Verification

- Focused Vitest verification: 315 tests pass across five files. The
updated terminal owner hold regression also passes. The files cover
issue service, recovery routes, activity authorization, sandbox callback
bridge, and native workspace finalization.
- `pnpm -r typecheck` and `pnpm build` pass locally. The full general
and serialized unit test matrix passes in CI on the latest commit. The
duplicate local `pnpm test:run` was stopped after about 30 minutes once
that full CI coverage passed; it did not report a local full-suite pass.
- `node scripts/check-maps.mjs` passes for all paired areas and
capability IDs.
- Apex is 5/5 on commit 44c3787888. Its
access map finding is fixed and resolved. All CI checks are green on
this commit.
- The new regression repeats reads and background sweeps for done and
cancelled tasks. It checks one unchanged hold, zero physical copyback,
zero personal inbox participation, and system cleanup after the exact
receipt is cleared.
- Historical cleanup tests preserve explicit recovery decisions,
nullable metadata, unread counts, and the original audit rows.
- Authorization tests permit an authorized agent and reject
cross-company or issue-boundary reads. Bridge tests reject write
methods, extra path segments, and company-wide audit routes.

## Risks

Existing inbox items can disappear if their only participation evidence
was automatic read cleanup. No data migration or audit deletion is
needed. An unresolved workspace owner remains visible after task
completion; this preserves the physical writer fence. A stale or cleared
owner does not retain the hold. Issue activity access expands only the
sandbox transport allowlist and still uses existing controller
authorization.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository inspection, code
editing, shell execution, and test tools. The session identifies the
model family as GPT-6. A more specific runtime model ID and context
window size are not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Amy (Opus) <noreply@anthropic.com>
2026-10-12 19:25:15 -07:00
DottaandPaperclip 14795136f5 fix(runtime): finalize and recover sandbox workspace exports safely (#14402)
Serialize native workspace finalization, validate streamed archives within bounded limits, and quietly recover unsafe exports from saved results. Preserve exact allocations for exhausted transient failures and provide export-only retry without rerunning the provider.

Consolidates #14314, #14315, #14329, and #14334 while preserving the already-merged finalization label changes.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-28 10:57:32 -05:00