Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-04 02:32:07 +01:00
Code Issues Packages Projects Releases Wiki Activity
6,165 Commits 1,406 Branches 2,251 Tags
f131ecd7e044499ea5287eac00a482c51b22df42
Commit Graph
8 Commits
Author SHA1 Message Date
DottaPaperclipCodex
 		4c56702e06 fix(server): schedule retained native-session cleanup (#17717) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs can retain a local session after an interrupted durable
suspend.
> - The orphan reaper repeatedly scans completed history for that
cleanup duty.
> - Idle scans keep the database active. Removing them requires complete
wake and ownership paths.
> - This pull request moves retained-session cleanup to committed
signals, local release events and saved deadlines.
> - Existing physical proof checks still decide whether cleanup is safe.
The shared scheduler does not grant execution authority.
> - The benefit is quiet retained history with recoverable cleanup and
owned shutdown.

## Linked Issues or Issue Description

Refs #17709 and #18135. This is slice 51 of the existing
recurring-server sleep stack. Both referenced dependencies have merged.
The target is `master`.

**What happened?**

The periodic orphan pass scans retained native sessions even after their
cleanup is complete or no automatic action is safe. Some eligible
sessions also depend on a local session, environment lease or original
maintenance callback releasing ownership. An earlier decision
notification can be consumed before the run's later terminal projection
commits.

**Expected behavior**

Reconstruct existing work at startup. Wake from committed writes and
actual owner release. Register the saved claim expiry or a probe for a
specific live process. Stop recurring discovery when no obligation
remains. Preserve original callback ownership and all existing cleanup
proof checks.

**Steps to reproduce**

1. Retain completed native runs with a durable-suspend close failure.
2. Let the periodic orphan pass run with no new work.
3. Observe repeated cleanup discovery. Separately, delay terminal
projection after its decision notification is consumed and observe the
missing cleanup wake.

**Paperclip version or commit**

Parent #17709 at reviewed head
`3b75f4bc1da1370304f069fc287af3d693575716`, landed as
`df6156c2e37f542a69f393dde31e357b82b7cac1`. Current integrated master:
`31b927e3013a8a4b3266854b740cf40a24753788`.

## What Changed

- Move scoped, paged discovery to the retained cleanup owner. Preserve
explicit manual recovery.
- Register the duty with the shared scheduler and remove it from the
periodic orphan pass. Keep concrete lease expiry and live-process probe
obligations.
- Use typed outcomes for settled/refused work and process, session,
environment, maintenance or lease waits. Keep physical proof and
quarantine decisions in the executor.
- Wake from committed finalization and terminal projection writes,
durable process/lease release, and local session or callback settlement.
Routine company-only finalization hints leave dormant cleanup quiet.
- Join duplicate cleanup and serialize inspections on one database lane.
Keep original Runner callbacks owned until settlement and shutdown.
- Preserve all master profile fixtures and the branch's historical
identities. Save the colliding sleep profile 79 separately. Generate
combined Copilot profile 80 from its source and retain master's other
provider profiles.
- Add the owner audit and regression coverage. No new table, migration
or independent maintenance timer is added.

## Verification

- Server acceptance: **2,342 distinct passing cases in 18 complete
files**. The 14-file unaffected group contributes 1,532 passes. Full
startup contributes 76, process recovery 410, environment service 51,
and external-chat response 273. The 29-case cleanup worker suite and
977-case executor suite run in full. No case in these files is skipped.
- Controlled scope regression: removing only the ordinary finalization
issue/run guard wakes dormant retained cleanup; the same assertion
passes after restoring it. A real local owner release still resumes the
durable candidate.
- Preserve the first broad failure: 1,607 passed and one startup
shutdown case failed because the coordinator mock lacked `stop()`. Add
only that mock method. All 76 unchanged startup cases then pass. Do not
treat the first broad command as green.
- Dedicated PostgreSQL 17 databases use the existing migration 413
schema. Embedded-only fixtures use the existing recovery helper with
fresh per-call databases. All temporary imports are restored byte-exact.
No assertion or timeout is relaxed.
- Current-master integration preserves both complete source histories.
The server, UI and database trees are identical to the server-tested
source. All master historical profile fixtures remain byte-exact. The
prepared sleep profile 79 remains independently inspectable.
- Current runtime validation at
`d08f764e3efe75e96929173afe7ee8739121ad9f` passes all 12 commands (22
installed ACPX/Grok contract cases and 26 native-provider Rust cases):
`pnpm install --frozen-lockfile --ignore-scripts`; Copilot/Grok identity
and generated profile checks; Runner and server typechecking; server
build; Copilot/Pi installation, profile compatibility and transport
settlement files (130 passed; one opt-in closed Pi snapshot case skipped
because its distribution root is not configured); installed ACPX
model/reconnect package contract and Grok identity tests; Rust
native-provider backend tests; module boundaries and migration order.
- Earlier 94-case installation/settlement acceptance applies to the
prepared profile 79 source. It is retained as history and does not
substitute for the current profile 80 validation above.
- Failure history and duty-specific evidence remain in
`doc/plans/2026-10-26-retained-native-session-cleanup-sleep-audit.md`.
Exact-current-head acceptance: all 60 checks are clean, including
protected CI and security; Greptile gives 5/5 for
`281d579de0d08e907194b8b790a0597b07d16b7b` with no actionable findings.
No Ponytail review is present under the authorized missing-review
policy.
- These checks do not claim a full local repository suite, live Copilot
or Daytona qualification, deployed host/database suspension, or measured
cost savings.

## Risks

- Missing a supported producer could strand cleanup. Tests exercise real
finalizer, heartbeat, environment and process writers. Process-local
signals cover in-process writers; no multi-replica or external SQL
notification transport is added.
- A timeout can leave an original callback active. Cleanup and shutdown
must keep waiting for it. The existing global maintenance drain is
conservative across roots.
- A retained live process still has a scoped 30-second absence probe. A
saved lease keeps its actual deadline. Operational errors retain retry
work. Immutable proof refusals and operator-required histories do not
gain automatic retry authority.
- Cleanup keeps all existing artifact, identity, claim, no-launch and
quarantine checks. Terminal status alone does not authorize physical
cleanup.
- This PR does not expand Cloud alarm coverage or relax host sleep
eligibility. It proves worker behavior, not deployed app/database
suspension.
- The Copilot source identity has changed; these checks do not claim new
live provider qualification.
- The parent has merged. This PR is merged only after its current-head
protected CI and review gates pass under the existing campaign
authorization.

## Model Used

OpenAI Codex, GPT-6 family, with code execution and repository tools.
The exact serving variant and context-window size are not exposed in
this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-29 04:24:57 -05:00
DottaPaperclipCodex
 		ec3b558277 fix(server): schedule accepted native response recovery (#17696) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs store accepted results and their finalization decisions.
> - A completed run can still owe its public answer after an
interruption.
> - Periodic recovery scans for that answer even when no new work
exists.
> - This pull request wakes recovery from committed changes and saved
retry deadlines.
> - The existing owner still verifies the result, source, and current
publication authority.
> - Settled or dormant response work no longer needs its own periodic
scan.

## Linked Issues or Issue Description

Refs #17690 (merged). This is slice 48 of the sleep migration stack, now
based on master. Related: #17661 handles GitHub terminal publication;
this change handles native accepted-response presentation.
Governed-review presentation remains a separate duty.

**What happened?**

Native finalization recovery repeatedly checks already accepted results
for a missing answer. Removing that check without complete notifications
would lose responses when evidence arrives late, an agent resumes, or
admission settles after execution.

**Expected behavior**

Recover the answer from its saved result when a supported change makes
it eligible. Retain real retry deadlines. Stop querying when the answer
is selected or cannot currently be published. Reconstruct work after
restart.

**Steps to reproduce**

1. Commit a native accepted result with no selected response comment.
2. Run finalization recovery while its source is unavailable or locked.
3. Restore that source or release the lock. The answer must appear once
without repeating provider execution.

## What Changed

- Honor recorded non-create decisions and skip-comment policy in both
discovery and recovery. Apply durable conversation/result/continuation
guards before recovery locks. Preserve genuine unanswered and legacy
recovery.
- Scope routine response wakes to the affected issue/run. Coalesced
input names its active run; pending-effect settlement names its issue.
Sixteen existing global authority/configuration hooks use one explicit
company invalidation boundary. No new transaction wrappers or tables are
added.
- Add an accepted-response worker using the existing delivery
coordinator and scoped deadline helper. Preserve saved retry deadlines
and pending-effect fences under the owner's lock.
- Share the finalization notification boundary. Wake on finalization
settlement, exact accepted-result/final-message evidence, agent
availability, retry admission, and existing chat authority changes.
- Keep lifecycle policy in services. A narrow pre-write status hook lets
lifecycle retain ownership of its transaction and status transitions.
- Register the worker during enabled maintenance startup. Remove its
authority subscription on shutdown. Periodic replay skips this duty on
both committed and fresh-commit paths; ordinary finalization and
explicit recovery retain their behavior.
- Add a partial index and expression statistics for unpresented native
runs. Migration 0412 supports concurrent replay and adds no queue table.
- Add real PostgreSQL and lifecycle coverage. Preserve the
implementation and producer audit in the plan document.

## Verification

- Additional real Photon callback acceptance on the published production
code disproves the security warning: ten active assertions cause zero
response-recovery hints/SELECTs, while three concurrent assertions
during an actual attention-to-active recovery cause exactly one
wake/SELECT. The existing locked status guard already enforces this
boundary. No production change was needed; the resolved finding now has
a successful current-head security check. The regression will be
retained with the next slice.
- Eligibility repair source `892f36e5f66ba1af123034936a7160c64c3ab883`
passes **1489 distinct server cases**: 1139 in six complete
recovery/finalizer/monitor/summary/workspace suites (81.02 seconds),
plus 350 selected chat cases (297.68s). Another 1463 chat cases are
outside the pattern; the complete 1813-case file is not claimed. All
eleven new regressions are included in that count. Five dedicated fresh
PostgreSQL 17 databases migrate through unchanged 0412, with no
fixture-source adaptations. Database build, server types/build,
boundaries and full-SHA migration ordering pass. All seven test source
hashes match.
- The exact published baseline first fails eight behavioral assertions
while the three genuine-answer cases pass. The same corrected test
source passes all eleven after repair: recorded none/reuse remain
suppressed; five ineligible history families take zero recovery row
locks across repeated issue hints; missing/legacy/create-null answers
still publish once; fresh periodic finalization defers presentation to
its worker. Earlier fixture setup and variable/compiler failures remain
explained in the audit, rather than erased by later success.
- Repair source `2fc8b0614aaf9ac7337045de4ced7325f95edc45` passes **1821
server cases**: 1496 in 13 complete producer/lifecycle/authority suites
(110.63 seconds), plus 325 selected chat cases (278.99 seconds).
Coverage includes all native-session cases, queue admission, status
decisions, membership/board access, the complete GitHub workflow,
accepted-response lifecycle and real authority restoration. The other
1477 chat cases are outside this selection; the complete 1802-case file
is not claimed for this repair run.
- Actual PostgreSQL regressions first reproduce a coalesced-input hint
missing its run scope and two routine hint paths producing 12
response-history SELECTs each after three notifications. The same
assertions pass with run-scoped coalescing and zero response-history
SELECTs for unscoped routine hints. Actual membership restoration still
publishes the waiting response once. Assertion time limits are
unchanged; this measures real driver SQL and persisted producer work.
- Eleven dedicated fresh PostgreSQL 17 databases migrate through 0412.
One embedded-only wake-queue fixture import temporarily uses its own
external database and restores byte for byte. All 14 selected source
files match pre-run hashes. This qualifies external-database behavior
rather than reproducing embedded CI locally.
- Master advanced independently with #18126's Pi archive retry fix.
Normal merge `2a535bc61259ef99387369e8b32498f774e5ba2c` preserves both
complete binary patches. All accepted server/database/UI/test/CLI trees
stay identical; the three independent runner installer/manifest files
are retained. All 13 installer tests pass, including stalled header/body
cancellation. Combined server types/build, boundaries and full-SHA
migration order pass. A subsequent normal merge of #18128 at
`e8b78cce5bb557b086bee9ca930eeaab1d09b244` retains its three test-only
file changes with both complete patches preserved. The changed HTTPS
backfill case passes on a fresh PostgreSQL fixture (1 selected case; 178
outside the pattern), and combined types/build, boundaries and migration
order pass again. All 14 repair source hashes remain unchanged. The
independent browser fixture changes are preserved; no local browser run
is claimed.
- The earlier source `9feec170ef8174011fc3c5d3d6d76f0a0f39fea0` passed
the 4171-case full baseline before this scope repair. That scope is
preserved in the audit rather than represented as a current full rerun.
All 0412 database bytes remain unchanged: original concurrent
index/statistics/ANALYZE, prior executable history, journal entries and
retained snapshots. Five snapshots 0408 through 0412 remain. The
populated 0411-to0412 upgrade preserved 410 prior receipts and 38
companies, appended one receipt and none on replay; index/statistic
checks passed.
- The earlier P2 security failure, Greptile's former conditional
dependency on #17702, and the subsequent actual Ponytail eligibility
findings are recorded in
`doc/plans/2026-10-25-native-response-presentation-sleep-audit.md`. This
PR now includes the issue-scoped pending-effect notification fix itself,
so it is independently correct before #17702. Fresh published-head CI,
Greptile 5/5 and a clean security scan remain required. No
deployed-sleep, external-wake or fleet-savings claim is made.


## Risks

- The writer contract is process-local. Independent SQL writers and
other replicas are not covered by these notifications.
- Unpresented backlog reconstruction still has memory and time cost
proportional to candidate count. Pages contain at most 100 rows;
execution uses the shared fair batch. Selected history is excluded.
Health-only signals do not trigger reconstruction.
- Publication policy remains with its current owner. Digest, company/run
binding, source authorization, selected-answer suppression, and
ownership fences remain required.
- The rest of native recovery and governed-review presentation still run
separately. This PR does not relax Cloud sleep eligibility or add
external wake coverage.
- The stack parent has merged and its actual ancestry is reconciled.
This publication still needs fresh current-head CI and review.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell
execution, and PostgreSQL tests. The runtime does not expose a more
specific model identifier or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-29 02:31:20 -05:00
DottaandPaperclip 4e715eb7ea fix(server): schedule historical unsafe export recovery (#17687) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs can retain accepted results after an unsafe workspace
export fails.
> - Historical recovery can retire an old notice or resume the accepted
result without another provider turn.
> - The periodic reaper repeatedly scans failures that cannot currently
recover.
> - Existing rows and committed changes can identify the actual recovery
obligation.
> - This pull request wakes recovery from those changes and retries only
owed work.
> - Dormant history becomes quiet while ownership and explicit operator
decisions remain protected.

## Linked Issues or Issue Description

**What happened?**

The native finalization reaper repeatedly visits historical unsafe
workspace-export failures. Cancelled tasks, newer runs, changed
contracts, and explicit operator decisions can make those rows
permanently ineligible. They still produce periodic database reads.

**What did you expect?**

Retire stale unsafe-export notices and recover eligible accepted
results. Stop reading dormant history until a relevant committed change
occurs. Keep a retry while a real obligation is blocked by ownership or
a transient failure.

**Steps to reproduce**

Retain an unsafe-export terminal failure whose task is cancelled or
superseded. Leave the instance idle after startup. The periodic reaper
continues reading that history.

Prerequisite #17680 is merged into master. Related: #14402 established
safe export finalization; #14329 was an earlier historical recovery
proposal. #17679 owns separate policy-review cleanup. This PR changes
only historical unsafe-export scheduling and its producers.

## What Changed

- Register a scoped worker through the existing coordinator and
scheduler. Remove this duty from periodic reaping. Preserve the
manual/startup recovery entry point.
- Distinguish stale notices, eligible accepted results, dormant history,
and concrete ownership contention. Reuse the existing mutation owner and
lock order. Preserve explicit settlement vetoes and exact
result/contract identity.
- Signal relevant task, run, action, and deletion writes before they
commit. Existing transaction boundaries handle nested commits and
uncertain outcomes. Notifications contain only scope IDs.
- Retain a retry only for owed work. Reconstruct work at startup,
preserve drain/resume, and stop through the shared lifecycle.
- Add concurrent, idempotent partial index migration 0410 for unsafe
terminal failures. Keep the five newest generated schema snapshots.
- Document producer coverage, query plans, supported writer boundaries,
and test failures in the unsafe-export audit.

## Verification

- Source `2a13dd08a6cc52f0adab09e3df9ce31e97b746b9` passes all 2188
cases across 24 complete files, without skips. The main selection passes
2129 cases in 192.58 seconds; safe-replacement passes 59 in 26.04
seconds. Coverage includes all 968 native-session cases, real
export/action producers, explicit continuation, workspace
reconstruction, interaction routing, Stop, queue and Slack consumers.
The audit gives per-file counts.
- Twenty separate fresh PostgreSQL 17 databases migrate through 0410
before setup. Four embedded-only fixture imports temporarily use their
own external database on this IPC-exhausted host, then are restored byte
for byte. Assertions and timing limits stay unchanged. This is qualified
external-database acceptance, not an unmodified embedded CI run.
- Database build, server typecheck/build, module boundaries and
exact-head migration order pass. Normal integration preserves the
complete original runtime/schema patch and unrelated upstream blobs.
Original concurrent index SQL, all prior executable history, journal
entries and retained snapshot bytes are unchanged; five snapshots 0406
through 0410 remain. Added comment removal preserves the full
non-comment TypeScript token streams.
- A populated 0409-to0410 upgrade preserves all 408 prior migration
receipts and 38 companies. It appends exactly one receipt; replay adds
none. The index is valid and ready on fresh and upgraded databases.
- The owning audit retains earlier fixture failures, skipped checks and
notice-retirement regression history. Older combined stack checks retain
their original scope in their owning audits. Actual parent
`73d24aa5acfcf17b3f676b4da7cd05eb70b60b99` is reconciled through a
normal merge. Its tree equals the reviewed ancestor and the complete
accepted child tree is unchanged. Fresh published-head CI and Greptile
5/5 remain required. No deployed sleep, fleet-cost or live provider
qualification claim is made.

- Inherited prefix correction: current integration
`8f43c9f81fac23b50997e7f0e1d0d9cf383a8109` changes only two values in a
previously unselected corpus fixture. All production code, the complete
package tree and all 24 accepted test files remain byte-identical to the
2188-case source above. The additional complete 72-case corpus passes in
14.00 seconds and again on the same database in 13.08 seconds. Retained
PSC/OUT and sentinel rows remain unchanged. The repeat is not
double-counted. The parent audit retains the original collision and
rejected cleanup attempt.

- Latest parent context: integration
`4d51231abe8b68c21ce896d0d2ac16865978c8ab` carries only the parent's
one-argument GitHub observation correction and an independent master
cleanup assertion. All production code, packages, 24 accepted suite
files and the additional corpus are unchanged. The two focused
native-presentation cases pass in 15.14 seconds and the complete
cleanup-removal suite passes all six cases in 8.31 seconds on two fresh
0410 PostgreSQL fixtures. This does not claim the full parent workflow
was rerun here.

Current head `b28399babac72ba9db1c4a4ef882c5e87aded38c` has passing CI
and Greptile 5/5 with no actionable findings. The first CI attempt lost
runners: fourteen jobs explicitly received simultaneous shutdown
signals, two reported lost runner communication, and two aggregate gates
failed downstream. One failed-job retry on unchanged source passes. The
underlying host shutdown cause remains unconfirmed; no code fix or extra
retry is claimed. [CI run and
attempts](https://github.com/paperclipai/paperclip/actions/runs/40819408528).
No Ponytail verdict is present.

## Risks

- A missing future producer can delay dormant-history discovery until
restart. The audit names current producers. Independent SQL writers and
other server processes remain outside the process-local signal contract.
- Physical ownership and uncertain operations retain retries. Time alone
does not authorize taking workspace ownership. No provider turn or
unsafe archive retry is added.
- The recovery owner remains authoritative if state changes after
discovery. Terminal UI status does not authorize resource destruction or
accepted-result replay.
- The partial index adds matching-row write/storage cost. It builds
concurrently through the existing restart-safe migration runner.
- This proves the changed worker becomes quiet. It does not prove host
suspension, external wake coverage, or fleet cost savings. Other sleep
migrations remain. This slice still requires fresh current-head CI and
reviews.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning and code-execution tools.
The exact serving model ID and context-window size are not exposed in
this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 00:13:22 -05:00
Devin FoleyandPaperclip ef9e17304c perf: narrow runtime progress database reads (#17355) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Active runs publish workspace and provider progress.
> - Each update must check the current run status before it publishes.
> - This check fetched the full run, including prompts and runner
configuration.
> - This pull request reads only the four fields that progress
publication uses.
> - The benefit is less database transfer while each update still checks
durable state.

## Linked Issues or Issue Description

Related: #17348 narrows authorization reads. This PR applies a separate
projection to runtime progress. GitHub searches for runtime progress and
run-status read PRs found no duplicate.

**What existing behavior does this improve?**

The database read before a runtime progress update.

**Current behavior**

Each update calls `getRun`. This reads large run context and runner
configuration even though progress publication only uses the run ID,
company ID, agent ID, and status.

**Proposed behavior**

Use a fresh four-column read for each update. Keep the existing
active-status check, event fields, error propagation, and late-progress
suppression.

**Reason and benefit**

Transfer progress can emit many updates for one run. A small status
check avoids repeatedly transferring payloads that it does not use.

**Breaking changes**

None. No API response, schema, scheduling, or cancellation policy
changes.

## What Changed

- Add `getRunStatus` to the run-state owner. The full reader remains
available to consumers that need execution data.
- Wire the narrow reader into the lifecycle progress check. Remove the
unused context field from that check's input contract.
- Cover the exact selected fields, missing runs, database errors, large
stored payloads, terminal transitions, and the public heartbeat service
wiring. Document the read boundary.

## Verification

- Run-state and lifecycle suites: 44 tests passed before the additional
public-service regression.
- Lifecycle, public-service wiring, and native cancellation suites: 36
tests passed.
- `pnpm -r typecheck`: passed.
- `pnpm build`: passed.
- The [full PR
workflow](https://github.com/paperclipai/paperclip/actions/runs/40013092548)
passed on `95bf496f689049812646b5ccddcdf2cca21efad0`. All 54 reported
checks passed; two conditional checks were skipped. This includes test
shards, browser checks, Runner verification, production bundle boot, and
clean npm installation.
- `GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=init.defaultBranch
GIT_CONFIG_VALUE_0=master pnpm test:run`: an additional local all-in-one
run is still in progress; no full-local-pass claim. The process-scoped
Git default matches existing master-only fixtures and does not change
user configuration.
- The initial description check failed because the dedup-search checkbox
was missing. Searches are complete and the template checkbox is now
included. The rerun passed on the same source commit.
- Greptile Apex: 5/5 on `95bf496f689049812646b5ccddcdf2cca21efad0`, no
findings or review threads.
- No hosted deployment or bill reduction is claimed. CI and Apex are
complete; this PR is merge-ready.

## Risks

- Low scope: the query selects fewer columns and preserves the existing
status read on every progress callback. It adds no worker, cache, timer,
or idle query.
- A run can still finish after the status read and before publication,
as it could before this change. This PR does not change that concurrency
boundary.
- No migration or operator configuration is required.

## Model Used

OpenAI Codex (GPT-6 family). The runtime does not expose the exact model
ID or context window.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused suites above; full
suite is still running)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-23 12:50:36 -07:00
DottaandPaperclip 30b753f614 perf(budgets): scope accounting and deliver enforcement asynchronously (#17046) 
## Thinking Path

> - Paperclip manages AI agents and their paid work.
> - Budgets must prevent new work and stop reported overruns.
> - Cost settlement also ran serial lifecycle reconciliation for every
eligible company agent.
> - That made request latency grow with unrelated idle agents and
delayed results on stop delivery.
> - This change separates atomic admission and accounting from durable
asynchronous enforcement.
> - Scoped live usage detects overruns without writing provisional
ledger charges.
> - The result is constant query scope for ordinary settlement and
recoverable stop delivery.

## Linked Issues or Issue Description

**What happened?**

Settlement and budget checks entered company-wide lifecycle
reconciliation. A company with 1,000 unrelated idle agents performed
1,001 accounting transactions for one ordinary cost evaluation. User
requests could also wait for provider termination. Running usage
checkpoints did not enforce reported monetary exposure.

**Expected behavior**

Keep the final reservation gate authoritative. Commit charges, scope
blocks, and enforcement intent before returning. Deliver stop requests
after commit. Read only applicable policies and execution scopes for
ordinary charges. Stop long-running work when supported reported or
priced usage establishes an overrun.

**Steps to reproduce**

1. Create a disposable company with one company budget and idle agents.
2. Evaluate a charge below the budget with 0, 17, and 1,000 unrelated
idle agents.
3. Compare query counts, company accounting transactions, and duration.
4. Use the controlled partial-usage and delayed-stop fixtures to check
enforcement and request responsiveness.

**Paperclip version or commit**

The measured baseline is `6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837` from
`master`.

Related public work: Refs #16976, #2006, #3539, #13763, #14296, #16853,
#13075. These cover the affected service charge, cancellation, and
accounting paths. This change removes their shared reconciliation
overhead.

## What Changed

- Batch applicable policy exposure under the existing company accounting
lock. Preserve exact fractional money and zero-reservation dispatch
fences.
- Commit a durable final budget dispatch claim before provider I/O.
Retain a separate final issue/run ownership transaction through
synchronous handoff, without holding the company accounting lock or
awaiting provider work. Preserve native recovery lease checks and
duplicate handoff fencing. Browser handoff rechecks current policy; it
reuses a consumed rate slot only for the identical policy generation.
- Separate ledger spend, live observed spend, and effective held
capacity. Use the maximum of reservation and live observation for each
unfinished operation.
- Deliver budget cancellation through the existing coordinator.
Revalidate generations and persist queued/running stop fences before
external termination. Bound stop concurrency to four.
- Apply agent holds only to the affected agent. Project budgets leave
other projects running. Project real company transitions in bounded
lifecycle batches.
- Replace periodic broad reconciliation with durable pending work,
startup repair, receipt replay, and concrete retry/window deadlines.
Retain drain and idle-safety checks.
- Enforce accepted cumulative usage checkpoints without provisional
charges. Final accounting atomically replaces live exposure and settles
reservations.
- Add operator-gated budget spans, regression coverage, map updates, and
a real-PostgreSQL comparison harness. No schema or first-party telemetry
change.

## Verification

- Full workspace `pnpm -r typecheck` passes on the final ownership fix,
rebased onto `bbcf5dfe98`. Production `pnpm build` passes on final head
`4c5f4de8b1`. Map, module-boundary, token, and Storybook catalog checks
pass.
- Focused real-PostgreSQL coverage includes the 292-case rebased
accounting/dispatch run, 63 browser cases, 52 run-control/cancellation
cases, 42 policy-generation cases, 77 lifecycle/commentary cases, and
the exact Muse confirmation regression. Native executor unit coverage
passes all 768 cases. Final ownership coverage passes: 74
adapter/use-case cases, nine public regression cases, and the additional
deleted-project race. These exercise competing ownership, budget changes
and charges between phases, zero/nonzero duplicate claims, rollback,
recovery contention, read-only capacity rechecks, and controlled HTTP
startup after commit. Another 74 pure/startup/tracing cases pass. The
complete CI matrix and both 5/5 reviews passed on `1ac9b58`; its
external security scan identified a stale-owner handoff race. The final
fix restores synchronous issue/run ownership fencing after the
accounting transaction commits. Final head `8e84f55ce8` has fresh
Greptile 5/5 with zero new comments, Paperclip review 5/5, and a passing
security scan. The complete latest-head [CI
matrix](https://github.com/paperclipai/paperclip/actions/runs/39535189808)
passes, including budget, browser, Runner, build, and canary jobs. The
unchanged Runner close fixture passed on its sole same-head job retry.
- Broad local `pnpm test:run` invocations were stopped before required
source updates (rebase and review fixes). They are not claimed as full
passes. Final full-suite coverage comes from the complete latest-head CI
matrix; local focused coverage and the incomplete broad invocations are
reported separately.
- Reproduce performance with `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-budget-enforcement.ts
--baseline-ref=6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837`; repeat with
`--reverse`. The harness uses disposable real PostgreSQL, 20 warm
samples per variant/count, and records raw timings, SQL/lock counts,
agent reads, unchanged agent state, and query plans. Final
production-source measurement revision:
`eff713a4f72c7f4b7a4002a701cb481d4bfccd95`; the extended harness is
committed in `4c5f4de8b1`.

| Unrelated idle agents | Baseline median / p95 | New median / p95 |
Baseline SQL / locks | New SQL / locks | New agent reads |
| --- | ---: | ---: | ---: | ---: | ---: |
| 0 | 1.80 / 2.33 ms | 2.99 / 3.88 ms | 5 / 1 | 7 / 1 | 0 |
| 17 | 15.53 / 19.55 ms | 2.24 / 2.60 ms | 90 / 18 | 7 / 1 | 0 |
| 1,000 | 978.35 / 1,790.43 ms | 2.16 / 2.52 ms | 5,005 / 1,001 | 7 / 1
| 0 |

- These measurements use local PostgreSQL 18.1 and Node 26.4.0 arm64.
They isolate the cost-evaluation core and complete admission path,
without hosted-region or provider latency. Prior-revision reversed-order
controls also preserved the constant query counts; the final
measurements above use the final production source.
- Complete reserve → final ownership gate → immediate offline provider
result is measured separately, with one responsible agent and one active
operation at each unrelated-agent count. Admission p95 baseline/new: 0
agents **11.54 / 29.77 ms**; 17 agents **9.31 / 27.62 ms**; 1,000 agents
**24.51 / 74.05 ms**. Baseline admission was already scoped; the
stronger final fence adds bounded overhead. New counts stay constant at
53 Drizzle statements, three transactions, two company-accounting locks,
and five agent-reading statements bound to the responsible agent. Zero
unrelated-agent reads, exactly one provider callback per sample, and no
leftover runs or holds. All admission scenarios stay below 250 ms. No
external provider calls are included.
- The partial-usage/delayed-termination fixture passed 20/20: receipt
persistence p95 13 ms, receipt return to observed durable Stop fence p95
28 ms, complete receipt-start-to-fence p95 50 ms, unrelated database
request p95 1 ms while termination was blocked. After releasing the
controlled termination fixture, acknowledgement p95 was 14 ms. These are
separately measured phases; fixture acknowledgement is not live-provider
termination latency.
- Models and provider deadlines are unchanged. No deployment, live
inference, or staging acceptance is claimed. A quiet 30-minute follow-up
waits for the merged revision on staging, then repeats classifier and
controlled budget-stop acceptance in staging, recording hosted
bookkeeping, queue delay, and actual provider acknowledgement
separately.
- Development failures retained: finalizer notification regressions
(fixed; 27/27 affected cases passed), invalid browser admission fixture
assumptions (fixed; browser coverage now passes 63/63), and first-head
lifecycle/dispatch CI regressions (reproduced and repaired). A
prior-head Cases E2E test hit its unchanged announcement-dismiss handler
race while opening a filter; the same journey passed on `1ac9b58`. The
external security review then found a stale-owner race between
accounting commit and provider entry; final verification covers a
competing run in that gap while retaining provider I/O outside the
accounting lock. Early PostgreSQL startup attempts hit local semaphore
pressure; serial reruns succeeded. The final-head Runner lane had one
failure among 1,823 tests: the unchanged `timeout-attachment` fixture
failed at its first process close after 6.28 seconds. The sole same-head
job retry passed all 1,819 active tests (four skipped), including that
fixture in 1.85 seconds. Three isolated local probes could not run
correctly because the Rust fake-provider fixture binary was absent;
cleanup masked the original startup failure. CI builds that fixture
through its normal lane. These probes are inconclusive and are not
counted as passes.

## Risks

- This changes accounting, lifecycle, and cancellation ordering. Durable
generations, source/sequence checks, ownership leases, and company
locking remain required for correctness.
- Provider stop acknowledgement can be slow or fail. Stop intent remains
pending until existing termination proof succeeds; cancelled row status
alone does not acknowledge delivery.
- Mid-run limits depend on provider usage reports or supported price
calculations. Providers that report only at completion cannot provide
exact mid-call detection.
- Signals are process-local wake hints for the supported
single-active-writer model. Durable state and startup recovery remain
authoritative.
- There are no migrations, new queue tables, or cached balances. Repeat
the classifier and controlled budget-stop acceptance after normal
rollout.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, code execution,
and parallel coding agents. The host does not expose the exact backend
model revision or context-window size, so those values are not asserted.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-21 07:23:00 -05:00
DottaandPaperclip 505e1a5582 feat(cases): add persistent workflows with same-task agent handoffs (#16959) 
## Thinking Path

> - Paperclip manages AI agents and their work.
> - An agent workflow needs persistent facts and a safe way to transfer
work.
> - Creating a task for each phase splits the objective, discussion,
workspace, and review policy.
> - A case can retain the facts while one primary task moves between
agents.
> - Transfers must survive process failure and preserve private access,
budgets, and completion gates.
> - This pull request adds durable case handoffs, native case tools, and
read-only case pages.
> - Prompts describe the workflow. Existing task controls decide when
work can run and finish.

## Linked Issues or Issue Description

**Problem or motivation**

Prompt-driven agent teams need a persistent record that can move through
triage, implementation, verification, and correction. Ordinary
reassignment does not provide a durable instruction, safe release
barrier, or recoverable delivery receipt for an agent handing off its
own running task.

**Proposed solution**

Extend standalone cases with one primary task, a descriptive phase,
retained access sources, and a handoff ledger. Apply assignments after
source execution and workspace settlement. Deliver each transfer through
the existing durable wake queue. Provide `create_case`, `read_case`,
`update_case`, and `handoff_case` in the native runner. Creation
requires kickoff and explicit workflow text. An explicit handoff to
`done` closes the case when normal task completion is accepted.

**Alternatives considered**

Do not add a workflow engine, filesystem queue, configured state graph,
or one task per phase. The workflow document combines transition prose
with an optional Mermaid `stateDiagram-v2` map. The server does not
execute the diagram. The Pipelines experiment removed on master stays
removed.

**Roadmap alignment**

This extends Agent Reviews and Approvals, Enforced Outcomes, Bounded Run
Recovery, and Activity Log attribution. Existing approval, review,
budget, privacy, workspace, and responsible-user rules remain
authoritative.

Related work: [ordinary handoff wake
recovery](https://github.com/paperclipai/paperclip/pull/15019). This
feature adds a separate persisted case-handoff receipt and release
barrier.

## What Changed

- Add case routing fields, retained access sources, durable handoff
history, delivery leases, and replay-safe migrations.
- Keep the same primary task across agent phases. Preserve each
returning agent's own session and the task's workspace.
- Fence outgoing mutations, wait for provider and workspace release, and
recover delivery without duplicate task comments or runs. Recheck
current assignment permissions as the original caller before applying a
delayed transfer. Scope a run Stop to that run’s own handoff intents, so
repeated old Stop requests preserve newer work. Cancelling terminal
intent without a source run also requires fenced current task ownership,
including native retries and preparation with an old checkout. For Cloud
callers, revalidate current stack membership with an uncached Cloud RPC
and derive owner elevation from the current managed setting.
- Keep case reads, documents, assets, Audit, and live events inside
retained task/project permissions.
- Add native case creation/read/update/handoff tools. Store required
workflow text and initial documents on the first task. Page oversized
case facts and referenced document bodies through the dedicated tool
with bounded UTF-8 JSON, current routing versions, snapshot conflicts,
and current source authorization on every page.
- Add read-only case lists and detail pages, task-style list controls,
compact dotted field tables, linked handoff tasks, and case filters in
Audit.
- Add native/legacy integration tests, migration/restart tests, browser
coverage, and bounded live receipt fixtures. Extend the trusted sandbox
condition to cover the new subscription fixture profile.
- Hide obsolete paused-agent warnings on completed or cancelled task
threads.
- Synchronize the company-switch settings test with React effects after
a CI failure. Preserve its security assertions and production remount
behavior.

## Verification

The branch directly includes master `9ef434fe79`. The required `ci /
verify` and `ci / e2e` gates pass on head `765fccd67c`. [Current-head
CI](https://github.com/paperclipai/paperclip/actions/runs/39464348614)
passes typecheck, build, tests, and all eight browser shards. Security
checks pass. [Greptile gives this exact head
5/5](https://github.com/paperclipai/paperclip/pull/16959#issuecomment-6291156549),
with no unresolved review threads. The optional Paperclip Review check
remains queued because its assigned agent is paused. It is not a
required branch check. No check result was overridden.

Local workspace typecheck and build pass after the master merge, along
with database migration checks, token gates, map checks, Storybook
catalog checks, and the Copilot profile identity check. Standard builds
refresh Runner vendor output. Focused checks pass 2,008
API/paging/context tests and 7 strict case protocol tests. The final
commit changes one test fixture only; its ten tests, UI typecheck, and
token gates pass. No production source changed in that commit.

On current head `765fccd67c`, all **194 critical workflow tests**
executed and passed with no skipped tests in those files:

- 38 Stop/ownership tests, including board completion intent, native
retries, stale Stop, and preparation with an old checkout.
- 2 actual legacy process/HTTP handoff tests, including scratch failure,
a closed-stdio child that continues writing, and recovery through a
fresh worker after the process group stops.
- 5 migration tests, including SQL column metadata, preserved null
authority context, and all six prior-numbering receipts.
- 3 process-settlement and 5 scratch tests.
- 88 handoff-service, 18 heartbeat handoff, and 21 native case-tool
integration tests.
- 7 bounded case/document-read projection tests and 7 strict
protocol-contract tests.

The current CI checkout was `fed65d49`, merging `765fccd67c` into master
`9ef434fe79`. Its new document regressions reconstruct exact
Unicode/escaped bodies and revisions through real native authority and
authenticated HTTP with generic API tools disabled. They cover
case-owned, primary-task, and linked-task documents; routing on every
page; stale document snapshots; access loss and unlinking;
unrelated/foreign/private sources; system documents; and quarantined
content. Active-task document tools retain their existing behavior.
These are current-head test results from the first CI attempt; the
browser-only retry did not rerun these suites.

The full `64fdd3403f` CI failed one separate settings test: after a
company switch, it observed an unsettled API-key draft. The production
component and test were unchanged from prior green source. Production
switches remount both keyed settings layers; the test directly reused an
unkeyed view and used a timer tick instead of waiting for React effects.
The final commit replaces that fixture synchronization with React `act`.
The exact secret-clearing assertions and all timeouts remain unchanged.
The original assertion failure remains recorded; no unchanged retry was
used. The narrow unchanged baseline and fixed fixture each pass 10
tests.

The upstream result-paging merge preserves case capture namespaces and
current asset privacy. Its API integration suite executed 39 of 41 tests
successfully, with two skipped; the optional assigned-result PRP
integration test was skipped. Neither skipped suite is counted in the
194 critical tests or as live provider proof.

All 361 upstream migration entries and SQL bodies remain unchanged. Six
case migrations follow master as 0363–0368 with identical SQL hashes.
Official Drizzle generators produced the combined metadata and five
retained snapshots. Generation reports no schema changes, and the
snapshot drift test passes. The earlier `ce361ae5d0` CI passed on
attempt 1, including its then-current 179 critical tests. Fresh review
later identified unusable document references with generic API tools
disabled; the bounded document reads and ten additional authenticated
HTTP tests address that finding.

The full local `pnpm test:run` attempt was cancelled after this Mac
exhausted PostgreSQL semaphore capacity; it is **not** counted as a
pass. Later database attempts also stopped before assertions or skipped,
including one sandbox socket denial. Two original stale-Stop regressions
did reproduce the bug against the old source. A REST serialization
assertion passed after two preserved database-startup failures. No test
timeout was increased. All 194 assertions listed above executed and
passed in current-head CI. The complete 385-file, 3,963-test UI shard
also passed after the fixture repair.

A configured-provider receipt workflow passed 13 backend checks across
triage → implementation → verification → correction → done, using one
task and four work runs. The saved receipt is revision 2 with total 6
USD; case and task both closed. The original canonical attempt remains
failed because its browser assertion confused the terminal destination
with completed-task tooltips. The corrected full browser verifier passed
on the same retained database and original runs, with no additional
provider calls. The last clean-build browser readback is on
`6a86c54a53`; later changes affect recovery, authorization, bounded
reads, and settlement. Accounted work cost is a $1.010574 rate-card
estimate; provider billing and setup-probe usage are unavailable.

Copilot profile 49 passed the unchanged seven-fixture private protocol
pack (46/46 checks, no retries) and the product file-edit canary after
master's completion-replay change. All 60 bound implementation hashes
remain unchanged on the current source. Historical profile-47/48
identities and results remain intact. See the [profile-49 compatibility
record](https://github.com/paperclipai/paperclip/blob/codex/persistent-case-handoffs/doc/plans/2026-10-20-copilot-completion-recovery-compatibility-v49.json).
This is bounded compatibility evidence, not a fresh full matrix,
installed-package product test, or Daytona rollout.

Preserved CI failures and retries:

- On `20662930ac`, Slack managed setup failed a five-second success
assertion after reload. Its test/application path was unchanged from a
prior green head. One unchanged-source retry passed all 25 tests in that
shard; no source or timeout changed.
- On `9402b4f17a`, two CI workers received shutdown signals at 15:24:19
UTC. Neither log had an assertion failure before interruption. One
unchanged-source retry of each passed all Runner static checks and all
32 tests in browser shard 6; both aggregates passed. The next head,
`7b4374edaf`, passed all CI without retries.
- On `295f289b5a`, serialized shard 9 timed out during its first
device-login test, then the next test found an active login lease.
Direct login/test paths were unchanged from a prior green head. Cold
import plus a timed-out async body using the next test's mutable
actor/store is the supported fixture-race explanation; no request trace
proves that sequence. The original failure remains recorded. One
unchanged-source retry passed all 294 tests in 19 files, including all
41 device-login tests. No login source or timeout changed. The
subsequent `ce361ae5d0` CI passed without retries.
- On `64fdd3403f`, one company-switch settings assertion observed an
unsettled draft. The final test-only commit uses React `act` without
changing assertions, timeouts, or production code. On `765fccd67c`, all
ten settings tests and the complete 385-file, 3,963-test UI shard
passed.
- On `765fccd67c`, browser shard 3 completed the split-handoff
assertions, then cleanup received HTTP 500 from run cancellation after a
one-second PostgreSQL row-lock timeout. The extra chat run was reporting
completed work while teardown began. Cleanup stopped before restoring a
global experiment flag, causing a later assertion to fail. Relevant
production and test paths are unchanged from an earlier green commit.
Stop's lock order is consistent, and no nested self-lock was found; the
exact PostgreSQL blocker was not recorded. One unchanged-shard retry
passed. The original failure, trace, and downstream flag failure remain
recorded. No Stop implementation, cleanup assertion, or timeout changed.
The retry supports intermittent cleanup contention; it does not identify
the original blocking transaction.


## Risks

- Local process transfers require host-owned PID/group absence. Missing
or uncertain termination evidence keeps delivery pending, independently
of scratch setup.
- Handoff admission and task cancellation share execution ownership. New
regression tests cover temporary holds, late source writes, and
uncertain provider termination.
- Retained private sources use an intersection of existing permissions.
This intentionally fails closed when a source is missing or an existing
task cannot preserve that boundary.
- Migrations add tables and indexes. They preserve master history and
can replay over prior feature installs. Migration 0368 adds nullable
internal authorization provenance with no default or authority backfill.
Historical transfers still awaiting release without this context fail
closed; an operator must cancel them and request a new handoff.
- Delayed Cloud handoffs depend on the current Cloud membership RPC. An
unavailable, ambiguous, expired, or mismatched response holds the
handoff for retry; it does not reuse cached authority or grant access.
- Bounded reads reload and reauthorize the selected projection per page.
Underlying work grows with the data size and number of pages. Changed
data or routing requires restarting from offset zero without the old
snapshot.
- Phase names and workflow diagrams remain advisory. They do not satisfy
review or approval gates.
- This is a large, cross-cutting PR. The maintainer requested one PR for
the whole feature; the commits separate data, runtime, UI, and
qualification work.

## Model Used

OpenAI Codex, GPT-6 family, with tool use, code execution, and delegated
code review. The exact deployment model ID and context-window size were
not exposed in the session. Live fixture agents used the explicitly
configured provider model recorded in their test evidence.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-20 12:46:14 -05:00
DottaandPaperclip 556daf9a99 feat(dot): run cloud assignments in managed sandboxes (#15691) 
## Thinking Path

> - Paperclip manages agents and governs their work.
> - The Runner owns execution and task authority.
> - Dot executes at OpenAI and uses Paperclip through MCP.
> - Dot previously required a self-hosted controller.
> - Cloud must admit Dot work in a managed sandbox and keep protocol
traffic in the tenant.
> - This pull request adds managed Dot execution and dedicated pairing
routes.
> - The same Runner path can later host sandboxed workspace tools.

## Linked Issues or Issue Description

**Agent or provider**

OpenAI Dot, through the existing dedicated MCP adapter. Related changes:
#15402, #15414, and #15581.

**Why this adapter is useful**

Cloud users need to assign work to Dot and let Dot request work while
idle. Neither path should execute agent commands on the control-plane
host or require an idle sandbox.

**How the agent is invoked**

The tenant serves OAuth, signed mailbox events, and scoped tools. Normal
admission creates a managed sandbox for each assignment. A verified Rust
Runner owns that assignment and relays its tools and results.

## What Changed

- Enable the external-agent invitation in Cloud behind the standalone
Dot flag.
- Launch Dot through the existing managed process launcher and
authenticated PRP transport.
- Verify the artifact digest and Dot capability before dispatch. Read
provider checkpoints from the execution target for recovery.
- Reject local execution in Cloud and reject managed workspace access
for this release.
- Add dedicated public Dot pairing aliases. Preserve the personal
assistant consent paths.
- Keep Dot events in the tenant when a personal Cloud broker is
configured.
- Use the app-configured heartbeat for Dot-initiated work. Report
terminal admission failures and fence stranded offers before new
admission.
- Offer normal comments through the active assignment's mailbox. Keep
their wakes deferred until a successful history receipt contains the
exact IDs; preserve unread comments through finish, batching and
recovery without replaying consumed input.
- Require a sandbox process fingerprint before dispatch and allow a
bounded five-minute controller reconnect gap. Preserve remote process
birth without inspecting a colliding controller PID.
- Recognize authenticated remote shutdown receipts even without an OS
exit code.
- Document dedicated pairing aliases and teach route coverage to inspect
literal path arrays.
- Add opt-in private Daytona qualification and focused contract tests.

## Verification

- Workspace typecheck, production build, and UI token gates passed.
- Focused onboarding, OAuth, managed environment, artifact metadata, and
terminal-fence tests passed. One environment-test timeout passed on an
isolated repeat.
- A private Daytona fixture passed with the real Rust Runner, signed
readiness, assigned app tools, document writes, duplicate receipts,
authority fences, completion, and sandbox deletion.
- A real OpenAI Dot completed both an operator-assigned task and a
Dot-initiated task in managed Daytona sandboxes. Both leases were
released.
- A real accepted Dot assignment survived a controller gap longer than
one minute with the same sandbox process, Runner, session, and
assignment. Dot consumed the normal task-comment follow-up and completed
exactly once; the lease was released.
- A real cancellation fenced an accepted assignment, released its lease,
and received Dot's acknowledgement without claiming a global provider
stop.
- Final local checks: 74 Dot execution, OAuth onboarding and API
contract tests after integrating the current agent-lifecycle changes, 9
real Rust driver tests, 21 remote-runtime tests plus the 3
detached-process cleanup tests, and 17 lifecycle tests passed. The final
live Daytona fixture passed and deleted its sandbox.
- A final real Dot follow-up stayed pending until its exact comment
appeared in a successful history receipt. Dot completed the original
managed assignment once; its lease was released.
- The broad local suite crossed a rebase and was stopped; its failures
are not a qualification of this final source. Focused repeats passed.
The final head passed all CI gates: workspace types, build, general and
serialized test suites, Rust and Runner verification, all browser
shards, and the canary dry run. Greptile is 5/5 with both findings
resolved.

## Risks

- Deploy the tenant app with its current Linux Runner artifact and the
corresponding front-door protocol routes together.
- This branch has not been deployed to a Cloud tenant. A deployed tenant
still needs an OAuth and assignment smoke test.
- Dot provider usage and cost are unmetered. Revocation fences Paperclip
tools and does not prove a global OpenAI stop.
- Missing or changed recovery identity fails closed. No replacement Dot
thread is invented.
- No schema migration is added. Managed Dot workspace access remains
unavailable.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, code execution,
and browser testing. The exact runtime model identifier and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-09 12:58:35 -05:00
DottaandPaperclip 0b80ea17ef refactor(heartbeat): extract run lifecycle handling (#15672) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The heartbeat service admits, executes, and settles agent runs.
> - Earlier extractions separated workspaces, preparation, state,
retries, recovery, and queue admission.
> - Run status, progress, liveness, and completion handling still live
in the main service.
> - These operations share status predicates, event writes, and
completion policies.
> - This pull request moves them into one lifecycle module and adds
boundary tests.
> - Reviewers can inspect lifecycle policy separately from adapter
execution.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The structure and testability of heartbeat run lifecycle handling.

**Subsystem affected**

server/ — orchestration services.

**Current behavior**

Run status writes, progress, events, liveness, completion handoffs,
issue-comment finalization, and runtime settlement live inside
`heartbeat.ts`.

**Proposed behavior**

Keep these operations in
`server/src/services/heartbeat/run-lifecycle.ts`. Bind the database and
reporting, recovery, and wakeup callbacks with
`createHeartbeatLifecycle`.

**Reason and benefit**

This removes 2,682 lines from `heartbeat.ts`, leaving 11,558 lines. The
lifecycle module has 2,982 lines and keeps related policy together in
one file.

**Breaking changes**

None. Existing function bodies, service methods, public exports, status
conditions, and side effects keep their behavior.

**Additional context**

Continues #15626 and the earlier merged heartbeat extractions. A search
of open and closed issues and PRs found no duplicate lifecycle
extraction. This does not add a roadmap feature.

## What Changed

- Move status transitions, run events and progress, liveness
classification, completion handoffs, plan-resume failure reporting,
issue-comment finalization, and runtime/cost settlement into
`heartbeat/run-lifecycle.ts`.
- Supply reporting, recovery, and admission callbacks through an
explicit dependency interface. Preserve construction order with
forwarding callbacks.
- Keep adapter execution, cancellation, and terminal telemetry reporting
in the main service.
- Add 16 tests for inert construction, continuation gates, late
progress, status races, native ownership, usage metadata, revoked wakes,
event sequencing, and redaction.
- Update the existing publication-order source assertion to read the
extracted event writer. Keep its persist-before-publish check.
- Document the lifecycle module boundary in `doc/DEVELOPING.md`.

## Verification

- Before extraction: 87 tests passed across seven existing suites.
- After extraction: 103 tests passed across eight files, including the
16 new module tests and real PostgreSQL coverage.
  ```sh
pnpm exec vitest run server/src/services/heartbeat/run-lifecycle.test.ts
server/src/__tests__/heartbeat-run-status-payload.test.ts
server/src/__tests__/heartbeat-run-event-sequencing.test.ts
server/src/__tests__/heartbeat-run-terminalize-before-release.test.ts
server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts
server/src/__tests__/heartbeat-cost-accounting.test.ts
server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts
server/src/__tests__/run-liveness.test.ts
  ```
- Additional recovery, cancellation, plan-resume, summary, and handoff
coverage: **595 tests passed across 14 files**. The module suite appears
in both runs. With the publication-order suite below, combined focused
coverage is **694 distinct tests across 22 files**, with no skips.
  ```sh
pnpm exec vitest run server/src/services/heartbeat/run-lifecycle.test.ts
server/src/services/heartbeat/recovery.test.ts
server/src/services/heartbeat/retries.test.ts
server/src/services/heartbeat/queue.test.ts
server/src/services/recovery/successful-run-handoff.test.ts
server/src/services/recovery/review-path-recovery.test.ts
server/src/services/heartbeat-run-runtime-status.test.ts
server/src/__tests__/heartbeat-process-recovery.test.ts
server/src/__tests__/heartbeat-native-runner-cancellation.test.ts
server/src/__tests__/heartbeat-native-cleanup-admission.test.ts
server/src/__tests__/heartbeat-accepted-plan-workspace-refresh.test.ts
server/src/__tests__/heartbeat-runtime-state.test.ts
server/src/__tests__/heartbeat-run-summary.test.ts
server/src/__tests__/heartbeat-context-summary.test.ts
  ```
- The publication-order and lifecycle suites passed together: **28 tests
across two files**. The first CI run exposed an assertion that still
read the old event-writer source path. That assertion now reads the
lifecycle module.
  ```sh
pnpm exec vitest run
server/src/services/chat-publication-reconciliation.test.ts
server/src/services/heartbeat/run-lifecycle.test.ts
  ```
- After resolving the import conflict with the Pi Runner integration and
rebasing onto `57e977be7`: **44 tests passed across four files**,
including the updated source assertion, lifecycle module, run-status
payloads, and accepted-plan workspace refresh.
  ```sh
pnpm exec vitest run
server/src/services/chat-publication-reconciliation.test.ts
server/src/services/heartbeat/run-lifecycle.test.ts
server/src/__tests__/heartbeat-run-status-payload.test.ts
server/src/__tests__/heartbeat-accepted-plan-workspace-refresh.test.ts
  ```
- Full `pnpm -r typecheck` and `pnpm build` passed on final head
`bca87df089c541450932f6b678ca776942d267b2`. Full local `pnpm test:run`
was restarted on this head and stopped after the entire CI suite passed.
It did not finish and is not counted as a full local pass.
- The earlier full local run had one transient public MCP Cloud
bootstrap failure. All **82 public MCP tests passed in isolation**. The
first local run was stopped before the source-assertion fix and rebase.
- All **54 latest-head checks passed**; two optional Storybook checks
were skipped. One general-test shard passed all 1,314 tests but failed
during runner temporary-directory cleanup with `EACCES`. One rerun
passed. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/37940997283).
- Greptile completed on final head
`bca87df089c541450932f6b678ca776942d267b2` with **5/5 and no actionable
findings**. There are no review threads or merge conflicts.
- Structural comparison confirms 63 module function bodies and ten
declarations match the originals. Remaining service function bodies and
all 140 public exports are preserved.

## Risks

Moving closures can break callback construction order or status
settlement. The factory receives explicit callbacks and starts no work
during construction. Compare-and-set predicates, native ownership holds,
revoked-wake guards, event sequencing, redaction, and completion
policies keep their original bodies. This PR has no schema or API
changes.

## Model Used

OpenAI GPT-6 via Codex. The exact serving model ID and context window
were not exposed in this session. Used reasoning, repository inspection,
shell tools, and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-09 09:26:45 -05:00