mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-31 22:24:16 +01:00
f53db0c70fe13db4471f253a7ddfee138cb6e393
15
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
50fe32684d |
fix(server): wake committed native run projection from durable changes (#17752)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run has separate result acceptance, task decision, run status, and resource settlement steps. > - A task decision can commit before its run status is saved. > - Recovery currently revisits committed decisions through the periodic orphan pass. > - This PR gives the existing run projection repair its own transaction-driven worker. > - Existing records reconstruct unfinished work after restart. Completed history becomes quiet. > - Other completion and resource obligations keep their existing recovery paths. ## Linked Issues or Issue Description Refs #17748 (stack parent: accepted-result commit), #17734 (workspace finalization), and #17717 (retained session cleanup). Related #17614 preserves terminal notifications. This slice is prepared on current parent #17748 at `b2efdb2dbbbef5b2227ffa2bd2a9a6cfe3867fe2`; the parent has merged as `28e38da7e6bdd6af4092dda56bc2be05bed47af5`. **Current behavior** The native recovery pass revisits committed decisions to repair the heartbeat-run status and metadata. A crash between decision and run writes creates real work, but periodic scanning also revisits retained history. **Proposed behavior** Wake projection repair after committed producer changes. Reconstruct mismatches on startup and settings activation. Retry concrete failures through the shared scheduler. Keep the run projection before resource settlement. **Reason and benefit** The repair worker performs no recurring SQL after its work settles. A real decision, effect settlement, or ownership release reactivates it. This is one worker migration; it does not establish host or database suspension. ## What Changed - Extract the existing projection predicate and write into their native-runtime owner. The finalizer keeps initial metadata enrichment and task decision authority. - Discover accepted, committed, mismatched run projections with company, issue and run bindings. Exclude pending status effects and held runner ownership. Read narrow fields instead of whole result payloads. - Use existing transaction hints, scoped delivery scheduling, failure backoff and lifecycle handling. Add no tables or per-feature timers. - Require issue/run scope for ordinary finalization fanout. Routine company-only hints leave dormant projection quiet; explicit settings and dedicated producer signals keep their existing contract. - Connect finalization, pending effects, lifecycle status writes, issue changes and worktree execution activation. Preserve outer transaction commit and rollback behavior. - Lock the current run before deciding which terminal event and completion time to save. Concurrent live and recovery projection must not announce the same transition twice. - Delegate only already-committed projection from the periodic pass. Its candidate scan uses the same pending predicate to defer remaining consequences and cleanup until projection finishes. - Add real PostgreSQL producer, race and lifecycle tests plus an owner audit. ## Verification - Current local source `a83b9449233a275176e637c765059f414305d7c1` incorporates the parent with a normal merge. All 1,208 parent-only, seven owned-only and ten automatic shared paths are preserved. The only manual conflict keeps new projection fanout inside the ordinary issue/run guard. The complete package tree matches parent profile 80 byte-for-byte. - Controlled real-database regression: the original projection-only unscoped fanout produces three candidate SELECTs for routine company hints. The same zero-query assertion passes with scoped fanout. A real committed result then repairs the run and emits one terminal event before returning to quiet. Production bytes restore exactly in both phases. Twenty cases outside that focused pattern are not counted as passing there. - The complete main command passes **1,830 cases in 19 files**, without skips, in 301.14 seconds. It includes all 21 projection cases, 85 startup cases, required quality review and heartbeat run lifecycle, plus physical-work and result-commit consumers. Separate process recovery passes 410 in 163.58 seconds and external-chat response passes 273 in 53.26 seconds. Total: **2,513 distinct passes in 21 complete files**, with no skips. Normal merge `d4b46fd29cb23126d556cc3aeb883e6b41f1a8a5` adopts the actual parent squash with this entire tested child tree unchanged. - Production server typecheck, build, module boundaries, migration order and source identity checks pass. Seventeen dedicated PostgreSQL 17 databases use the unchanged migration 413 schema. Embedded-only calls use the existing fresh-per-call recovery helper. All temporary imports restore byte-exact and all 21 selected test hashes match after completion. No assertion or timeout is relaxed. - Parent package acceptance remains applicable: 130 passing installation/compatibility/transport cases, 22 installed ACPX/Grok contract cases and 26 native-provider Rust cases. One pre-existing opt-in closed Pi snapshot case is skipped because its distribution root is not configured. No live provider qualification is claimed. - The owner audit at `doc/plans/2026-10-26-native-run-projection-sleep-audit.md` retains prior failures and their disposition. Current-head acceptance: all 57 checks are clean, including protected CI and security. Greptile gives 5/5 on `a49a2dac13b0e1c19da5d3d6b435c781703b1c0b`, with no actionable findings. Old scores do not substitute. No Ponytail verdict is present under the authorized missing-review policy. - These checks do not claim a full local repository suite, deployed suspension, expanded Cloud wake coverage or measured cost savings. ## Risks - Process-local signals cover instrumented application writers. Independent SQL writers and replicas without supported ingress are not newly covered. Durable startup reconstruction remains enabled. - Projection must precede remaining committed resource settlement. The real reconciler test proves the periodic path defers cleanup while projection is pending and resumes it after repair. - Status decisions, external conversation settlement, watchdog resolution, Muse reconciliation and resource release keep their own authority and recovery. A committed result alone is not proof of completed cleanup. - Candidate discovery uses narrow bounded pages. There is no large-fleet performance or deployed sleep claim. Existing Cloud wake coverage and sleep blockers remain unchanged. - Parent #17748 has merged. This PR targets master at `28e38da7e6bdd6af4092dda56bc2be05bed47af5`. Current-head CI and review must pass before the authorized protected merge. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The exact serving model ID and context-window size are not exposed in this session. No subagent was used for this slice. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
DottaandPaperclip
|
8252974141 |
fix(server): schedule stopped instruction collection (#17674)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runs can stop while their private instruction edits still need collection. > - Existing copy rows and stop evidence describe that unfinished work. > - The orphan reaper still queries for it on every maintenance pass. > - This PR activates collection from committed copy and run transitions. > - It uses saved retry deadlines when stop proof or collection remains pending. > - Active copies and completed history no longer cause collection polling. ## Linked Issues or Issue Description Refs #17671 and #17670. Both prerequisites are merged into master. Captured saves and post-save release remain separate reasons with their own workers. **What needs to be improved?** `recoverStopped` queries instruction copies on every orphan-reaper pass, even when no terminal run owes collection. A run can become terminal without changing its copy row, so copy-only notifications would miss real work. **Why is this improvement important?** Repeated empty queries prevent idle tenant instances from becoming quiet. Removing them must preserve edits, stop proof, current ownership, and recovery after restart. **Proposed solution** Use the existing transaction signals, scoped deadline worker, and scheduler. Reconstruct eligible copies at startup, signal supported copy and terminal-run writers, and retain saved retry deadlines for unresolved work. Preserve the existing collection and directory owners. Related PR search found the warm agent-file ownership work (#14695), verified stop boundary (#14311), and task-workspace cleanup (#16464). This PR does not replace those contracts. ROADMAP.md has no duplicate migration. ## What Changed - Add scoped stopped-instruction collection with keyset reconstruction and fair batches of 20. - Signal before copy preparation, mutation and adoption writes, and before terminal run writes in their existing transactions. - Cover heartbeat, native finalization/restart/session recovery, execution-control, queue and dispatch producers. - Signal settings changes and company/agent deletion. Preserve drain, standby, restart and shutdown ownership. - Recheck terminal-run eligibility in the due-time claim. Keep terminal status separate from process-stop proof. - Retain existing stop checks, attempt limits, saved candidate bytes and remote-resource fencing. - Skip only this lane in periodic orphan recovery. Preserve manual/startup recovery. - Add concurrent, replay-safe partial index 0408 and retain the five newest snapshots. Add no table. - Record the owner and producer audit in `doc/plans/2026-10-25-stopped-instruction-collection-sleep-audit.md`. ## Verification - Source `cb2818a8f46ef961f744977a258f0e9042e31a42`: all 1469 cases across 14 complete files pass without skips in 141.69 seconds. This includes all 968 native-session cases, filesystem saves, startup, terminal-run producers, dispatch, queue, execution-control, finalization and lifecycle. - Eleven separate fresh PostgreSQL 17 databases migrated through 0408. The embedded-only lifecycle fixture temporarily used its own external database on this IPC-exhausted host and was restored byte for byte. Assertions and timing limits stayed unchanged. This qualifies external-database behavior; it does not claim an unmodified embedded CI run. - Database build, server typecheck/build, module boundaries and exact-current migration order pass. The original concurrent index SQL is unchanged. All prior executable migration history, journal entries and retained snapshot bytes are preserved; the five newest snapshots are 0404 through 0408. - A populated 0407-to0408 upgrade preserves all 406 prior migration receipts and 38 companies. It appends exactly one receipt. Replay adds none. The collection index is valid and ready on fresh and upgraded databases. - The normal merge of actual parent `267842fc3c6e6542deed8a1bcc2c78bec8df7862` preserves both complete patches, all independent sidebar changes from #18117, and the full accepted server/package/test trees. The complete sidebar test file passes all 36 cases on the combined tree. React act warnings remain. - Prior failures and fixture qualifications remain in the audit. Published head `d176d731f083c7edd2ede688cb901d1e66d8f1d8` has Greptile 5/5, all 57 checks passing or skipped, and no unresolved review threads. No Ponytail review is present. These local tests do not establish deployed host/database suspension or Cloud alarm delivery. ## Risks - Process-local signals cover instrumented writers. Independent SQL writers and additional replicas still need integration. - Terminal-run signals can cause one scoped inspection even if no instruction copy exists. They do not create a recurring scan. - Reconstruction and memory scale with actual pending candidates; active candidates are filtered by the run join. - Admitted file/provider operations can outlive a later drain request. Shutdown joins them. Uncertain ownership remains pending rather than being discarded. - The partial index adds storage and write cost. Its concurrent migration uses the existing interrupted-index repair path. - Cloud alarms, sleep eligibility and provider suspension are unchanged. Worker quiet is not a claim of deployed host/database sleep or measured savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL/filesystem tests. The runtime does not expose a more precise model ID or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
ef0a8db502 |
perf(native): narrow reconciliation candidate payloads (#17531)
Select the complete workspace-sync record and native status wake identifier for reconciliation candidates. Preserve candidate policy, JSON types, full later evidence, and stored data. The committed replay regression includes the real reconciliation scan. Full build, typecheck, 114 focused tests, complete CI, and exact-head Greptile5/5 pass. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
64a8b927c1 |
fix(native): narrow committed finalization and cleanup reads (#17525)
Select fresh metadata for committed finalization and cleanup. Preserve full saved evidence for new finalization, cancelled workspace retention, and legacy Stop. PostgreSQL byte-budget and unchanged-row regressions pass, full CI is green, and Greptile scored the exact head 5/5. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
30b753f614 |
perf(budgets): scope accounting and deliver enforcement asynchronously (#17046)
## Thinking Path > - Paperclip manages AI agents and their paid work. > - Budgets must prevent new work and stop reported overruns. > - Cost settlement also ran serial lifecycle reconciliation for every eligible company agent. > - That made request latency grow with unrelated idle agents and delayed results on stop delivery. > - This change separates atomic admission and accounting from durable asynchronous enforcement. > - Scoped live usage detects overruns without writing provisional ledger charges. > - The result is constant query scope for ordinary settlement and recoverable stop delivery. ## Linked Issues or Issue Description **What happened?** Settlement and budget checks entered company-wide lifecycle reconciliation. A company with 1,000 unrelated idle agents performed 1,001 accounting transactions for one ordinary cost evaluation. User requests could also wait for provider termination. Running usage checkpoints did not enforce reported monetary exposure. **Expected behavior** Keep the final reservation gate authoritative. Commit charges, scope blocks, and enforcement intent before returning. Deliver stop requests after commit. Read only applicable policies and execution scopes for ordinary charges. Stop long-running work when supported reported or priced usage establishes an overrun. **Steps to reproduce** 1. Create a disposable company with one company budget and idle agents. 2. Evaluate a charge below the budget with 0, 17, and 1,000 unrelated idle agents. 3. Compare query counts, company accounting transactions, and duration. 4. Use the controlled partial-usage and delayed-stop fixtures to check enforcement and request responsiveness. **Paperclip version or commit** The measured baseline is `6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837` from `master`. Related public work: Refs #16976, #2006, #3539, #13763, #14296, #16853, #13075. These cover the affected service charge, cancellation, and accounting paths. This change removes their shared reconciliation overhead. ## What Changed - Batch applicable policy exposure under the existing company accounting lock. Preserve exact fractional money and zero-reservation dispatch fences. - Commit a durable final budget dispatch claim before provider I/O. Retain a separate final issue/run ownership transaction through synchronous handoff, without holding the company accounting lock or awaiting provider work. Preserve native recovery lease checks and duplicate handoff fencing. Browser handoff rechecks current policy; it reuses a consumed rate slot only for the identical policy generation. - Separate ledger spend, live observed spend, and effective held capacity. Use the maximum of reservation and live observation for each unfinished operation. - Deliver budget cancellation through the existing coordinator. Revalidate generations and persist queued/running stop fences before external termination. Bound stop concurrency to four. - Apply agent holds only to the affected agent. Project budgets leave other projects running. Project real company transitions in bounded lifecycle batches. - Replace periodic broad reconciliation with durable pending work, startup repair, receipt replay, and concrete retry/window deadlines. Retain drain and idle-safety checks. - Enforce accepted cumulative usage checkpoints without provisional charges. Final accounting atomically replaces live exposure and settles reservations. - Add operator-gated budget spans, regression coverage, map updates, and a real-PostgreSQL comparison harness. No schema or first-party telemetry change. ## Verification - Full workspace `pnpm -r typecheck` passes on the final ownership fix, rebased onto `bbcf5dfe98`. Production `pnpm build` passes on final head `4c5f4de8b1`. Map, module-boundary, token, and Storybook catalog checks pass. - Focused real-PostgreSQL coverage includes the 292-case rebased accounting/dispatch run, 63 browser cases, 52 run-control/cancellation cases, 42 policy-generation cases, 77 lifecycle/commentary cases, and the exact Muse confirmation regression. Native executor unit coverage passes all 768 cases. Final ownership coverage passes: 74 adapter/use-case cases, nine public regression cases, and the additional deleted-project race. These exercise competing ownership, budget changes and charges between phases, zero/nonzero duplicate claims, rollback, recovery contention, read-only capacity rechecks, and controlled HTTP startup after commit. Another 74 pure/startup/tracing cases pass. The complete CI matrix and both 5/5 reviews passed on `1ac9b58`; its external security scan identified a stale-owner handoff race. The final fix restores synchronous issue/run ownership fencing after the accounting transaction commits. Final head `8e84f55ce8` has fresh Greptile 5/5 with zero new comments, Paperclip review 5/5, and a passing security scan. The complete latest-head [CI matrix](https://github.com/paperclipai/paperclip/actions/runs/39535189808) passes, including budget, browser, Runner, build, and canary jobs. The unchanged Runner close fixture passed on its sole same-head job retry. - Broad local `pnpm test:run` invocations were stopped before required source updates (rebase and review fixes). They are not claimed as full passes. Final full-suite coverage comes from the complete latest-head CI matrix; local focused coverage and the incomplete broad invocations are reported separately. - Reproduce performance with `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-budget-enforcement.ts --baseline-ref=6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837`; repeat with `--reverse`. The harness uses disposable real PostgreSQL, 20 warm samples per variant/count, and records raw timings, SQL/lock counts, agent reads, unchanged agent state, and query plans. Final production-source measurement revision: `eff713a4f72c7f4b7a4002a701cb481d4bfccd95`; the extended harness is committed in `4c5f4de8b1`. | Unrelated idle agents | Baseline median / p95 | New median / p95 | Baseline SQL / locks | New SQL / locks | New agent reads | | --- | ---: | ---: | ---: | ---: | ---: | | 0 | 1.80 / 2.33 ms | 2.99 / 3.88 ms | 5 / 1 | 7 / 1 | 0 | | 17 | 15.53 / 19.55 ms | 2.24 / 2.60 ms | 90 / 18 | 7 / 1 | 0 | | 1,000 | 978.35 / 1,790.43 ms | 2.16 / 2.52 ms | 5,005 / 1,001 | 7 / 1 | 0 | - These measurements use local PostgreSQL 18.1 and Node 26.4.0 arm64. They isolate the cost-evaluation core and complete admission path, without hosted-region or provider latency. Prior-revision reversed-order controls also preserved the constant query counts; the final measurements above use the final production source. - Complete reserve → final ownership gate → immediate offline provider result is measured separately, with one responsible agent and one active operation at each unrelated-agent count. Admission p95 baseline/new: 0 agents **11.54 / 29.77 ms**; 17 agents **9.31 / 27.62 ms**; 1,000 agents **24.51 / 74.05 ms**. Baseline admission was already scoped; the stronger final fence adds bounded overhead. New counts stay constant at 53 Drizzle statements, three transactions, two company-accounting locks, and five agent-reading statements bound to the responsible agent. Zero unrelated-agent reads, exactly one provider callback per sample, and no leftover runs or holds. All admission scenarios stay below 250 ms. No external provider calls are included. - The partial-usage/delayed-termination fixture passed 20/20: receipt persistence p95 13 ms, receipt return to observed durable Stop fence p95 28 ms, complete receipt-start-to-fence p95 50 ms, unrelated database request p95 1 ms while termination was blocked. After releasing the controlled termination fixture, acknowledgement p95 was 14 ms. These are separately measured phases; fixture acknowledgement is not live-provider termination latency. - Models and provider deadlines are unchanged. No deployment, live inference, or staging acceptance is claimed. A quiet 30-minute follow-up waits for the merged revision on staging, then repeats classifier and controlled budget-stop acceptance in staging, recording hosted bookkeeping, queue delay, and actual provider acknowledgement separately. - Development failures retained: finalizer notification regressions (fixed; 27/27 affected cases passed), invalid browser admission fixture assumptions (fixed; browser coverage now passes 63/63), and first-head lifecycle/dispatch CI regressions (reproduced and repaired). A prior-head Cases E2E test hit its unchanged announcement-dismiss handler race while opening a filter; the same journey passed on `1ac9b58`. The external security review then found a stale-owner race between accounting commit and provider entry; final verification covers a competing run in that gap while retaining provider I/O outside the accounting lock. Early PostgreSQL startup attempts hit local semaphore pressure; serial reruns succeeded. The final-head Runner lane had one failure among 1,823 tests: the unchanged `timeout-attachment` fixture failed at its first process close after 6.28 seconds. The sole same-head job retry passed all 1,819 active tests (four skipped), including that fixture in 1.85 seconds. Three isolated local probes could not run correctly because the Rust fake-provider fixture binary was absent; cleanup masked the original startup failure. CI builds that fixture through its normal lane. These probes are inconclusive and are not counted as passes. ## Risks - This changes accounting, lifecycle, and cancellation ordering. Durable generations, source/sequence checks, ownership leases, and company locking remain required for correctness. - Provider stop acknowledgement can be slow or fail. Stop intent remains pending until existing termination proof succeeds; cancelled row status alone does not acknowledge delivery. - Mid-run limits depend on provider usage reports or supported price calculations. Providers that report only at completion cannot provide exact mid-call detection. - Signals are process-local wake hints for the supported single-active-writer model. Durable state and startup recovery remain authoritative. - There are no migrations, new queue tables, or cached balances. Repeat the classifier and controlled budget-stop acceptance after normal rollout. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, code execution, and parallel coding agents. The host does not expose the exact backend model revision or context-window size, so those values are not asserted. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
36e4c6657f |
fix(sentry): report a native retry failure only when the retry ends (#16865)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server sends a Sentry `run_failure` event when an agent run
reaches a terminal failure.
> - A native run that waits for a same-run retry is stored as `failed`
with `nativePhase: "retryable_failure"`. That state is not terminal.
> - `reportRunFailure` reported that pending state. Runs that later
recovered, for example after a workspace copy-back retry, still sent a
run failure event.
> - These events make one catch-all Sentry group noisy and hide the real
terminal failures in it.
> - This pull request skips the pending retry state and reports once
when the retry ends as a terminal failure.
> - The benefit is that run failure events match runs that really
failed.
## Linked Issues or Issue Description
**What happened?**
Two staging runs had a workspace copy-back failure during settlement
(`Daytona syncOut tar command failed ... No space left on device`). The
executor stored each run as `failed` with `nativePhase:
"retryable_failure"` and `errorCode: native_session_interrupted`. Sentry
received a `run_failure` event for each run. About two minutes later,
the copy-back retry succeeded and both runs finished as `succeeded` with
`workspaceFinalizeStatus: "succeeded"`.
**Expected behavior**
A run that waits for a same-run retry must not send a terminal run
failure event. If the retry ends as a terminal failure, the server must
send one event at that time.
**Steps to reproduce**
1. Set `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected`.
2. Start a native run that fails retryably. For example, make the
workspace copy-back fail once.
3. See the `run_failure` event in Sentry, and then see the run finish as
`succeeded` after the retry.
## What Changed
- `reportRunFailure` returns without a report for a native run that is
`failed` with `nativePhase: "retryable_failure"`.
- New `isNewTerminalRunFailure(before, after)`: a status change counts
as before. A failed run that leaves the pending retry state but stays
`failed` also counts.
- Three terminal writers can keep the status `failed` → `failed`. They
now use `isNewTerminalRunFailure`, so they still report once:
- the resume reconciler's ambiguous-state write;
- the restart recovery `blocked` write;
- the committed-result projection (`projectCommittedRun`).
- Result-less retries already reset the run to `running` before they
start, so their terminal write still reports through the existing
status-change guard.
- `doc/observability.md` describes the new rule.
## Verification
- `pnpm exec vitest run
src/services/__tests__/run-failure-report.test.ts
src/__tests__/run-failure-sentry.test.ts
src/__tests__/native-session-resumption.test.ts
src/services/native-runtime/native-run-finalizer-telemetry.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/__tests__/heartbeat-status-preserving-telemetry.test.ts
src/services/execution-control-reconciliation.test.ts` (from `server/`):
7 files, 363 passed, 9 skipped.
- New tests:
- `run-failure-report.test.ts`: a pending native retry does not report,
and a `terminal_failure` run does. Unit cases cover
`isNewTerminalRunFailure`.
- `native-session-resumption.test.ts`: a sweep that ends an unreported
pending retry sends one event.
- `native-run-finalizer-telemetry.test.ts`: a committed failure for a
pending-retry row sends one event, and a replay sends none.
- `native-runner-restart-recovery.integration.test.ts`: a restart
recovery `blocked` write for a pending-retry row sends one event.
- I removed each of the three writer changes one at a time. The matching
new test failed each time (`expected [] to have a length of 1` / `to
deeply equal [ [ ObjectContaining{…} ] ]`).
- I did not run the full server `tsc --noEmit` locally. The local
container has a 4 GiB memory limit, and the process was killed (exit
137). CI runs the typecheck.
## Risks
- A run that stays in the pending retry state and never resumes no
longer sends a run failure event. The run stays visible on the board as
failed with a scheduled retry. A later terminal write still reports it.
- The finalizer's `failed` + `retryable_failure` write for a failed
provider result also stops reporting. The terminal write that ends that
retry reports instead: retry exhaustion in `recordRetryableFailure`
reports unconditionally, and commit goes through `finalizeNativeRun` or
`projectCommittedRun`.
- A run that was reported in the pending state before this change can
report once more when it becomes terminal after deploy. This affects
only runs in flight during the upgrade.
- There are no schema, migration, or API changes. Run status, retries,
and recovery do not change.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, code search, test runs) and a read-only code
exploration subagent. Context window: the standard window for that
model.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandPaperclip
|
21ee5c9c85 |
feat(sentry): add opt-in run failure reporting with saved run references (#16300)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent runs retain execution results and output in the instance. > - Operators can report run failures to Sentry for investigation. > - A backend Sentry DSN currently enables those reports with no separate choice. > - A repair bot also needs the exact instance, company, agent, and saved run. > - This pull request adds an explicit reporting mode and authenticated run references. > - The benefit is that operators can investigate every failure on selected deployments while keeping other deployments quiet. ## Linked Issues or Issue Description **What existing behavior does this improve?** Optional server reporting of terminal agent run failures to Sentry. **Current behavior** A backend DSN enables run failure reports. Known account, quota, and connection failures are excluded. Reports omit company and agent references. Runs without a task are skipped. Tool-name collision errors omit the conflicting bindings. **Proposed behavior** Set `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected` to retain the existing filters. Set it to `all` to report every failed or timed-out run, including known setup and quota failures. The default is `off`. Expected stops stay excluded. Each report supplies identifiers and API paths that a bot can use with its own authorized Paperclip connection. **Reason and benefit** Operators can enable broad failure reporting on one staging deployment. A bot can fetch the saved run, events, and logs instead of relying on a short error label or copying complete transcripts into Sentry. **Breaking changes** A backend DSN alone no longer enables run failure reports. Existing operators must set `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected` to retain the previous policy. Ordinary server exception capture remains enabled by the DSN. Failure grouping gains the phase and original cause code when available. Refs #16281, #16297, #16298. These cover related tool discovery and failure diagnostics. This change retains their existing boundaries and does not depend on their open changes. ## What Changed - Add `off`, `unexpected`, and `all` modes. Missing and invalid values keep reporting off before diagnostic reads. - Keep the Sentry backend DSN and audited optional SDK requirements. - Include company, agent, run, and optional task identifiers. Include versioned paths for the saved run, events, log, and task. Add absolute run URLs only for a valid public HTTP(S) origin. - Report company runs without a task. Prefer the first-class issue binding when selecting the task reference. - Select saved native provider/model settings, timestamps, and original failure codes. Preserve gateway reason codes through redacted exception projection. - Group by error code, adapter, and available phase/cause. Keep run IDs and error text out of grouping. - Include the tool names and connection bindings in collision errors without copying tool schemas or provider metadata. - Document configuration, upgrade behavior, bot access, redaction, retention, and best-effort delivery. ## Verification - All 56 checks passed for `cccb47a721eaa1cc36e63278fc177b9101477541`. Two unrelated Storybook jobs were skipped by their path filters. No checks remain pending. - Complete [Paperclip CI run](https://github.com/paperclipai/paperclip/actions/runs/38591422284): passed. This covers general and serialized tests, Runner checks, browser E2E shards, typecheck, build, release-registry checks, canary dry run, and Docker checks. - Greptile reviewed that exact head: 5/5, with no unresolved threads or actionable findings. Superagent, Socket, and Snyk checks passed. - Local focused reporting, diagnostics, collision, and real SDK contract tests: 410 passed across five suites. This includes the external-wake filter in both reporting modes. - The audited `@sentry/node@10.71.0` tests use an in-memory transport. No events were sent to Sentry. - `pnpm -r typecheck` and `pnpm build`: passed locally after the final rebase. - `node scripts/check-module-boundaries.mjs` and `git diff --check`: passed. - The full local `pnpm test:run` command was stopped to avoid duplicating the complete CI shards. It did not finish locally. The full suite passed in CI. ## Risks - The new default disables existing run failure exports until the operator selects a mode. The upgrade setting is documented. - `all` can increase event volume and include known user-action failures. Secret redaction and diagnostic limits still apply. - Reporting remains asynchronous and best effort. This PR adds no durable delivery queue and does not install a bot or Sentry alert rule. - API references grant no authority. Existing company, private-task, and run-output checks still apply. Complete transcripts stay in Paperclip. - This changes optional Sentry error reporting. It changes neither first-party Telemetry nor OpenTelemetry span exports. It adds no database migration. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, shell, file editing, and test execution. The exact backend deployment ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
a65ca09508 |
fix(runner): settle accepted results after shutdown failures (#15217)
## Thinking Path > - Paperclip manages AI agents and the tasks they perform. > - The native runner saves tool results and completion reports before it releases a session. > - Large project discovery responses can exceed the durable command limit. > - A shutdown failure can leave a saved answer waiting for workspace repair. > - Recovery reused the old assessment for a different status decision, which violated a database constraint. > - This pull request bounds discovery responses and lets recovery commit the saved result after workspace repair. > - The benefit is a task that reaches its correct final status without another provider turn. ## Linked Issues or Issue Description **What happened?** A native run can save its final answer, fail during shutdown, and leave the task In Progress after workspace repair succeeds. Reconciliation tries to reuse the failed-workspace assessment for a new decision. The one-decision-per-assessment constraint rejects the write. Replaying the old decision can also retain a fresh coordinator lease. Separately, retained session cleanup only recognizes the old `adapter_failed` label. The project-list tool returns full project records, including large descriptions and workspace configuration. A large response exceeds the runner's durable command limit. The settlement diagnostic previously recorded only a failure flag. **Expected behavior** Project discovery stays within the command limit. Recovery finishes the saved result after workspace repair, releases its lease, and preserves the original error for inspection. It does not repeat provider work or relax session ownership checks. **Steps to reproduce** 1. Return large project records from `list_projects` and observe an oversized semantic result. 2. Persist an accepted native completion result, then record a shutdown failure. 3. Finalize with a failed workspace, repeat that attempt, then record successful workspace repair. 4. Reconcile the run. Before this fix, the issue stays In Progress. **Paperclip version or commit** Reproduced on `a386a599983519eb1d399f8b770bfccdb2a74762`. **Deployment mode** Self-hosted server with Paperclip Runner. Related transport work: #12208 drains queued events; #12241 resumes interrupted semantic calls. This change addresses bounded project discovery and accepted-result finalization. ## What Changed - Read bounded project summary projections from the database and return at most 50 authorized summaries with a continuation cursor and explicit description truncation. Agent and run trust boundaries narrow the database candidates; project-specific policies still receive full authorization. Only visible projects determine continuations. The default project-list API remains unchanged. - Record bounded, content-free settlement failure causes for command limits, storage errors, and rejected dispatch. - Include workspace state in assessment identity. Preserve the initial assessment for interrupted finalization, and commit replacement assessment and decision references together. - Release the coordinator lease when an existing decision is replayed, without repeating its effects. - Clear stale errors when recovery succeeds and retain them in `recoveredExecutionFailure`. - Accept both current and legacy close-failure labels in the existing exact-state cleanup path. - Add regression tests and update the tool contracts and recovery documentation. ## Verification - Red: the new project paging, settlement diagnostic, current cleanup label, and repaired-workspace regressions failed on the original implementation. - Green: protocol/catalog/tool checks (117 tests), the full project-tool and finalizer suites (47 tests), cleanup ownership cases (70 tests), and cleanup sweep cases (4 tests) pass. Database-backed pagination covers large descriptions/configuration, complete enumeration, uppercase cursors, agent/run restrictions, project-policy scope contributions, and identical results/cursors when hidden projects are added. - The initial CI failures in interrupted Board waits, contended endpoint proof, and semantic schema validation were reproduced and fixed; all affected cases pass locally. - `pnpm -r typecheck` — passed. - `pnpm build` — passed. - `pnpm test:run` — started before the review corrections; it spanned several source revisions and was stopped after reporting old-behavior and timing failures. It is not claimed green. Fresh project and recovery suites pass; the recovery suite also passes all 25 cases with the broad runner’s isolated home/config. The Slack timing case passed in isolation. Latest-head CI is the authoritative complete test matrix. - Existing authorization suite — 66 tests passed. - Latest-head CI on `8e5763d915aa6f375bdab6601996899ea01496fc` — 55 checks passed, 4 intentionally skipped, no pending or failing checks. - Greptile — 5/5, zero unresolved threads on the same commit. - `git diff --check` — passed. ## Risks - `list_projects` now returns summaries. Callers must follow `nextCursor` and use the authorized project API for full records. Candidate narrowing is only an optimization: project policy and responsible-user authorization remain authoritative. - Assessment identity changes for workspace finalization. Existing evidence remains intact; no migration is needed. - Cleanup still requires matching identities, settled tool evidence, and verified process ownership. Unknown tool outcomes remain blocked from session reuse. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, code execution, and GitHub tools. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
14795136f5 |
fix(runtime): finalize and recover sandbox workspace exports safely (#14402)
Serialize native workspace finalization, validate streamed archives within bounded limits, and quietly recover unsafe exports from saved results. Preserve exact allocations for exhausted transient failures and provide export-only retry without rerunning the provider. Consolidates #14314, #14315, #14329, and #14334 while preserving the already-merged finalization label changes. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
fd071748ee |
fix: stop repeated notifications for finalized run failures (#13739)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native run reconciliation repairs saved execution outcomes after
interruptions.
> - The sweep also visits runs whose final results are already
committed.
> - An unchanged failed run still received a new status delivery ID on
every sweep.
> - The browser treated each delivery as a new failure after its short
duplicate window expired.
> - This pull request makes unchanged projections a no-op and suppresses
repeated or historical run toasts.
> - Operators receive fresh failure alerts without repeated alerts for
old work.
## Linked Issues or Issue Description
**What happened?**
An old failed run repeatedly produced failure toasts while the browser
remained open. The task could already be cancelled. Reconciliation
rewrote the same failed outcome and queued another status broadcast.
**Expected behavior**
An unchanged committed run must not queue a new status notification.
Repeated deliveries must still refresh cached state without another
toast.
**Steps to reproduce**
1. Finalize a native run with a failed result and successful workspace
finalization.
2. Deliver its pending execution status and cancel its task.
3. Replay finalization and status delivery on each periodic sweep.
4. Observe another failure broadcast for every sweep before this fix.
**Paperclip version or commit**
Reproduced against source commit
|
||
|
DottaandPaperclip
|
9fd2e50310 |
feat: create company skills from runner tasks (#13538)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Runner gives agents tools to change company resources. > - Users need agents to save reusable skills during a task. > - A saved skill needs a visible result that users can inspect and edit. > - This pull request adds `create_skill` and a task feed card linked to Skill Studio. > - Users can open the saved skill from the task and edit the same resource. ## Linked Issues or Issue Description **Subsystem affected** Runner tools, company skill storage, task feed, and Skill Studio. **Problem or motivation** The Runner has no dedicated tool to create a company skill. A user cannot follow a creation result from the task feed to the saved skill. **Proposed solution** Add a company-scoped `create_skill` tool. Save the skill with the existing company policy. Add one creation card to the task. Open a named sidebar tab from that card. Let the user open the same skill in Skill Studio. **Alternatives considered** An agent can write a local file, but that file is not a company skill. A second document copy in the task would become stale after a Studio edit. The sidebar therefore reads the saved skill directly. **Roadmap alignment** This extends the shipped Skills Manager, Skill Studio, and Skills Store milestone. The maintainer requested and approved this scope. Search found no duplicate `create_skill` PR or issue. Related UI validation work: #8715. This PR does not change that validation display. ## What Changed - Add the real Runner tool, its contract, and its mock implementation. - Validate the complete SKILL.md and derive company, task, agent, and run identity from authentication. - Apply the existing company skill policy. Do not assign the skill to an agent. - Make keyed retries return one skill and one creation event. Reject conflicting retries. - Make concurrent file creation safe. Never replace an existing published skill during creation. - Add a creation card, a named sidebar tab, and an Open in Skill Studio action. - Show saved Studio edits when the user returns to the task. - Add storage, policy, mode, retry, UI, and Product E2E tests. Document the tool. - Fix deleted-name reuse, onboarding panel persistence, immediate feed refresh, and mock validation parity from review. - Serialize Studio file edits and renames with skill deletion and recreation. Reject stale editor requests before they can change a replacement skill. - Generate the standalone mock parser and validator from the production contract. Use portable UUIDs so the browser scenario bundle builds. ## Verification - All latest-head PR checks pass on `145dd76a5`, including all server shards, browser E2E, Runner verification, build, typecheck, and release dry run. Greptile: 5/5 with no open findings. An interrupted CI runner was retried successfully. - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - `pnpm check:token-gates`: passed. - Review regressions: 73 storage tests, 6 real API tests, 63 UI tests, and 61 semantic runtime tests passed. Parser synchronization passed. - CI exposed existing fire-and-forget Sentry test races. Reproduced the resumption race locally, then synchronized the related sweep and finalizer assertions on the actual report; all 27 tests across the three affected files pass. - Runner scenario browser build and strict content-security-policy check: passed. - Runner suite: 2,012 tests passed; 10 skipped. - `pnpm test:run`: the general-server batch had 12,416 passes and two failures. The old tool-count assertion was fixed; all 16 authority tests then passed. The chat webhook test had a socket error; it passed four isolated reruns. - Both workspace test groups passed. The isolated route suites completed. Two socket failures in the initial route batches passed on individual reruns; all remaining 61 files passed. - Product E2E `create-skill-studio`: passed with local Codex and local ACPX Claude. - Manual browser test: submit a task, observe the real tool call and creation card, open the sidebar, edit in Studio, save, and return. The task reached Done. The saved second revision and sidebar tab survived a server restart. - The new companion headless Runner Eval passed. Companion coverage PR: https://github.com/paperclipai/paperclip-evals/pull/23. Daytona was not run because no immutable runner image was configured. ## Risks - Database writes and local file writes cannot share one transaction. Recovery accepts only an exact file-for-file retry after a database rollback. Conflicting files remain untouched. - The sidebar displays the current skill. The feed card remains the historical creation receipt. - No database migration, dependency, or workflow change is included. - Remote Daytona behavior still needs a run with a configured immutable image. ## Model Used OpenAI GPT-6 (`gpt-6-astra`) handled design, integration, review, and browser verification. OpenAI `gpt-5.6-luna` assisted with bounded implementation and eval work. Both used code execution and tool access. The host did not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
c276d3fdc3 |
feat(observability): report terminal run failures to Sentry (#13446)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server records the state of each agent run. > - Terminal run failures need clear error tracking for operators. > - The server did not report terminal `failed` or `timed_out` transitions to Sentry. > - This pull request reports each genuine terminal failure transition with safe diagnostic data. > - The benefit is faster diagnosis without changing run control flow or exposing credentials. ## Linked Issues or Issue Description **What happened?** The server wrote terminal run failures but did not report them to Sentry. Operators could not see these failures in error tracking. **Expected behavior** The server should report each genuine transition to `failed` or `timed_out` to Sentry. **Steps to reproduce** 1. Run an agent task that reaches a terminal failure state. 2. Inspect the Sentry events for the server. 3. Observe that the terminal run failure has no matching Sentry event. **Paperclip version or commit** The change targets the current `master` branch. No public GitHub issue or pull request covers this change. ## What Changed - Add `captureRunFailure()` as a fail-open Sentry entry point. - Add `reportRunFailure()` to filter status, resolve the adapter, redact text, and report the failure. - Call `reportRunFailure()` beside each of the eight terminal status writers. - Report six diagnostic values: the instance host, task identifier, run identifier, error message, error code, and agent adapter. - Group events by error code and agent adapter while keeping the redacted message in the event. - Report only genuine transitions and avoid duplicate finalization events. - Keep Sentry failures outside run control flow. ## Verification - `pnpm vitest run server/src/services/__tests__/run-failure-report.test.ts server/src/__tests__/run-failure-sentry.test.ts server/src/__tests__/native-session-resumption.test.ts` - `pnpm vitest run server/src/services/execution-control-reconciliation.test.ts` - `pnpm --filter @paperclipai/server typecheck` - The full continuous-integration suite must run on this pull request. ## Risks - The report path can add diagnostic events when Sentry is configured. - The report path returns without action when Sentry is not configured. - Redaction runs before length limits and before the event leaves the process. - The change has no migration and no schema change. ## Model Used OpenAI Codex, GPT-5, tool use and code review support. The exact context window and reasoning configuration are not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
889947c238 |
feat: add experimental native chat connectors (#13038)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People also ask agents for work in their existing chat tools. > - Each external conversation needs one task and a current authorized source. > - Retries, Stop, and provider failures must not duplicate work or expose private data. > - The first chat PR establishes the opt-in provider and data contracts. > - This PR adds experimental channel integration and its durable control plane. > - Users can request work from connected channels and inspect delivery in Paperclip. ## Linked Issues or Issue Description Refs #13100 and #13092. This is the second of exactly two chat PRs. Foundation #13100 is merged and changed 143 files. Runner prerequisite #13092 is also merged. This PR changes 400 files against master, below the 500-file review limit. It contains no wireframe images or HTML galleries. ## What Changed - Add native Slack, GitHub, Microsoft Teams, Telegram, and Discord chat connections. Keep chat disabled unless the operator enables experimental chat connectors. Preserve the production GitHub tool connection and its normal setup path. - Bind each provider bot identity to one immutable Paperclip agent. Bind each admitted external conversation to one task. Paperclip owns tasks, runs, permissions, and audit records. - Add durable admission, per-conversation queues, questions, task controls, progress, final replies, images, files, and delivery receipts. Board comments remain internal unless explicitly sent to the channel. - Check current identity, provider reach, resource access, credentials, runtime generation, and exact source before provider effects. Keep private responses private. Never send raw reasoning, private logs, credentials, or tool arguments. - Hold uncertain sends for explicit audited resolution. Make Board Send-to-channel atomic and idempotent. Keep reconnect and setup credentials in Paperclip secret storage. - Preserve current native-runner authority across retries, lost acknowledgements, and recovery. Keep immutable input and completion contracts separate from newer user input. Receipt reconciliation cannot launch a provider. - Reconcile chat close/new ordering and provider-effect lock order. Audit resource access changes in the same transaction. Submit only the selected resource from each UI toggle so stale pages cannot undo unrelated access changes. - Drain Codex stdout before certifying process exit. Bound the drain with the existing shutdown grace. Preserve observed terminal authority without treating an undrained process as successful or reusable. - Incorporate master `018ca5da` with its ACP Stop, mobile task layout, runner packaging, and official lock changes. Preserve dedicated chat-answer continuations in both directions when ordinary queued comments are adopted after Stop. - Fence late adapter readiness behind an earlier Stop for the same run. Preserve verified cleanup for registered adapters. Handle single Stop, agent pause, duplicate Stops, and failure release without creating a false cancellation receipt. - Incorporate master's `6dd48cad4` wake-queue extraction. Preserve exact failed-chat retry authorization and lineage, retired question-source suppression, and the block on generic recovery that would discard the admitted source. Fresh deferred input retains its separate promotion path. - Incorporate master `2a05b5ed3` and its queue-admission extraction, simplified transaction ports, and separate runner CI job. Preserve exact durable receipts, actor separation, and dedicated-answer isolation through the new module. A failed receipt insert rolls back the accompanying deferred-wake merge. ## Verification Current head: `afe19299d06253cb628eb398e91d1200ea9f412a`, incorporating master `2a05b5ed3457ea33efd6895520447d1d97fe98d8`. The conflicts are resolved. This successor fixes two test-harness boundaries exposed by CI: per-case route-module preparation and actual durable-save completion before intentional runner termination. Production code and all existing test/turn deadlines are unchanged. [Exact-head Greptile review](https://github.com/paperclipai/paperclip/pull/13038#issuecomment-5587250594) is **5/5**, completed September 10 at 13:20:55 UTC, with no actionable findings or open review threads. [Fresh exact-head CI](https://github.com/paperclipai/paperclip/actions/runs/34481724341) passes **all 24 jobs**, including Build and both required aggregates. Normal exact-head guarded merge was attempted and rejected by the remaining branch approval policy: CODEOWNER review is required and no human approval is present. Normal **squash auto-merge is enabled** as of September 10 at 13:36:26 UTC. Requested CODEOWNERS have been notified; no approval bypass or self-approval was used. Earlier-head results below remain historical evidence, not qualification of this successor. - Final exact-head Linux evidence: 995/995 chat integration cases; 36/36 agent-skills routes; 35/35 runner live-session cases, including real process kill/resume; 1948 runner Vitest cases with three existing benchmark/platform guards; 870/870 API-authority cases; and 104 browser cases with four existing optional skips. Rust, conformance/replay, full repository build, typecheck, canary, all server/workspace shards, and both required aggregates pass with normal CI concurrency. Earlier failed attempts remain recorded below. - Latest test-only qualification: 141/141 route/permissions/authentication cases pass in separate cold forks, with plain server types and independent review clear. The real-runner suite passes 35/35, with plain runner types and independent review clear. A controlled premature-save acknowledgement fails as expected; matching ownership/effect/process evidence, rejected saves, real turn outcome, test abort, and pre-kill liveness are covered. No local reproduction of the original CI scheduling failure is claimed. The preceding [CI run](https://github.com/paperclipai/paperclip/actions/runs/34479680858) passes 21/24 jobs, including all 995 Linux chat cases and browser aggregate (104 passed, four existing optional skips); only Build, the skills serialized shard, and the required verification aggregate fail. Its exact-head Greptile review was 5/5. Both failed job logs are retained. - Final fixture qualification: all eight focused Discord cases and all 995 chat integration cases pass. The exact modal statement/PID is observed before taking the real connection lock; the test then proves its actual blocking relationship before mutation. Original SQL execution, provider behavior, negative assertions, and 1s/15s timeouts remain unchanged. Independent review is clear and test/production hashes remain frozen. The preceding [CI attempt](https://github.com/paperclipai/paperclip/actions/runs/34477184777) passed 22 jobs, including Build/runner, typecheck, canary, all other test shards, and browser aggregate (104 passed, four existing optional skips); the two fixture failures and failed verification aggregate remain recorded, not relabeled as a pass. - Current queue-module composition: 308/308 recovery/batching/queue/Stop tests; 995/995 full chat integration; 89/89 module tests, including real PostgreSQL receipt-insert rollback; 24/24 workflow/module-boundary tests; plain server and UI types. All four actual local process/ACP browser paths pass in 1.4 minutes. Fresh databases, no skips or retries, stable reviewed source hashes. The initial boundary failure is retained; its no-op service wrapper was removed without changing recovery context or weakening the check. An exploratory standalone test-directory typecheck fails because its new upstream transformation config is not a standalone typechecking project; standard CI/build does not invoke it, and no configuration was weakened to suppress those diagnostics. - The preceding head `e02a63d462ce5d47433b0aeb632bb6fd20aab1ba` passed [all 24 CI jobs](https://github.com/paperclipai/paperclip/actions/runs/34436462958) and exact-head Greptile review at 5/5. Required CODEOWNER review prevented its normal merge before master advanced again. - Final extracted-module composition: 307/307 recovery, batching, queue and Stop-control tests; 995/995 full chat integration; 49/49 module tests including eight PostgreSQL adapter cases; and 19/19 issue-update tests. Plain server types pass. All four actual local process/ACP browser paths pass in 1.3 minutes. Fresh databases, no skips or retries in these cohorts, frozen source hashes, and independent review clear. - The preceding head `3e4e1c1c` passes [all PR CI jobs](https://github.com/paperclipai/paperclip/actions/runs/34415826820), including Build and required `ci / verify` and `ci / e2e`. Both the original Rust failure and the previously load-sensitive lineage fixture pass with unchanged Linux concurrency. Master advanced afterward and required this reconciliation. - Final master composition: 448/448 focused UI tests, 186/186 adapter tests, 24/24 queue/control tests, and 11/11 packaging tests. Plain UI, server, shared, and adapter types pass. Token gates and diff checks pass. Independent server and UI reviews are clear. - Stop-registration regression: both real-service cases fail against exact `a95` source and pass with the fix. The full corrected recovery/control suite passes 265/265. Duplicate-owner and failed-Stop controls also pass. Plain server types pass. The readiness barrier prevents provider startup without adding an acknowledgment to an already terminal run. - Final qualification strengthens terminal-field equality and repeats both affected cases successfully on a fresh database. All four actual local process/ACP browser paths pass again in 1.3 minutes, without skips or retries. The final screenshot shows Cancelled, a paused subtree, retained input, and no error toast. - Two new actual-service regressions fail before the merge fix. They prove that queued-comment adoption could consume a dedicated chat answer or add unrelated input to that answer. The fixed four-case cohort passes, including ordinary upstream continuation and adapter Stop controls. Full recovery passes 257/257. All four actual local process/ACP Stop browser flows pass in 1.4 minutes, without skips or retries, on a fresh database. - The unchanged runner artifact was qualified with 171/171 transport tests, 870/870 API-authority tests, conformance 1/1, and replay 11/11. Six controlled reader tests prove the exit/drain repair. Its local serial Rust workspace passed 546 top-level cases plus two invoked helpers; the later passing Linux CI supplies default-concurrency evidence. - Prior exact-source full chat integration passes 995/995. Settings regressions cover concurrent stale pages, 501 destinations, pending state, rejected updates, and explicit retry. These deterministic tests do not prove live provider behavior. - Retained failed attempts and their causes are in the [qualification log](https://github.com/paperclipai/paperclip/blob/afe19299d06253cb628eb398e91d1200ea9f412a/doc/plans/chat-adapters/2026-09-08-chat-queue-and-webhook-repair.md). The first merge adapter run timed out while macOS slept for 290 seconds. Its unchanged repeat passed with a temporary sleep guard. No assertion, deadline, or CI gate was weakened. Review commands include `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-process-recovery.test.ts src/__tests__/issue-queued-comments-routes.test.ts` and `pnpm exec playwright test --config tests/e2e/playwright.config.ts tests/e2e/acp-stop-continuation.spec.ts`. Database suites require fresh disposable databases. See the [browser runbook](https://github.com/paperclipai/paperclip/blob/afe19299d06253cb628eb398e91d1200ea9f412a/doc/plans/chat-adapters/2026-09-04-chat-adapters-browser-e2e-runbook.md) for provider setup and separate live acceptance steps. ## Risks - This remains experimental. Deterministic tests and bounded live evidence do not establish every provider feature, tenant, permission layout, or media shape. Teams work-tenant qualification is still open. - Failed and uncertain provider effects remain visible and can require operator action. A transport receipt does not prove recipient visibility. - Native controller and runner artifacts must remain compatible. Preserve lease ownership, terminal authority, source binding, and quarantine during future changes. - Access and audit rows commit together, but activity notifications remain best-effort. This is not a new durable event outbox. - The PR operation does not deploy a live server, replace its runner, or change provider permissions. Remaining live qualification is documented in the [temporary handoff](https://github.com/paperclipai/paperclip/blob/afe19299d06253cb628eb398e91d1200ea9f412a/doc/plans/chat-adapters/2026-09-08-open-qualification-followups.md). ## Model Used OpenAI Codex assisted with implementation, tool execution, testing, and review. The work records `gpt-6-astra` assistance. The environment does not report a context-window size. No private reasoning traces are included. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
1dceee9a4e |
fix(runner): persist warm Daytona workspaces (#12901)
## Thinking Path > - Paperclip manages AI agent work and the execution state for each task. > - Remote agents run in sandbox environments such as Daytona. > - Daytona keeps files while a sandbox is stopped, but deletion removes those files. > - Runner Codex did not copy successful remote workspace changes back to the host workspace. > - A warm sandbox could therefore hide data loss until Daytona replaced or deleted the sandbox. > - This pull request makes the host workspace durable after every successful turn and keeps verified reusable sandboxes warm. > - The benefit is reliable multi-turn work across warm reuse, restart, stop, and sandbox replacement. ## Linked Issues or Issue Description **What happened?** A successful native Codex turn in Daytona could leave workspace changes only in the remote sandbox. A later warm turn appeared to work because it reused that filesystem. A replacement sandbox could start from stale host data and lose the successful changes. **Expected behavior** Paperclip must merge each successful remote turn into the authoritative host workspace before it completes the run. A verified warm lease may reuse its remote files. A replacement lease must reconstruct the exact durable workspace seed. **Steps to reproduce** 1. Run Codex in a reusable Daytona environment. 2. Write a file during one successful turn. 3. Replace the Daytona sandbox before the next turn. 4. Observe that the next turn can start without the prior file on the unpatched code. Related remote workspace foundation: #10070. ## What Changed - Added explicit `host_current`, `durable_seed`, and `adopt_remote` workspace preparation modes. - Added atomic, versioned native workspace descriptors and seed archives under `PAPERCLIP_HOME`. - Added real native sandbox export and three-way host merge before terminal result completion. - Added workspace-only recovery after a proposed result. Recovery does not submit another provider turn or consume the provider retry budget. - Added fail-closed handling when a sandbox with unexported changes is gone. - Kept healthy reusable Daytona sandboxes started for legacy Codex and Runner Codex. - Kept the Runner Codex process and provider session across verified warm turns. - Added the paid `daytona-warm-continuity` browser suite. It contains exactly the legacy Codex and Runner Codex cells. Each cell performs three measured turns. - Documented `pnpm test:e2e:runner -- --suite daytona-warm-continuity`. No package script was added. - Added no database migration. The metadata format is backward compatible and idempotent. ## Verification - `pnpm typecheck` - `pnpm test:e2e:runner:unit` — 114 passed - Native workspace, finalizer, session, and environment tests — 232 passed - Daytona provider tests — 150 passed - Workspace staging and merge tests — 98 passed - Runner transport tests — 63 passed - Legacy Codex restore tests — 5 passed - Rust format and compile checks pass through root typecheck - The paid Daytona suite was not run locally because the required Daytona, OpenAI, and immutable image credentials are not present. ## Risks - The main risk is an incorrect workspace identity or merge after a crash. Durable descriptors bind the run, workspace, lease, provider lease, local root, remote root, and baseline digest. Ambiguous evidence fails closed. - The host merge may conflict with concurrent host edits. The existing three-way merge and exclusion rules handle this case and surface failures. - A deleted sandbox cannot recover unexported bytes. Paperclip now blocks with `workspace_sync_out_unrecoverable` instead of reporting success or rerunning the provider. - There is no database migration. Descriptor writes and recovery are atomic and idempotent. ## Model Used OpenAI Codex with GPT-5. The run used agentic reasoning, repository inspection, code execution, test execution, Git, and GitHub CLI tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
Nicky LeachandPaperclip
|
184b014c25 |
feat(telemetry): add the agent.task_run event and emit it at every terminal run transition (#12809)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip records agent run outcomes through telemetry and run lifecycle services > - Terminal run transitions need one consistent event for outcome analysis > - The current paths do not report every terminal transition through one event > - This pull request adds the agent.task_run event and emits it at each terminal transition > - The benefit is complete run outcome data without exposing raw task identifiers ## Linked Issues or Issue Description **What existing behavior does this improve?** Paperclip telemetry reports agent activity, but it does not report every terminal task run through one event. **Subsystem affected** Cross-cutting (multiple of the above): packages/shared telemetry and server run lifecycle services. **Current behavior** Several run paths write a terminal status without a matching agent.task_run telemetry event. **Proposed behavior** Each terminal run transition emits one agent.task_run event. The event records the terminal state and uses the existing pseudonym helper for the optional task identifier. **Reason and benefit** Complete terminal-run data helps operators measure agent outcomes. The pseudonym helper prevents the raw task identifier from leaving the installation. **Breaking changes** None. The change adds an event and keeps existing event behavior compatible. ## What Changed - Add the agent.task_run telemetry contract and client helper. - Reuse the existing pseudonym helper for the task identifier. The helper hashes the identifier with a per-installation salt and returns 16 hexadecimal characters. The raw identifier never leaves the installation. Existing identifiers do not move. - Emit one event from each legacy, native, recovery, and issue terminal transition. - Keep emissions outside database transactions and make delivery best-effort. - Add regression tests for event shape, hashing, terminal transitions, and emission failures. - Document the event and its privacy rule in the telemetry data contract. ## Verification - `npx tsc --noEmit` in `server/` passes at the submitted commit. - The pull-request CI suite must pass. CI is the authority because local Vitest has a known dependency artifact. - The added regression tests cover event output shape, per-installation hash divergence, raw identifier handoff, omitted identifiers, and non-throwing emits. ## Risks - A missed terminal path could reduce event coverage. - Telemetry delivery remains best-effort and cannot change run finalization. - The pseudonym helper uses installation-specific state, so identifiers differ between installations. ## Model Used OpenAI Codex, GPT-5, tool use and code execution. Context window details were not provided. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I have addressed all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |