mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-26 18:31:34 +01:00
5c3dce09b5cb80911a65a03ec09d20bf8fbf78dd
41
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
75fcb0b3fc |
fix(sentry): keep typed ACP provider access failures local (#17366)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server sends unexpected run failures to Sentry through
`reportRunFailure`
> - A Claude ACP run can fail because the provider rejects the stored
credential ("Invalid API key"). The ACP agent then sends a typed
`access` terminal failure.
> - The Claude adapter already classifies this failure as
`claude_auth_required`. The run is blocked, and the responsible user
gets a reconnect request.
> - Sentry also records each of these runs as "ACP agent reported a
terminal access failure.". This duplicates a signal that the user
already gets, and it hides real defects.
> - This pull request adds one narrow exclusion for this proven provider
auth state, next to the existing quota and Codex capacity exclusions.
> - The benefit is less Sentry noise, and all other access-shaped
failures stay visible.
## Linked Issues or Issue Description
Refs #14886 (related: it routes ACP access failures in recovery, not in
Sentry reporting).
**What happened?**
Staging Sentry recorded a run failure "ACP agent reported a terminal
access failure." for 8 events. All events had
`agent_adapter=claude_local`, `error_code=claude_auth_required`, and a
typed provider access failure ("Invalid API key · Fix external API
key"). The failures stopped when the provider accepted the credential
again.
**Expected behavior**
A provider auth failure that the adapter types and classifies is a user
action state. The product already shows it to the user. It must not go
to Sentry in `unexpected` mode.
**Steps to reproduce**
1. Configure a `claude_local` agent with an ACP runtime and a credential
that the provider rejects.
2. Start a run.
3. The ACP agent ends the turn with `category: "access"`. The run fails
with `claude_auth_required`.
4. See a new Sentry run-failure event.
**Agent adapter(s) involved**
Claude Code (`claude_local`, ACP lane).
## What Changed
- `server/src/services/run-failure-report.ts`: add
`isAcpProviderAccessFailure`. It skips a report only when all of these
are true: the reporting mode is `unexpected`, the run status is
`failed`, the error code is `claude_auth_required` or
`acpx_auth_required`, `resultJson.terminalSessionFailure.category` is
`access`, the phase is not `setup`, and the run has no signal.
- `server/src/services/__tests__/run-failure-report.test.ts`: add
regression tests for the staging shape, eight near-miss shapes that must
still report, and the setup and `all` mode paths.
## Verification
- Red: without the predicate, the two new "keeps ... local" tests fail
(`2 failed | 266 passed`).
- Green: `cd server && pnpm exec vitest run
src/services/__tests__/run-failure-report.test.ts
src/services/__tests__/run-failure-diagnostics.test.ts` gives `383
passed`.
- `pnpm exec tsc --noEmit -p server` shows no errors in the changed
files.
## Risks
- Low risk. The change only removes Sentry events. It does not change
run status, recovery, or user notices.
- A real defect that the ACP agent wrongly types as `access` with a
Claude auth code is not reported in `unexpected` mode. Set
`PAPERCLIP_SENTRY_RUN_FAILURES=all` to see these events again.
- Codex and other ACP adapters do not map `access` to an auth code, so
their access failures still report.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), run through Claude Code
with tool use (shell, file edit, Sentry and GitHub connectors).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Devin Foley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
36e4c6657f |
fix(sentry): report a native retry failure only when the retry ends (#16865)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server sends a Sentry `run_failure` event when an agent run
reaches a terminal failure.
> - A native run that waits for a same-run retry is stored as `failed`
with `nativePhase: "retryable_failure"`. That state is not terminal.
> - `reportRunFailure` reported that pending state. Runs that later
recovered, for example after a workspace copy-back retry, still sent a
run failure event.
> - These events make one catch-all Sentry group noisy and hide the real
terminal failures in it.
> - This pull request skips the pending retry state and reports once
when the retry ends as a terminal failure.
> - The benefit is that run failure events match runs that really
failed.
## Linked Issues or Issue Description
**What happened?**
Two staging runs had a workspace copy-back failure during settlement
(`Daytona syncOut tar command failed ... No space left on device`). The
executor stored each run as `failed` with `nativePhase:
"retryable_failure"` and `errorCode: native_session_interrupted`. Sentry
received a `run_failure` event for each run. About two minutes later,
the copy-back retry succeeded and both runs finished as `succeeded` with
`workspaceFinalizeStatus: "succeeded"`.
**Expected behavior**
A run that waits for a same-run retry must not send a terminal run
failure event. If the retry ends as a terminal failure, the server must
send one event at that time.
**Steps to reproduce**
1. Set `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected`.
2. Start a native run that fails retryably. For example, make the
workspace copy-back fail once.
3. See the `run_failure` event in Sentry, and then see the run finish as
`succeeded` after the retry.
## What Changed
- `reportRunFailure` returns without a report for a native run that is
`failed` with `nativePhase: "retryable_failure"`.
- New `isNewTerminalRunFailure(before, after)`: a status change counts
as before. A failed run that leaves the pending retry state but stays
`failed` also counts.
- Three terminal writers can keep the status `failed` → `failed`. They
now use `isNewTerminalRunFailure`, so they still report once:
- the resume reconciler's ambiguous-state write;
- the restart recovery `blocked` write;
- the committed-result projection (`projectCommittedRun`).
- Result-less retries already reset the run to `running` before they
start, so their terminal write still reports through the existing
status-change guard.
- `doc/observability.md` describes the new rule.
## Verification
- `pnpm exec vitest run
src/services/__tests__/run-failure-report.test.ts
src/__tests__/run-failure-sentry.test.ts
src/__tests__/native-session-resumption.test.ts
src/services/native-runtime/native-run-finalizer-telemetry.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/__tests__/heartbeat-status-preserving-telemetry.test.ts
src/services/execution-control-reconciliation.test.ts` (from `server/`):
7 files, 363 passed, 9 skipped.
- New tests:
- `run-failure-report.test.ts`: a pending native retry does not report,
and a `terminal_failure` run does. Unit cases cover
`isNewTerminalRunFailure`.
- `native-session-resumption.test.ts`: a sweep that ends an unreported
pending retry sends one event.
- `native-run-finalizer-telemetry.test.ts`: a committed failure for a
pending-retry row sends one event, and a replay sends none.
- `native-runner-restart-recovery.integration.test.ts`: a restart
recovery `blocked` write for a pending-retry row sends one event.
- I removed each of the three writer changes one at a time. The matching
new test failed each time (`expected [] to have a length of 1` / `to
deeply equal [ [ ObjectContaining{…} ] ]`).
- I did not run the full server `tsc --noEmit` locally. The local
container has a 4 GiB memory limit, and the process was killed (exit
137). CI runs the typecheck.
## Risks
- A run that stays in the pending retry state and never resumes no
longer sends a run failure event. The run stays visible on the board as
failed with a scheduled retry. A later terminal write still reports it.
- The finalizer's `failed` + `retryable_failure` write for a failed
provider result also stops reporting. The terminal write that ends that
retry reports instead: retry exhaustion in `recordRetryableFailure`
reports unconditionally, and commit goes through `finalizeNativeRun` or
`projectCommittedRun`.
- A run that was reported in the pending state before this change can
report once more when it becomes terminal after deploy. This affects
only runs in flight during the upgrade.
- There are no schema, migration, or API changes. Run status, retries,
and recovery do not change.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, code search, test runs) and a read-only code
exploration subagent. Context window: the standard window for that
model.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandPaperclip
|
51c81d0129 |
feat(sentry): link run errors to task and run pages (#16803)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Operators use Sentry to inspect run and server failures. > - Reports identify the run and task, but the existing links open API records. > - Operators need a direct path from an error to the normal task and run pages. > - This change adds navigation links through the shared server Sentry hook. > - Saved company and task records determine the links. A failed lookup preserves the error report. ## Linked Issues or Issue Description Related work: #16300 added saved run API references. **What existing behavior does this improve?** Sentry event context for run failures, Stop timeouts, and server errors from an authenticated agent run. **Current behavior** Reports show task and run UUIDs. The existing run URL opens an API record. The operator must find the normal task page separately. **Proposed behavior** Add `task_url`, `run_url`, and `task_identifier` tags when saved run metadata is available. Add `taskUrl` and `runPageUrl` to the event context. Use the company prefix and saved task identifier in normal board URLs. **Reason and benefit** An operator can open the task or run directly from the error report. ## What Changed - Add a shared Sentry hook for events with a known run identity. - Read only the saved company prefix, task identifier, and run identity. Keep the lookup within the recorded company and agent. - Include authenticated run identity on request errors. Use the existing run identity on Stop errors. - Bound optional lookups to one second and eight unfinished reads. Preserve the original event on failure. - Keep the existing API references and error grouping. - Document the fields and add SDK, timeout, and company-boundary coverage. ## Verification - Passed: `pnpm -r typecheck` and the final server TypeScript check. - Passed: `pnpm build`. - Passed: 332 tests across the affected server suites, plus 25 final link and real Sentry SDK tests after review fixes. The SDK test checks outgoing event envelopes for run failures, request errors, and Stop errors. It checks that unrelated errors receive no links. The database test verifies saved metadata and company boundaries. It passes with embedded PostgreSQL and with an explicitly configured disposable test database. - Passed in CI on `bec0cdac853efdef71041d59a66261941827653e`: the full test matrix, typecheck, build, Runner checks, browser suites, and the real Sentry SDK contract. All 56 active checks pass. Two Storybook jobs are skipped. - The full local `pnpm test:run` attempt first failed because PostgreSQL library links were missing. The package setup script repaired the install. The affected database suites then passed. The repaired full local run was stopped after all CI test jobs passed; it did not complete locally. - Greptile: 5/5 on the final commit. All three review threads are resolved. - Read-only browser inspection confirms that Sentry displays context URL values with an Open link control. - No live test event was posted. Validation uses the real SDK with an in-memory transport. ## Risks - A slow or unavailable database can prevent optional links. It does not discard the error. At most eight unfinished lookups can remain active. - Missing or unsafe public origins omit the links. URLs longer than the tag limit remain in context. - This change applies to newly captured events. It does not update existing Sentry events. - The change adds no schema, permission, or first-party Telemetry contract change. ## Model Used OpenAI Codex, `gpt-6.1-sol`, xhigh reasoning, with tool use and code execution. The exact context window is not exposed in the session metadata. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
aaeff418cf |
fix(acpx): add a positive control to stalled-tool permission evidence (#16769)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The ACPX engine runs Claude agents. A watchdog ends a turn when a pending tool call gets no update for 30 minutes. > - #16688 added a field to stalled runs. The field records if the stalled call's permission request reached Paperclip. > - Recent stalled runs show `acpStalledToolPermissionRequested: false`. We can read this as "the call stopped before its permission request". That reading is correct only if calls of the same kind usually send a request. > - We have no data that shows if finished calls of that kind sent requests. Thus we cannot tell if the missing request is a real signal. > - This pull request records a positive control: the number of finished calls of the stalled call's kind, and how many of them sent a permission request. > - The benefit: the next stall event tells us if the missing request locates the stall, or if it gives no information. ## Linked Issues or Issue Description Refs #16688 **What happened?** Runs stop with the stalled-tool watchdog error. The diagnostics show that the stalled call sent no permission request. The diagnostics do not show if finished calls of the same kind sent requests. **Expected behavior** The stalled-run diagnostics contain enough data to show if a missing permission request is a real signal. **Steps to reproduce** A pending tool call gets no ACP update for 30 minutes. The watchdog ends the turn. The run result contains the stalled-tool fields. ## What Changed - `packages/adapter-utils/src/acpx-engine/execute.ts`: when the watchdog fires, count the finished calls of the stalled call's kind, and count how many of them sent a permission request. Write the counts to the run result as `acpSettledSameKindToolCount` and `acpSettledSameKindToolPermissionRequestedCount`. - `server/src/services/run-failure-diagnostics.ts`: allow the two new fields as non-negative safe integers. Other values are dropped. - `doc/observability.md`: explain how to read the two fields with the existing permission-request field. - Tests: extend the existing stalled-tool tests in `execute.test.ts` and `run-failure-diagnostics.test.ts`. ## Verification - `pnpm exec vitest run src/acpx-engine/execute.test.ts` in `packages/adapter-utils`: 244/244 pass. - `pnpm exec vitest run src/services/__tests__/run-failure-diagnostics.test.ts` in `server`: 115/115 pass. - Mutation check: remove the kind filter, or count every finished call as "requested". Each mutation makes the stalled-tool test fail. - `tsc --noEmit` passes in `packages/adapter-utils` and in `server`. Note: the full `server` `typecheck` script did not run locally, because the runner vendor build step failed in this sandbox. I ran `tsc --noEmit` directly. ## Risks - Low risk. The change only adds two counts to the run result after the watchdog fires. It does not change the decision to stop the turn, or any permission decision. - The fields hold only counts. They contain no tool titles, inputs, or identifiers. This is the run-log and run-result path. It is not Telemetry. The fields reach Sentry through the existing allowlist in `run-failure-diagnostics.ts`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
63306f6660 |
fix(acpx): record whether a stalled tool call's permission request arrived (#16688)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Hosted `claude_local` runs use the ACPX engine. Claude can start
background subagents, and those subagents call the runtime MCP tools
that Paperclip attaches.
> - #16671 added a stalled-tool watchdog. It ends a Claude turn when a
pending tool call gets no ACP update for 30 minutes. That PR said the
most likely cause of the stall is a lost ACP permission round trip, but
it could not prove it.
> - A staging run tripped the watchdog. Two background subagents started
12 MCP calls. ACP reported each call as `pending`, and no call got
another update. The MCP gateway logs show the 9 earlier calls from the
parent agent, but no request from the 12 subagent calls.
`MCP_TOOL_TIMEOUT` (15 minutes) did not end them. So the calls stalled
before Claude sent them to the MCP server.
> - The run record cannot say if the permission request for a stalled
call reached Paperclip. Without that fact, the next stall gives no new
evidence about the cause.
> - This pull request records, for a stalled run, whether the stalled
call's permission request reached Paperclip, and how many pending calls
had a permission request.
> - The benefit is that the next stall tells us which side of the
permission round trip stopped, so the real fix can go to the correct
layer.
## Linked Issues or Issue Description
- Refs #16671 (the stalled-tool watchdog; this PR tests the cause that
PR suspected)
- Refs #16360 (the MCP client deadline that cannot reach a call that was
never sent)
**What happened?**
A hosted staging `claude_local` run started two background subagents.
Each subagent started six `execute-sentry-tool` MCP calls. ACP reported
all 12 calls as `pending`. None got a result, an update, or an error.
The MCP gateway received no request for them. The parent turn finished
its answer but stayed open for the subagents. After 31 minutes the
watchdog ended the turn with `acpx_tool_stalled`. A single background
subagent with six parallel calls to the same tool did not stall in a
later test.
**Expected behavior**
When a tool call stalls, the run record shows if the call's permission
request reached Paperclip. Then the stall can be put on the agent side
or on the relay side of the round trip.
**Steps to reproduce**
1. Start an ACP `claude_local` run in a sandbox that uses two background
subagents to call runtime MCP tools.
2. Wait until a subagent's tool calls stay `pending` with no further ACP
update (in staging this happened without a known trigger).
3. See that the watchdog fails the run with `acpx_tool_stalled`, but the
record has no evidence about the permission round trip.
**Paperclip version or commit**
`master` at
|
||
|
|
938a441944 |
fix(acpx): end Claude turns whose pending tool call stops receiving updates (#16671)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Hosted `claude_local` runs use the ACPX engine. Paperclip sends
runtime MCP servers to Claude, and Claude can start subagents (the Task
tool) that call those MCP tools.
> - #16360 set `MCP_TOOL_TIMEOUT` for Claude. That bound works: in a
local test with the same Claude binary, a direct MCP call and a subagent
MCP call to a server that never answers both ended at the bound.
> - A hosted run still hit the 4h sandbox timeout on a release that has
#16360. Its run log shows a subagent that started four MCP calls. ACP
reported the four calls as `pending`, and they never got another update.
About 40 other MCP calls in the same minutes finished in 3–25 seconds.
So the four calls most likely never reached the MCP server, and
`MCP_TOOL_TIMEOUT` did not apply to them.
> - The subagent's Task call got an `in_progress` heartbeat every 30
seconds for the full 4 hours. The turn looked active, so no existing
bound stopped it.
> - This pull request adds a stalled-tool watchdog to the ACPX engine.
It ends the turn when a pending tool call gets no update for 30 minutes
and no other tool call finishes in that time.
> - The benefit is that a stalled run fails in about 30 minutes with a
clear error code, not after 4 hours with a general timeout.
## Linked Issues or Issue Description
- Refs #16360 (MCP client deadline; this PR covers the stall that the
deadline cannot reach)
- Refs #15962 (stop-time diagnostics that showed the pending tools and
their age)
**What happened?**
A hosted `claude_local` run gave lookups to two subagents. One subagent
started four MCP tool calls. ACP reported each call as `pending`. No
call got a result, an update, or an error. The other subagent finished
in 3 minutes. The stalled subagent's Task call got a heartbeat every 30
seconds, so the last-event age at stop was only 28 seconds. The oldest
pending tool was 3h58m old. The sandbox execution timeout ended the run
after 4 hours.
**Expected behavior**
A tool call that never gets a result ends the turn after a bounded time.
The run fails with an error that names the stall.
**Steps to reproduce**
1. Start an ACP `claude_local` run that uses a subagent to call runtime
MCP tools.
2. Make one of the subagent's tool calls stay `pending` with no further
ACP update (in production this happened without a known trigger).
3. See that the run continues until the adapter execution timeout,
because the Task heartbeats keep the turn active.
**Paperclip version or commit**
`master` at
|
||
|
|
8f1cb11a03 |
fix(workspaces): name the owner action when a shared source is restricted (#16545)
A reused task workspace in a shared project checkout keeps its source denial when another task's restricted worktree shares that checkout (#16095 deliberately does not clone over existing workspaces). The run then failed as an opaque `setup_failed` 403 and the task was blocked with a generic "no live execution path" notice. Convert that exact denial into a `configuration_incomplete` failure with reason `workspace_source_restricted`, give the task notice and the recovery action the owner step, and keep this pre-dispatch owner action out of the unexpected Sentry run-failure stream. The reason and predicate live in a dependency-free module. Access checks are unchanged. Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin Foley
|
8365872743 |
fix(codex-local): retry ACP model capacity failures as transient upstream (#16407)
Classify typed Codex ACP `service` failures about model capacity or server overload as `codex_transient_upstream` / `transient_upstream`, so the existing bounded transient retry and continuation recovery apply. In `unexpected` Sentry mode, skip only runs carrying that full classified evidence; `all` mode still reports them. |
||
|
Devin FoleyandClaude Opus 5.5
|
11d69d04c7 |
fix(native-runtime): report remote recovery labels thrown before coordinator save (#16406)
Restart reattachment can fail in the executor before the native coordinator saves its result. The run failure report now reads the closed remote recovery stage and reason from the caught error when the saved result lacks them. The saved result keeps priority and the same closed-vocabulary validation applies. Add a regression for thrown, wrapped-transport, saved-priority, and out-of-vocabulary cases. Full current-head CI and Greptile 5/5 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
275189d8d6 |
fix(native): report why an unclassified Codex turn failed (#16354)
<!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs use the Paperclip runner to drive a provider session, for example the Codex app server > - When a Codex turn fails, two classifiers look for known failures: a model rejection and a server capacity failure > - When no classifier matches, the run fails with the generic message `Native session failed` and the error code `adapter_failed` > - The run failure report to Sentry then has no data about the cause, because the server drops the Codex `codexErrorInfo` value > - This pull request records a closed-vocabulary diagnostic for that failed turn and sends it to the Sentry `provider_failure` context > - The benefit is that operators can see which Codex failure kind causes these generic failures, and then write a targeted fix ## Linked Issues or Issue Description **What happened?** Hosted staging reports many run failures with the title `Error: Native session failed`. The events have `agent_adapter=paperclip_runner`, `error_code=adapter_failed`, `driverKind=codex_app_server`, and `exitCode=1`. The events do not show why the Codex turn failed. **Expected behavior** A run failure report for a failed Codex turn shows the failure kind, so that an operator can find the cause. The report must not contain provider response text, prompts, or model names. **Steps to reproduce** 1. Start a native Codex run. 2. Make the Codex app server end the turn with `status: "failed"` and an error that is not a model rejection or `serverOverloaded`. For example, use `codexErrorInfo: "contextWindowExceeded"`. 3. Look at the Sentry event for the run failure. Before this change, the `provider_failure` context is empty. **Agent adapter(s) involved** Codex (native runner, `codex_app_server` driver). ## What Changed - `native-provider-failure.ts`: The provider failure observation now records a diagnostic for each failed Codex turn. The diagnostic has only closed values: `provider: "codex"`, `category: "turn_failed"`, a known `CodexErrorInfo` kind (or `unrecognized` / `absent`), an HTTP status from 100 to 599 for the HTTP variants, and a runner failure code from an explicit allowlist of the codes the runner itself writes on a failed Codex turn (any other string becomes `unrecognized`, because a provider error object can carry its own `code`). It never copies the provider message. - New `unclassifiedTurnFailure()` gives this diagnostic only for a failed terminal turn that `forTerminal()` does not classify. `forTerminal()`, error codes, error messages, retries, and finalizer decisions do not change. - New `readNativeTurnFailureDiagnostic()` rebuilds a persisted value from the closed vocabulary and rejects all other values. - `native-session-executor.ts`: The adapter result keeps the generic message. It adds `resultJson.nativeTurnFailure` when the turn failure is unclassified. - `run-failure-diagnostics.ts`: The run failure report copies the validated diagnostic into the `provider` context. Sentry sends this as `provider_failure`. - Tests: unit tests for the vocabulary, HTTP status bounds, unknown kinds, runner codes, and privacy; a diagnostics test; and an update to the end-to-end executor test for the `unknown-error` scenario. ## Verification - `pnpm exec vitest run src/services/native-runtime/native-provider-failure src/services/__tests__/run-failure-diagnostics.test.ts src/services/__tests__/run-failure-report*.test.ts src/__tests__/sentry*.test.ts` (in `server/`): 6 files, 432 passed, 5 skipped. - `pnpm exec vitest run src/services/native-runtime/native-session-executor.test.ts` (in `server/`): 698 passed. - I could not run the full `tsc --noEmit` in my sandbox: the memory limit stopped the process, and the runner vendor build needs `cargo`. CI runs the full typecheck. - To check by hand: make a Codex turn fail with an unclassified error. Then make sure that the Sentry event `provider_failure` context shows `category: turn_failed` and the `codexErrorInfo` kind, and no provider text. ## Risks - Low risk. The change adds diagnostic data only. Run status, error code, error message, retry, and finalizer logic do not change. - Privacy: the diagnostic uses an allowlist of Codex enum names, a bounded integer, and an allowlist of runner-issued failure codes. Tests make sure that provider text does not get through. - If Codex adds a new `CodexErrorInfo` kind, the diagnostic shows `unrecognized` until the allowlist gets the new kind. - This change does not fix the root cause. It gives the data that is necessary to find the cause. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #15347 (capacity classifier), #15304 (model rejection diagnostics). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ccc77c5b5e |
fix(server): record legacy controller lease-loss timings (#16359)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat runner executes legacy agent runs, and a controller lease watchdog keeps durable ownership of each run > - The watchdog renews the lease every 10 seconds and stops the run when a renewal does not finish within the 60-second lease > - A staging run failed in setup with `LegacyControllerLeaseLostError` and the `renewal_deadline` cause, after about eight minutes of preparation and with no server restart > - The setup failure path did not record the cause, and no evidence showed why the renewal did not finish: a stalled database query or a blocked event loop > - This pull request records bounded watchdog timings with each lease loss, on the setup failure path and on the cancellation path > - The benefit is that the next event shows which of the two causes stops renewal, which is the missing fact to fix the root cause ## Linked Issues or Issue Description Refs #16084 (added the closed lease-loss cause for cancelled runs) Refs #16355 (retries a run that a restart stopped before dispatch; a different path) **What happened?** A legacy `paperclip_runner` run on staging failed with `setup_failed` and the message "Legacy controller lease lost". The stack shows the watchdog deadline timer (`renewal_deadline`). The run was in `executionStage: "preparing"` for about 467 seconds. The server started about 11 minutes before the run, so this was not a restart. The run record and the Sentry event do not show the lease-loss cause, because the setup failure path does not read it. Nothing shows if a renewal query was in flight at the deadline, or if the deadline callback ran late. **Expected behavior** When the watchdog loses the lease, the run records the closed cause and the watchdog timings. This applies when the loss fails setup and when it stops the adapter. The failure report contains only the closed cause and bounded numbers. **Steps to reproduce** 1. Start a legacy run with a controller lease. 2. During setup, make a lease renewal query hang for more than 60 seconds, or block the event loop past the deadline. 3. Let the watchdog abort the run. 4. Read `resultJson` on the failed run. Before this change, it has no lease-loss evidence. ## What Changed - `legacy-controller-lease.ts`: The watchdog tracks the last successful renewal, the renewals in flight, the scheduled deadline, and the renewal count. `LegacyControllerLeaseLostError` now carries `leaseLossTiming`: `sinceRenewalMs`, `renewalPendingMs` (or `null`), `deadlineLateMs` (only for `renewal_deadline`), and `renewalCount`. Ownership behavior does not change. - `run-cancellation.ts`: Add the `LegacyControllerLeaseLossTiming` type and `readControllerLeaseLoss()`. The reader accepts only a closed cause and non-negative finite numbers, and clamps each value to 24 hours. - `heartbeat/run-completion.ts`: Persist `resultJson.controllerLeaseLoss` with the cause and timings. This applies to a lease loss that stops the adapter and to a lease loss that fails setup (the thrown abort reason). - `run-failure-diagnostics.ts`: Report `controllerLeaseLossCause` and the four bounded timing fields from `controllerLeaseLoss`. A setup failure now gets the cause label too. - `doc/run-log-events.md`: Document the new record and how to read the timings. - Tests: watchdog timings for a stalled query, a blocked event loop, and a failing query; completion records the evidence for an adapter stop and for a setup failure; the failure report clamps values, drops unknown keys, and drops a record without a closed cause. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/legacy-controller-lease.test.ts src/services/heartbeat/run-completion.test.ts src/services/__tests__/run-failure-report.test.ts src/services/run-cancellation.test.ts` - I could not run these tests locally. The sandbox has a 4 GB memory limit, and `pnpm install` was stopped by the out-of-memory killer during lockfile resolution. CI is the test run for this change. - After deployment, a new lease loss shows `controllerLeaseRenewalPendingMs` and `controllerLeaseDeadlineLateMs` in the Sentry execution context. A pending time near 50 seconds or more points at the database. A large deadline lateness points at the event loop. ## Risks - Low risk. This change adds diagnostics. It does not change when the watchdog stops a run, and it does not prevent the lease loss. - The new `resultJson.controllerLeaseLoss` field is server-written. Readers ignore unknown values. Old rows do not have it. - The failure report gets five closed or numeric fields. It gets no reason text, IDs, or user content. - The event-loop test blocks the loop for about 80 ms to make the deadline late. It uses real timers, so a very slow CI host makes the lateness larger, not smaller. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, Sentry and GitHub connectors). The exact context-window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass (not possible in this sandbox; see Verification) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Devin Foley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
ff0f976de1 |
fix(native-runtime): retain remote recovery failure reasons (#16341)
Preserve closed marker-read and process-verification reasons through native recovery failures and operator reports. Keep existing transport errors and reattachment ownership checks. Add shell, process-probe, persistence and privacy regression coverage. All current-head CI checks pass and Greptile is 5/5. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandClaude Opus 5.5
|
5e8956c207 |
fix(heartbeat): retry legacy runs a restart stopped before dispatch (#16355)
A hard server restart can stop a legacy run after it is claimed but before the adapter is dispatched. The next boot reaped it as failed/process_lost with no retry and held it for operator reconciliation, so the task wake was lost.
The legacy controller only writes executionStage "dispatching" in the lease renewal that admits the adapter. The orphan reaper now revokes the expired foreign-boot controller first, and uses the stage and process identifiers returned by that same UPDATE ... RETURNING. If the run is still "preparing" with no process identifiers, it records bootstrap evidence (providerWorkStarted: false) and uses the existing one-time process-loss retry. In "unexpected" mode, run-failure reporting does not send this retried restart interruption to Sentry. Runs that reached dispatching, native runs, and runs owned by the current boot are unchanged.
Verified exact head
|
||
|
DottaandPaperclip
|
21ee5c9c85 |
feat(sentry): add opt-in run failure reporting with saved run references (#16300)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent runs retain execution results and output in the instance. > - Operators can report run failures to Sentry for investigation. > - A backend Sentry DSN currently enables those reports with no separate choice. > - A repair bot also needs the exact instance, company, agent, and saved run. > - This pull request adds an explicit reporting mode and authenticated run references. > - The benefit is that operators can investigate every failure on selected deployments while keeping other deployments quiet. ## Linked Issues or Issue Description **What existing behavior does this improve?** Optional server reporting of terminal agent run failures to Sentry. **Current behavior** A backend DSN enables run failure reports. Known account, quota, and connection failures are excluded. Reports omit company and agent references. Runs without a task are skipped. Tool-name collision errors omit the conflicting bindings. **Proposed behavior** Set `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected` to retain the existing filters. Set it to `all` to report every failed or timed-out run, including known setup and quota failures. The default is `off`. Expected stops stay excluded. Each report supplies identifiers and API paths that a bot can use with its own authorized Paperclip connection. **Reason and benefit** Operators can enable broad failure reporting on one staging deployment. A bot can fetch the saved run, events, and logs instead of relying on a short error label or copying complete transcripts into Sentry. **Breaking changes** A backend DSN alone no longer enables run failure reports. Existing operators must set `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected` to retain the previous policy. Ordinary server exception capture remains enabled by the DSN. Failure grouping gains the phase and original cause code when available. Refs #16281, #16297, #16298. These cover related tool discovery and failure diagnostics. This change retains their existing boundaries and does not depend on their open changes. ## What Changed - Add `off`, `unexpected`, and `all` modes. Missing and invalid values keep reporting off before diagnostic reads. - Keep the Sentry backend DSN and audited optional SDK requirements. - Include company, agent, run, and optional task identifiers. Include versioned paths for the saved run, events, log, and task. Add absolute run URLs only for a valid public HTTP(S) origin. - Report company runs without a task. Prefer the first-class issue binding when selecting the task reference. - Select saved native provider/model settings, timestamps, and original failure codes. Preserve gateway reason codes through redacted exception projection. - Group by error code, adapter, and available phase/cause. Keep run IDs and error text out of grouping. - Include the tool names and connection bindings in collision errors without copying tool schemas or provider metadata. - Document configuration, upgrade behavior, bot access, redaction, retention, and best-effort delivery. ## Verification - All 56 checks passed for `cccb47a721eaa1cc36e63278fc177b9101477541`. Two unrelated Storybook jobs were skipped by their path filters. No checks remain pending. - Complete [Paperclip CI run](https://github.com/paperclipai/paperclip/actions/runs/38591422284): passed. This covers general and serialized tests, Runner checks, browser E2E shards, typecheck, build, release-registry checks, canary dry run, and Docker checks. - Greptile reviewed that exact head: 5/5, with no unresolved threads or actionable findings. Superagent, Socket, and Snyk checks passed. - Local focused reporting, diagnostics, collision, and real SDK contract tests: 410 passed across five suites. This includes the external-wake filter in both reporting modes. - The audited `@sentry/node@10.71.0` tests use an in-memory transport. No events were sent to Sentry. - `pnpm -r typecheck` and `pnpm build`: passed locally after the final rebase. - `node scripts/check-module-boundaries.mjs` and `git diff --check`: passed. - The full local `pnpm test:run` command was stopped to avoid duplicating the complete CI shards. It did not finish locally. The full suite passed in CI. ## Risks - The new default disables existing run failure exports until the operator selects a mode. The upgrade setting is documented. - `all` can increase event volume and include known user-action failures. Secret redaction and diagnostic limits still apply. - Reporting remains asynchronous and best effort. This PR adds no durable delivery queue and does not install a bot or Sentry alert rule. - API references grant no authority. Existing company, private-task, and run-output checks still apply. Complete transcripts stay in Paperclip. - This changes optional Sentry error reporting. It changes neither first-party Telemetry nor OpenTelemetry span exports. It adds no database migration. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, shell, file editing, and test execution. The exact backend deployment ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
bcfc03bc94 |
fix(sentry): keep provider-rejected Grok wakes out of run failure reports (#16297)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Paperclip Runner can run an external Grok bot. Paperclip offers the bot an assignment and sends a webhook wake. The bot must accept the assignment in 10 minutes. > - When the Grok endpoint rejects every wake with HTTP 4xx, the bot never sees the work. The offer expires and the run fails with `Dot assignment acceptance or execution authority expired.` > - Paperclip already records each wake in `grok_bot_dispatches` with its HTTP status. The operator sees the rejection and can retry it. Thus Paperclip operates correctly in this case. > - But `reportRunFailure` sends this run to Sentry as an unexplained backend error. This makes noise and hides real defects. > - This pull request skips the Sentry report only when durable receipts prove that the Grok endpoint rejected every wake. > - The benefit is a cleaner Sentry signal. Local defects, uncertain delivery, and accepted work stay visible. ## Linked Issues or Issue Description Sentry issue `PAPERCLIP-BACKEND-3M` (staging, 1 event). **What happened?** A Grok bot run on staging failed after 10 minutes in the `preparing` stage. Sentry got the error `Dot assignment acceptance or execution authority expired.` with the code `native_provider_terminal_failed`. The stored receipts show two `mailbox_updated` wakes for the assignment. The Grok endpoint rejected both wakes with HTTP 400: the first wake and one operator retry. The assignment was never accepted. Wakes with the same shape got HTTP 200 before and after this run. **Expected behavior** The run fails, and the operator sees the rejected delivery. Sentry does not get a backend error for a proven external rejection. **Steps to reproduce** 1. Connect a Grok bot agent and verify its routine. 2. Make the Grok endpoint return HTTP 400 for work wakes. 3. Assign a task to the agent and wait 10 minutes. 4. See that the run fails and that a Sentry event is sent. **Paperclip version or commit** `f69c6a8` (master). ## What Changed - `server/src/services/run-failure-report.ts`: Add `isRejectedExternalBotWake`. It returns true only when all of these conditions are true: - The run failed in native mode with `native_provider_terminal_failed`. - The finalization `failureDetail.originalFailureCode` is `dot_assignment_expired`. - The run has exactly one Dot assignment, and that assignment was never accepted. - The assignment has at least one `mailbox_updated` wake, and each wake has the status `rejected` with an HTTP 4xx status. - `captureTerminalRunFailure` skips the Sentry capture when the check is true. If the receipt check itself fails, it logs a warning and the failure stays reportable. The run status, the task, and the operator recovery do not change. - `packages/db`: Add the index `grok_dispatch_assignment_kind_idx` on `grok_bot_dispatches (assignment_id, kind)` (migration `0346`). The receipt lookup and the dispatcher's per-assignment checks use this index. - `server/src/services/__tests__/run-failure-report.test.ts`: Add one test for the skipped case. Add nine tests for cases that must stay reportable: no wake, a local rejection with no HTTP status, a delivered wake, uncertain delivery, accepted work, a different provider failure code, no finalization, more than one offer, and a receipt read that fails. ## Verification - `pnpm --dir server exec vitest run src/services/__tests__/run-failure-report.test.ts`: 208 tests pass. - I removed the new catch and ran the tests again. The receipt-read-failure test failed. - `pnpm --filter @paperclipai/db run check:migrations` and `src/migration-snapshot-drift.test.ts` pass. - I removed the source change and ran the new tests again. The skipped-case test failed and the six reportable cases passed. This shows that the test proves the change. - I did not run the full `pnpm -r typecheck` locally. The server `typecheck` script needs `cargo` to build the Runner binary, and the sandbox stopped `tsc` with SIGKILL. CI runs the full typecheck. ## Risks - Low risk. The change only removes one Sentry report. It does not change run state, recovery, or what users see. - The check uses durable receipts, not message text. A wake with no HTTP status (local validation or credential failure) stays reportable, so a Paperclip defect cannot hide behind this rule. - For failed native runs with `native_provider_terminal_failed`, the check adds up to three indexed reads. Other runs do no extra reads. - Migration `0346` builds one index in a transaction. `grok_bot_dispatches` was created in `0344` and is small, so the build lock is short. The table is not on the migration gate's large-table list. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic tool use with shell, code search, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
03fcd3c1fb |
fix(native-runtime): report which close barrier failed (#16298)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agent runs use the Paperclip runner. When a run ends, the controller closes the session. Close must prove that the runner durably suspended before the controller releases its checkpoint. > - If close cannot prove this, the run fails with `provider_transport_failed: runner did not durably suspend before checkpoint`. > - The runner knows which barrier was not proven. It writes that settlement only to the instance run log. > - The Sentry report for the failed run has no settlement evidence. Operators cannot tell a stuck suspend command from an undrained provider, an unreadable provider state, or unsettled tool results. > - This pull request sends that settlement to the run failure report as closed labels and counts. > - The benefit is that the next occurrence shows which barrier failed, so the root cause can be fixed instead of guessed. ## Linked Issues or Issue Description No public issue exists. This describes the bug. **What happened?** Hosted staging native Codex runs failed with `provider_transport_failed: runner did not durably suspend before checkpoint`. This happened on two runs, on 2026-09-30 and 2026-10-14. The second run used a release from 2026-10-14. The Sentry events contain only the error code, run phase, driver kind, and duration. They contain no stack and no settlement data. The native executor calls `reportRunFailure(db, run)` without the caught error, and the runner settlement is never read. **Expected behavior** A failed close report should tell an operator which barrier was not proven: runner suspension, provider drain, provider state, or semantic tool settlement. It must not expose tool identities, digests, provider turn IDs, or unredacted error text. **Steps to reproduce** 1. Run a native Codex session whose close cannot prove durable runner suspension. For example, the runner suspend command does not complete before the close deadline. 2. Observe the Sentry run failure event. Before this change, `run_execution` has no close settlement fields. ## What Changed - `collectRunFailureDiagnostics` accepts a new `nativeCloseFailure` option. It reads only a `native_session_close_unrecoverable` error, directly or up to four `cause` levels deep. - It adds closed labels and counts to `run_execution`: `closeRunnerSuspended`, `closeProviderDrained`, `closeSuspendCommandStatus`, `closeRunnerLifecycle`, `closeRunnerIdentityMatches`, `closeProviderState`, `closeProviderPendingEventCount`, `closeProviderTurnActive`, `closeSemanticResultPersistenceFailed`, `closeSemanticToolsPending`, `closeSemanticToolDeliveriesOpen`, and `closeSemanticResultFailures`. - Values outside the known vocabularies are dropped. Getters that throw are contained. - The native session executor passes the caught error as `nativeCloseFailure` when it reports a terminal run failure. It does not pass the error as `error`, because the raw error text and stack bypass the executor's identity redaction. - `doc/observability.md` documents the new fields and the privacy boundary. ## Verification - `cd server && npx vitest run src/services/__tests__/run-failure-diagnostics.test.ts src/__tests__/run-failure-sentry.test.ts`: 130 tests pass. - New tests check the full label set, a wrapped cause, null suspension and provider states, unrelated error codes, hostile and unknown values, and a missing settlement. They also check that no tool identity, digest, turn ID, or error text leaves the module. - Negative control: with the reader disabled, the new main test fails. - A scoped `tsc` run on `run-failure-diagnostics.ts`, its test, and `run-failure-report.ts` passes. The full server `tsc` and `pnpm build` were not run locally. The sandbox has a 4 GiB memory limit and no `cargo` for the runner binary. CI runs the full checks. ## Risks - Low risk. The change only adds Sentry context fields for one error code. Close, quarantine, retry, and recovery behavior do not change. - Sentry grouping does not change. The fingerprint stays `[errorCode, agentAdapter]`, and no exception chain is added. - The data path is Observability (Sentry error reporting, gated by the operator DSN). It does not change Telemetry. The fields contain only booleans, fixed labels, and counts. The DSN gate is unchanged. - These fields are observations. They do not prove a root cause by themselves. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), run through Claude Code as an agent with tool use (shell, file edits, test execution, Sentry and GitHub connectors). Context window and reasoning mode are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Devin Foley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a64969b288 |
fix(server): attribute legacy runs stopped by controller lease loss (#16084)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat runner executes legacy agent runs. A controller lease watchdog keeps durable ownership of each legacy run. > - When the watchdog loses the lease, it aborts the adapter. The abort reason was a plain `Error`, and nothing recorded why the run stopped. > - Completion then saved the cancellation as source `unknown`. Sentry got an unexpected cancel that it could not tell apart from other unrecorded stops. > - We saw this on a sandboxed Grok run. The system cancelled it after about 22 minutes with `cancellationSource: unknown`. Workspace copy-back then failed. Every `cancelRun` path records a stop request, so an unrecorded stop this late points to the lease watchdog. > - This pull request records a lease loss as its own cancellation source, with a closed cause label. > - The benefit is that the next event shows whether the lease loss caused the stop, and which renewal failure caused it. That is the missing fact to fix the root cause. ## Linked Issues or Issue Description Refs #15790 Refs #15920 **What happened?** A legacy sandbox run was stopped by the system while it was working. The run record and the Sentry diagnostics show `cancellationSource: unknown`, with no initiator. The controller lease watchdog aborts the adapter with a plain `Error("Legacy controller lease lost")`. Completion does not read the abort reason, so it records the stop as `unknown`. **Expected behavior** When the host stops its own adapter because it lost the controller lease, the run records that cause. Diagnostics name the cause with a closed label. A stop that a user or the control plane requested earlier keeps its recorded source. **Steps to reproduce** 1. Start a legacy run that has a controller lease. 2. Make lease renewal fail: the renewal query errors, the update matches no row, or the lease deadline passes. 3. Let the adapter return its cancelled result. 4. Read `resultJson.cancellation` on the run. Before this change, `source` is `unknown`. **Agent adapter(s) involved** Not adapter-specific (core bug). We saw it with `grok_local` in a sandbox. **Additional context** This change adds diagnostics. It does not prevent the lease loss. #15920 retries a transient copy-back failure. It does not help here, because the stopped sandbox rejects every retry. ## What Changed - `legacy-controller-lease.ts`: The watchdog aborts with a new `LegacyControllerLeaseLostError`. The error carries a closed cause: `renewal_rejected`, `renewal_deadline`, or `renewal_error`. - `run-cancellation.ts`: Add the `controller_lease` cancellation source. Add `controllerLeaseLossCancellation()` and `readControllerLeaseLossCause()`. The reader accepts only the server-written reason text. - `heartbeat/run-completion.ts`: When the outcome is `cancelled` and the abort reason is a lease loss, record a `controller_lease` cancellation (`expected: false`). This covers both an adapter that returns a cancelled result (`completeRun`) and one that throws after the abort, for example through `signal.throwIfAborted()` (`failRun`). A stop request recorded earlier still takes precedence. - `run-failure-diagnostics.ts`: Add `controllerLeaseLossCause` to the execution diagnostics. Only the closed label goes to Sentry, never the reason text. - `heartbeat/run-state.ts`: preserve `controller_lease` in the bounded SQL projection when saved results exceed 64 KB. A database regression verifies the default run-detail read. - `canContinueCancelledRun` stays unchanged. It does not accept `controller_lease`, so a lease loss does not authorize automatic continuation. - `doc/run-log-events.md`: Document the new source and cause labels. - Tests: unit tests for the cause on each watchdog path, completion tests that record the source when the adapter returns and when it throws the abort reason (both fail without the change), tests that a recorded stop request takes precedence, and a test that the Sentry report contains the closed label and not the reason text. ## Verification - `cd server && npx vitest run src/services/run-cancellation.test.ts src/services/legacy-controller-lease.test.ts src/services/__tests__/run-failure-report.test.ts src/services/heartbeat/run-completion.test.ts`: 238 passed, including the embedded Postgres suites. - I reverted only `run-completion.ts` and ran the new completion test. It failed with `source: "unknown"`. With the change, it records `controller_lease`. - I removed only the `failRun` attribution and ran the new thrown-error test. It failed without the `controller_lease` record, and passes with it. - Final full repository `pnpm -r typecheck` and `pnpm build` pass. - Final local regression verification: all 247 tests pass across run-state, cancellation, watchdog, failure reporting, and completion suites. The large-result test fails before the SQL allowlist change. All latest-head CI checks pass (54 successes and two intentional skips). Fresh Apex is 5/5 on `708856c568d07d7b446a4a4af2edc6d9eec7d155` with no unresolved findings. ## Risks - Low risk. The change adds a new source value to server-owned JSON. Old rows keep their values. The readers accept the new value, and no UI code switches on this field. - A lease-loss cancel stays `expected: false`, so it is still reported as unexpected, as before. Now it has a more specific source. - The Sentry payload gets one new closed label (`controllerLeaseLossCause`). It does not include reason text or IDs. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs). OpenAI GPT-6 (`gpt-6`) through Codex added the large-result regression and SQL projection fix, with shell, file-edit, and test tools. The exact context-window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
4b958ecbd8 |
fix(server): keep classified provider quota runs out of Sentry (#16029)
A run that stops because the user's provider account hit its usage limit is an expected wait, not a Paperclip defect. The adapter already classifies it (errorCode and resultJson.errorFamily are both provider_quota) and recovery schedules a retry at the reset time. Skip the Sentry report only when both adapter-owned fields agree, the run status is failed, no signal stopped it, and the failure did not happen during setup. Timeouts, cancellations, signals, other codes and missing or mismatched families still report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Dotta <cryppadotta@users.noreply.github.com> |
||
|
|
4b766b8729 |
fix(acpx): record turn activity at the moment a stop fires (#15962)
Record stop-time ACP turn diagnostics (last-event age, oldest pending tool age, pending tool kinds) so wall-clock timeout events show whether the turn was stalled. Diagnostics only. Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
657e71584a |
fix(daytona): record provider state when lease creation hits its deadline (#15946)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents can run in Daytona sandboxes. The Daytona plugin acquires a
lease before each run.
> - Some acquisitions time out. The report then says
`environmentAcquisitionPhase=create` and that the whole 300 s budget was
used.
> - The Daytona SDK create timeout is equal to the acquisition budget.
The outer timer always fires first. Thus the report never shows the
state that Daytona left the sandbox in.
> - Without that state, we cannot choose a fix. A stuck snapshot pull, a
slow start, and a missing allocation need different fixes.
> - This pull request adds one bounded, read-only provider lookup at the
create deadline. It records a fixed-enum provider state in the existing
acquisition diagnostic.
> - The benefit is that the next timeout identifies where the provider
stopped. Retry, budget, and cleanup behavior do not change.
## Linked Issues or Issue Description
Refs #15736 (bounded Daytona acquisition failure observations). This
pull request extends those observations.
**What happened?**
A Daytona lease acquisition timed out with `Daytona lease acquisition
timed out; allocation cleanup is pending`. The run-failure report gave
`environmentAcquisitionPhase=create`,
`environmentAcquisitionElapsedMs=300004`, and
`environmentAcquisitionBudgetMs=300000`. The report did not show the
Daytona sandbox state. The SDK create timeout is equal to the outer
budget, so the outer timer always fires first. The SDK never reports the
stuck state.
**Expected behavior**
When creation is still pending at the deadline, the report should show
where the provider left the allocation. Examples are `pulling_snapshot`,
`starting`, `error`, or `absent`.
**Steps to reproduce**
1. Configure a Daytona environment.
2. Start a run when Daytona does not finish create and start within the
acquisition budget.
3. Inspect the setup-failure report. Only the phase, the elapsed time,
and the budget are present.
**Paperclip version or commit**
`master` at
|
||
|
Devin FoleyandPaperclip
|
9befc66a44 |
Retain bounded Daytona acquisition failure observations (#15736)
## Thinking Path > - Paperclip manages AI-agent work and reports failed runs. > - Some runs acquire a Daytona sandbox before model execution. > - Acquisition has one budget for creation, setup, and inline cleanup. > - A timeout currently carries cleanup ownership but loses the step that failed. > - This pull request carries a bounded observation through the existing error envelope. > - Operators can identify the first failed step without exposing provider responses or changing recovery. ## Linked Issues or Issue Description **What happened?** An acquisition timeout reports pending allocation cleanup without the first failed step. The provider already knows its current step, but that evidence is discarded at the worker boundary. A setup failure followed by slow cleanup is also indistinguishable from slow creation. **Expected behavior** The immediate setup-failure report should contain a fixed phase and bounded elapsed/budget values. Cleanup ownership, original messages, deadlines, retry policy, and task behavior should remain unchanged. **Steps to reproduce** Use the provider regression fixtures to stall creation, workspace setup, shell detection, expiry configuration, or sentinel upload. Also fail setup early and let cleanup consume the remaining budget. The previous source emits no phase diagnostic. **Paperclip version or commit** Base commit: `98e3cec78e8461774022a5ba0a777b09d4d16b96`. **Deployment mode** Hosted and self-hosted installations that use the Daytona sandbox provider and enable Sentry. Related public changes: #13979 established failed-allocation cleanup ownership; #14097 preserves the provider acquisition budget; #15637 carries bounded workspace-transfer evidence. Duplicate searches found no existing acquisition-phase diagnostic patch. ## What Changed - Capture the first acquisition failure before inline cleanup. Keep the observation if cleanup later reaches the deadline. - Add an optional, validated observation to the existing creation-cleanup envelope for acquire calls. Old ownership envelopes remain valid. Malformed observations do not alter cleanup. - Bind immediate reporting to an actual RPC error with matching company, environment, and run ownership. Project only three bounded fields from private host evidence. - Add provider, SDK, actual worker/client, PostgreSQL cleanup, collector, and real-Sentry regressions. Document the evidence limits. ## Verification - Passed 404 focused tests across the SDK cleanup envelope, complete Daytona provider suite, diagnostic collector, actual built worker/client subprocess, and mandatory real-Sentry SDK suite. - Passed 10 focused PostgreSQL cleanup-boundary cases. The filter excluded 152 unrelated cases. - Passed `pnpm -r typecheck` and `pnpm build` on the committed source. - Full `pnpm test:run` is running. Final outcomes will be recorded before merge. - Independent source review passed. Replacing the provider with the original source fails all 8 new phase/timing assertions. Replacing the host boundary with the original source fails all 3 owned diagnostic cases while 7 negative cases remain clean. - CI passed: 54 successful checks and 2 expected Storybook skips. This includes all server, workspace, serialized-server, Runner, browser, build, canary, and real-Sentry SDK lanes. - Greptile scored the committed source 5/5 with no review threads. - No live provider operation, task replay, runtime configuration change, or deployment was performed. ## Risks Low risk: this only adds optional diagnostic evidence. The first observed failing phase does not prove an upstream cause. Elapsed time ends at that observation, not at completion of later cleanup. Evidence is process-local and accompanies immediate failure reporting; post-restart reports cannot reconstruct it. No historical run or lease is backfilled. The wire schema remains v1 and cleanup ownership parsing is unchanged. ## Model Used OpenAI Codex, GPT-6, with repository inspection, code editing, local execution, and independent agent review. The exact model snapshot and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
7333e4eba8 |
Capture bounded workspace base-ref failure diagnostics (#15719)
## Thinking Path > - Paperclip manages AI agents and their work. > - A run can prepare a Git worktree before an adapter starts. > - That preparation must resolve the configured base ref. > - The same unresolved-ref error can follow a missing ref, failed fetch, or failed credential lookup. > - Current failure reports do not distinguish these paths. > - This PR records bounded outcomes from the operations that already run. > - Maintainers can use this evidence without changing task recovery or hiding failures. ## Linked Issues or Issue Description Related: #11892 established remote-only base-ref resolution and pre-adapter recovery. I found no open PR for these diagnostics. **What happened?** A fresh worktree can fail with `UnresolvedWorkspaceBaseRefError`. Its failure report has no closed fields that distinguish remote lookup, credential lookup, fetch, and local ref-probe outcomes. An empty warning list does not prove that a fetch ran or that a remote ref is absent. **Expected behavior** Keep the task-visible failure and owner recovery action. Report bounded observations from the existing operations. Keep unknown causes reportable. **Steps to reproduce** Use a local Git repository with a configured base branch that is absent. Repeat with no origin, an inaccessible local origin, or a rejected credential resolver. Start fresh workspace preparation. Before this change, each path lacks the new typed observations in the failure report. Tests use local repositories and intercepted fetch commands; they require no external Git service. **Paperclip version or commit** Reproduced against source commit `1602512820`, built from source. This affects core workspace preparation before adapter dispatch. ## What Changed - Record closed remote lookup, credential lookup, fetch, and local ref-probe outcomes. Limit new output parsing to 8 KiB. - Attach evidence privately to the resolver's own error. Reject copied, forged, inherited, accessor, and malformed evidence. - Persist allowed values in the existing setup result and export them through run-failure diagnostics. - Preserve Git commands, credential fallback, warning text, recovery actions, and Sentry reporting. - Add real Git, heartbeat, reporter, and real Sentry envelope tests. Document the fields and their limits. ## Verification - Current head: `015b2737df004e971eeeb134ebc9a1024e244494`, based on `be1bc5d494d`. The only rebase conflict was in heartbeat test imports. All three imports were retained; `git range-diff` confirms no other patch change. - Full `pnpm -r typecheck` and `pnpm build` passed on the current head. - Fresh focused tests passed: 13 real-Git producer cases, 279 complete collector/reporter cases, two actual heartbeat recovery cases, and all nine real Sentry SDK cases. The required-SDK gate used version 10.71.0. Actual envelopes contain only allowed fields, with no private fixture text or cross-event leakage. - Independent review passed all 16 combined heartbeat cases from both integrated changes and five real-Git probes with identical baseline command sequences and outcomes. The original producer negative control failed on the absent receipt; the candidate passed. No review findings remain. - Current-head CI is complete: 54 successful checks and two intentional Storybook skips. Greptile is 5/5 on this exact head, with no unresolved threads or merge conflicts. - Local aggregate limitation: the earlier full run reported one failure in an unchanged tool-access case and was stopped with SIGINT (exit 130) to isolate it. Vitest did not print the deferred assertion summary. The unchanged complete file passed 420/420 alone. That aggregate is not claimed as a pass. Before the mechanical rebase, the complete workspace-runtime file also passed 172/172. The fresh full CI result covers the integrated head. - An initial independent combined reporter run hit its database setup timeout; the unchanged complete reporter file then passed 186/186 alone. Earlier-head CI lost four runners and recovered on one verification retry; current-head CI passed separately. ## Risks - These observations do not prove that a remote ref is absent. All affected failures remain reportable, including resumed and provider-started runs. - The new fields include no repository URL, ref text, customer path, raw command output, or credential. Existing task-visible details are unchanged. - No migration, dependency, authorization, retry, or deployment change. ## Model Used OpenAI GPT-6 through Codex, with tool execution and parallel independent review. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
1602512820 |
fix(sentry): classify expected credential sharing denials (#15714)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A run can use only an AI credential shared with its responsible user. > - The selection service correctly rejects a credential outside that user's sharing permissions. > - That known configuration rejection still becomes a Sentry error because it has no owned selection marker. > - This pull request marks only that verified sharing rejection and reuses the existing pre-provider reporting gate. > - The task remains blocked with its recovery action, while unexpected authorization and runtime failures remain reportable. ## Linked Issues or Issue Description Refs #15618, which introduced owned provenance for expected AI account selection blockers. The credential-sharing subtype still used a plain HTTP 403. No duplicate open fix was found. **What happened?** An active responsible user selects an AI credential they cannot use. Selection correctly throws `This credential is not shared with the responsible user`. A fresh task remains blocked, but the broad `ConfigurationIncompleteFailure` wrapper is also sent to Sentry. **Expected behavior** Preserve the denial, HTTP 403, task error and owner recovery action. Keep this proven pre-provider configuration rejection out of Sentry without suppressing unrelated permission failures. **Steps to reproduce** Assign a fresh run an explicit personal credential owned by another user, or a shared credential whose human audience excludes its active responsible user. The existing selection check denies access before adapter dispatch. ## What Changed - Add a dedicated sharing-rejection factory that preserves the existing 403, message and absent details while retaining one private bounded reason. - Use it only after active responsible-user membership and the existing credential checks have passed. Missing identity and other authorization checks remain unmarked. - Keep the existing failed/preparing/setup/bootstrap reporting conditions and provider/resume exclusions unchanged. - Add real selection, heartbeat recovery, reporter-boundary and forged-error regressions. Update monitoring documentation. ## Verification - Producer, private-marker and run-failure reporter suites: 310 tests passed. - Actual heartbeat recovery cases: 6 passed; 377 unrelated cases filtered. These verify failed runs, blocked tasks, active owner recovery actions, failed wakeups and no adapter dispatch. - Coverage includes personal and restricted shared credentials, allowed audience changes, missing identity, inactive credential owners, wrong binding kinds, membership in another company, agent permission checks, forged 403 markers, resumed native input and provider work already started. - `pnpm -r typecheck` and `pnpm build` passed. All 53 PR checks passed; two unrelated Storybook jobs were skipped. - The full local aggregate was **not green**: it reported an existing browser-sign-in case and a real-runner damaged-resume case before the broad run was stopped (exit 130). The full affected AI-connections file then passed 111/111 in isolation, including the browser case. The unchanged native-session-resume file reproduced 47 passed / 1 failed on both this commit and an untouched base checkout with the same local dependencies and runner binary: the fixture expected a succeeded terminal result and received a failed one. No broader local-suite pass is claimed; full sharded CI passed. - Independent review passed the same 310 focused tests and six heartbeat cases from a clean checkout. Replacing only the producer with its original implementation fails both new marker assertions while preserving 403; restoring this change passes. - Greptile scored this exact commit 5/5 with no review threads. - Source, tests and documentation were scanned for credentials, personal data, private URLs and generated artifacts; none found. ## Risks - A known sharing rejection no longer emits a Sentry run-failure event only when the existing fresh setup proof is present. It remains visible in the task and recovery UI. - No credential-use permission, HTTP contract, execution or retry behavior changes. - This is not a blanket filter for HTTP 403 or `ConfigurationIncompleteFailure`. Unknown causes and resumed or post-provider failures remain reportable. ## Model Used OpenAI GPT-6 through Codex, with tool execution and parallel independent review. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run focused tests locally and they pass; broader local limitations are documented above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
1c4ce44e13 |
fix: retain bounded workspace transfer failure evidence (#15637)
## Thinking Path > - Paperclip manages AI agents and the work they produce. > - Sandbox work must return to the host when an agent run ends. > - A failed transfer must keep its source available for recovery. > - Some transfer errors lose their useful fields before restore diagnostics reach Sentry. > - This change records fixed transfer stages, failure kinds, and numeric RPC codes. > - The next failure can identify the operation that failed without exposing workspace contents. ## Linked Issues or Issue Description **What happened?** A workspace transfer can fail after model execution succeeds. Several command, archive validation, and download errors become plain errors. The existing RPC envelope then reports `unknown`, with no transfer stage or command exit code. A host RPC timeout also loses its code from restore diagnostics. **Expected behavior** Keep bounded evidence through the provider, worker RPC, restore result, and Sentry context. Keep the same exception, error message, retry rules, and source retention policy. A transfer timeout must remain separate from the model execution timeout flag. **Steps to reproduce** Return a nonzero exit code from the sandbox archive command, return a per-file download error, exceed a tar listing limit, or time out the sync-out RPC. Observe the missing fields in the saved restore diagnostic. The added tests use local fixtures for these cases. Related public work: #15479 preserves the source after restore failure. #15481 carries bounded sync-out diagnostics through RPC. This change supplies missing producer evidence and extends that same envelope. I checked the roadmap and searched open PRs for duplicate transfer diagnostic work. ## What Changed - Add optional transfer stage and failure kind fields to the existing diagnostic envelope. Capture command exits and listing deadlines at their producers. - Record only known codes from typed RPC errors at the sync-out boundary. Revalidate every field before persistence and Sentry projection. - Keep annotations private to each outbound request and restore settlement. Preserve frozen error identity and prevent evidence from leaking across concurrent or later calls. - Document the fields. Cover producer failures, quota limits, old workers, concurrent error reuse, and redaction through the real Sentry SDK. ## Verification - Focused SDK, provider, host, persistence, and real Sentry tests: 449 passed, no skips. Independent review also ran the focused contracts and all provider tests. The optional Sentry SDK is required for this check, so the contract tests cannot skip. - A compiled Daytona transfer and compiled SDK worker pass a local RPC round trip. This uses the development TypeScript loader for workspace dependency exports. The transfer and capture modules are compiled JavaScript. - `pnpm -r typecheck` and `pnpm build` passed. The excluded Daytona package also passed `tsc --noEmit`. - The initial local `pnpm test:run` stopped in the general-server group: 60 suites could not initialize embedded PostgreSQL because the isolated install skipped its Darwin library-link setup. The existing package postinstall repairs this; `initdb --version` now passes. All 60 affected suites then passed: 1,342 tests, no skips. No local full aggregate pass is claimed. - Independent source review covers privacy, concurrency, packaging, and recovery behavior. ## Risks - These diagnostics help identify future failures. They do not establish or repair the cause of a past transfer failure. - New fields are optional. Older workers remain compatible. Unknown values are omitted. - The change adds a small request-local diagnostic scope. It keeps the original thrown errors, archive confinement, cleanup order, timeout values, retry count, and source retention policy. - No schema or deployment changes are required. No live provider actions are part of validation. ## Model Used OpenAI Codex, based on GPT-6, with repository inspection, code execution, and independent agent review. The exact deployed model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
65b5581805 |
Keep proven AI account selection blockers out of Sentry (#15618)
Preserve trusted evidence from explicit AI account selection rejections so known setup blockers stay out of Sentry. Keep task failure, blocked state, recovery actions, HTTP behavior, and unexpected error reporting unchanged. Validated with full typecheck/build, focused PostgreSQL lifecycle and reporting tests, independent reporting tests, and all PR CI checks. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
3d0e74e743 |
Keep proven external connection failures out of Sentry (#15590)
Apply the reviewed change for Keep proven external connection failures out of Sentry. Validation: local typecheck/build and focused regression tests, passing exact-head CI, Greptile 5/5, and independent source review. The PR records full-suite evidence and any local environment limitations. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
5717523b9e |
fix: retain the original repository for reused task workspaces (#15528)
Apply the reviewed change for fix: retain the original repository for reused task workspaces. Validation: required local typecheck and tests, passing CI, Greptile 5/5, and independent review of the exact source head. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
1d19f9b562 |
Distinguish failed Git inspection from missing worktree registration (#15525)
Apply the reviewed change for Distinguish failed Git inspection from missing worktree registration. Validation: required local typecheck and tests, passing CI, Greptile 5/5, and independent review of the exact source head. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
6b171c9616 |
Keep proven local workspace configuration conflicts out of Sentry (#15521)
Apply the reviewed change for Keep proven local workspace configuration conflicts out of Sentry. Validation: required local typecheck and tests, passing CI, Greptile 5/5, and independent review of the exact source head. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
128c95837b |
Retain bounded orphan process-loss diagnostics (#15475)
Record bounded observer uptime, run and output ages, process-check observations and retry eligibility before orphan cleanup changes the evidence. Preserve existing recovery and reporting behavior and omit process IDs, raw paths and credentials. Verified focused helper, actual reaper and real SDK regressions, full typecheck/build, exact-head CI and independent review. An unchanged Cursor timeout passed its isolated retry; unrelated local Slack timing failure is documented separately. Greptile 5/5 with no unresolved comments. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
43b0aff54b |
Keep pre-dispatch secret configuration blockers out of Sentry (#15472)
Keep confirmed pre-dispatch missing-secret setup blockers out of Sentry while preserving failed runs and owner recovery actions. Runtime, provider and ambiguous failures remain reportable. Verified full exact-head CI, focused local regressions, workspace typecheck and independent review; Greptile 5/5 with no unresolved comments. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b67db12d90 |
feat: add durable storage for private tasks (#14717)
Add private task and project ownership, downward access grants, and immutable run/workspace provenance. Apply migration 0313 with bounded batch commits and concurrent indexes. Keep the enforcement and sharing changes in their dependent PRs. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
1d5a439e2b |
Diagnose native model rejections and preserve repairable reviews (#15304)
## Thinking Path > - Paperclip manages agent execution and recovery. > - A native provider can return a failed turn with an accepted result. > - This path replaced a structured model rejection with a generic failure. > - Finalization could also queue the same failed request for recovery. > - This change retains a bounded model and authentication diagnosis. > - Workers wait for a configuration fix; reviews retain their assignment. ## Linked Issues or Issue Description **Describe the bug** A native Codex turn can report HTTP 400 because the selected model is not supported by its ChatGPT connection. When the turn also returns a failed semantic result, the run shows `Native session failed` and `adapter_failed`. The native failed-result finalizer can create a retry intent before the adapter error is saved. **To Reproduce** Use a native Codex session that returns a structured `invalid_request_error` model/ChatGPT rejection and an accepted failed result. The regression tests reproduce this with a stub provider and a real isolated PostgreSQL database. No provider account or live request is needed. **Expected behavior** Show an actionable error and stop automatic retries of the rejected request. Block the current worker for a configuration fix. Preserve a pending review so its reviewer can explicitly retry after repair and resolve the original assignment. A stale run must not block a new task owner or override a completed review. **Version and deployment mode** The affected source path is present at `858094ba81`. This change applies to native Codex execution in local and remote environments. Related work: #13853 covers CLI protocol compatibility, #14942 refreshes runtime/model pins, and #12767 proposes legacy adapter model validation. This change concerns accepted native failed results and does not replace those changes. ## What Changed - Recognize only a bounded structured HTTP 400 provider error for the exact selected model and failed turn. - Save a fixed actionable error and four fixed diagnostic fields. Exclude raw provider text and model names from the new diagnostic. - Re-derive finalization evidence from the pinned execution and committed runner event, including its canonical hash and identity bindings. - Block only the current worker. For a current pending reviewer, preserve the original review decision and status version, release execution, and create no automatic recovery action. Recheck execution ownership under the issue lock. Preserve superseded and completed reviews. - Keep unknown failures, accepted results, cleanup, and ownership fences intact. Advance the status policy version to `phase6-v8`. - Route this exact failure code through the existing configuration recovery path. Permit an authenticated Board retry of this native failure only for its saved, still-pending review; discard caller review markers and revalidate at dispatch. ## Verification - Node 24: `pnpm -r typecheck` and `pnpm build` passed. - Independent focused validation: 222 tests passed across the new observer and PostgreSQL integration suites, status arbiter, diagnostic redaction, recovery classification, wake policy, and wake use cases. - The integration tests use actual result persistence, finalization, and the wake dispatcher. They cover zero retry dispatch for an authorized rejection, repeated reconciliation, pending/superseded/resolved reviews, reassignment, and a same-agent successor at an unchanged status version. - Final-commit verification: full typecheck and build passed; independent route/integration/arbiter/review validation passed 136 tests with no remaining source findings. This includes authenticated retry refusals, actual heartbeat retry admission, atomic queued-review claim, and acceptance of the original review. - The local aggregate test run was stopped to avoid duplicating the canonical Linux CI aggregate on this development host. Its partial log showed no failures; no full local-suite pass is claimed. The full hosted Linux CI aggregate passed on the final commit. - No live provider calls, runtime changes, or schema changes were made. ## Risks The classifier is intentionally narrow. A changed provider response format remains an unknown failure. No global model restriction is introduced. The worker blocker requires current task authority and does not replace newer assignments or completed review decisions. A pending review remains in review and requires an explicit retry after configuration repair; it is not rebound to a new status decision. The retry decision is durable. There remains a small existing gap between status/effect commit and diagnostic metadata persistence; a crash there can leave a generic diagnostic while preserving the blocked decision. No schema migration is required. ## Model Used OpenAI Codex, GPT-6, with reasoning, tool use, and code execution. The exact serving model suffix and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #123` / `Refs #123` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal ticket id or instance-derived details - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
efac8ff2f4 |
Add bounded Git integration restore diagnostics (#15291)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Sandbox runs must restore their workspace before finalization can succeed. > - Restore diagnostics identify the failed phase and operation. > - A Git integration exit code can still describe several different failures. > - This pull request adds fixed command labels and supported failure classes. > - Operators can distinguish these failures without collecting private Git output. ## Linked Issues or Issue Description Refs #15005. This branch includes merged #15268. It labels that change's locked ref transaction and nested branch probe without changing their behavior. **What happened?** A failed Git integration can report only `git_integration`, `unknown`, and an exit code. That evidence does not identify the failed command. Some Git versions also return exit 1 for both a merge conflict and an invalid object. **Expected behavior** Record a fixed command family and a supported failure class. Keep unknown cases as `unknown`. Exclude command arguments, process output, paths, repository URLs, filenames, and ref names. **Steps to reproduce** The tests create local repositories with conflicting commits, a missing object, and an expected-old ref mismatch. They call the real Git operations and inspect the resulting diagnostic. No hosted workspace or external provider is used. **Paperclip version or commit** Base: `e99854249c`. **Deployment mode** Built from source. The diagnostic applies to sandbox workspace restore. ## What Changed - Label Git integration calls with a closed command enum. Preserve arguments, options, errors, and retry behavior. - Recognize supported object and ref errors and OS permission codes. Require both exit 1 and completed tree output for a merge conflict. - Carry the closed fields through the existing restore receipt, saved adapter result, and Sentry projection. Revalidate saved metadata before projection. - Test nested wrappers, parallel failures, reused errors, handled probes, result settlement, and privacy with the real Sentry SDK. - Document the field contract and its limits. ## Verification - Eight focused suites pass: 295 tests. These cover Git sync, restore diagnostics, result settlement, teardown, sandbox runtime, failure projection, the real Sentry SDK, and native warm-workspace Git history. - `PAPERCLIP_REQUIRE_SENTRY_TEST_SDK=1 pnpm exec vitest run packages/adapter-utils/src/workspace-restore-diagnostics.test.ts packages/adapter-utils/src/workspace-restore-result.test.ts packages/adapter-utils/src/git-workspace-sync.test.ts packages/adapter-utils/src/workspace-restore-teardown.test.ts packages/adapter-utils/src/sandbox-managed-runtime.test.ts server/src/services/__tests__/run-failure-diagnostics.test.ts server/src/__tests__/run-failure-sentry-real-sdk.test.ts server/src/__tests__/native-workspace-sync-history.test.ts` - Local checks use Node 24.21.0 and the pinned pnpm 9.15.4. The optional Sentry SDK is pinned to the declared 10.71.0 and uses an in-memory transport. - `pnpm -r typecheck` and `pnpm build`: pass on the rebased head. - The full local `pnpm test:run` was stopped before source changes when #15268 merged and required a rebase. It reported three pre-existing company-skill cache test failures on macOS. The same three cases fail on clean bases `2c43b39167` and `e99854249c` and the earlier head with `EACCES` when publishing a read-only cache directory. The relevant test, service, and cache source blobs are identical. No cache changes are included. Later local test groups were not reached. - All 54 checks pass on exact head `8448896bc6`, including the post-ready security scan, with two intentional Storybook skips. Greptile scores this head 5/5 with no review threads. Full Linux CI covers the local groups that were not reached. - Independent review found no blocker. Its diagnostics, Git workspace, and native history suites pass 115/115 on this head. - A source comparison confirms that removing only diagnostic wrappers yields the merged upstream Git integration code exactly, including ref locks, transaction protocol, arguments, options, and retry decisions. - The clean base has stale dependency overrides in its lockfile. Local installation resolved them as the existing PR CI fallback does. No manifest or lockfile change is included. - `git diff --check` and a local secrets/PII review pass. ## Risks A Git version or localized message may not match a known failure form. Such cases remain `unknown`. Up to 16 KiB of stderr and the bounded tree-ID prefix of stdout are inspected only in memory. The saved fields contain enum values only. These diagnostics do not establish workspace recovery or authorize retries. Restore decisions and Git mutations are unchanged. No schema migration is required. ## Model Used OpenAI Codex, GPT-6, with reasoning, tool use, and code execution. The service does not expose the exact model deployment ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
3b47a6befd |
Record bounded workspace restore failure stages (#15005)
Capture allowlisted restore substep and failure metadata for future diagnosis while preserving workspace recovery behavior, error identity, cleanup ordering, and privacy. Validation: exact-head Greptile 5/5, passing CI, no unresolved review threads, and a clean merge. Detailed verification and limitations are recorded in the pull request. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7a52dcdc74 |
fix: repair MCP validation and cancelled execution recovery (#14951)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The tool gateway gives agents access to connected services. Recovery controls what happens when a run stops. > - Generated tool names can exceed the provider limit after the MCP client adds its prefix. > - The same invalid definition can fail each automatic retry. A cancelled run can also hold saved messages without showing its cause. > - This pull request bounds tool names, stops configuration retries, and retains cancellation evidence. > - It shows the stopped run and admits saved input only after the existing safety checks pass. > - The benefit is a clear recovery path that preserves operator Stop and prevents duplicate message delivery. ## Linked Issues or Issue Description **What happened?** A long connected MCP tool name makes the provider reject the entire request. Automatic recovery repeats the invalid request. Separately, unexpected legacy cancellations can leave saved input behind a recovery hold. The notice does not identify the stopped run or its cause. **Expected behavior** Complete MCP names fit the provider limit. Tool-definition errors require configuration repair. Cancelled runs retain their source and reason. The recovery notice shows the cause and saved-message count. Verified unexpected cancellations can start a fresh turn through the existing admission checks. **Steps to reproduce** 1. Assign an App gallery connection with a long application key and tool name to a Claude agent. 2. Start a run. The provider rejects a name over 128 characters, including its MCP prefix. 3. For cancellation recovery, stop a legacy provider turn without an operator Stop request and send a user message while the recovery hold is active. 4. Inspect the recovery notice and the deferred message queue. **Paperclip version or commit** Rebased onto master at `cf8ad63c806685bfd7c48e3ed4a919d61a7c55f1`. **Deployment mode** Hosted or self-hosted server with legacy Claude or Codex execution. Related public work: - Refs #14017. That PR caps name segments. This PR preserves existing short names and uses stable hash aliases for long complete names. It also covers classification and recovery. - Refs #4510. That PR adds a cancellation-source column. This PR records bounded evidence in the existing run result, without a migration. - Refs #12552 and #4506. Those PRs suppress recovery after operator cancellation. This PR preserves operator intent and uses the existing continuation gates. ## What Changed - Bound gateway names with the full provider prefix in the 128-character budget. Retain the original upstream tool name for dispatch and permissions. - Classify invalid tool definitions as configuration failures before diagnostic redaction. Stop automatic retries and continuation attempts for that error code. - Persist cancellation source, expectedness, initiator, reason, and time. Preserve recorded Stop intent when adapter results arrive. Report unexpected started cancellations with closed diagnostic labels. - Show the run cause, saved-message count, and Inspect run link. Offer Continue for eligible unexpected cancellations. Require verified provider stop, empty tool inventory, ownership, and the existing pause, budget, approval, and dependency gates. Use the existing queue for single delivery. - Add regression coverage and update the execution, MCP gateway, and run-log documentation. ## Verification - `pnpm -r typecheck` and `pnpm build` passed. - `pnpm check:token-gates` passed. - Ran `pnpm test:run` and completed its workspace and serialized groups. Initial resource and timing failures passed on isolated reruns. All 149 serialized route suites passed. - Reran the changed server, adapter, and UI suites after the rebase. Coverage includes long-name upstream dispatch, configuration retry suppression, cancellation evidence retention, privacy labels, oversized run projection, and concurrent saved-message delivery. - `pnpm test:e2e tests/e2e/legacy-failure-continuation.spec.ts` passed all six browser scenarios. The recovery notice shows the run cause and inspection link, and each recovery entry point reaches one new response. - Added database-backed checks for active, removed, paused, unavailable, and disabled chat connections. The final continuation and recovery-notice suites passed 167 tests. Externally bound chats hide board Continue and show a usable next action. - All 55 GitHub checks passed on `42afbf1371dcaeb72646e3d8f65c19ff7cddf8de`. Two unrelated Storybook jobs were skipped by their normal conditions. Greptile reviewed that commit at 5/5 with no findings and no open review threads. ## Risks - Long tool names change to aliases. Existing short names stay compatible. The original connection and upstream name remain the dispatch authority. - Invalid tool definitions no longer get automatic retries. An operator must repair the configuration before a new attempt. - Continuation changes apply only to positively identified unexpected legacy cancellations with complete empty tool inventory. Operator Stop, unknown historical cancellations, outstanding tools, and unverified provider termination keep their holds. - No database migration. The added projection fields are optional. Cancellation reason and initiator IDs remain local run evidence; Sentry receives only closed source and initiator-type labels and expectedness. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, repository editing, shell execution, and GitHub tool use. The runtime does not expose the exact model variant or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
4b9a6000f7 |
Add bounded evidence for directory lock timeouts (#14787)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent files use directory locks during collection and cleanup. > - A lock timeout can fail finalization after the model turn completes. > - The timeout currently identifies no owner state or waiting operation. > - This pull request adds bounded evidence to the existing run failure report. > - Operators can distinguish a known local holder from a possible old lock without changing lock safety. ## Linked Issues or Issue Description **What happened?** A directory lock timeout does not distinguish active local work from an owner record left by an earlier process. The stored execution stage can also precede the cleanup operation that failed. **Expected behavior** The failure report should identify the waiting operation and expose bounded ownership clues. It must preserve the timeout and keep unknown ownership protected. **Steps to reproduce** Hold a directory merge lock while a second caller reaches its acquisition deadline. The regression tests exercise a live holder and an older owner record with a live PID. Related: #9667 proposes stale-lock recovery under a single-server assumption. This change only adds evidence and does not adopt that assumption. #14575 and #14665 add other run failure diagnostics. ## What Changed - Record lock owner state, capped age and wait duration, same-process and process-age comparisons, and whether this module holds the lock. - Label agent-directory release, collection, checkpoint, and warm handoff timeouts with a fixed operation code. - Validate each field before the existing event-local Sentry report accepts it. Exclude owner records, PIDs, paths, and absolute timestamps. - Limit the extra diagnostic owner read to 100 ms with best-effort abort; malformed JSON is `invalid` and unreadable owner records remain `unknown`. - Document the diagnostic limits and verify that contenders never reclaim protected locks. ## Verification - Focused lock, diagnostic, real Sentry SDK, and database-backed agent-directory tests: 126 passed, including stalled-read and malformed/missing/unreadable-owner regression coverage. - Final revision `0691613dcc`: all 54 reported checks successful, with two intentionally skipped Storybook checks. Greptile: 5/5, zero unresolved review threads; no merge conflicts. - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - `pnpm test:run`: complete suite coverage ran with the existing repository shard flags: four general-server shards, four serialized shards, two general-workspaces-a shards, and general-workspaces-b. The full run is not green because of the base failures below. - The broad run found 13 failures in the unchanged macOS skill-cache tests. All 13 reproduce on the clean base revision. Open PR #14290 covers that existing failure. - Two unchanged CLI archive tests hit their five-second limits during the broad run; all 17 tests in that file pass on recheck. A CLI auth socket error also cleared on recheck (19 tests), and its full serialized shard passed on rerun. ## Risks This is a diagnostic change, not a stale-lock fix. Owner observations can race with release. Wall-clock shifts can affect the age comparison. A local-holder flag covers only this module instance. None of these fields authorizes reclamation or proves a file save. Lock acquisition, release, retries, task status, and recovery guards retain their current behavior. No schema change or deployment action is required. ## Model Used OpenAI Codex, based on GPT-6, with code execution and repository tools. The exact model build and context window were not exposed to this agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass (focused checks pass; existing base failures are documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
0ea6b10967 |
Record ACP activity and workspace restore failure evidence (#14665)
## Thinking Path Paperclip records terminal run failures for operators. A timeout's own log and cleanup output update the run's last-output timestamp, so that timestamp can make a long-silent provider look active. Snapshot runtime activity before finalization and include the saved workspace restore classification to make the next failure actionable without copying tool payloads. ## Linked Issues or Issue Description **What existing behavior does this improve?** Terminal run diagnostics in the existing opt-in Sentry integration. **Current behavior** Reports cannot distinguish runtime events from finalization logging and omit the already-persisted workspace restore code. A later successful run also does not establish that earlier workspace files were restored. **Proposed behavior** Record runtime-event age/count and pending-tool inventory at finalization, before status reads and cleanup. Forward only finite counts, a completeness boolean, and known restore codes through the existing reporter. **Reason and benefit** Operators can distinguish a silent turn with unfinished tools from recent runtime activity and see restore failures without retrieving private run output. Neither signal certifies productive work or successful recovery. **Breaking changes** None. Error grouping, execution deadlines, cancellation, recovery policy, and the Sentry opt-in remain unchanged. Related diagnostic work: #14573, #14575, #14639. ## What Changed - Snapshot ACP activity before success/failure finalization, including thrown relay failures. - Forward bounded numeric/boolean evidence and shared workspace restore codes; exclude commands, tool IDs, paths, and arbitrary result data. - Document limitations and test silence, empty streams, timeout, cleanup delay, incomplete tool inventory, and privacy. ## Verification - `pnpm -r typecheck` passed after the final implementation. - Changed suites: 252 tests passed; all 29 database reporter tests subsequently passed after restoring the embedded-Postgres package library symlinks. The migration test also passed (30 database cases total). - `pnpm build` passed during implementation. Final head `fe78dba6f592b1abccac7cdbf341bd2e0b0d30cb` passed all 53 CI checks, including complete test coverage, typecheck, build, and browser/runner gates; two unrelated checks intentionally skipped. - Full local test attempts initially hit missing embedded-Postgres library symlinks; the dependency setup was repaired and database tests passed. Duplicate local full-suite runs were stopped after full CI completed. This PR does not claim a completed full local suite. - Review regression: completed, failed, and cancelled tools are excluded from the pending count; focused activity/timeout tests and adapter-utils typecheck passed. - Reviewed the diff for secrets, customer data, and internal references. ## Risks Low risk, diagnostic-only. The existing tool inventory is incomplete for some runtime events, so the report carries its completeness flag. Event age is measured at finalization and does not prove useful work or identify the underlying provider failure. No schema changes or new capture gate. ## Model Used OpenAI GPT-6 via Codex, with repository inspection, code execution, and tests. Exact model build identifier is not exposed by this session. ## Checklist - [x] Thinking path and model are specified - [x] Checked ROADMAP.md; this is a maintenance correction, not planned feature work - [x] Searched for duplicate and related PRs - [x] Described the issue using the enhancement template - [x] No internal issue references, customer data, or private instance links - [x] Descriptive branch name - [x] Focused regression tests pass - [x] Added tests and updated documentation - [x] Risks documented - [x] Required validation and CI gates are green (full suite validated in CI; local scope documented above) - [x] Greptile is 5/5 with no unresolved findings - [x] I will address review comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b3eb03fcba |
fix(sentry): preserve run failure stacks and diagnostic context (#14585)
Builds on merged #14575 and targets `master`. ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Operators use optional Sentry reports to diagnose failed agent runs. > - The shared reporter currently converts each saved error message into a new exception. > - This loses the original stack and cause chain, and omits available adapter and provider diagnostics. > - This pull request selects, redacts, and bounds useful diagnostics before it sends them to Sentry. > - Operators can locate the failing operation and correlate upstream failures without copying arbitrary run data. ## Linked Issues or Issue Description Refs #14573. That change preserves ACP provider failure details in the run record. Refs #14575. That merged PR adds recorded exit codes and signals. This PR covers stacks, causes, and structured diagnostics without duplicating those fields. A thrown setup or adapter error currently appears in Sentry with the reporter's stack. A saved provider failure can contain useful details that never reach the Sentry event. Both cases use the same shared reporting path. ## What Changed - Pass caught setup and execution exceptions, and structured adapter error metadata, through successful terminal status transitions. - Snapshot a fixed set of execution, adapter, provider, and exception fields. Preserve up to four exceptions in the cause chain. - Remove registered secret values, declared runtime environment credentials, unknown inherited environment values and encoded credential forms, and credential patterns before truncation. Mark truncated fields and bound provider details and stacks. - Rebuild sanitized Sentry exceptions with the original stacks and causes. Use an adapter stack preview when available. Omit a fabricated reporting stack when the source has no stack. - Keep contexts local to each event. Preserve the optional DSN gate and existing error-code/adapter fingerprint. - Document the fields, limits, and omitted data. - Settle leftover chat fixture outbox rows only after assertions and worker shutdown, so subsequent tests cannot claim earlier cases’ pending actions or provider I/O. This fixes the CI shard contamination exposed during verification; production chat behavior and test timeouts are unchanged. ## Verification - `pnpm -r typecheck` passed. - `pnpm build` passed. - 86 focused diagnostic, Sentry, and startup tests passed after the rebase, including the real `@sentry/node@10.71.0` SDK with an in-memory transport. - Tests cover cause chains, HTTP status and request IDs, long provider details, secret redaction, failed secret resolution, cyclic causes, size limits, and context isolation. - Five targeted heartbeat integration cases passed, including thrown and returned errors containing an opaque environment-bound credential. The earlier full heartbeat integration run also passed its integration cases. - The focused suite also passed with an unknown inherited environment value set to `1`; reporting tests use a controlled environment and separately verify short-secret redaction. - Before the fixture cleanup, the full chat shard reproduced the CI Telegram timeout at `pending recovery before restart` (354 passed, 1 failed). After the cleanup, the same shard passed all 355 tests. No timeout or production behavior changed. - Latest-head CI (`0ae9e70df321d66dda025c3a7ba4787e169e0a4f`) passed typecheck, build, all 12 server shards, all 3 chat shards, workspace and serialized suites, browser shards, Runner checks, canary validation, the real Sentry SDK contract, and security checks. Required `ci / verify` and `ci / e2e` passed. - The branch has been rebased onto `master` after #14575 merged. All 55 applicable checks passed on this head (2 unrelated Storybook checks skipped). Greptile re-reviewed the current 11-file diff at 5/5 with no findings or unresolved review threads. - Local monolithic full-suite attempts were interrupted to apply fixes; full-suite success is not claimed from those runs. ## Risks - Error messages and stacks can contain credentials. The reporter uses existing redactors and the run's encrypted secret registry, reads only known fields, and skips capture if registered-secret resolution fails. - Unknown environment values remain private by default. Unrecognized short values can mask benign matches; known public settings are explicitly allowed. - Diagnostic text is bounded and can be truncated. Truncation is explicit. Data discarded upstream cannot be recovered. - These additional fields go to the operator's configured Sentry endpoint. Arbitrary request/response objects, headers, configuration, prompts, and stdout/stderr are not copied. ## Model Used OpenAI GPT-6 through Codex, with tool use and code execution. The exact serving model ID, context window, and configured reasoning level are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
e5bf9d49a5 |
fix(sentry): retain recorded process exit details (#14575)
## Thinking Path > - Paperclip manages agents and records their task runs. > - Operators can enable Sentry reports for terminal run failures. > - A failed adapter can leave only the generic message “Adapter failed.” > - The run already records a process exit code and signal, but the report omits them. > - This pull request carries those two values through a strict capture boundary. > - Operators can distinguish a nonzero exit from signal termination when the message is generic. ## Linked Issues or Issue Description **What happened?** A failed run can store `exitCode: 1` or `signal: "SIGTERM"` while its Sentry event contains only `adapter_failed` and “Adapter failed.” The existing reporter drops both recorded fields. This occurs on the current master reporting path. **Expected behavior** The opt-in report preserves bounded process exit evidence without exporting adapter output or changing run behavior. **Steps to reproduce** Enable the backend Sentry DSN and report a failed run whose message is “Adapter failed” and whose stored signal is `SIGTERM`. Before this change, the event has no signal field. After this change, `run_failure.signal` is `SIGTERM` and the existing fingerprint stays the same. Related: #12105 and #8222 describe missing adapter/HTTP failure details. #13152 changes terminal-result cleanup classification, and #12886 adds process-failure classification and runtime URL checks. None forwards these stored fields through the Sentry reporter. This change does not resolve those broader issues. ## What Changed - Forward the stored exit code and signal from the terminal run reporter. - Accept only signed 32-bit integer exit codes; use `null` for missing or malformed values. - Accept only the reporting host's Node signal constants; use `null` for missing values and `unknown` for unrecognized values. - Keep the added fields in event-local context, outside tags and fingerprints. - Cover database-backed reporting, malformed input, privacy, and isolation through the real Sentry SDK. - Document the fields and their limits. - Give the dedicated Sentry job the normal PR dependency-resolution fallback, with lifecycle scripts disabled on every install and the required real-SDK test retained. ## Verification - Before the change: 16 report-shape/exit-field assertions failed in the focused capture suite. - After the change: 91 focused Sentry, DSN, and database-backed reporting tests passed. The real SDK uses an in-memory transport. - Final real-SDK test also passed with malformed metadata; it verifies that arbitrary signal text is absent from captured events and unrelated errors inherit no run context. - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - `pnpm test:run`: exited nonzero after 718 general-server files: 14,085 tests passed, 14 failed, 86 skipped. Thirteen skill-service/cache failures reproduce on the unchanged base commit on this macOS host. One comment-wake test timed out; the complete 29-test suite passes on the unchanged base and in final-head Linux CI. Local isolated rechecks skipped because embedded PostgreSQL could not start; these are not counted as passes. The remaining local workspace/serialized lanes did not run after the failing first lane; all CI lanes passed. - Initial Sentry CI failed before tests with `ERR_PNPM_LOCKFILE_CONFIG_MISMATCH`. Its install step lacked the normal PR fallback. The repaired real-SDK job passed. Security review then requested disabling lifecycle scripts for resolved dependencies; every install now uses `--ignore-scripts`. A fresh isolated checkout passed the exact script-disabled fallback and real-SDK contract. Final-head real-SDK CI and the security scan passed. - GitHub CI: all 54 checks passed on `37e0a836e50660f7753d367bcf5a4959eaf89b90`, including required `ci / verify` and `ci / e2e`; two unrelated checks skipped. - Greptile: 5/5 on that commit. No unresolved review comments. - Merge status: conflict-free; required CODEOWNER approval for the workflow change is still pending. ## Risks Low risk: this only adds two validated fields to existing opt-in error reports. It changes no database schema, run status, retry, fingerprint, or suppression rule. Process output and adapter result payloads remain excluded. A recorded signal does not identify its sender or prove an out-of-memory kill. Missing exit evidence stays unknown; this change does not establish the cause of a historical generic adapter failure. ## Model Used OpenAI GPT-6 (Codex), with reasoning, terminal tools, and code execution. The context window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes:` / `Closes` / `Refs` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal ticket id or instance-derived details - [x] I have run focused tests locally and they pass; broader validation is recorded above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
c276d3fdc3 |
feat(observability): report terminal run failures to Sentry (#13446)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server records the state of each agent run. > - Terminal run failures need clear error tracking for operators. > - The server did not report terminal `failed` or `timed_out` transitions to Sentry. > - This pull request reports each genuine terminal failure transition with safe diagnostic data. > - The benefit is faster diagnosis without changing run control flow or exposing credentials. ## Linked Issues or Issue Description **What happened?** The server wrote terminal run failures but did not report them to Sentry. Operators could not see these failures in error tracking. **Expected behavior** The server should report each genuine transition to `failed` or `timed_out` to Sentry. **Steps to reproduce** 1. Run an agent task that reaches a terminal failure state. 2. Inspect the Sentry events for the server. 3. Observe that the terminal run failure has no matching Sentry event. **Paperclip version or commit** The change targets the current `master` branch. No public GitHub issue or pull request covers this change. ## What Changed - Add `captureRunFailure()` as a fail-open Sentry entry point. - Add `reportRunFailure()` to filter status, resolve the adapter, redact text, and report the failure. - Call `reportRunFailure()` beside each of the eight terminal status writers. - Report six diagnostic values: the instance host, task identifier, run identifier, error message, error code, and agent adapter. - Group events by error code and agent adapter while keeping the redacted message in the event. - Report only genuine transitions and avoid duplicate finalization events. - Keep Sentry failures outside run control flow. ## Verification - `pnpm vitest run server/src/services/__tests__/run-failure-report.test.ts server/src/__tests__/run-failure-sentry.test.ts server/src/__tests__/native-session-resumption.test.ts` - `pnpm vitest run server/src/services/execution-control-reconciliation.test.ts` - `pnpm --filter @paperclipai/server typecheck` - The full continuous-integration suite must run on this pull request. ## Risks - The report path can add diagnostic events when Sentry is configured. - The report path returns without action when Sentry is not configured. - Redaction runs before length limits and before the event leaves the process. - The change has no migration and no schema change. ## Model Used OpenAI Codex, GPT-5, tool use and code review support. The exact context window and reasoning configuration are not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |