mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 16:27:18 +02:00
6f045207bb0347dbde952886f0da8bb287cdc208
79
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
36e4c6657f |
fix(sentry): report a native retry failure only when the retry ends (#16865)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server sends a Sentry `run_failure` event when an agent run
reaches a terminal failure.
> - A native run that waits for a same-run retry is stored as `failed`
with `nativePhase: "retryable_failure"`. That state is not terminal.
> - `reportRunFailure` reported that pending state. Runs that later
recovered, for example after a workspace copy-back retry, still sent a
run failure event.
> - These events make one catch-all Sentry group noisy and hide the real
terminal failures in it.
> - This pull request skips the pending retry state and reports once
when the retry ends as a terminal failure.
> - The benefit is that run failure events match runs that really
failed.
## Linked Issues or Issue Description
**What happened?**
Two staging runs had a workspace copy-back failure during settlement
(`Daytona syncOut tar command failed ... No space left on device`). The
executor stored each run as `failed` with `nativePhase:
"retryable_failure"` and `errorCode: native_session_interrupted`. Sentry
received a `run_failure` event for each run. About two minutes later,
the copy-back retry succeeded and both runs finished as `succeeded` with
`workspaceFinalizeStatus: "succeeded"`.
**Expected behavior**
A run that waits for a same-run retry must not send a terminal run
failure event. If the retry ends as a terminal failure, the server must
send one event at that time.
**Steps to reproduce**
1. Set `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected`.
2. Start a native run that fails retryably. For example, make the
workspace copy-back fail once.
3. See the `run_failure` event in Sentry, and then see the run finish as
`succeeded` after the retry.
## What Changed
- `reportRunFailure` returns without a report for a native run that is
`failed` with `nativePhase: "retryable_failure"`.
- New `isNewTerminalRunFailure(before, after)`: a status change counts
as before. A failed run that leaves the pending retry state but stays
`failed` also counts.
- Three terminal writers can keep the status `failed` → `failed`. They
now use `isNewTerminalRunFailure`, so they still report once:
- the resume reconciler's ambiguous-state write;
- the restart recovery `blocked` write;
- the committed-result projection (`projectCommittedRun`).
- Result-less retries already reset the run to `running` before they
start, so their terminal write still reports through the existing
status-change guard.
- `doc/observability.md` describes the new rule.
## Verification
- `pnpm exec vitest run
src/services/__tests__/run-failure-report.test.ts
src/__tests__/run-failure-sentry.test.ts
src/__tests__/native-session-resumption.test.ts
src/services/native-runtime/native-run-finalizer-telemetry.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/__tests__/heartbeat-status-preserving-telemetry.test.ts
src/services/execution-control-reconciliation.test.ts` (from `server/`):
7 files, 363 passed, 9 skipped.
- New tests:
- `run-failure-report.test.ts`: a pending native retry does not report,
and a `terminal_failure` run does. Unit cases cover
`isNewTerminalRunFailure`.
- `native-session-resumption.test.ts`: a sweep that ends an unreported
pending retry sends one event.
- `native-run-finalizer-telemetry.test.ts`: a committed failure for a
pending-retry row sends one event, and a replay sends none.
- `native-runner-restart-recovery.integration.test.ts`: a restart
recovery `blocked` write for a pending-retry row sends one event.
- I removed each of the three writer changes one at a time. The matching
new test failed each time (`expected [] to have a length of 1` / `to
deeply equal [ [ ObjectContaining{…} ] ]`).
- I did not run the full server `tsc --noEmit` locally. The local
container has a 4 GiB memory limit, and the process was killed (exit
137). CI runs the typecheck.
## Risks
- A run that stays in the pending retry state and never resumes no
longer sends a run failure event. The run stays visible on the board as
failed with a scheduled retry. A later terminal write still reports it.
- The finalizer's `failed` + `retryable_failure` write for a failed
provider result also stops reporting. The terminal write that ends that
retry reports instead: retry exhaustion in `recordRetryableFailure`
reports unconditionally, and commit goes through `finalizeNativeRun` or
`projectCommittedRun`.
- A run that was reported in the pending state before this change can
report once more when it becomes terminal after deploy. This affects
only runs in flight during the upgrade.
- There are no schema, migration, or API changes. Run status, retries,
and recovery do not change.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, code search, test runs) and a read-only code
exploration subagent. Context window: the standard window for that
model.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandPaperclip
|
98ca6ff7c5 |
perf: reuse verified provider files during sandbox startup (#16493)
Build provider-pack transport layers once, reuse verified image files in private directories, and upload only missing layers. Preserve archive integrity checks and directory fallback on unsupported hosts. Verified with 44 focused tests, all 48 main CI jobs, cloud Docker Runner checks, and Greptile 5/5 on the merged head. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
51c81d0129 |
feat(sentry): link run errors to task and run pages (#16803)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Operators use Sentry to inspect run and server failures. > - Reports identify the run and task, but the existing links open API records. > - Operators need a direct path from an error to the normal task and run pages. > - This change adds navigation links through the shared server Sentry hook. > - Saved company and task records determine the links. A failed lookup preserves the error report. ## Linked Issues or Issue Description Related work: #16300 added saved run API references. **What existing behavior does this improve?** Sentry event context for run failures, Stop timeouts, and server errors from an authenticated agent run. **Current behavior** Reports show task and run UUIDs. The existing run URL opens an API record. The operator must find the normal task page separately. **Proposed behavior** Add `task_url`, `run_url`, and `task_identifier` tags when saved run metadata is available. Add `taskUrl` and `runPageUrl` to the event context. Use the company prefix and saved task identifier in normal board URLs. **Reason and benefit** An operator can open the task or run directly from the error report. ## What Changed - Add a shared Sentry hook for events with a known run identity. - Read only the saved company prefix, task identifier, and run identity. Keep the lookup within the recorded company and agent. - Include authenticated run identity on request errors. Use the existing run identity on Stop errors. - Bound optional lookups to one second and eight unfinished reads. Preserve the original event on failure. - Keep the existing API references and error grouping. - Document the fields and add SDK, timeout, and company-boundary coverage. ## Verification - Passed: `pnpm -r typecheck` and the final server TypeScript check. - Passed: `pnpm build`. - Passed: 332 tests across the affected server suites, plus 25 final link and real Sentry SDK tests after review fixes. The SDK test checks outgoing event envelopes for run failures, request errors, and Stop errors. It checks that unrelated errors receive no links. The database test verifies saved metadata and company boundaries. It passes with embedded PostgreSQL and with an explicitly configured disposable test database. - Passed in CI on `bec0cdac853efdef71041d59a66261941827653e`: the full test matrix, typecheck, build, Runner checks, browser suites, and the real Sentry SDK contract. All 56 active checks pass. Two Storybook jobs are skipped. - The full local `pnpm test:run` attempt first failed because PostgreSQL library links were missing. The package setup script repaired the install. The affected database suites then passed. The repaired full local run was stopped after all CI test jobs passed; it did not complete locally. - Greptile: 5/5 on the final commit. All three review threads are resolved. - Read-only browser inspection confirms that Sentry displays context URL values with an Open link control. - No live test event was posted. Validation uses the real SDK with an in-memory transport. ## Risks - A slow or unavailable database can prevent optional links. It does not discard the error. At most eight unfinished lookups can remain active. - Missing or unsafe public origins omit the links. URLs longer than the tag limit remain in context. - This change applies to newly captured events. It does not update existing Sentry events. - The change adds no schema, permission, or first-party Telemetry contract change. ## Model Used OpenAI Codex, `gpt-6.1-sol`, xhigh reasoning, with tool use and code execution. The exact context window is not exposed in the session metadata. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
023f4a5d0c |
fix: preserve repository cache negotiation refs (#16687)
## Thinking Path > - Paperclip manages agent work and its execution environments. > - A fresh repository task needs an independent checkout before its runner can start. > - The object cache from #16487 should reduce repeated network transfer. > - The cache kept Git objects but dropped remote-tracking refs during its bare clone. > - Git needs those tips to negotiate which objects it already has. > - This pull request preserves the tips in a versioned mirror cache and records explicit cache outcomes. > - The benefit is less repeated transfer and clearer evidence when a cache does not help. ## Linked Issues or Issue Description Refs #16487 and #16454. No duplicate cache-negotiation fix was found in the public PR search. **What happened?** A valid object cache could make a fresh clone slower. A controlled staging probe took 45.87 seconds and received 184.39 MB even though the reference already held the objects. An earlier probe took 52.33 seconds. An uncached clone took 20.94 seconds. `git clone --bare` copied the pristine checkout's objects, local branches, and tags, but omitted its remote-tracking refs. The source had about 1,200 remote branches. Git could not use the missing tips to negotiate those objects. **Expected behavior** Keep the pristine reference's negotiation tips. Each task must still authenticate online, use the current remote refs, and end with independent Git objects. **Steps to reproduce** 1. Clone a repository with many remote branches into a fresh checkout. 2. Publish a reference with `git clone --bare --no-hardlinks` from that checkout. 3. Clone the remote with `--reference-if-able` and `--dissociate`. 4. Compare with a reference that retains the remote-tracking tips. Use Git Trace2 and count received pack bytes. **Paperclip version or commit** The staging runtime was `27ab902d3db2770ada79c9f42a9c06e1d346f34b`, with Git 2.47.3. The cache implementation was unchanged from #16487. **Deployment mode** Docker on a persistent application volume. The probe used the application file owner and the public Paperclip repository. It did not start agent tasks or alter the application cache. ## What Changed - Publish pristine object caches with `--mirror` to retain remote-tracking tips. - Put cache format version 2 in the key and receipt. The first checkout builds a new generation. Existing generations stay under the current storage limits. - Keep the existing online authentication, current remote refs, credential and company isolation, independent objects, quotas, and fallback behavior. - Add closed Honeycomb cache outcomes and a boolean that states whether a reference was supplied to clone. - Document that a lookup hit or supplied reference does not prove Git avoided transfer. ## Verification - Controlled staging component experiment: a disposable reference copy held exactly the same 210,057 objects as the original. It added 1,187 advertised branch tips whose commits were already in those objects. - Before: 45.87 seconds and 184.39 MB received, one sample in this comparison. - After: 12.20 and 14.58 seconds; median 13.39 seconds. Received pack sizes were 1.99 and 1.95 MB. - All comparison clones used the same current remote HEAD and ended without an alternates file. - The remaining repack took 9.72–9.81 seconds. Checkout file writing took 0.86–0.96 seconds. - The staging experiment restored tips on a disposable cache copy. It did not deploy this patch or measure complete agent latency. - Confirmed both staging scratch locations were empty afterward. The original cache timestamps stayed unchanged. The temporary operator SSH key was revoked and its local files removed. - TypeScript 5.9.3 transpilation reported zero syntax diagnostics across the four changed TypeScript files. `git diff --check` passed. - The new negotiation regression fails with the old `--bare` publisher and passes with the mirror publisher. - Focused Vitest suites: 26 tests passed across repository-object-cache and execution-preparation-timing. These cover independent objects, current refs, unavailable/revoked remotes, scope separation, invalid storage, tracing isolation, and endpoint-off behavior. - The first full focused run passed 24 tests but failed its existing localhost HTTP fixture because the sandbox denied `listen` on `127.0.0.1`. The rerun with local loopback permission passed all 26 tests. - Server semantic typechecking passed with `tsc -p server/tsconfig.json --noEmit` after building the required workspace declarations. Initial attempts lacked generated dependency outputs; the completed dependency build and rerun passed. - A full application build was not run locally. CI passed: 55 checks passed and two optional checks were skipped. Greptile is 5/5 on the current head, with both findings resolved. ## Risks - Version 2 causes one cold cache rebuild. Old cache generations retain their disk usage, and the existing quota can prevent publication. - A cache still needs a full repack to produce an independent task checkout. This PR does not remove that cost. - The component experiment establishes the negotiation effect on staging. It does not establish an end-to-end agent speedup or the under-30-second goal. - The new trace fields remain behind the existing OTLP gate. Their values are closed enums or a boolean. They export no refs, paths, URLs, cache keys, credentials, or error text. ## Model Used OpenAI Codex, GPT-6 family, with code and tool use. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
aaeff418cf |
fix(acpx): add a positive control to stalled-tool permission evidence (#16769)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The ACPX engine runs Claude agents. A watchdog ends a turn when a pending tool call gets no update for 30 minutes. > - #16688 added a field to stalled runs. The field records if the stalled call's permission request reached Paperclip. > - Recent stalled runs show `acpStalledToolPermissionRequested: false`. We can read this as "the call stopped before its permission request". That reading is correct only if calls of the same kind usually send a request. > - We have no data that shows if finished calls of that kind sent requests. Thus we cannot tell if the missing request is a real signal. > - This pull request records a positive control: the number of finished calls of the stalled call's kind, and how many of them sent a permission request. > - The benefit: the next stall event tells us if the missing request locates the stall, or if it gives no information. ## Linked Issues or Issue Description Refs #16688 **What happened?** Runs stop with the stalled-tool watchdog error. The diagnostics show that the stalled call sent no permission request. The diagnostics do not show if finished calls of the same kind sent requests. **Expected behavior** The stalled-run diagnostics contain enough data to show if a missing permission request is a real signal. **Steps to reproduce** A pending tool call gets no ACP update for 30 minutes. The watchdog ends the turn. The run result contains the stalled-tool fields. ## What Changed - `packages/adapter-utils/src/acpx-engine/execute.ts`: when the watchdog fires, count the finished calls of the stalled call's kind, and count how many of them sent a permission request. Write the counts to the run result as `acpSettledSameKindToolCount` and `acpSettledSameKindToolPermissionRequestedCount`. - `server/src/services/run-failure-diagnostics.ts`: allow the two new fields as non-negative safe integers. Other values are dropped. - `doc/observability.md`: explain how to read the two fields with the existing permission-request field. - Tests: extend the existing stalled-tool tests in `execute.test.ts` and `run-failure-diagnostics.test.ts`. ## Verification - `pnpm exec vitest run src/acpx-engine/execute.test.ts` in `packages/adapter-utils`: 244/244 pass. - `pnpm exec vitest run src/services/__tests__/run-failure-diagnostics.test.ts` in `server`: 115/115 pass. - Mutation check: remove the kind filter, or count every finished call as "requested". Each mutation makes the stalled-tool test fail. - `tsc --noEmit` passes in `packages/adapter-utils` and in `server`. Note: the full `server` `typecheck` script did not run locally, because the runner vendor build step failed in this sandbox. I ran `tsc --noEmit` directly. ## Risks - Low risk. The change only adds two counts to the run result after the watchdog fires. It does not change the decision to stop the turn, or any permission decision. - The fields hold only counts. They contain no tool titles, inputs, or identifiers. This is the run-log and run-result path. It is not Telemetry. The fields reach Sentry through the existing allowlist in `run-failure-diagnostics.ts`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
63306f6660 |
fix(acpx): record whether a stalled tool call's permission request arrived (#16688)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Hosted `claude_local` runs use the ACPX engine. Claude can start
background subagents, and those subagents call the runtime MCP tools
that Paperclip attaches.
> - #16671 added a stalled-tool watchdog. It ends a Claude turn when a
pending tool call gets no ACP update for 30 minutes. That PR said the
most likely cause of the stall is a lost ACP permission round trip, but
it could not prove it.
> - A staging run tripped the watchdog. Two background subagents started
12 MCP calls. ACP reported each call as `pending`, and no call got
another update. The MCP gateway logs show the 9 earlier calls from the
parent agent, but no request from the 12 subagent calls.
`MCP_TOOL_TIMEOUT` (15 minutes) did not end them. So the calls stalled
before Claude sent them to the MCP server.
> - The run record cannot say if the permission request for a stalled
call reached Paperclip. Without that fact, the next stall gives no new
evidence about the cause.
> - This pull request records, for a stalled run, whether the stalled
call's permission request reached Paperclip, and how many pending calls
had a permission request.
> - The benefit is that the next stall tells us which side of the
permission round trip stopped, so the real fix can go to the correct
layer.
## Linked Issues or Issue Description
- Refs #16671 (the stalled-tool watchdog; this PR tests the cause that
PR suspected)
- Refs #16360 (the MCP client deadline that cannot reach a call that was
never sent)
**What happened?**
A hosted staging `claude_local` run started two background subagents.
Each subagent started six `execute-sentry-tool` MCP calls. ACP reported
all 12 calls as `pending`. None got a result, an update, or an error.
The MCP gateway received no request for them. The parent turn finished
its answer but stayed open for the subagents. After 31 minutes the
watchdog ended the turn with `acpx_tool_stalled`. A single background
subagent with six parallel calls to the same tool did not stall in a
later test.
**Expected behavior**
When a tool call stalls, the run record shows if the call's permission
request reached Paperclip. Then the stall can be put on the agent side
or on the relay side of the round trip.
**Steps to reproduce**
1. Start an ACP `claude_local` run in a sandbox that uses two background
subagents to call runtime MCP tools.
2. Wait until a subagent's tool calls stay `pending` with no further ACP
update (in staging this happened without a known trigger).
3. See that the watchdog fails the run with `acpx_tool_stalled`, but the
record has no evidence about the permission round trip.
**Paperclip version or commit**
`master` at
|
||
|
|
938a441944 |
fix(acpx): end Claude turns whose pending tool call stops receiving updates (#16671)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Hosted `claude_local` runs use the ACPX engine. Paperclip sends
runtime MCP servers to Claude, and Claude can start subagents (the Task
tool) that call those MCP tools.
> - #16360 set `MCP_TOOL_TIMEOUT` for Claude. That bound works: in a
local test with the same Claude binary, a direct MCP call and a subagent
MCP call to a server that never answers both ended at the bound.
> - A hosted run still hit the 4h sandbox timeout on a release that has
#16360. Its run log shows a subagent that started four MCP calls. ACP
reported the four calls as `pending`, and they never got another update.
About 40 other MCP calls in the same minutes finished in 3–25 seconds.
So the four calls most likely never reached the MCP server, and
`MCP_TOOL_TIMEOUT` did not apply to them.
> - The subagent's Task call got an `in_progress` heartbeat every 30
seconds for the full 4 hours. The turn looked active, so no existing
bound stopped it.
> - This pull request adds a stalled-tool watchdog to the ACPX engine.
It ends the turn when a pending tool call gets no update for 30 minutes
and no other tool call finishes in that time.
> - The benefit is that a stalled run fails in about 30 minutes with a
clear error code, not after 4 hours with a general timeout.
## Linked Issues or Issue Description
- Refs #16360 (MCP client deadline; this PR covers the stall that the
deadline cannot reach)
- Refs #15962 (stop-time diagnostics that showed the pending tools and
their age)
**What happened?**
A hosted `claude_local` run gave lookups to two subagents. One subagent
started four MCP tool calls. ACP reported each call as `pending`. No
call got a result, an update, or an error. The other subagent finished
in 3 minutes. The stalled subagent's Task call got a heartbeat every 30
seconds, so the last-event age at stop was only 28 seconds. The oldest
pending tool was 3h58m old. The sandbox execution timeout ended the run
after 4 hours.
**Expected behavior**
A tool call that never gets a result ends the turn after a bounded time.
The run fails with an error that names the stall.
**Steps to reproduce**
1. Start an ACP `claude_local` run that uses a subagent to call runtime
MCP tools.
2. Make one of the subagent's tool calls stay `pending` with no further
ACP update (in production this happened without a known trigger).
3. See that the run continues until the adapter execution timeout,
because the Task heartbeats keep the turn active.
**Paperclip version or commit**
`master` at
|
||
|
DottaandPaperclip
|
ce0eff7d1f |
perf: reduce fresh workspace clone and sandbox staging waits (#16487)
## Thinking Path > - Paperclip manages AI agents and their work. > - Each run needs a private workspace and safe persistence. > - Fresh isolated repository sources download upstream Git history again. > - Sandbox staging also verifies an optional seed cache before it uploads the workspace. > - That verification holds a company lock during a full extraction and hash scan. > - This pull request reuses pristine Git objects through an authenticated remote clone. > - It also overlaps seed verification with upload and moves verification outside the publication lock. > - The benefit is less repeated transfer and less serial preparation work, with the same task isolation and recovery contract. ## Linked Issues or Issue Description **What existing behavior does this improve?** Fresh task-private repository preparation and inbound sandbox workspace staging. **Current behavior** Each isolated remote source clones the full upstream history. Optional seed verification then extracts and hashes another full tree before upload. One workspace's verification can hold the company cache lock while other workspaces wait. **Proposed behavior** Use a company and credential scoped cache of pristine upstream objects for GitHub HTTPS sources. Each task still clones from the current remote. It disconnects from the cache before publication. Run seed verification alongside upload. Hold the company lock only for final capacity checks and publication. **Reason and benefit** These operations lie on the workspace preparation path. Removing repeated Git pack transfer and serial verification can reduce startup time. Local measurements below cover these boundaries. They do not establish hosted agent latency. Related: #16425, #16454, #16398, #16461, #16468, #16473. The cache inspection and durable folder proposal in #13376 covers separate product work. ## What Changed - Add a pristine upstream Git object cache for fresh isolated GitHub HTTPS sources. Key it by company, repository and current credential fingerprint. Do not store credentials. - Keep online clone authorization and current remote refs. Use `--reference-if-able --dissociate` so task objects remain independent. A reference failure retries once online without the cache and resolves credentials again. - Cap cache storage at 64 entries and 4 GiB per company, with 2 GiB per entry. Skip contended publication. Bound priming by a cooperative five-second budget and a 4,096-entry scan allowance. Keep the existing free-space reserve. - Overlap optional seed verification and required workspace upload. Settle both before returning or deleting temporary files. Recheck quotas and company directory identity under the publication lock. - Keep the independent fresh upstream reproducibility proof, task commits, dirty source copies, ignored files, nested repositories and durable recovery behavior. - Suppress optional Git server-info generation during dissociation so generated storage passes the existing safe-expiration checks. This uses [the supported Git setting](https://git-scm.com/docs/git-config/2.43.0#Documentation/git-config.txt-repackupdateServerInfo). - Add bounded timing phases, regression coverage and two repeatable local benchmarks. No first-party Telemetry or dependency changes. ## Verification - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - Affected adapter suites: 240 tests passed. Native workspace persistence/history and tracing: 33 tests passed. Server checkout/isolation/tracing batch: 53 tests passed. Final object-cache authority, path and capacity suite: 12 tests passed. These batches overlap. - Cache and expiration/restoration lifecycle batch: 34 tests passed. A direct generated-storage check covers cached clones. - Real HTTP Git fixture: a matching cache does not bypass revoked remote access. Other fixtures cover current refs/tags, hidden cached commits, independent objects after cache removal, corruption fallback, credential/company scope, quota limits and publication contention. - Initial restricted runs hit SQLite write and Unix socket permission failures. The adapter batch passed with disposable writable instance state and socket access. The first typecheck rejected the test use of `Promise.withResolvers`; typed deferred gates now use the supported library contract. - Repository benchmark on final head `9047368bf3b7fa2061e00d698b8b51cf4ae1da32`: five alternating samples, real local `file://` Git transport, synthetic 32 MiB history, no injected latency. Median 1143.8 ms to 240.5 ms; ranges 1085.7–1486.6 ms and 188.0–254.8 ms. Received Git pack bytes fell from 33,569,788 to zero. Initial cache priming took 1208.4 ms and is reported separately. - Sandbox benchmark, five alternating samples per mode on 10,000 files with real packing, extraction and verification: with a controlled 3-second transfer delay, median 9.450 s to 7.620 s (ranges 9.280–12.115 s and 7.296–7.983 s). With no injected delay, median 6.808 s to 6.450 s (ranges 6.255–7.029 s and 5.709–6.902 s). The no-delay ranges overlap; local copying competes with verification for disk. Both modes verify final tree contents, ignored-file exclusion and Git HEAD. These are synthetic local fixtures, not hosted agent runs. - Reproduce with `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-repository-object-cache.ts` and `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-workspace-staging.ts --baseline-ref=307eb16c0c --transfer-delay-ms=3000`. - Prior head `af766bc820` received Greptile 5/5 with no findings. Its E2E shard 7 failed in `plan-task-fixture` cleanup: company deletion raced an `activity_log` foreign key. That fixture has no provider run or Git workspace. The failure is retained here; final-head CI must pass. The corrected corruption fixture now forces a pack and verifies a real damaged archive. Current head `9047368bf3b7fa2061e00d698b8b51cf4ae1da32` has Greptile 5/5 with no review threads or findings. Final head has 55 passing checks and two optional Storybook skips. [The first workspace-shard attempt](https://github.com/paperclipai/paperclip/actions/runs/38802441857/job/116655485350) passed all 1,778 cases, then reported a late `EPIPE` from unchanged `server-utils.ts` while the ACP reuse test was active. That case mocks workspace preparation and passed locally in isolation. [Its single retry on the same commit](https://github.com/paperclipai/paperclip/actions/runs/38802441857/job/116659728034) passed, as did the dependent verification gate. The earlier E2E cleanup fixture passed on this final head. Both initial failures remain recorded here. A full local suite was not duplicated; the affected boundaries were run locally and the complete suite runs in CI. ## Risks - First cache population adds local copying. Cold clones, credential rotation, capacity exhaustion and a corrupt cache retain the online clone path. No eviction is added. Cached data is bounded, but many companies can each consume their allowance. - Individual filesystem calls and cleanup can exceed the cooperative priming budget. Failed reference clones make one additional online attempt. - Parallel verification and upload compete for disk and CPU. The local no-delay benchmark shows a smaller gain than a slow-transfer case. Verification still checks contents before cache publication, and required upload failures remain failures. - Local fixtures do not prove hosted throughput or the 15–30-second end-to-end target. The fresh checkout still writes its working tree, and model startup, copyback and other preparation costs remain. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context window are not exposed in this session. Used reasoning, source editing, shell tools and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal task identifiers - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0e58f29eb7 |
fix(observability): trace pre-dispatch controller waits (#16515)
Measure usage startup, MCP handoff, dispatch admission, and budget reservation with the existing endpoint-gated tracer. Preserve outer attempt context across asynchronous provider dispatch.
Validated with 135 focused tests, full typecheck/build, fresh full CI/E2E, Superagent, and Apex 5/5 on
|
||
|
DottaandPaperclip
|
becdc3b08e |
perf(slack): remove delays before mention acknowledgements (#16491)
Slack mentions now start eyes reactions after the authorized task/comment commits, independently of attachments, subscription and scheduler acceptance. Omit unused optional URL-preview polling while retaining message text, links and files. Keep receipt failures and missing-file warnings independent, including terminal cleanup of older multi-line diagnostics. Verified with full current-head CI and E2E, Superagent and legacy security checks, Apex 5/5 with no unresolved findings, 19 selected real-PostgreSQL receipt/authorization/cleanup cases, and full frozen-source typecheck/build. The first CI attempt's unchanged Cursor probe timeout is retained in the PR; the one failed-only retry passed under unchanged deadlines. Live Slack latency still requires deployed acceptance. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5ec93dae01 |
fix(connectors): avoid redundant run preparation discovery (#16461)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Run preparation selects connector skills and tools under current permissions. > - An assignment check can build an unrelated company tool catalog and wait for a shared discovery slot. > - Native startup repeats assignment discovery to obtain static tool definitions. > - Slack also locks and reconciles an unchanged catalog, then repeats task authorization. > - This pull request removes that redundant startup work and preserves live checks at tool execution. > - The benefit is less preparation work, less contention, and unchanged permission and approval enforcement. ## Linked Issues or Issue Description Related: #16454 (detailed preparation tracing), #16425 (workspace preparation performance). **What happened?** Connector configuration added avoidable work to run preparation. Browser Use assignment authorized the whole company catalog, including other connector families. Four connector resolvers ran in series. Native controller startup repeated all assignment resolution when any connector skill was pinned. Slack reconciled an unchanged catalog under an advisory lock and checked task authority twice. **Expected behavior** Prepare only the connector data needed by the run. Reuse static tool definitions from pinned skills. Preserve current company, assignment, identity, profile, policy, grant, and approval checks when an operation executes. **Steps to reproduce** 1. Configure Browser Use and an unrelated MCP catalog with 500 tools for one company. 2. Resolve Browser Use resources for an authorized task. The old path reads and authorizes the unrelated catalog. 3. Occupy both full discovery slots. The old assignment check waits for those slots. 4. Resolve a linked Slack task repeatedly with an existing current catalog. The old path still acquires a catalog lock and repeats authority checks. **Paperclip version or commit** Base: `68a09dcb5f50e3fb6ceaaeac2706d96db97fc9b3`. **Deployment mode** Observed in hosted native runs. The local reproduction uses a source checkout and real embedded PostgreSQL. ## What Changed - Restrict Browser Use catalog reads and policy decisions to active Browser Use connections. Preserve guest restrictions, reviewed catalog filters, live grants, and deterministic resource order. Keep assignment checks out of the full discovery queue. - Resolve the four independent connector families concurrently. Wait for outstanding work to settle before propagating a failure, and retain registration order. - Register native direct connector tool definitions from pinned connector skills. Keep live assignment resolution and governed execution on every tool call. - Compare Slack catalog hashes before reconciliation. Current catalogs skip the catalog transaction and lock. Check task authority once after current profile checks. Initial setup and upgrades retain locked reconciliation and checks before and after it. - Precompute immutable bundled Slack descriptor hashes. Preserve disabled entries, profile state, removed bindings, and review metadata. - Document the startup paths and add focused contention, revocation, catalog upgrade, and failure-settlement regressions. - Repair a pre-existing CI guard false positive: the filename `docker-compose-secrets.test.mjs` matched `secrets.`. Check GitHub secret expressions instead, with dot, bracket, and whole-context controls. No workflow changes. ## Verification - Conflict resolution: merge `b1566ca494f4af6e8d4418fa37393c5caee02582` from master into head `55140b1609ae7bd21a038eb75a48c2683299e4b4`. Combine both secret-expression guards and regression cases. Preserve current workspace, native recovery, and Daytona changes. All 467 policy tests, 787 focused native/connector/Browser Use tests, and server typecheck passed locally after integration. Fresh full CI passed all 55 checks on this head, with two optional Storybook jobs skipped. Greptile reviewed this exact head at 5/5 with no unresolved threads. - `pnpm -r typecheck` — passed. - `pnpm build` — passed. - Focused native startup, connector concurrency, and AgentMail suites — 763 tests passed. - Complete Browser Use suite — 33 tests passed. Focused Slack catalog, linked task, retained provenance/revocation, and pinned native tool tests passed. The complete runner tool authority suite passed (26 tests). The 1,337-case chat suite passed 1,335 tests and had two failures: the original new transaction-count assertion and a 15-second timeout in the publication-batch case. Both cases passed an isolated rerun. The publication timeout occurred during concurrent local checks; the passing rerun does not erase it. CI must still pass the chat shards. - The first focused run found two incorrect new assertions: policy evaluation performs a second catalog lookup, and an explicit allow policy overrides a profile default. The tests now distinguish the scoped catalog query and revoke the governing policy. The focused Browser Use rerun passed all 33 cases after these assertion fixes. - Full-suite test verification used the CI shards of `pnpm test:run`. Local runs focused on the changed boundaries; the complete repository test command was not duplicated locally. - `node --test '.github/scripts/tests/*.test.mjs'` — all 467 policy tests passed. The CI failure reproduced on the unchanged base guard before this correction. Actual GitHub secret expressions still match the guard. - CI on the final head passed all chat shards, typecheck, build, Runner checks, browser shards, and other test lanes. One existing workspace-runtime case hit `EADDRINUSE` for a randomly selected test listener in server shard 1/12 (1,162 tests passed in that shard). The shard passed its single retry on the same SHA with no source changes. The first failure remains recorded here. Previous reviewed head `7191ac40691ddb1ef38b86665c95cc14c57ea390`: all 55 checks passed, two optional Storybook jobs skipped, and no pending or failed checks. Greptile is 5/5 on this head with no unresolved review threads. The external security scan passed. Local before/after measurements use the old and new implementations against the same fixture. Both paths are warmed. Seven samples alternate execution order. The database is real embedded PostgreSQL. No latency is injected. These measurements do not establish hosted speed. | Local path | Before median (range) | After median (range) | Work reduction | | --- | --- | --- | --- | | Browser Use assignment with 500 unrelated MCP tools | 44.020 ms (33.994–107.328) | 26.502 ms (20.385–99.037) | 26 → 22 statements; unrelated catalog and discovery queue removed | | Slack assignment with current catalog and linked responsible user | 49.723 ms (43.624–80.653) | 22.141 ms (19.781–30.818) | 5 → 2 database transactions; no catalog lock on the current path | The contention regression fails against the old implementation after a bounded five-second wait, then passes against the optimized implementation. It occupies the real global discovery scheduler and verifies that Browser Use assignment completes without releasing those slots. It records catalog SQL to verify connection scope. Revocation cases remove policy, catalog availability, and grants between calls. Existing Slack integration cases exercise retained provenance, removed profile bindings, identity revocation, approved writes, and current responsible-user restrictions. ## Risks - Pinned native tool definitions can remain visible after access is revoked. They confer no authority. Invocation rechecks current assignments and fails closed before an operation executes. Newly available connector families still require a newly admitted skill selection. - Concurrent resolution can increase instantaneous database reads to four connector families per configuration call. It is a fixed bound. Outstanding work settles before a failed preparation returns. - Slack skips reconciliation only when all bundled version hashes and an existing profile match. Missing or changed definitions still use the existing locked path. Operator revocations remain effective. - This PR addresses connector preparation. Workspace transfer, provider startup, copyback, and queue delay can still keep total latency above the user's goal. Hosted improvement needs a deployed before/after trace. - No schema, lockfile, workflow, telemetry event, or authorization cache changes. ## Model Used OpenAI GPT-6 (Codex), with reasoning, repository inspection, code execution, and GitHub tools. The runtime does not expose the exact serving model ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
25b16e65fc |
fix(observability): detail workspace and connector preparation (#16454)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server prepares a repository, sandbox workspace, and connector access before an agent can reply. > - Existing traces measure large preparation phases but leave important work inside those phases unclear. > - Repository source materialization and Browser Use discovery can do more work than their callers suggest. > - New detail must preserve access checks, run ownership, and the existing work order. > - This pull request adds fixed OpenTelemetry spans inside those operations. > - Operators can use one deployed trace to select the next performance fix. ## Linked Issues or Issue Description Refs: #16419, #16425, #16428. **What existing behavior does this improve?** Startup traces for server workspace preparation, native sandbox synchronization, and connector assignment resolution. **Current behavior** Workspace and connector aggregate spans can take many seconds. The trace does not separate private source cloning, seed reuse, archive work, progress persistence, or each connector resolver. Browser Use resource selection builds the full authorized tool catalog. The configuration and native backend can each resolve connector assignments. **Proposed behavior** Show the awaited work as fixed child spans. Separate discovery queue wait from catalog construction. Preserve the initiating run context when another run releases a scheduler slot. Add no new lifecycle event writes for workspace detail callbacks. **Reason and benefit** Operators need evidence about the hosted critical path before another performance change. These spans distinguish filesystem work, provider transfer, authorization, transaction entry, locks, and reporting overhead. **Breaking changes** None. Authorization, query order, transaction ownership, synchronization, existing progress events, and task results retain their contracts. The change adds opt-in Observability spans under the existing OTLP endpoint gate. ## What Changed - Trace private source directory setup, Git credentials, clone, requested ref, source identity, and checkout publication. - Trace sandbox source capture, seed cache/replay, shallow clone and archive, overlay copy/archive, verification, upload, warm stamps, and descriptor persistence. - Separate existing awaited progress reporting from useful workspace work. Keep new detail callbacks trace-only. - Trace each connector resolver and Browser Use catalog/grant work. Split discovery queueing, descriptor sources, and policy evaluation. - Trace Slack authority, credential reads, catalog synchronization, transaction entry/settlement, and advisory locks. - Update the real filesystem staging trace expectations and the Honeycomb investigation guide. - Add a real scheduler regression for a queued job released by another run. Assert run ownership, parent spans, and exactly one completed queue span. ## Verification - Passed: server and adapter-utils TypeScript checks (`pnpm --filter @paperclipai/server exec tsc --noEmit` and `pnpm --filter @paperclipai/adapter-utils exec tsc --noEmit`). - Passed: 252 checks across 9 focused files. These cover real filesystem staging, native workspace synchronization, trace ownership, endpoint gating, and workspace preparation. - Passed after review: 26 trace-context, tracing-failure, and discovery-scheduler checks across 3 files (these overlap the first run). The new queued-discovery regression fails when context binding is removed and passes after restoration. An initial submillisecond timestamp assertion was replaced with explicit span-completion order to avoid SDK clock precision assumptions. - Passed on implementation commit `0e68968a4bd61001893d833f281f12bb110edbe1`: `pnpm -r typecheck` and `pnpm build`. The later commit updates only an engine test expectation. - The first CI workspace lane found an old engine trace-tree expectation. It expected the earlier span set. The updated assertion now checks the new detail spans, their parents, and single completion. Its targeted executor check passes. - The full test suite, build, release registry, and end-to-end gates pass in CI on final head `40e37d89ef679109d60a96bb5731a0616efeaa43`: 55 passing checks and 2 optional Storybook skips. The full suite was not duplicated locally after the focused run. - Greptile reviewed the final head at 5/5. No unresolved review threads remain. - After deployment, filter Honeycomb by the hashed run ID and verify `service.version`. Inspect `heartbeat.prepare.workspace`, `runtime_selection.workspace_sync`, and the connector assignment parents. The guide in `doc/observability.md` explains each new family and the nested intervals. - No hosted run has used this new instrumentation yet. Local checks cannot establish hosted latency improvement. ## Risks - Tracing adds spans when an operator configures an OTLP endpoint. Detail remains capped at 16 spans per phase per execution attempt. - Parent and child durations overlap. The documentation explains how to avoid counting them twice. - A discovery job can start in another run's scheduler continuation. The bound callback preserves the initiating async context. - New spans export fixed names and the existing hashed run ID. They add no paths, repository URLs, connector resource IDs, credentials, tool schemas, or exception text. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The exact served model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
57fa6b358d |
fix(observability): separate workspace finalization and copyback spans (#16428)
Give native workspace finalization its own span name. Keep copyback and commit as separate child measurements, with an exported-span regression that preserves the workspace operation result. Validation: 40 runtime/tracing tests and server typecheck passed locally. Final-head CI, E2E and security checks passed. Apex review: 5/5, zero findings and no unresolved threads. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
cccde7122b |
Trace execution preparation and durable response persistence (#16419)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server prepares an execution environment before it starts an agent. > - Operators need to distinguish preparation, provider wait, and response persistence. > - Existing preparation phases start separate traces, and the native session trace ends before the final response is saved. > - This pull request adds an execution-attempt trace and detailed spans at the existing work boundaries. > - The benefit is a measurable startup and settlement path without changing execution policy. ## Linked Issues or Issue Description **What existing behavior does this improve?** Operator-configured OpenTelemetry tracing for agent preparation, startup, and durable response storage. **Subsystem affected** Server orchestration, native Runner startup, and the Honeycomb run link. **Current behavior** Preparation phases start independent roots. Workspace lock waits, authorization groups, runtime asset work, provider-pack fallback, and the gap after provider completion are hard to attribute. **Proposed behavior** Use a server-owned async context for each execution attempt. Parent nested work to that attempt. Link the separately sampled native session because its ingress and queue history can begin earlier. Record transaction entry, lock waits, grouped authorization, asset preparation, artifact fallback, copyback, and response commit. **Reason and benefit** Operators can identify expensive work from recorded boundaries instead of inferring the cause from elapsed time. Nested and concurrent durations remain visible and must not be added together. **Breaking changes** No API or execution-policy change. New Observability span names and closed count attributes are documented. The existing local run-performance schema also records the additional provider-pack and harness measurements. Refs: #16178, #16205, #16330. ## What Changed - Add `task.execution_attempt` with same-run async ownership, independent root sampling, and native-session links. - Measure workspace transaction entry and settlement, advisory and row waits, source scans, reader groups, notifications, and writes. - Measure runtime selection and each concurrent instructions, skills, and MCP branch. Separate manifest, cache, materialization, and verification work. - Measure provider-pack discovery, verified reuse or upload, final verification, and remote harness state checks. - Measure workspace settlement, accounting, liveness, response authorization, comment commit, and publication enqueue. - Preserve the silent endpoint-off path, including when another library has registered a tracer. Bound repeated detail spans to 16 per name per attempt. - Extend the Honeycomb run link and document the actual coverage and remaining gaps. ## Verification - After resolving the conflict with retained-reader batching on `b98cdaea531c8447637cd4dcc19c61b3f5f41844`, all 61 targeted query, authorization, workspace-preparation and trace-boundary tests passed. The server typecheck passed. Reader tests still bound query count, avoid loading retained run payloads and enforce fresh project grants on every publication attempt. - Before that base integration, the tracing/exporter group passed 855 tests with 2 optional-SDK skips across 10 files and no inherited OTLP endpoint. It covers real PostgreSQL waits/rollback, durable response commit, assembled heartbeat preparation, concurrent parentage, sampling and tracer failure isolation. Native workspace/history tests passed 23 cases. These results remain supporting evidence for the unchanged tracing paths. - Full workspace typecheck and build passed during implementation. Full sharded CI, release validation, E2E and Apex 5/5 passed on prior head `8cebad80f3856dbcff6f3a6d5d34880ff77a20f2`. Fresh CI, security and Apex review are required on the current head before merge. - `pnpm check:token-gates` passed. No styles changed. - Earlier CI runs exposed exact-source-expression assertions changed by wrappers, missing run hashes on transaction-boundary spans, and sampling fixtures without an enabled endpoint. The fixes preserve the calls, add the declared hash and span cap, and enable tracing explicitly in the sampling fixtures. Focused repairs passed; the full prior-head CI passed after those repairs. Earlier failed run: https://github.com/paperclipai/paperclip/actions/runs/38729131366. - The full local `pnpm test:run` was stopped after the base advanced during that process. Its mixed-version results are not final-head evidence. A full local-suite pass is not claimed. - The old-head Superagent scan stalled at 35%. The current head has a fresh scan; no security pass is inferred from the old pending result. - No deployed Honeycomb acceptance is claimed by this code change. ## Risks - Enabled tracing creates more spans. Repeated detail spans are bounded, and inventories use group spans. - The attempt and native session remain linked traces. Historical queue time can precede the attempt, and nested intervals overlap. - A committed comment proves durable server storage. Publication enqueue and realtime dispatch do not prove external delivery or browser paint. - New custom attributes use only the existing run hash and validated file/byte counts. No SQL, paths, permissions, content, or exception text is added. ## Model Used OpenAI GPT-6 through Codex. The serving model ID and context window size are not exposed in this session. The agent used reasoning, tools, code editing, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin Foley
|
8365872743 |
fix(codex-local): retry ACP model capacity failures as transient upstream (#16407)
Classify typed Codex ACP `service` failures about model capacity or server overload as `codex_transient_upstream` / `transient_upstream`, so the existing bounded transient retry and continuation recovery apply. In `unexpected` Sentry mode, skip only runs carrying that full classified evidence; `all` mode still reports them. |
||
|
Devin FoleyandClaude Opus 5.5
|
11d69d04c7 |
fix(native-runtime): report remote recovery labels thrown before coordinator save (#16406)
Restart reattachment can fail in the executor before the native coordinator saves its result. The run failure report now reads the closed remote recovery stage and reason from the caught error when the saved result lacks them. The saved result keeps priority and the same closed-vocabulary validation applies. Add a regression for thrown, wrapped-transport, saved-priority, and out-of-vocabulary cases. Full current-head CI and Greptile 5/5 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
ff0f976de1 |
fix(native-runtime): retain remote recovery failure reasons (#16341)
Preserve closed marker-read and process-verification reasons through native recovery failures and operator reports. Keep existing transport errors and reattachment ownership checks. Add shell, process-probe, persistence and privacy regression coverage. All current-head CI checks pass and Greptile is 5/5. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
cda77edc60 |
perf(server): stop readiness waiting on startup housekeeping (#16329)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On restart, `/api/health` reports `starting` until startup recovery finishes, and deploy tooling waits for `ok` > - After heartbeat recovery, readiness also waits for housekeeping that every scheduler tick repeats and that serving requests does not need > - #16330 added OpenTelemetry spans for the main startup phases; native recovery and orphan reaping are the slow ones, and both run the native finalization reconciliation pass, which has no inner breakdown > - This pull request stops readiness waiting on that housekeeping, and records the phases of slow reconciliation passes as spans > - The benefit is faster readiness on every restart, and traces that show where a slow reconciliation pass spends its time ## Linked Issues or Issue Description Builds on #16330, which records `controller.startup.<phase>` spans. Related to #16328. **What existing behavior does this improve?** Server readiness time after a restart, and diagnosis of slow native recovery. **Current behavior** - After heartbeat recovery, readiness waits for the custom-image setup cleanup, tool connection health (remote MCP checks), decision expiry, the adapter and setup-token login reapers, environment lease cleanup (provider calls) and the retention sweep. Each also runs on every scheduler tick. On one instance they added 11–28 s to every measured restart. A throw in any of them fails startup. - On the same instance, native recovery took from 33 s to 11.5 min and orphan reaping from 30 s to 4 min across restarts in one hour. Both run the native finalization reconciliation pass. The #16330 spans time each step as a whole, but not the phases inside that pass. **Proposed behavior** - The housekeeping steps run after readiness. The scheduler interval still starts after them. - With an OTLP endpoint, a reconciliation pass of at least one second records a span with one child span per phase. **Reason and benefit** Readiness depends only on what serving requests and run recovery need. Slow reconciliation passes become attributable to a phase in the existing traces, without adding trace volume for the periodic pass. **Breaking changes** None. A failed housekeeping step no longer fails startup; it is logged and the next step runs. ## What Changed - `server/src/index.ts`: moves the housekeeping steps into `runStartupHousekeeping`, which runs after readiness under `trackHeartbeatSchedulerWork` (idle sleep and shutdown wait for it). Each step catches and logs its own failure (`startup housekeeping step failed`). The heartbeat scheduler interval starts when housekeeping finishes, unless the scheduler was stopped, so the reapers still finish before the first timer tick. - New `server/src/services/native-runtime/native-reconciliation-timing.ts`: records the phase boundaries of one pass. When the pass took at least one second and an OTLP endpoint is configured, it emits a `controller.native_reconciliation` root span (`paperclip.native-reconciliation`) with child spans `legacy_workspace_exports`, `obsolete_policy_reviews`, `automatic_completion_reviews`, `candidate_scan` and `candidates`, using the recorded start and end times. It records `scoped` and the candidate count only. A failed pass marks the pass span and the running phase. Tracing failures are swallowed. - `native-finalization-reconciler.ts`: the exported function wraps the pass in that timing and finishes it in `finally`, so failed passes are recorded; the original error is rethrown. - `doc/execution-semantics.md`: what readiness waits for, and the housekeeping order. `doc/observability.md`: the reconciliation spans. ## Verification - `pnpm exec vitest run src/__tests__/server-startup-feedback-export.test.ts src/services/native-runtime/native-reconciliation-timing.test.ts src/services/native-runtime/native-finalization-reconciler-timing.test.ts src/startup-recovery-timing.test.ts` (in `server/`): 52 passed. New tests: - Readiness is `ready` while a housekeeping step hangs, and the heartbeat interval starts only after the step settles. - A failed housekeeping step is logged, and the scheduler still starts after the remaining steps. - Span timing: nothing for a fast pass or without an endpoint; a slow pass emits the pass span and one child per phase with the recorded times; a failed pass marks the running phase and the pass; a failing tracer does not throw. - The reconciler records its phases and a failed outcome when a pass throws, and rethrows. - Three existing startup tests settle housekeeping before they read the scheduler interval or the setup cleanup. - `pnpm exec vitest run src/services/native-runtime`: 4,399 passed. - `pnpm typecheck` in `server/`: passed. ## Risks - Housekeeping now runs concurrently with the first requests and with `waitForExternalAdapters`. None of these steps is needed to serve requests, and each one also runs on every scheduler tick. - Readiness is reported earlier; the heartbeat timer starts at the same point relative to housekeeping as before. - The new spans exist only with an OTLP endpoint and only for passes of at least one second. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
ebceb2a744 |
trace(server): time workspace, native setup and startup recovery (#16330)
## Thinking Path > - Paperclip manages agents and their work. > - The controller prepares workspaces before it starts a Runner. > - A slow run can spend 30–80 seconds inside the combined workspace phase. > - Startup recovery can also delay queued work for minutes. > - Native runs also have gaps before Runner startup and after the provider result. > - Existing traces do not identify the awaited steps inside those intervals. > - This change adds closed timing boundaries around the existing operations. > - Operators can locate the wait without changing execution or exposing content. ## Linked Issues or Issue Description **What existing behavior does this improve?** Operator-configured OpenTelemetry diagnosis of controller preparation and startup recovery. Related: #16178, #16236 and #16285 introduced the existing preparation timing and sampling contract. #16306 and #16312 reduce separate configuration and copyback work. **Current behavior** The workspace interval combines source resolution, trust checks, provisioning, database persistence, binding and repository admission. Startup awaits accounting and native recovery before queued work resumes, but neither has a separate duration. Native backend setup and post-turn cleanup also contain unmeasured awaits. CPU profiles and a fast database probe cannot identify which await is slow. **Proposed behavior** Record workspace steps and native controller setup and cleanup with the existing hashed run ID. Record startup phases as independent, operator-sampled root spans without custom attributes. Keep the same operation order, results, failures and authorization. **Reason and benefit** These boundaries distinguish source work, database writes, repository admission and restart recovery. This is a diagnosis change. It does not claim a response-time improvement or a solved latency budget. ## What Changed - Add 14 closed workspace step names around 16 existing awaits. Repeated updates and bindings produce separate spans. - Time startup tool reviews, gateway sessions, delivery-worker readiness, accounting, plugin readiness, native recovery, hot restart, orphan reaping, retries, queued work and goal recovery. - Time native GitHub setup, wake binding, backend construction, connector assignments, MCP discovery, tool authority, companion selection and binary resolution. Time selected post-turn GitHub, coordinator and instruction cleanup awaits. - Move preparation timing to a pure shared module. Keep the existing heartbeat import as an alias. Both heartbeat and native timing skip tracer lookup when the endpoint is unset or blank, including when another library registered a provider. - Tracing lookup, callback, status and end faults cannot repeat work or replace its value or error. - Add real PostgreSQL workspace timing coverage and silent no-op/fault controls. Update the tracing contract and paired run maps. - Keep span attributes closed. Startup spans add none; workspace steps use only the existing run hash. No commands, paths, returned records or exception messages are recorded. ## Verification - Diagnostic-gap baseline: the new real workspace test fails against the unchanged workspace implementation because the expected step spans are absent. It passes with the wrappers. - Final affected coverage: **804 distinct cases pass** across seven complete suites: 698 native-executor cases, 39 real PostgreSQL workspace cases, 61 timing/context/startup cases, and six execution-configuration integration cases. The final native and unit invocations pass with no selected skips. The workspace suite passed in the initial six-file invocation on the same final workspace source. - Preserve the initial six-file attempt as failed: **7 failed and 789 passed**. All seven errors occurred during embedded PostgreSQL startup. The log reports local macOS shared-memory exhaustion. A fresh 15-case database group passes, followed by the complete 698-case native suite. These runs overlap; do not add their counts. - The new native timing assertion fails against unchanged native code because the expected spans are absent. It passes with the wrappers and retains the credential-isolation assertions. - Full `pnpm -r typecheck` and `pnpm build`: passed on the final production source. Earlier nullable-closure errors are fixed with erased assertions inside existing guards. The failed attempts remain recorded. - Normalized AST comparison: removing only timing wrappers, timing imports/local handles and erased assertions restores the original workspace, startup and native executor programs. This checks call order and policy expressions; it is not live acceptance. - The expanded head first failed two configuration integration assertions because their test providers had no endpoint. Configure the endpoint explicitly in those two tracing tests, and restore it after each test. All six cases then pass with the original credential cleanup, redaction and exact span assertions. Production is unchanged by this repair. - The first PR head failed one unchanged attachment test at its 15-second deadline in CI. The complete 32-case attachment suite passes locally with that deadline. Do not label the failed CI attempt green. The expanded head needs fresh full CI. - `node scripts/check-maps.mjs`: passed, 35 areas and 176 capability IDs. `node scripts/storybook-catalog.mjs`: 283 files and 2,728 stories. Structural checks only; no browser, play or Storybook build claim. - `git diff --check`: passed. No added prose comments, lockfile changes or new 1,000-line crossings. - A duplicate full local test run was not started. Fresh exact-head full CI and E2E pass on `15a683bb2419960f611584f8c07cdcfc9ce3af0b`. All 56 check runs are successful or optional skipped. Superagent and legacy Snyk pass. Completed Apex review is 5/5 on this same head, with zero unresolved review threads. Commands: ```sh pnpm exec vitest run --project @paperclipai/server --maxWorkers=1 --fileParallelism=false server/src/services/native-runtime/native-session-executor.test.ts server/src/services/execution-preparation-timing.test.ts server/src/startup-recovery-timing.test.ts server/src/services/heartbeat/preparation-timing.test.ts server/src/services/heartbeat/preparation-timing-context.test.ts server/src/services/heartbeat/workspace-preparation.test.ts pnpm -r typecheck pnpm build node scripts/check-maps.mjs node scripts/storybook-catalog.mjs ``` ## Risks - Configured operators receive more spans. They still follow the configured sampler and exporter. - Startup spans measure the awaited phase, including remaining readiness waits. They do not measure earlier worker construction or a provider turn. - Workspace and native setup intervals overlap their containing phases. Do not add them to the parent duration. Native timing covers named awaits, not every statement or every error path. - These spans are not yet deployed or tested against a live latency run. No first-party Telemetry event, run-log write, schema or API changes are included. ## Model Used OpenAI GPT-6 through Codex, with tool use and code execution. The exact served variant and context-window size are not exposed to this task. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
21ee5c9c85 |
feat(sentry): add opt-in run failure reporting with saved run references (#16300)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent runs retain execution results and output in the instance. > - Operators can report run failures to Sentry for investigation. > - A backend Sentry DSN currently enables those reports with no separate choice. > - A repair bot also needs the exact instance, company, agent, and saved run. > - This pull request adds an explicit reporting mode and authenticated run references. > - The benefit is that operators can investigate every failure on selected deployments while keeping other deployments quiet. ## Linked Issues or Issue Description **What existing behavior does this improve?** Optional server reporting of terminal agent run failures to Sentry. **Current behavior** A backend DSN enables run failure reports. Known account, quota, and connection failures are excluded. Reports omit company and agent references. Runs without a task are skipped. Tool-name collision errors omit the conflicting bindings. **Proposed behavior** Set `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected` to retain the existing filters. Set it to `all` to report every failed or timed-out run, including known setup and quota failures. The default is `off`. Expected stops stay excluded. Each report supplies identifiers and API paths that a bot can use with its own authorized Paperclip connection. **Reason and benefit** Operators can enable broad failure reporting on one staging deployment. A bot can fetch the saved run, events, and logs instead of relying on a short error label or copying complete transcripts into Sentry. **Breaking changes** A backend DSN alone no longer enables run failure reports. Existing operators must set `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected` to retain the previous policy. Ordinary server exception capture remains enabled by the DSN. Failure grouping gains the phase and original cause code when available. Refs #16281, #16297, #16298. These cover related tool discovery and failure diagnostics. This change retains their existing boundaries and does not depend on their open changes. ## What Changed - Add `off`, `unexpected`, and `all` modes. Missing and invalid values keep reporting off before diagnostic reads. - Keep the Sentry backend DSN and audited optional SDK requirements. - Include company, agent, run, and optional task identifiers. Include versioned paths for the saved run, events, log, and task. Add absolute run URLs only for a valid public HTTP(S) origin. - Report company runs without a task. Prefer the first-class issue binding when selecting the task reference. - Select saved native provider/model settings, timestamps, and original failure codes. Preserve gateway reason codes through redacted exception projection. - Group by error code, adapter, and available phase/cause. Keep run IDs and error text out of grouping. - Include the tool names and connection bindings in collision errors without copying tool schemas or provider metadata. - Document configuration, upgrade behavior, bot access, redaction, retention, and best-effort delivery. ## Verification - All 56 checks passed for `cccb47a721eaa1cc36e63278fc177b9101477541`. Two unrelated Storybook jobs were skipped by their path filters. No checks remain pending. - Complete [Paperclip CI run](https://github.com/paperclipai/paperclip/actions/runs/38591422284): passed. This covers general and serialized tests, Runner checks, browser E2E shards, typecheck, build, release-registry checks, canary dry run, and Docker checks. - Greptile reviewed that exact head: 5/5, with no unresolved threads or actionable findings. Superagent, Socket, and Snyk checks passed. - Local focused reporting, diagnostics, collision, and real SDK contract tests: 410 passed across five suites. This includes the external-wake filter in both reporting modes. - The audited `@sentry/node@10.71.0` tests use an in-memory transport. No events were sent to Sentry. - `pnpm -r typecheck` and `pnpm build`: passed locally after the final rebase. - `node scripts/check-module-boundaries.mjs` and `git diff --check`: passed. - The full local `pnpm test:run` command was stopped to avoid duplicating the complete CI shards. It did not finish locally. The full suite passed in CI. ## Risks - The new default disables existing run failure exports until the operator selects a mode. The upgrade setting is documented. - `all` can increase event volume and include known user-action failures. Secret redaction and diagnostic limits still apply. - Reporting remains asynchronous and best effort. This PR adds no durable delivery queue and does not install a bot or Sentry alert rule. - API references grant no authority. Existing company, private-task, and run-output checks still apply. Complete transcripts stay in Paperclip. - This changes optional Sentry error reporting. It changes neither first-party Telemetry nor OpenTelemetry span exports. It adds no database migration. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, shell, file editing, and test execution. The exact backend deployment ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
03fcd3c1fb |
fix(native-runtime): report which close barrier failed (#16298)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agent runs use the Paperclip runner. When a run ends, the controller closes the session. Close must prove that the runner durably suspended before the controller releases its checkpoint. > - If close cannot prove this, the run fails with `provider_transport_failed: runner did not durably suspend before checkpoint`. > - The runner knows which barrier was not proven. It writes that settlement only to the instance run log. > - The Sentry report for the failed run has no settlement evidence. Operators cannot tell a stuck suspend command from an undrained provider, an unreadable provider state, or unsettled tool results. > - This pull request sends that settlement to the run failure report as closed labels and counts. > - The benefit is that the next occurrence shows which barrier failed, so the root cause can be fixed instead of guessed. ## Linked Issues or Issue Description No public issue exists. This describes the bug. **What happened?** Hosted staging native Codex runs failed with `provider_transport_failed: runner did not durably suspend before checkpoint`. This happened on two runs, on 2026-09-30 and 2026-10-14. The second run used a release from 2026-10-14. The Sentry events contain only the error code, run phase, driver kind, and duration. They contain no stack and no settlement data. The native executor calls `reportRunFailure(db, run)` without the caught error, and the runner settlement is never read. **Expected behavior** A failed close report should tell an operator which barrier was not proven: runner suspension, provider drain, provider state, or semantic tool settlement. It must not expose tool identities, digests, provider turn IDs, or unredacted error text. **Steps to reproduce** 1. Run a native Codex session whose close cannot prove durable runner suspension. For example, the runner suspend command does not complete before the close deadline. 2. Observe the Sentry run failure event. Before this change, `run_execution` has no close settlement fields. ## What Changed - `collectRunFailureDiagnostics` accepts a new `nativeCloseFailure` option. It reads only a `native_session_close_unrecoverable` error, directly or up to four `cause` levels deep. - It adds closed labels and counts to `run_execution`: `closeRunnerSuspended`, `closeProviderDrained`, `closeSuspendCommandStatus`, `closeRunnerLifecycle`, `closeRunnerIdentityMatches`, `closeProviderState`, `closeProviderPendingEventCount`, `closeProviderTurnActive`, `closeSemanticResultPersistenceFailed`, `closeSemanticToolsPending`, `closeSemanticToolDeliveriesOpen`, and `closeSemanticResultFailures`. - Values outside the known vocabularies are dropped. Getters that throw are contained. - The native session executor passes the caught error as `nativeCloseFailure` when it reports a terminal run failure. It does not pass the error as `error`, because the raw error text and stack bypass the executor's identity redaction. - `doc/observability.md` documents the new fields and the privacy boundary. ## Verification - `cd server && npx vitest run src/services/__tests__/run-failure-diagnostics.test.ts src/__tests__/run-failure-sentry.test.ts`: 130 tests pass. - New tests check the full label set, a wrapped cause, null suspension and provider states, unrelated error codes, hostile and unknown values, and a missing settlement. They also check that no tool identity, digest, turn ID, or error text leaves the module. - Negative control: with the reader disabled, the new main test fails. - A scoped `tsc` run on `run-failure-diagnostics.ts`, its test, and `run-failure-report.ts` passes. The full server `tsc` and `pnpm build` were not run locally. The sandbox has a 4 GiB memory limit and no `cargo` for the runner binary. CI runs the full checks. ## Risks - Low risk. The change only adds Sentry context fields for one error code. Close, quarantine, retry, and recovery behavior do not change. - Sentry grouping does not change. The fingerprint stays `[errorCode, agentAdapter]`, and no exception chain is added. - The data path is Observability (Sentry error reporting, gated by the operator DSN). It does not change Telemetry. The fields contain only booleans, fixed labels, and counts. The DSN gate is unchanged. - These fields are observations. They do not prove a root cause by themselves. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), run through Claude Code as an agent with tool use (shell, file edits, test execution, Sentry and GitHub connectors). Context window and reasoning mode are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Devin Foley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
d4892d4761 |
fix(observability): sample native runs independently of triggers (#16285)
Start native background run traces under the operator's root sampling policy and link them to valid trigger contexts. Preserve child parenting and silent unconfigured tracing. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c80d93e14e |
fix(observability): time execution configuration steps (#16236)
Record optional execution-configuration timings before provider dispatch, with success, failure, sampling, cleanup, and silent-unconfigured controls. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
ccdc6754b2 |
fix: retain background preparation traces from unsampled requests (#16205)
Start preparation phase root traces independently of request sampling and link each phase to its trigger. Preserve operator root sampling, concurrent child context, and silent unconfigured behavior. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
e3029c632b |
feat: trace controller preparation before agent dispatch (#16178)
Trace eleven controller preparation operations with the existing hashed run correlation and preserve errors without exporting exception text. Verify asynchronous child-span parenting with a real OpenTelemetry SDK and context manager. Full exact-head CI and E2E passed after one documented fixture-startup retry; Greptile Apex 5/5. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f329b24ee1 |
feat: give tasks durable project-optional workspaces (#15749)
Give tasks durable company-scoped workspace bindings independent of project assignment, channel defaults and authorized repository preparation. Reuse verified input seeds and sparse output recovery while preserving trust, privacy and controller provisioning boundaries.
Full CI and E2E, Superagent, legacy checks and Greptile Apex 5/5 passed on reviewed head
|
||
|
|
4b766b8729 |
fix(acpx): record turn activity at the moment a stop fires (#15962)
Record stop-time ACP turn diagnostics (last-event age, oldest pending tool age, pending tool kinds) so wall-clock timeout events show whether the turn was stalled. Diagnostics only. Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
657e71584a |
fix(daytona): record provider state when lease creation hits its deadline (#15946)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents can run in Daytona sandboxes. The Daytona plugin acquires a
lease before each run.
> - Some acquisitions time out. The report then says
`environmentAcquisitionPhase=create` and that the whole 300 s budget was
used.
> - The Daytona SDK create timeout is equal to the acquisition budget.
The outer timer always fires first. Thus the report never shows the
state that Daytona left the sandbox in.
> - Without that state, we cannot choose a fix. A stuck snapshot pull, a
slow start, and a missing allocation need different fixes.
> - This pull request adds one bounded, read-only provider lookup at the
create deadline. It records a fixed-enum provider state in the existing
acquisition diagnostic.
> - The benefit is that the next timeout identifies where the provider
stopped. Retry, budget, and cleanup behavior do not change.
## Linked Issues or Issue Description
Refs #15736 (bounded Daytona acquisition failure observations). This
pull request extends those observations.
**What happened?**
A Daytona lease acquisition timed out with `Daytona lease acquisition
timed out; allocation cleanup is pending`. The run-failure report gave
`environmentAcquisitionPhase=create`,
`environmentAcquisitionElapsedMs=300004`, and
`environmentAcquisitionBudgetMs=300000`. The report did not show the
Daytona sandbox state. The SDK create timeout is equal to the outer
budget, so the outer timer always fires first. The SDK never reports the
stuck state.
**Expected behavior**
When creation is still pending at the deadline, the report should show
where the provider left the allocation. Examples are `pulling_snapshot`,
`starting`, `error`, or `absent`.
**Steps to reproduce**
1. Configure a Daytona environment.
2. Start a run when Daytona does not finish create and start within the
acquisition budget.
3. Inspect the setup-failure report. Only the phase, the elapsed time,
and the budget are present.
**Paperclip version or commit**
`master` at
|
||
|
Devin FoleyandPaperclip
|
0ba88d2195 |
fix(opencode-local): preserve deadlines in Daytona process sessions (#15905)
## Thinking Path > - Paperclip manages work done by AI agents. > - The OpenCode adapter can run its CLI in a provider sandbox. > - A long CLI run must remain observable until it exits or the host stops it. > - The legacy main command does not request a provider process session. A launch without an active run step uses one synchronous command request. > - That request can lose its response before the resolved execution budget ends. > - This PR requests the existing asynchronous session path for the main Daytona OpenCode command. It also carries the resolved command deadline through the host and provider contracts. The host still applies the effective capability gate. > - The change removes that synchronous request from eligible sandbox runs. ## Linked Issues or Issue Description Related reporting fix: #15789. A search of open and closed GitHub issues and PRs found no duplicate transport fix. **What happened?** A hosted Linux legacy OpenCode run retained a gateway-timeout response about one hour after CLI startup. It had a four-hour resolved budget and recent agent output. The earlier reporting fix correctly kept the configured policy separate from the timeout message. The original response owner and specific timer were not retained, so this PR does not claim to prove that provider cause. **Expected behavior** A Daytona target that supports persistent process sessions should dispatch the long agent command asynchronously. Paperclip should observe its output and exit through the existing session path. A single command HTTP request should not need to stay open for the full agent run. **Steps to reproduce** 1. Use legacy OpenCode with a Daytona sandbox target and a four-hour execution budget. 2. Launch outside an active measured step. The current main CLI call does not request a persistent process session. 3. In the SDK fixture, make the synchronous command request return a provider timeout before the budget ends. Keep the asynchronous session healthy for more than one hour. 4. The regression tests require session dispatch, incremental output, one command dispatch, a real exit result, and session cleanup. 5. Set a shorter command deadline and stall dispatch, a status read or a log snapshot. The command must return a timeout at its own deadline. Healthy output must not extend that deadline. **Paperclip version or commit** Observed source: `e733bddc45e00efb1819fffa8677f7b0d07802c5`. PR base: `4b59e2a2ecc67e98b336b8eb4c09959681672d54`. **Deployment mode** Hosted Linux server, legacy OpenCode adapter, Daytona sandbox. The host gate retains the old path when a provider cannot keep process sessions. ## What Changed - Add an optional sandbox session choice to the shared process helper. Leave the choice absent for existing callers. - Request that session only at the main Daytona OpenCode CLI invocation. Keep probes, preparation and other providers on their existing paths. - Carry a separate optional command deadline through the shared runner, host service and plugin SDK. Existing session callers can omit it. Zero leaves the command deadline disabled. - Bound dispatch, live output and recovery observations with one command timer. Clear it on settlement. Fence late stream callbacks. Remove deadline listeners after each read and clear interrupted polling waits. Preserve a known exit code if only the final logs time out. A timeout result does not claim that the process stopped. - Add adapter and process-helper regression coverage. Test the existing Daytona session for a virtual run beyond one hour, without command replay, with release cleanup. - Document transport selection and the limits of the timeout evidence. ## Verification - `pnpm -r typecheck` passed on Node 25.6.1 and pnpm 9.15.4 with the command deadline changes. - `pnpm build` passed with the updated contracts. The separately installed Daytona package also passes its own typecheck and build against the pinned SDK 0.203.0 and TypeScript 5.9.3. - Five focused process, adapter, host and provider suites passed all 349 tests. They cover a run beyond one hour, five stalled observation phases, healthy-stream deadlines, zero/absent budgets, late callbacks, session release, SSH and capability fallback. - The real environment-runtime forwarding tests passed: 19 selected tests, including the separate command budget and forced session flag in the worker RPC. - An earlier full local `pnpm test:run` began before a clean build and overlapped the correction. That mixed-version run was stopped and is not passing evidence. After the build, the two previously uncollected suites pass (1,659 tests). Two unchanged workspace fixtures fail with this machine's Git default branch `main`; both pass with a test-process-only `master` override. One unchanged native recovery fixture fails on both this branch and an independently installed and built checkout of base `4b59e2a2ecc67e98b336b8eb4c09959681672d54`, with the same terminal-result mismatch. It is not introduced by this change. The full local suite is not claimed green. These limitations do not replace fresh hosted CI. - Final head `8e96638cdef8ca6e91f7a42a52c83a075180c137` passes all hosted checks: 54 passed and 2 expected Storybook skips. Greptile reviewed this exact head at 18:12 UTC and scored it 5/5. Both actionable findings are fixed and resolved. GitHub reports no conflicts and a clean merge state. Earlier-head results are not used as final-head proof. - The standalone provider test command passes 373 tests, with 14 expected live-probe skips. - SDK tests use a controlled virtual clock. No live provider experiment, task replay, manual deployment, or public PR merge was performed. ## Risks The main legacy OpenCode command uses Daytona's existing session route when the effective capability gate permits it. A quiet log stream can use the existing status and snapshot fallback. Those observations add provider reads. The new optional command budget is separate from each read's timeout and returns a timeout without the host RPC's extra 30-second allowance. Existing session callers that omit it keep their previous lifetime. Callback bridge control commands retain their explicit session bypass. The capability gate retains the previous path for unsupported targets. Other providers, local and SSH execution keep their existing path. Retry policy, Stop acknowledgement and lease ownership rules stay the same. The SDK exposes no cancellation handle for an abandoned stream; existing Stop and lease teardown still own command termination. This fix does not establish the original gateway owner or prove recovery in a live environment. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools and code execution. The exact backend identifier and context window size are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (changed paths; the independently reproduced local baseline failure is documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
caa7895f9a |
fix(opencode): report timeouts without claiming deadline expiry (#15789)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The OpenCode adapter reports the outcome of a local or remote execution. > - A provider can report a timeout before the configured execution deadline. > - The adapter treated every timeout as proof that the configured deadline expired. > - This pull request uses a generic timeout message and retains the resolved policy. > - Operators can compare the run duration with its actual configured limit without a false cause claim. ## Linked Issues or Issue Description **What happened?** A remote execution can return `timedOut: true` after a short interval while the sandbox policy is four hours. OpenCode reported `Timed out after 14400s`. It also dropped the resolved policy on that result. Run finalization then used the raw `timeoutSec: 0` configuration, so diagnostics incorrectly reported an unlimited timeout. **Expected behavior** Keep the timeout outcome. Report the resolved policy separately from the failure message. Do not infer which timer fired from the provider's boolean or stderr. **Steps to reproduce** Call the OpenCode adapter with a sandbox execution target and `timeoutSec: 0`. Return a timed-out process result before four hours. Inspect the message and the policy that finalization sends to the run-failure collector. The new tests reproduce this with local fixtures and an in-memory Sentry transport. **Paperclip version or commit** Reproduced on `de042e79efbb7bb88e9716835407882e3ebab188`. **Deployment mode** Legacy OpenCode execution on a local instance or remote sandbox. Refs #14832 for related timeout result preservation; this PR does not change session state. Related PRs #14964, #15659 and #15715 cover silent-run limits, gateway requests and OpenCode v2 support. They do not fix this policy/message path. ## What Changed - Retain the complete existing timeout resolution on both OpenCode result branches. - Report `OpenCode execution timed out` without copying provider output into exception text. - Document that configured policy and the existing `timeoutFired` outcome field do not prove timer origin. - Test policy overrides, sandbox defaults, opt-out, ordinary exits, real local timer expiry, cleanup and real Sentry SDK delivery. ## Verification - 142 focused adapter, stop-metadata and diagnostic tests passed on Node 25.6.1. - The real Sentry SDK contract suite passed 11/11 with the audited optional peer 10.71.0 and an in-memory transport. The new test runs the actual adapter, finalization policy merge and collector. It verifies that raw HTML and credential canaries stay out of the event and unrelated captures stay isolated. - Full workspace `pnpm -r typecheck` and `pnpm build` passed. All hosted CI checks passed on `f4df5cba1d2223d444559de960bbf88dfa985bbf` (54 passed, 2 expected skips). - The local aggregate is not a pass: 15 existing PostgreSQL fixture cases initially failed because this fresh macOS install lacked bundled ICU symlinks. The dependency's existing hydration script repaired the isolated install; all 15 affected cases then passed with real PostgreSQL. No test or production source was changed for that setup issue. The aggregate was stopped after hosted test coverage passed; it is incomplete, not a local full-suite pass. All owned test processes exited. - Greptile scored this exact head 5/5 with no review threads. Independent review found no issues. It ran 58 positive cases and adapter typechecking. Three negative controls failed as expected when it restored the old producer, removed timeout policy, or copied raw stderr into the message. ## Risks The timeout message changes for local and remote OpenCode runs. The resolved policy is additive result data. The numeric timeout, grace period, process result, retry rules, session handling, cleanup and recovery behavior are unchanged. This fixes reporting and does not identify or fix an upstream provider timeout. No schema change or migration is required. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and independent agent review. The exact service model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
32745f4e84 |
feat(issues): keep follow-up parents and show the creation source (#15762)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents delegate work by creating follow-up tasks through the issue API, the CLI, and the chat `create_task` tool > - An agent can omit `parentId` when it creates a follow-up from an ordinary execution task. The API accepts the task as a top-level task. The task hierarchy does not show where the task came from, even though the run provenance is recorded > - The operator must read run history to learn why the task exists. This is slow and it hides delegated work from the parent task > - This pull request keeps the parent link for delegated follow-ups by default, keeps intentional standalone tasks possible, and shows an access-checked "Created from <task> by <agent>" note on the task page from the recorded run provenance > - The benefit is that each agent-created task shows its source without log inspection, while structural parentage, provenance, and blockers stay separate concepts ## Linked Issues or Issue Description Refs #13241 (shows tasks created from a task on the source task page; this PR shows the reverse direction on the created task) Refs #9727 (open proposal for creation provenance tracking) **What existing behavior does this improve?** Issue creation by agent runs, and the task detail page. **Subsystem affected** Server (`POST /api/companies/:companyId/issues`, `GET /api/issues/:id`), shared types, UI task detail page, CLI `issue create`, Paperclip skill docs. **Current behavior** When an agent run bound to an execution task creates a task and omits `parentId`, the task is created with no parent. The task page shows no source. The operator must inspect the run log to find the task that the agent was working on. The source run and the privacy parent are recorded, but the UI does not use them. **Proposed behavior** - When an agent run bound to an ordinary execution task omits `parentId`, the server sets `parentId` to that task. The `issue.created` activity records `parentDefaultedFromRunIssue: true`. - An explicit `parentId: null` keeps the task standalone. Conversation runs (chat handoffs), board or user creates, watchdog follow-ups, a run task the agent may not mutate, a default parent that would cause a delegation cycle, and a default parent the agent may not create a child under (for example a protected assignment policy without a grant) also stay standalone. The server runs the same `tasks:assign` decision with the defaulted parent in scope before it accepts the default. No request that succeeded before now fails. An explicit `parentId` is still checked as before. - `GET /api/issues/:id` returns `createdFrom` with the source task, the run, and the agent. The server resolves it from `origin_run_id`, or from the recorded creation activity for historical rows. It returns `null` when the viewer may not read the source task or the run, when the run is from another company, or when provenance is missing. - The task page shows "Created from <task> by <agent>" in the classic breadcrumb area and in the streamlined Tasks tab. The note is not a blocker and it is shown with or without a parent. - The CLI `issue create` command gets `--standalone`, which sends `parentId: null`. **Reason and benefit** Operators see why a task exists on the task itself. Delegated follow-ups stay inside the parent task tree, so parent completion and child wake-ups keep working. Private and cross-company sources stay hidden. **Breaking changes** None for the API schema. Agents that create tasks from a run and relied on the omitted `parentId` to mean "top-level" must now send `parentId: null` (CLI: `--standalone`). The skill and the spec document this. ## What Changed - `server/src/services/issue-creation-source.ts` (new): resolves a task's creation source from the origin run or the creation activity, scoped to the company, and loads the run-bound task by UUID or identifier. - `server/src/routes/issues.ts`: `POST /companies/:companyId/issues` defaults `parentId` to the run's task when the body omits it (with the exclusions above) and records the default in the activity log. `GET /issues/:id` adds the access-checked `createdFrom` field. - `packages/shared/src/types/issue.ts` and index exports: new `IssueCreationSource` type and the optional `createdFrom` field on `Issue`. - `ui/src/components/task-detail/IssueCreatedFromNote.tsx` (new), `TaskDetailTasksPanel.tsx`, and `IssueDetail.tsx`: show the creation source in both task page layouts and count it in the Tasks tab badge. - `cli/src/commands/client/issue.ts`: `issue create --standalone`. - Docs: `doc/SPEC-implementation.md`, `doc/CLI.md`, `skills/paperclip/SKILL.md`. - Tests: `server/src/__tests__/issue-creation-source-routes.test.ts` (new), `TaskDetailTasksPanel.test.tsx`, `IssueDetail.test.tsx`. ## Verification Run the new server route test (embedded Postgres): ```sh cd server && pnpm exec vitest run src/__tests__/issue-creation-source-routes.test.ts ``` It covers: ordinary delegation gets the parent and `createdFrom`; explicit `parentId: null` stays standalone but still shows the source; conversation handoff and board create stay standalone; a would-be delegation cycle falls back to standalone; a protected source task without a grant falls back to standalone while an explicit parent still returns 403; an explicit cycle still returns 409; historical provenance from the creation activity; missing provenance, a private source for an outsider, and a cross-company run all return `null`. Related existing suites that passed locally (11 files, 313 tests): `issue-created-from-routes`, `issue-create-deduplication-routes`, `issue-agent-mutation-ownership-routes`, `chat-project-tools`, `issue-assignee-invokability-routes`, `issue-watchdogs-routes`, `issue-dependency-wakeups-routes`, `public-mcp`, `native-runtime/task-creation-backlog`, `issue-access-grants-routes`, `issue-list-assignee-filter-routes`. UI: ```sh cd ui && pnpm exec vitest run src/components/task-detail/TaskDetailTasksPanel.test.tsx src/pages/IssueDetail.test.tsx pnpm check:token-gates ``` Manual check: as an agent run bound to a task, `POST /api/companies/:companyId/issues` with only a title. The response has `parentId` set to the run's task. Open the new task; the page shows "Created from <task> by <agent>". The related server suites pass locally (`issue-created-from`, `issue-create-deduplication`, `issue-agent-mutation-ownership`, `chat-project-tools`, `issue-watchdogs`, `public-mcp`, `task-creation-backlog`, `issue-access-grants`, `permissions-upgrade-boundary`). The full `pnpm -r typecheck`, `pnpm test:run`, and `pnpm build` were not run locally because the sandbox has a 4 GB memory limit and the TypeScript compiler is killed on the server project; the CI gates on this PR cover them. ## Risks - Behavior shift for agent clients: a run-created task without `parentId` now becomes a child of the run's task. Clients that want a top-level task must send `parentId: null`. The watchdog, conversation, and board paths are excluded, and the default never adds a blocker. - The default parent inherits the parent's execution workspace, which run-created tasks already inherited from the run's task. - `createdFrom` adds one run lookup, one issue lookup, one access decision, and one agent lookup to the task detail request. It is skipped when the task has no origin run and no creation activity with a run. - Privacy: the field returns `null` unless the viewer can read the source task and the run, so a private source is indistinguishable from missing provenance. ## Model Used Claude Fable 5.1 (`claude-fable-5-1`), Anthropic, run through Claude Code in a Paperclip-managed execution workspace with tool use, shell execution, and a local embedded Postgres for tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
9befc66a44 |
Retain bounded Daytona acquisition failure observations (#15736)
## Thinking Path > - Paperclip manages AI-agent work and reports failed runs. > - Some runs acquire a Daytona sandbox before model execution. > - Acquisition has one budget for creation, setup, and inline cleanup. > - A timeout currently carries cleanup ownership but loses the step that failed. > - This pull request carries a bounded observation through the existing error envelope. > - Operators can identify the first failed step without exposing provider responses or changing recovery. ## Linked Issues or Issue Description **What happened?** An acquisition timeout reports pending allocation cleanup without the first failed step. The provider already knows its current step, but that evidence is discarded at the worker boundary. A setup failure followed by slow cleanup is also indistinguishable from slow creation. **Expected behavior** The immediate setup-failure report should contain a fixed phase and bounded elapsed/budget values. Cleanup ownership, original messages, deadlines, retry policy, and task behavior should remain unchanged. **Steps to reproduce** Use the provider regression fixtures to stall creation, workspace setup, shell detection, expiry configuration, or sentinel upload. Also fail setup early and let cleanup consume the remaining budget. The previous source emits no phase diagnostic. **Paperclip version or commit** Base commit: `98e3cec78e8461774022a5ba0a777b09d4d16b96`. **Deployment mode** Hosted and self-hosted installations that use the Daytona sandbox provider and enable Sentry. Related public changes: #13979 established failed-allocation cleanup ownership; #14097 preserves the provider acquisition budget; #15637 carries bounded workspace-transfer evidence. Duplicate searches found no existing acquisition-phase diagnostic patch. ## What Changed - Capture the first acquisition failure before inline cleanup. Keep the observation if cleanup later reaches the deadline. - Add an optional, validated observation to the existing creation-cleanup envelope for acquire calls. Old ownership envelopes remain valid. Malformed observations do not alter cleanup. - Bind immediate reporting to an actual RPC error with matching company, environment, and run ownership. Project only three bounded fields from private host evidence. - Add provider, SDK, actual worker/client, PostgreSQL cleanup, collector, and real-Sentry regressions. Document the evidence limits. ## Verification - Passed 404 focused tests across the SDK cleanup envelope, complete Daytona provider suite, diagnostic collector, actual built worker/client subprocess, and mandatory real-Sentry SDK suite. - Passed 10 focused PostgreSQL cleanup-boundary cases. The filter excluded 152 unrelated cases. - Passed `pnpm -r typecheck` and `pnpm build` on the committed source. - Full `pnpm test:run` is running. Final outcomes will be recorded before merge. - Independent source review passed. Replacing the provider with the original source fails all 8 new phase/timing assertions. Replacing the host boundary with the original source fails all 3 owned diagnostic cases while 7 negative cases remain clean. - CI passed: 54 successful checks and 2 expected Storybook skips. This includes all server, workspace, serialized-server, Runner, browser, build, canary, and real-Sentry SDK lanes. - Greptile scored the committed source 5/5 with no review threads. - No live provider operation, task replay, runtime configuration change, or deployment was performed. ## Risks Low risk: this only adds optional diagnostic evidence. The first observed failing phase does not prove an upstream cause. Elapsed time ends at that observation, not at completion of later cleanup. Evidence is process-local and accompanies immediate failure reporting; post-restart reports cannot reconstruct it. No historical run or lease is backfilled. The wire schema remains v1 and cleanup ownership parsing is unchanged. ## Model Used OpenAI Codex, GPT-6, with repository inspection, code editing, local execution, and independent agent review. The exact model snapshot and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
7333e4eba8 |
Capture bounded workspace base-ref failure diagnostics (#15719)
## Thinking Path > - Paperclip manages AI agents and their work. > - A run can prepare a Git worktree before an adapter starts. > - That preparation must resolve the configured base ref. > - The same unresolved-ref error can follow a missing ref, failed fetch, or failed credential lookup. > - Current failure reports do not distinguish these paths. > - This PR records bounded outcomes from the operations that already run. > - Maintainers can use this evidence without changing task recovery or hiding failures. ## Linked Issues or Issue Description Related: #11892 established remote-only base-ref resolution and pre-adapter recovery. I found no open PR for these diagnostics. **What happened?** A fresh worktree can fail with `UnresolvedWorkspaceBaseRefError`. Its failure report has no closed fields that distinguish remote lookup, credential lookup, fetch, and local ref-probe outcomes. An empty warning list does not prove that a fetch ran or that a remote ref is absent. **Expected behavior** Keep the task-visible failure and owner recovery action. Report bounded observations from the existing operations. Keep unknown causes reportable. **Steps to reproduce** Use a local Git repository with a configured base branch that is absent. Repeat with no origin, an inaccessible local origin, or a rejected credential resolver. Start fresh workspace preparation. Before this change, each path lacks the new typed observations in the failure report. Tests use local repositories and intercepted fetch commands; they require no external Git service. **Paperclip version or commit** Reproduced against source commit `1602512820`, built from source. This affects core workspace preparation before adapter dispatch. ## What Changed - Record closed remote lookup, credential lookup, fetch, and local ref-probe outcomes. Limit new output parsing to 8 KiB. - Attach evidence privately to the resolver's own error. Reject copied, forged, inherited, accessor, and malformed evidence. - Persist allowed values in the existing setup result and export them through run-failure diagnostics. - Preserve Git commands, credential fallback, warning text, recovery actions, and Sentry reporting. - Add real Git, heartbeat, reporter, and real Sentry envelope tests. Document the fields and their limits. ## Verification - Current head: `015b2737df004e971eeeb134ebc9a1024e244494`, based on `be1bc5d494d`. The only rebase conflict was in heartbeat test imports. All three imports were retained; `git range-diff` confirms no other patch change. - Full `pnpm -r typecheck` and `pnpm build` passed on the current head. - Fresh focused tests passed: 13 real-Git producer cases, 279 complete collector/reporter cases, two actual heartbeat recovery cases, and all nine real Sentry SDK cases. The required-SDK gate used version 10.71.0. Actual envelopes contain only allowed fields, with no private fixture text or cross-event leakage. - Independent review passed all 16 combined heartbeat cases from both integrated changes and five real-Git probes with identical baseline command sequences and outcomes. The original producer negative control failed on the absent receipt; the candidate passed. No review findings remain. - Current-head CI is complete: 54 successful checks and two intentional Storybook skips. Greptile is 5/5 on this exact head, with no unresolved threads or merge conflicts. - Local aggregate limitation: the earlier full run reported one failure in an unchanged tool-access case and was stopped with SIGINT (exit 130) to isolate it. Vitest did not print the deferred assertion summary. The unchanged complete file passed 420/420 alone. That aggregate is not claimed as a pass. Before the mechanical rebase, the complete workspace-runtime file also passed 172/172. The fresh full CI result covers the integrated head. - An initial independent combined reporter run hit its database setup timeout; the unchanged complete reporter file then passed 186/186 alone. Earlier-head CI lost four runners and recovered on one verification retry; current-head CI passed separately. ## Risks - These observations do not prove that a remote ref is absent. All affected failures remain reportable, including resumed and provider-started runs. - The new fields include no repository URL, ref text, customer path, raw command output, or credential. Existing task-visible details are unchanged. - No migration, dependency, authorization, retry, or deployment change. ## Model Used OpenAI GPT-6 through Codex, with tool execution and parallel independent review. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
1602512820 |
fix(sentry): classify expected credential sharing denials (#15714)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A run can use only an AI credential shared with its responsible user. > - The selection service correctly rejects a credential outside that user's sharing permissions. > - That known configuration rejection still becomes a Sentry error because it has no owned selection marker. > - This pull request marks only that verified sharing rejection and reuses the existing pre-provider reporting gate. > - The task remains blocked with its recovery action, while unexpected authorization and runtime failures remain reportable. ## Linked Issues or Issue Description Refs #15618, which introduced owned provenance for expected AI account selection blockers. The credential-sharing subtype still used a plain HTTP 403. No duplicate open fix was found. **What happened?** An active responsible user selects an AI credential they cannot use. Selection correctly throws `This credential is not shared with the responsible user`. A fresh task remains blocked, but the broad `ConfigurationIncompleteFailure` wrapper is also sent to Sentry. **Expected behavior** Preserve the denial, HTTP 403, task error and owner recovery action. Keep this proven pre-provider configuration rejection out of Sentry without suppressing unrelated permission failures. **Steps to reproduce** Assign a fresh run an explicit personal credential owned by another user, or a shared credential whose human audience excludes its active responsible user. The existing selection check denies access before adapter dispatch. ## What Changed - Add a dedicated sharing-rejection factory that preserves the existing 403, message and absent details while retaining one private bounded reason. - Use it only after active responsible-user membership and the existing credential checks have passed. Missing identity and other authorization checks remain unmarked. - Keep the existing failed/preparing/setup/bootstrap reporting conditions and provider/resume exclusions unchanged. - Add real selection, heartbeat recovery, reporter-boundary and forged-error regressions. Update monitoring documentation. ## Verification - Producer, private-marker and run-failure reporter suites: 310 tests passed. - Actual heartbeat recovery cases: 6 passed; 377 unrelated cases filtered. These verify failed runs, blocked tasks, active owner recovery actions, failed wakeups and no adapter dispatch. - Coverage includes personal and restricted shared credentials, allowed audience changes, missing identity, inactive credential owners, wrong binding kinds, membership in another company, agent permission checks, forged 403 markers, resumed native input and provider work already started. - `pnpm -r typecheck` and `pnpm build` passed. All 53 PR checks passed; two unrelated Storybook jobs were skipped. - The full local aggregate was **not green**: it reported an existing browser-sign-in case and a real-runner damaged-resume case before the broad run was stopped (exit 130). The full affected AI-connections file then passed 111/111 in isolation, including the browser case. The unchanged native-session-resume file reproduced 47 passed / 1 failed on both this commit and an untouched base checkout with the same local dependencies and runner binary: the fixture expected a succeeded terminal result and received a failed one. No broader local-suite pass is claimed; full sharded CI passed. - Independent review passed the same 310 focused tests and six heartbeat cases from a clean checkout. Replacing only the producer with its original implementation fails both new marker assertions while preserving 403; restoring this change passes. - Greptile scored this exact commit 5/5 with no review threads. - Source, tests and documentation were scanned for credentials, personal data, private URLs and generated artifacts; none found. ## Risks - A known sharing rejection no longer emits a Sentry run-failure event only when the existing fresh setup proof is present. It remains visible in the task and recovery UI. - No credential-use permission, HTTP contract, execution or retry behavior changes. - This is not a blanket filter for HTTP 403 or `ConfigurationIncompleteFailure`. Unknown causes and resumed or post-provider failures remain reportable. ## Model Used OpenAI GPT-6 through Codex, with tool execution and parallel independent review. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run focused tests locally and they pass; broader local limitations are documented above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
1c4ce44e13 |
fix: retain bounded workspace transfer failure evidence (#15637)
## Thinking Path > - Paperclip manages AI agents and the work they produce. > - Sandbox work must return to the host when an agent run ends. > - A failed transfer must keep its source available for recovery. > - Some transfer errors lose their useful fields before restore diagnostics reach Sentry. > - This change records fixed transfer stages, failure kinds, and numeric RPC codes. > - The next failure can identify the operation that failed without exposing workspace contents. ## Linked Issues or Issue Description **What happened?** A workspace transfer can fail after model execution succeeds. Several command, archive validation, and download errors become plain errors. The existing RPC envelope then reports `unknown`, with no transfer stage or command exit code. A host RPC timeout also loses its code from restore diagnostics. **Expected behavior** Keep bounded evidence through the provider, worker RPC, restore result, and Sentry context. Keep the same exception, error message, retry rules, and source retention policy. A transfer timeout must remain separate from the model execution timeout flag. **Steps to reproduce** Return a nonzero exit code from the sandbox archive command, return a per-file download error, exceed a tar listing limit, or time out the sync-out RPC. Observe the missing fields in the saved restore diagnostic. The added tests use local fixtures for these cases. Related public work: #15479 preserves the source after restore failure. #15481 carries bounded sync-out diagnostics through RPC. This change supplies missing producer evidence and extends that same envelope. I checked the roadmap and searched open PRs for duplicate transfer diagnostic work. ## What Changed - Add optional transfer stage and failure kind fields to the existing diagnostic envelope. Capture command exits and listing deadlines at their producers. - Record only known codes from typed RPC errors at the sync-out boundary. Revalidate every field before persistence and Sentry projection. - Keep annotations private to each outbound request and restore settlement. Preserve frozen error identity and prevent evidence from leaking across concurrent or later calls. - Document the fields. Cover producer failures, quota limits, old workers, concurrent error reuse, and redaction through the real Sentry SDK. ## Verification - Focused SDK, provider, host, persistence, and real Sentry tests: 449 passed, no skips. Independent review also ran the focused contracts and all provider tests. The optional Sentry SDK is required for this check, so the contract tests cannot skip. - A compiled Daytona transfer and compiled SDK worker pass a local RPC round trip. This uses the development TypeScript loader for workspace dependency exports. The transfer and capture modules are compiled JavaScript. - `pnpm -r typecheck` and `pnpm build` passed. The excluded Daytona package also passed `tsc --noEmit`. - The initial local `pnpm test:run` stopped in the general-server group: 60 suites could not initialize embedded PostgreSQL because the isolated install skipped its Darwin library-link setup. The existing package postinstall repairs this; `initdb --version` now passes. All 60 affected suites then passed: 1,342 tests, no skips. No local full aggregate pass is claimed. - Independent source review covers privacy, concurrency, packaging, and recovery behavior. ## Risks - These diagnostics help identify future failures. They do not establish or repair the cause of a past transfer failure. - New fields are optional. Older workers remain compatible. Unknown values are omitted. - The change adds a small request-local diagnostic scope. It keeps the original thrown errors, archive confinement, cleanup order, timeout values, retry count, and source retention policy. - No schema or deployment changes are required. No live provider actions are part of validation. ## Model Used OpenAI Codex, based on GPT-6, with repository inspection, code execution, and independent agent review. The exact deployed model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
65b5581805 |
Keep proven AI account selection blockers out of Sentry (#15618)
Preserve trusted evidence from explicit AI account selection rejections so known setup blockers stay out of Sentry. Keep task failure, blocked state, recovery actions, HTTP behavior, and unexpected error reporting unchanged. Validated with full typecheck/build, focused PostgreSQL lifecycle and reporting tests, independent reporting tests, and all PR CI checks. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
3d0e74e743 |
Keep proven external connection failures out of Sentry (#15590)
Apply the reviewed change for Keep proven external connection failures out of Sentry. Validation: local typecheck/build and focused regression tests, passing exact-head CI, Greptile 5/5, and independent source review. The PR records full-suite evidence and any local environment limitations. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
5717523b9e |
fix: retain the original repository for reused task workspaces (#15528)
Apply the reviewed change for fix: retain the original repository for reused task workspaces. Validation: required local typecheck and tests, passing CI, Greptile 5/5, and independent review of the exact source head. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
1d19f9b562 |
Distinguish failed Git inspection from missing worktree registration (#15525)
Apply the reviewed change for Distinguish failed Git inspection from missing worktree registration. Validation: required local typecheck and tests, passing CI, Greptile 5/5, and independent review of the exact source head. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
6b171c9616 |
Keep proven local workspace configuration conflicts out of Sentry (#15521)
Apply the reviewed change for Keep proven local workspace configuration conflicts out of Sentry. Validation: required local typecheck and tests, passing CI, Greptile 5/5, and independent review of the exact source head. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
465140596f |
Preserve bounded workspace sync diagnostics across RPC (#15481)
Preserve bounded error codes and HTTP/exit statuses across the environmentSyncOut worker RPC boundary, and revalidate that method-scoped envelope before attaching host restore diagnostics. Keep the original error and all recovery policy unchanged; do not transmit provider payloads or credentials. Verified real RPC roundtrip and privacy regressions, 95 focused tests, 68 independent tests, full typecheck/build and all exact-head CI. Greptile 5/5 with no unresolved comments. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
128c95837b |
Retain bounded orphan process-loss diagnostics (#15475)
Record bounded observer uptime, run and output ages, process-check observations and retry eligibility before orphan cleanup changes the evidence. Preserve existing recovery and reporting behavior and omit process IDs, raw paths and credentials. Verified focused helper, actual reaper and real SDK regressions, full typecheck/build, exact-head CI and independent review. An unchanged Cursor timeout passed its isolated retry; unrelated local Slack timing failure is documented separately. Greptile 5/5 with no unresolved comments. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
43b0aff54b |
Keep pre-dispatch secret configuration blockers out of Sentry (#15472)
Keep confirmed pre-dispatch missing-secret setup blockers out of Sentry while preserving failed runs and owner recovery actions. Runtime, provider and ambiguous failures remain reportable. Verified full exact-head CI, focused local regressions, workspace typecheck and independent review; Greptile 5/5 with no unresolved comments. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
90182b4f8b |
Report bounded Cloud portfolio failure diagnostics (#15340)
## Thinking Path > - Paperclip manages work for AI agents and their companies. > - Cloud instances proxy a trusted user's portfolio through the control plane. > - A failed request returns a generic error to the client. > - That replacement error loses the failure phase and network code in Sentry. > - Operators need bounded evidence without upstream messages or credentials. > - This PR adds safe diagnostics while preserving the existing request behavior. ## Linked Issues or Issue Description Refs #10850, which added the portfolio proxy. No open PR for this diagnostic gap was found. **What happened?** A rejected portfolio fetch becomes a generic 502 in Sentry. The event cannot distinguish a connection reset, deadline, HTTP response failure, or body failure. The route's previous warning also included the original error and a stack identifier. **Steps to reproduce** Make the portfolio proxy's fetch reject with a TypeError whose cause has `code: ECONNRESET`. The client correctly receives the generic 502, but the captured replacement error loses that code. **Expected behavior** Keep the existing client response. Attach only bounded server-side diagnostic fields to the failure event. Do not retry the request or expose the original error. ## What Changed - Add a typed portfolio error with a private frozen diagnostic record: phase, upstream HTTP status, elapsed milliseconds, and an allowlisted network code. - Read at most four error/cause objects through own data properties. Unknown codes, messages, getters, and out-of-range values do not enter the record. - Replace the route's raw-error warnings with safe fields. Send a plain error plus event-local context through the existing optional Sentry gate. Keep the route callsite and default fingerprint policy. - Preserve authentication, trusted headers, cookies, exact HTTP error bodies, cache behavior, the ten-second deadline, and one fetch per request. Public responses receive no diagnostic fields. - Document the fields and test HTTP behavior, privacy, and event isolation with the real Sentry SDK. ## Verification - `PAPERCLIP_REQUIRE_SENTRY_TEST_SDK=1 pnpm exec vitest run server/src/__tests__/cloud-portfolio-error.test.ts server/src/__tests__/cloud-routes.test.ts server/src/__tests__/sentry.test.ts server/src/__tests__/run-failure-sentry-real-sdk.test.ts`: 65 passed, with the audited optional SDK installed. - Full local `pnpm -r typecheck` and `pnpm build` passed on Node 24.21.0 and pnpm 9.15.4. - Independent review found no blockers and independently passed all 65 tests, including the real SDK checks, on this exact commit. - Full Linux CI passed on this exact commit and provides aggregate suite coverage (54 successful checks, 2 intentional skips). A duplicate full local aggregate was not run. - The first SDK contract job failed before tests when npm could not resolve an OpenTelemetry transitive package. A subsequent empty-cache install first encountered a missing tarball, then succeeded after the registry artifact became available. All 6 real-SDK tests passed against that fresh install; the single unchanged-head CI retry passed. The SDK pin, workflow, and dependency files are unchanged. - The first browser shard 8 run timed out waiting for the inbox retry reply after 45 seconds. The unchanged isolated case passed (1/1), and the test, UI handler, fixture, and recovery files match the base commit. The failed log contains no wakeup POST before the test's immediate navigation; a navigation/request timing race is suspected but unproven without a trace. The single unchanged-head shard retry passed (20 passed, 1 skipped); no timeout or source change was made. - Greptile reviewed this exact commit at 5/5 with no unresolved review threads. - No live portfolio request was replayed. Route tests use controlled local upstream responses. - The added diff passed the secret and PII scan and `git diff --check`. ## Risks This is a diagnostic change. It does not identify or repair the origin of a connection reset. Unknown transport failures remain `unknown`. Elapsed values outside 0–60,000 ms become null. Default Sentry fingerprinting remains enabled; exact historical group membership is not guaranteed. No retry, migration, deployment, or configuration change is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository editing, code execution, and independent agent review. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
1d5a439e2b |
Diagnose native model rejections and preserve repairable reviews (#15304)
## Thinking Path > - Paperclip manages agent execution and recovery. > - A native provider can return a failed turn with an accepted result. > - This path replaced a structured model rejection with a generic failure. > - Finalization could also queue the same failed request for recovery. > - This change retains a bounded model and authentication diagnosis. > - Workers wait for a configuration fix; reviews retain their assignment. ## Linked Issues or Issue Description **Describe the bug** A native Codex turn can report HTTP 400 because the selected model is not supported by its ChatGPT connection. When the turn also returns a failed semantic result, the run shows `Native session failed` and `adapter_failed`. The native failed-result finalizer can create a retry intent before the adapter error is saved. **To Reproduce** Use a native Codex session that returns a structured `invalid_request_error` model/ChatGPT rejection and an accepted failed result. The regression tests reproduce this with a stub provider and a real isolated PostgreSQL database. No provider account or live request is needed. **Expected behavior** Show an actionable error and stop automatic retries of the rejected request. Block the current worker for a configuration fix. Preserve a pending review so its reviewer can explicitly retry after repair and resolve the original assignment. A stale run must not block a new task owner or override a completed review. **Version and deployment mode** The affected source path is present at `858094ba81`. This change applies to native Codex execution in local and remote environments. Related work: #13853 covers CLI protocol compatibility, #14942 refreshes runtime/model pins, and #12767 proposes legacy adapter model validation. This change concerns accepted native failed results and does not replace those changes. ## What Changed - Recognize only a bounded structured HTTP 400 provider error for the exact selected model and failed turn. - Save a fixed actionable error and four fixed diagnostic fields. Exclude raw provider text and model names from the new diagnostic. - Re-derive finalization evidence from the pinned execution and committed runner event, including its canonical hash and identity bindings. - Block only the current worker. For a current pending reviewer, preserve the original review decision and status version, release execution, and create no automatic recovery action. Recheck execution ownership under the issue lock. Preserve superseded and completed reviews. - Keep unknown failures, accepted results, cleanup, and ownership fences intact. Advance the status policy version to `phase6-v8`. - Route this exact failure code through the existing configuration recovery path. Permit an authenticated Board retry of this native failure only for its saved, still-pending review; discard caller review markers and revalidate at dispatch. ## Verification - Node 24: `pnpm -r typecheck` and `pnpm build` passed. - Independent focused validation: 222 tests passed across the new observer and PostgreSQL integration suites, status arbiter, diagnostic redaction, recovery classification, wake policy, and wake use cases. - The integration tests use actual result persistence, finalization, and the wake dispatcher. They cover zero retry dispatch for an authorized rejection, repeated reconciliation, pending/superseded/resolved reviews, reassignment, and a same-agent successor at an unchanged status version. - Final-commit verification: full typecheck and build passed; independent route/integration/arbiter/review validation passed 136 tests with no remaining source findings. This includes authenticated retry refusals, actual heartbeat retry admission, atomic queued-review claim, and acceptance of the original review. - The local aggregate test run was stopped to avoid duplicating the canonical Linux CI aggregate on this development host. Its partial log showed no failures; no full local-suite pass is claimed. The full hosted Linux CI aggregate passed on the final commit. - No live provider calls, runtime changes, or schema changes were made. ## Risks The classifier is intentionally narrow. A changed provider response format remains an unknown failure. No global model restriction is introduced. The worker blocker requires current task authority and does not replace newer assignments or completed review decisions. A pending review remains in review and requires an explicit retry after configuration repair; it is not rebound to a new status decision. The retry decision is durable. There remains a small existing gap between status/effect commit and diagnostic metadata persistence; a crash there can leave a generic diagnostic while preserving the blocked decision. No schema migration is required. ## Model Used OpenAI Codex, GPT-6, with reasoning, tool use, and code execution. The exact serving model suffix and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #123` / `Refs #123` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal ticket id or instance-derived details - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
efac8ff2f4 |
Add bounded Git integration restore diagnostics (#15291)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Sandbox runs must restore their workspace before finalization can succeed. > - Restore diagnostics identify the failed phase and operation. > - A Git integration exit code can still describe several different failures. > - This pull request adds fixed command labels and supported failure classes. > - Operators can distinguish these failures without collecting private Git output. ## Linked Issues or Issue Description Refs #15005. This branch includes merged #15268. It labels that change's locked ref transaction and nested branch probe without changing their behavior. **What happened?** A failed Git integration can report only `git_integration`, `unknown`, and an exit code. That evidence does not identify the failed command. Some Git versions also return exit 1 for both a merge conflict and an invalid object. **Expected behavior** Record a fixed command family and a supported failure class. Keep unknown cases as `unknown`. Exclude command arguments, process output, paths, repository URLs, filenames, and ref names. **Steps to reproduce** The tests create local repositories with conflicting commits, a missing object, and an expected-old ref mismatch. They call the real Git operations and inspect the resulting diagnostic. No hosted workspace or external provider is used. **Paperclip version or commit** Base: `e99854249c`. **Deployment mode** Built from source. The diagnostic applies to sandbox workspace restore. ## What Changed - Label Git integration calls with a closed command enum. Preserve arguments, options, errors, and retry behavior. - Recognize supported object and ref errors and OS permission codes. Require both exit 1 and completed tree output for a merge conflict. - Carry the closed fields through the existing restore receipt, saved adapter result, and Sentry projection. Revalidate saved metadata before projection. - Test nested wrappers, parallel failures, reused errors, handled probes, result settlement, and privacy with the real Sentry SDK. - Document the field contract and its limits. ## Verification - Eight focused suites pass: 295 tests. These cover Git sync, restore diagnostics, result settlement, teardown, sandbox runtime, failure projection, the real Sentry SDK, and native warm-workspace Git history. - `PAPERCLIP_REQUIRE_SENTRY_TEST_SDK=1 pnpm exec vitest run packages/adapter-utils/src/workspace-restore-diagnostics.test.ts packages/adapter-utils/src/workspace-restore-result.test.ts packages/adapter-utils/src/git-workspace-sync.test.ts packages/adapter-utils/src/workspace-restore-teardown.test.ts packages/adapter-utils/src/sandbox-managed-runtime.test.ts server/src/services/__tests__/run-failure-diagnostics.test.ts server/src/__tests__/run-failure-sentry-real-sdk.test.ts server/src/__tests__/native-workspace-sync-history.test.ts` - Local checks use Node 24.21.0 and the pinned pnpm 9.15.4. The optional Sentry SDK is pinned to the declared 10.71.0 and uses an in-memory transport. - `pnpm -r typecheck` and `pnpm build`: pass on the rebased head. - The full local `pnpm test:run` was stopped before source changes when #15268 merged and required a rebase. It reported three pre-existing company-skill cache test failures on macOS. The same three cases fail on clean bases `2c43b39167` and `e99854249c` and the earlier head with `EACCES` when publishing a read-only cache directory. The relevant test, service, and cache source blobs are identical. No cache changes are included. Later local test groups were not reached. - All 54 checks pass on exact head `8448896bc6`, including the post-ready security scan, with two intentional Storybook skips. Greptile scores this head 5/5 with no review threads. Full Linux CI covers the local groups that were not reached. - Independent review found no blocker. Its diagnostics, Git workspace, and native history suites pass 115/115 on this head. - A source comparison confirms that removing only diagnostic wrappers yields the merged upstream Git integration code exactly, including ref locks, transaction protocol, arguments, options, and retry decisions. - The clean base has stale dependency overrides in its lockfile. Local installation resolved them as the existing PR CI fallback does. No manifest or lockfile change is included. - `git diff --check` and a local secrets/PII review pass. ## Risks A Git version or localized message may not match a known failure form. Such cases remain `unknown`. Up to 16 KiB of stderr and the bounded tree-ID prefix of stdout are inspected only in memory. The saved fields contain enum values only. These diagnostics do not establish workspace recovery or authorize retries. Restore decisions and Git mutations are unchanged. No schema migration is required. ## Model Used OpenAI Codex, GPT-6, with reasoning, tool use, and code execution. The service does not expose the exact model deployment ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
3b47a6befd |
Record bounded workspace restore failure stages (#15005)
Capture allowlisted restore substep and failure metadata for future diagnosis while preserving workspace recovery behavior, error identity, cleanup ordering, and privacy. Validation: exact-head Greptile 5/5, passing CI, no unresolved review threads, and a clean merge. Detailed verification and limitations are recorded in the pull request. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
1815474597 |
fix: report pending execution phase at Stop timeout (#14990)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server owns each adapter execution and waits for it to settle after Stop. > - A Stop timeout reports that termination remains unverified. > - Existing phase timings arrive only after their work completes, so a stalled await has no timing. > - This pull request samples the pending phase when the Stop timer expires. > - Operators can identify the pending operation without treating diagnostics as stop proof. ## Linked Issues or Issue Description **What existing behavior does this improve?** The opt-in Sentry context for an unconfirmed adapter Stop timeout. **Current behavior** The timeout includes execution identity but no pending phase. A session close, instruction collection, workspace restore, or diagnostic write can remain pending without producing its completion timing. **Proposed behavior** Add a closed-list phase and elapsed milliseconds from the exact live execution control. Sample them when the timeout fires. Report `unknown` and a null age when the control or attribution is unavailable. **Reason and benefit** The next timeout can identify which operation is still pending. It does not require task text, paths, provider output, or additional database writes. **Breaking changes** No API or execution behavior change. The existing opt-in error context gains two fields. Related public work: #14639 added Stop identity diagnostics; #14866 and #14945 cover instruction cleanup and teardown outcomes. This change adds pending attribution to those paths. The native Stop work in #14802 remains separate. ## What Changed - Add a bounded tracker per execution control. Token scopes support nested and overlapping awaits. A late release cannot clear a newer scope. - Track adapter execution, ACP cancellation and settlement, diagnostic writes, and host cleanup. Keep a coarse host scope until the executor finishes. - Sample only the matching current control and settlement promise at timeout. Freeze the sanitized result. Use a monotonic clock and cap elapsed time at one day. - Test stalled operations, repeated Stop calls, stale and wrong-run controls, callback failures, scope bounds, and the real Sentry SDK context. ## Verification - `pnpm -r typecheck` passed. - `pnpm build` passed. - Six focused suites passed: 79 tests. They cover pending scopes, Stop control ownership, real ACP settlement stalls, and Sentry context isolation. - The real Sentry SDK contract ran with the audited optional peer `@sentry/node@10.71.0` installed outside the workspace. Valid phase and elapsed values were exported; arbitrary labels and nonfinite elapsed values were rejected. - An independent agent reviewed the production diff and ran the focused tests without blockers. - `git diff --check` and a redacted Gitleaks scan passed. The local full `pnpm test:run` was stopped during its large serial server batch to avoid duplicating the sharded CI suite. No complete local broad-suite pass is claimed. - All CI gates passed on `c223237b58aa8d479d1c66d7d399de30270bac4f`: 54 successful checks and two expected Storybook skips. This includes the full sharded test suite, typecheck, build, real Sentry SDK isolation, browser tests, canary dry run, and the security scan after the PR became ready for review. - Greptile scored the same commit 5/5 with no actionable findings or unresolved review threads. ## Risks This is diagnostic instrumentation. Cancellation, deadlines, teardown order, termination proof, and file recovery proof remain unchanged. Unsupported or uninstrumented work uses a coarse phase. Tracker overflow fails closed to `unknown`. The tracker emits no new run-log or Telemetry event. The existing Sentry opt-in gate remains in place. ## Model Used OpenAI Codex, GPT-6. The agent used code inspection, local command execution, automated tests, and an independent agent review. The runtime did not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
4abff286c2 |
fix: retain resolved ACP execution timeout metadata (#14986)
## Thinking Path > - Paperclip manages agent work through adapters and heartbeat runs. > - ACP adapters resolve a timeout for the selected execution target. > - An untouched sandbox timeout uses a four-hour default. > - Heartbeat finalization rebuilt the metadata from the stored zero. > - This made a timed-out sandbox run report an effective timeout of zero. > - This change retains the adapter's resolved policy for accurate run diagnostics. ## Linked Issues or Issue Description **What happened?** ACP sandbox runs with `timeoutSec: 0` use the four-hour default. Their terminal metadata reports `effectiveTimeoutSec: 0` and `timeoutSource: config` because heartbeat finalization only reads the stored agent configuration. **Expected behavior** The result must report the policy the adapter used: `14400` and `sandbox_default`. Explicit limits, fractional limits, explicit unlimited overrides, and local defaults must keep their resolved values. **Steps to reproduce** Run an ACP adapter on a sandbox target with `timeoutSec: 0`. Compare the start log's four-hour policy with the terminal result's effective timeout. The new tests exercise the adapter result and the heartbeat metadata merge without waiting four hours. **Paperclip version or commit** Reproduced from `6eaf218924f0a89faf1c02eb0d6877a6c5c8a2cb`. **Deployment mode** Self-hosted server with an ACP sandbox execution target. Searched open timeout and metadata issues and PRs. Related #14804 exposes timeout configuration in forms; #14496 proposes a default policy; #14833 addresses CLI session retention. This PR changes only ACP result metadata. ## What Changed - Retain the resolved timeout in the ACP result after settlement. - Use validated adapter resolution when merging terminal timeout metadata. Preserve config fallbacks for older adapters and the HTTP millisecond policy. - Test sandbox defaults, explicit and fractional limits, explicit unlimited overrides, local defaults, malformed metadata, and unchanged cancellation fields. - Document the result fields and their meaning. ## Verification - `pnpm -r typecheck` passed. - `pnpm build` passed. - Stop metadata tests: 23 passed. - Reporter and diagnostic suites: 84 passed; two real-Sentry-SDK tests skipped because the optional SDK is not installed. - ACP engine suite: all 206 tests passed with a deterministic local `gemini --version` shim. Ambient host CLI probes made the existing Gemini session-resume fixture intermittent (one assertion failure in each of two broad runs); an isolated 15-case rerun and the clean-base 206-test suite also passed. No assertion or timeout was changed. - `pnpm test:run` is in progress. This PR does not claim a complete local suite pass. - Independent review found no blocking issues. The tests cover real adapter emission and the real metadata merge separately. ## Risks Low runtime risk: this changes result diagnostics. It does not change timeout values, cancellation acknowledgement, cleanup, checkpoint safety, retries, or provider operations. It does not fix the cause of a quiet or long-running tool. Older stored results are not rewritten. The new source values apply only when an adapter returns a valid resolution. ## Model Used OpenAI GPT-6 with reasoning, repository inspection, code editing, and test execution. The deployment-specific model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
4b9a6000f7 |
Add bounded evidence for directory lock timeouts (#14787)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent files use directory locks during collection and cleanup. > - A lock timeout can fail finalization after the model turn completes. > - The timeout currently identifies no owner state or waiting operation. > - This pull request adds bounded evidence to the existing run failure report. > - Operators can distinguish a known local holder from a possible old lock without changing lock safety. ## Linked Issues or Issue Description **What happened?** A directory lock timeout does not distinguish active local work from an owner record left by an earlier process. The stored execution stage can also precede the cleanup operation that failed. **Expected behavior** The failure report should identify the waiting operation and expose bounded ownership clues. It must preserve the timeout and keep unknown ownership protected. **Steps to reproduce** Hold a directory merge lock while a second caller reaches its acquisition deadline. The regression tests exercise a live holder and an older owner record with a live PID. Related: #9667 proposes stale-lock recovery under a single-server assumption. This change only adds evidence and does not adopt that assumption. #14575 and #14665 add other run failure diagnostics. ## What Changed - Record lock owner state, capped age and wait duration, same-process and process-age comparisons, and whether this module holds the lock. - Label agent-directory release, collection, checkpoint, and warm handoff timeouts with a fixed operation code. - Validate each field before the existing event-local Sentry report accepts it. Exclude owner records, PIDs, paths, and absolute timestamps. - Limit the extra diagnostic owner read to 100 ms with best-effort abort; malformed JSON is `invalid` and unreadable owner records remain `unknown`. - Document the diagnostic limits and verify that contenders never reclaim protected locks. ## Verification - Focused lock, diagnostic, real Sentry SDK, and database-backed agent-directory tests: 126 passed, including stalled-read and malformed/missing/unreadable-owner regression coverage. - Final revision `0691613dcc`: all 54 reported checks successful, with two intentionally skipped Storybook checks. Greptile: 5/5, zero unresolved review threads; no merge conflicts. - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - `pnpm test:run`: complete suite coverage ran with the existing repository shard flags: four general-server shards, four serialized shards, two general-workspaces-a shards, and general-workspaces-b. The full run is not green because of the base failures below. - The broad run found 13 failures in the unchanged macOS skill-cache tests. All 13 reproduce on the clean base revision. Open PR #14290 covers that existing failure. - Two unchanged CLI archive tests hit their five-second limits during the broad run; all 17 tests in that file pass on recheck. A CLI auth socket error also cleared on recheck (19 tests), and its full serialized shard passed on rerun. ## Risks This is a diagnostic change, not a stale-lock fix. Owner observations can race with release. Wall-clock shifts can affect the age comparison. A local-holder flag covers only this module instance. None of these fields authorizes reclamation or proves a file save. Lock acquisition, release, retries, task status, and recovery guards retain their current behavior. No schema change or deployment action is required. ## Model Used OpenAI Codex, based on GPT-6, with code execution and repository tools. The exact model build and context window were not exposed to this agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass (focused checks pass; existing base failures are documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |