mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 22:13:59 +01:00
07a2941f0c6b2747b26c73df2c8e46dff414adde
1813
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Devin FoleyandPaperclip
|
07a2941f0c |
fix(recovery): reclaim verified inactive provider caches on full disks (#19192)
Recover from a full-disk workspace save by reclaiming only a verified inactive provider installation under the existing source and maintenance authority. Commit the preparation inventory in Core before changing cache contents. Bind interrupted cleanup to stable filesystem identities and keep small staged generation markers when directory birth times are unavailable. Skip read-only or permission-denied assets and stop freeing blocks once marker space is available. Preserve task files, Git, session history and uncertain action receipts. Only the normal saved-workspace and provider-stop receipts permit continuation. Validated with 446 complete recovery regressions on the repair revision, subsequent focused recovery and transport suites, typecheck/build, and all exact-head hosted CI. Greptile reviewed the final head at 5/5 with no unresolved findings. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
037c58bb22 |
fix(daytona): keep the provider auto-stop strictly after the warm runner idle window (#19199)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm remote agent keeps its sandbox running after a turn. The host keeps a warm runner for an idle window (`runnerIdleTimeoutMs`, or the agent's `idleTimeoutMs`) > - The Daytona provider also stops an idle sandbox after `autoStopInterval` minutes. The default is 15 minutes > - On staging, the warm idle window was 900,000 ms (15 minutes). The two windows were equal > - A follow-up turn at the end of the warm window resumed a sandbox that Daytona was stopping. Daytona refused the resume in about 4 seconds, and the run failed at setup > - #19195 added a bounded retry for that refusal. It did not remove the cause > - This pull request makes the two windows non-overlapping by construction. Daytona derives its auto-stop for a warm reusable sandbox from the same warm idle value that the host uses > - The benefit is that a follow-up inside the warm window always finds a running sandbox, and idle sandboxes still stop ## Linked Issues or Issue Description Refs #19195 (bounded retry for a refused resume; this change removes the common cause). **What happened?** On a staging soak, three follow-up runs failed at setup with a refused sandbox resume. Each resume began 15 minutes 8 to 25 seconds after the previous turn released the sandbox. The warm idle timeout was 900,000 ms. Daytona's default `autoStopInterval` is 15 minutes. Nothing in the code kept these two values apart. **Expected behavior** A reusable sandbox that keeps a warm runner must not auto-stop inside the warm idle window or at its edge. The provider backstop must fire after the host's own idle handling, with a margin. Idle sandboxes must still stop. **Steps to reproduce** 1. Configure a Daytona sandbox environment with `reuseLease` and a warm runner with `runnerIdleTimeoutMs: 900000` (or an agent with `lifecycleMode: warm`, `idleTimeoutMs: 900000`). 2. Run a turn. Do not set `autoStopInterval`. 3. Before this change: the sandbox is created with `autoStopInterval: 15`, equal to the warm window. A follow-up near minute 15 races the provider stop. **Paperclip version or commit** `c656b95b6b` (current `master`). ## What Changed - Daytona plugin (`plugin.ts`): add `effectiveAutoStopIntervalMinutes`. For a reusable lease that keeps a warm runner, the effective auto-stop is at least `ceil(warm idle / 1 min) + 5` minutes. The plugin applies the floor only when the host forwards `warmRunnerIdleTimeoutMs`, or when the environment itself selects `runnerLifecycleMode: warm`. An explicit `0` (disabled), a longer configured value, `inherit` with a per-turn agent, per-turn, and non-reusable sandboxes keep the configured value. - Daytona plugin: use the effective value at sandbox creation. On resume, raise a shorter auto-stop on an existing sandbox. This update is best effort and bounded to 5 seconds; the #19195 retry still covers a refusal. The stale-handle refresh threshold keeps the configured interval, so a cached handle is not trusted for longer before the provider confirms the new interval. The warm window is not written to the saved normalized config. - Host (`runner-environment-lifecycle.ts`): add `resolveWarmRunnerIdleTimeoutMsForRun`. It returns the run's effective warm idle window (environment `runnerIdleTimeoutMs`, else the agent `idleTimeoutMs`) for a warm `paperclip_runner` sandbox run. It shares the lifecycle-mode resolution with `resolveRunnerEnvironmentForRun`. - Host (`environment-run-orchestrator.ts`, `environment-runtime.ts`): pass that value as `warmRunnerIdleTimeoutMs` to the sandbox-provider driver on acquisition and on remote recovery resume. The driver adds it to the provider worker config only. Lease identity, reuse matching, and lease metadata are unchanged. - Daytona manifest: describe the warm floor on `autoStopInterval`. - `doc/execution-semantics.md`: document the contract. ## Verification - Red (before the fix, same tests, with the floor disabled): 5 failures in `plugin.test.ts` "warm idle window versus Daytona auto-stop". The staging case failed with `expected 900000 to be greater than or equal to 1200000`. The resume case failed because `setAutostopInterval` was not called. - Red (host): the new `environment-runtime.test.ts` case first failed because the provider worker config had no `runnerIdleTimeoutMs`. The service wrapper dropped the field. The fix forwards it. - Green: `npx vitest run --config vitest.config.ts` in `packages/plugins/sandbox-providers/daytona`: 563 passed. This includes a setter that never settles (resume completes after the 5-second bound) and `inherit` with no forwarded window (explicit `autoStopInterval: 2` stays 2). - Green: `npx vitest run src/services/runner-environment-lifecycle.test.ts src/__tests__/environment-runtime.test.ts src/__tests__/environment-run-orchestrator.test.ts` in `server/`: 242 passed. This includes recovery resume forwarding the admitted warm window. - The regression tests assert the relationship `autoStop >= warm idle + margin` for several warm windows (125 s, 300 s default, 900 s, 1,800 s) and for an explicit `autoStopInterval: 15`. They are not constant checks. - `node scripts/typecheck-low-memory.mjs` passes. `pnpm typecheck` in the Daytona package passes. ## Risks - A warm reusable Daytona sandbox now stays running up to the warm window plus 5 minutes after its last activity, instead of 15 minutes. With a 15-minute window this is 20 minutes. Cost is bounded and idle sandboxes still stop. - The resume-time `setAutostopInterval` call is best effort. If it fails, the old interval stays and the #19195 retry still applies. - Other providers are unchanged. They receive one extra host-owned config key (`warmRunnerIdleTimeoutMs`) only for warm `paperclip_runner` runs, and they ignore unknown keys. - No lease identity change, so existing retained sandboxes are still reused. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c656b95b6b |
fix(environments): retry a refused sandbox resume with the bounded bootstrap budget (#19195)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote agent keeps its sandbox after a turn, so a follow-up
turn resumes the same sandbox and its files
> - The sandbox provider auto-stops an idle sandbox (Daytona: 15 minutes
by default). The warm native idle window is also 15 minutes
> - A follow-up that reaches the sandbox during that stop gets a
provider refusal. The host keeps the sandbox ("the lease was preserved
and no replacement was created"), which is correct
> - But setup then records a generic `setup_failed`. No retry is
scheduled, no cause is recorded, and the reply that woke the agent is
effectively dropped
> - This pull request gives that refusal its own failure code and routes
it through the existing bounded bootstrap retry. The retry resumes the
same recorded sandbox
> - The benefit is that a follow-up continues a few seconds later on the
same saved sandbox, instead of stopping the task with no next action
## Linked Issues or Issue Description
Refs #18852 (the bounded bootstrap retry that this change reuses).
Related: #19152 and #19187 (copy-back causes), #19178 (Daytona
errored-sandbox recovery before stop-only cleanup), #19182 and #19180
(recovery holds). This change does not overlap with them: they cover
cleanup, copy-back, and recovery after a run. This change covers a
resume refusal before provider work starts.
**What happened?**
On a staging soak, two follow-up turns failed at setup after a
successful first turn:
- A board reply woke the agent. The run failed after 4.9 seconds with
`setup_failed`: `Reusable sandbox lease "…" could not be resumed; the
lease was preserved and no replacement was created.` The issue stayed in
review with no recovery action. The reply was not delivered.
- An earlier run on another task failed with the same message.
Both resume attempts started 15 minutes 8–25 seconds after the previous
turn released the sandbox. The first turns used `keep_running` with a
warm idle timeout of 900,000 ms. The Daytona default `autoStopInterval`
is also 15 minutes. So the resume reached the sandbox while the provider
was stopping it. The provider refused within seconds; this was not a
long resume timeout. The provider cause was not recorded, so this timing
is the strongest evidence. The exact provider state remains an
inference.
**Expected behavior**
A refused resume before provider dispatch is a temporary infrastructure
condition. The host retries it with a small, bounded budget. Each retry
resumes the same recorded sandbox. A refusal does not prove that the
sandbox is lost, so the host never destroys it or creates a replacement.
Only an explicit provider not-found or a workspace identity mismatch
with a verified backup permits replacement (unchanged). The run records
why the provider refused, without provider text.
**Steps to reproduce**
1. Run a warm remote turn that keeps its sandbox.
2. Make the provider's `environmentResumeLease` throw a non-transient
error (for example, a 400/409 while the sandbox stops).
3. Wake the agent with a reply.
4. Before this change: the run fails with `setup_failed`, no retry is
scheduled, and the error has no cause.
**Paperclip version or commit**
Seen on staging at `5b99bdae`. The touched code is unchanged on current
`master` (`f7a4c5678f`).
## What Changed
- Move `ReusableSandboxResumeError` to
`server/src/services/reusable-sandbox-resume-error.ts` (re-exported from
`environment-runtime.ts`). Give it the code
`reusable_sandbox_resume_failed`.
- Add the provider refusal to the error message by closed labels only:
`(cause=<label> status=<http> code=<errno>)`. The labels come from the
existing transport classifier, now exported as
`sandboxProviderFailureLabel`. Provider text, paths, and IDs are never
copied.
- Carry the provider status across the plugin worker RPC. A JSON-RPC
error has only a message and a numeric RPC code. The worker host now
attaches a validated `paperclip/environment-resume-error/v1` envelope
(HTTP status and an allowlisted errno only) to `environmentResumeLease`
failures. The host re-validates it and adds it to the recorded cause.
- In setup-failure settlement, find this error through the orchestrator
wrapper. Record `reusable_sandbox_resume_failed` only when provider
dispatch did not start and the run has no native session or runner
identity (the same guard as `remote_workspace_prepare_timeout`).
- Add the code to `isTransientWorkspacePreparationCode`. The existing
scheduler then gives two retries (30 s, 60 s) with its issue gate, Stop,
pause, and dependency checks.
- Exclude the code from generic continuation recovery and from legacy
reconciliation (no unknown provider actions; bootstrap evidence proves
none started), like the template.
- Add a specific stranded-recovery notice: "Saved sandbox could not be
resumed".
- Document the contract in `doc/execution-semantics.md`.
## Verification
- Red (before the fix, same tests): 6 failures. The `fresh` case
recorded `setup_failed` and did not schedule a retry. The error had no
cause. Legacy reconciliation held the run. Generic recovery granted
another budget.
- Green: `npx vitest run src/services/heartbeat/run-completion.test.ts
src/services/legacy-execution-recovery.test.ts src/services/recovery/
src/__tests__/heartbeat-retry-scheduling.test.ts
src/__tests__/environment-runtime.test.ts
src/services/remote-workspace-transport.test.ts
src/services/heartbeat/retries.test.ts` (in `server/`): 619 tests pass.
Also `src/services/recovery/stranded-notice` and
`src/services/remote-workspace*`: 411 pass.
- Safety regressions:
- `run-completion.test.ts`: only a pre-dispatch legacy failure with no
native identity gets the code and a retry. `started`, `retained`
(native), and an untyped copy of the message stay `setup_failed` with no
retry. The recorded error contains the closed cause and not the provider
path.
- `heartbeat-retry-scheduling.test.ts` (real PostgreSQL): two concurrent
schedulers across a restart create exactly one successor per attempt.
The budget ends after two attempts. Stop, pause, and a new dependency
cancel the scheduled retry. Generic stranded recovery creates no extra
run after exhaustion.
- `packages/plugins/sdk/tests/environment-resume-error.test.ts`: through
the real worker RPC, the `JsonRpcCallError` has no status, and the
envelope gives 409 and the errno. Provider text does not cross. Other
methods get no envelope. Malformed envelopes are rejected.
- `environment-runtime.test.ts`: a throwing resume still preserves the
exact lease. The runtime does not call destroy or acquire. The resume
throws a `JsonRpcCallError` whose message has no status. The thrown
message names `cause=provider_error status=409` from the envelope, and
not the provider text.
- `node scripts/typecheck-low-memory.mjs` passes. `git diff --check`
passes.
- Not verified: a live staging run at this revision. The original tasks
were not retried. This PR is not deployed.
## Risks
- Low. The change affects only runs that fail with this typed error
before provider dispatch. Each retry resumes the same lease and inherits
the source wake context, so a reply is delivered once and no second
executor starts.
- If the provider cannot start the sandbox for longer than about 90
seconds, the two retries end and the task gets the specific recovery
notice. The sandbox and its saved files are kept.
- The error message now has a parenthetical cause before the semicolon.
No code parses this message.
- An issue in review does not get the immediate stranded notice
(unchanged existing behavior). The retry, not the notice, delivers the
reply in that case.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, browser reads of staging diagnostics).
Extended reasoning enabled.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandPaperclip
|
245b4d3b79 |
Preserve cancelled workspaces after controllerless Stop (#18820)
Admit cancelled workspace preservation after an audited Stop acknowledgement when no live controller received an interrupt. Retain the strict dispatched receipt for provider termination and operator handoff. Existing source, lease, coordinator and result ownership checks remain enforced. Validated with complete Stop/preservation suites (134 tests), real-file recovery, workspace typecheck and build, and green hosted CI. Greptile and Paperclip both reviewed the exact head at 5/5 with no unresolved findings. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
3b17b5ce15 |
fix(native): keep a reassigned-away run from closing the new owner's task (#19186)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native agent run reports "done" in an accepted result. A finalizer then decides the task status. > - The board can reassign a task while the run is in flight. The old run is cancelled, but its accepted result still goes to the finalizer. > - The finalizer did not check who owns the task now. The old owner's "done" closed the task of the new owner. > - The new owner never did the work, and the task shows done with no reply from the current owner. > - This pull request makes a done report apply only when the reporting run's agent still owns the task. > - The benefit is that reassignment keeps its meaning: the new owner keeps the open task, and the old result stays as history. ## Linked Issues or Issue Description **What happened?** On a hosted staging instance, agent A started a native run on a task. Agent A finished the work and submitted a "done" result. Then the board reassigned the task to agent B, and Paperclip cancelled agent A's run with `issue_reassigned`. About 45 seconds later, the native finalizer committed agent A's done decision with reason `completion_claim_policy_accepted`. The task became done. Agent B's runs were then cancelled because the task was terminal. Agent B never worked on the task. **Expected behavior** A done report from a run that no longer owns the task must not close the task. The accepted result stays as history. The task status stays as it is for the current owner. **Steps to reproduce** 1. Assign a low-risk native task to agent A and start a run. 2. Submit a done result for the run. 3. Reassign the task to agent B before finalization. 4. Finalize the run. Before this change, the task becomes done. **Paperclip version or commit** Observed on staging at `b9fbbf8d30`. Reproduced on `master` at `06826915d7`. Related: #19130 adds a publication check for requested documents at finalization. It does not check task ownership. ## What Changed - `status-arbiter.ts`: add `ownsNativeTaskCompletion` and `supersedeStaleOwnerCompletion`. A done decision from an agent that is not the current assignee (another agent, a user, or nobody) becomes `preserve` with reason `completion_owner_superseded` and no effects. Other decisions do not change. This is the same ownership rule that the provider-failure path already uses (`ownsProviderFailureDecision`). - `native-run-finalizer.ts`: apply the rule after arbitration and before the deliverable check. Conversation tasks keep their existing path. - `native-status-reassessment.ts`: apply the same rule, so new evidence from a reassigned-away run cannot close the task later. - The status committer does not change. A reassignment during finalization already increments `statusVersion` (migration 0358). The committer then replans, and the retry applies the new rule. - `doc/execution-semantics.md`: describe the rule next to the existing "a late finalizer cannot change a reassigned task" statement. ## Verification New file `server/src/services/native-runtime/native-stale-owner-completion.test.ts`. It uses embedded PostgreSQL, the real coordinator store and the real finalizer. - Control: the owning run's done result closes the task with `completion_claim_policy_accepted`. - Reassigned before finalization: the task stays `in_progress` with agent B. The decision is `completion_owner_superseded`. The accepted result row is unchanged. - Reassigned after arbitration and before the status commit: the commit replans and the task stays `in_progress`. - Reassessment control: later approved evidence closes the owner's task with `decision_superseded_by_new_evidence`. - Reassessment after reassignment: later approved evidence from the old run does not close the task. The decision is `completion_owner_superseded`. This case fails when the reassessment rule is disabled. - Pure policy cases: another agent, a board user and no assignee all preserve status. A non-done decision passes through. Red before the fix (same tests, fix not applied): ```sh cd server && npx vitest run src/services/native-runtime/native-stale-owner-completion.test.ts # Tests 2 failed | 1 passed — both reassignment cases: expected "in_progress", received "done" ``` Green after the fix: ```sh cd server && npx vitest run src/services/native-runtime/native-stale-owner-completion.test.ts # Tests 10 passed cd server && npx vitest run src/services/native-runtime/paperclip-control-plane-port.test.ts \ src/services/native-runtime/native-completion-feedback.test.ts src/services/native-runtime/native-deliverable-feedback.test.ts \ src/services/native-runtime/native-skill-test-status.test.ts src/services/native-runtime/native-routine-status.test.ts \ src/__tests__/native-status-arbiter-corpus.test.ts src/__tests__/native-status-reassessment-work.test.ts \ src/services/native-runtime/agent-chat-response-recovery.test.ts src/__tests__/native-run-projection-work.test.ts \ src/services/native-runtime/task-reassignment.test.ts src/services/native-runtime/native-provider-failure.integration.test.ts # 11 files, 614 tests passed ``` I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm build` locally. CI runs them. ## Risks - A native run that reassigns its own task to a user or to another agent, and then reports done, no longer closes the task through arbitration. The status that the agent set through the API stays. This matches the existing provider-failure ownership rule. - A native run on a task with no assignee cannot close it through arbitration. Native task runs start from an assignment, so this is not expected in normal use. - Review runs and conversation tasks keep their existing paths. Review outcomes already preserve status before this rule. - No schema change and no migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), in the Claude Code agent harness. - Capabilities used: tool use, shell and code execution, repository search, browser reads of staging, and extended reasoning. - The model wrote the change, the tests and this description. A human requested the fix. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
46fb933b01 |
fix(recovery): keep the stop receipt when a file-preserving restart fails (#19181)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs can execute in a remote sandbox. When a run fails, Paperclip stops the sandbox and records the provider's stop receipt on the environment lease. > - System workspace recovery can restart that stopped sandbox to save its files. Before the restart, it removed the stop receipt from the lease. > - If the restart failed and the provider could not stop the sandbox after that, the lease had no receipt. Continuation admission requires a stop receipt on every remote lease of the failed run. So every Retry and every comment waited for an environment that never stopped. > - This pull request keeps the original receipt until the provider confirms the restart. It restores the receipt when the restart fails and the sandbox cannot stop. A restart that fails because the sandbox disk is full now ends recovery with a fresh-workspace outcome. > - The benefit is that a failed file-preserving restart no longer blocks the task permanently. Saved input and Retry continue in a fresh isolated workspace, and the original sandbox stays retained. ## Linked Issues or Issue Description No public issue exists. This is a bug report in the format of `bug_report.yml`. **What happened?** 1. A native run failed because its Daytona sandbox filled its 10 GB disk. 2. Paperclip stopped the sandbox normally and recorded the stop receipt on the environment lease. 3. Some minutes later, system workspace recovery tried to restart the sandbox to preserve its files. Before the restart, `prepareStoppedSandboxForRecovery` deleted the stop receipt. 4. The restart failed on the full disk. The sandbox went into the provider's `error` state. 5. The stop-only teardown after the failed restart also failed, because the provider cannot stop a sandbox in that state. The lease stayed in `pending_cleanup` with no receipt. The cleanup sweep retried the same failing stop with backoff. 6. `admitExplicitNativeContinuation` requires a provider stop receipt on every remote lease of the failed run. Every Retry and every comment was deferred with "Waiting for the previous environment to stop. Your message will start automatically." The message never started. Also, after recovery was exhausted, a Retry replaced the exhausted generation with a new one. That generation repeated the same restart, and the Retry still reported success. **Expected behavior** A lease must not lose its stop receipt while the real sandbox state is unknown or not running. When the file-preserving restart cannot succeed, recovery must end in a state that lets the task continue, for example in a fresh isolated workspace. **Steps to reproduce** 1. Use a native run with a retained remote sandbox lease that has a `stopped` receipt and a pending workspace recovery. 2. Make the provider resume fail (for example with `no space left on device`), and make the provider stop fail. 3. Run `recoverRetainedWorkspace` until recovery is exhausted. 4. Save a user comment on the task and run the deferred-comment admission. 5. Before this change, the lease is `pending_cleanup` without a receipt. The comment stays deferred with reason `remote_cleanup`, and no successor run starts. **Paperclip version or commit** Observed on a hosted staging build. Reproduced on `master` at `6452e964a36` with the new tests in this PR. **Deployment mode** Hosted, with the Daytona sandbox provider plugin. The defect is in core server recovery code, so it is not specific to one adapter. ## What Changed - `server/src/services/sandbox-stop-and-retain.ts` - `prepareStoppedSandboxForRecovery` still writes the stop-only intent and moves the lease to `pending_cleanup` before the provider call. It now keeps the prior `remoteExecutionTermination` (and `sandboxStopAndRetainReceipt`, if present) as `priorStop` inside the `sandboxStopAndRetain` intent. - New `confirmSandboxRecoveryRestart` removes `priorStop` when the provider resume returns. From that point, the sandbox may be running, and only a fresh stop can settle the lease. - New `restoreSandboxStopAfterFailedRestart` restores the prior receipts and the `released` / `success` lease state. It works only while `priorStop` is present, so only when no restart was confirmed. It checks the restored receipt with `hasRemoteTerminationReceipt` first. It uses the same attempt fence as `settleStopOnlyCleanup`, and it writes through `writeEnvironmentCleanup` so dispatch and cleanup wake up. - `settleStopOnlyCleanup` now shares that attempt fence (`stopOnlyAttempt`). Its behavior does not change. - `server/src/services/workspace-recovery.ts` - After `resumeWorkspaceRecoveryLease` returns, recovery confirms the restart before any other check. - If the restart is not confirmed, recovery still tries the stop-only teardown first. A successful stop records a fresh receipt, as before. If the stop also fails, recovery restores the prior receipt instead of leaving the lease without one. - A later attempt that finds a pending stop-only lease (for example after a process restart during the reopen) uses the same rule when its stop fails. - A resume error that reports a full sandbox disk (`No space left on device` or `Disk quota exceeded`) becomes `source_unavailable` / `use_alternative_strategy`. The generation is exhausted at once, and the existing fresh-workspace continuation applies. Other resume errors keep the existing bounded retry policy (`provider_unavailable`, at most six attempts in 30 minutes). Because the receipt stays, that policy also ends in a state that lets the task continue. - `server/src/services/task-retry.ts`: Retry binds to an exhausted generation with a `use_alternative_strategy` failure and no fresh attempt, the same way it binds to a complete or running generation. It does not create a new generation that repeats a step the policy already rejected. The Retry continues in a fresh isolated workspace. An explicit workspace recovery request (`requestWorkspaceRecovery`) still starts a new generation. - `doc/task-recovery.md`: describes the receipt handling around the reopen and the Retry binding. - `server/src/services/workspace-recovery.test.ts`: four new tests (five cases) with the real database and mocked provider calls. ### Design choice I kept the existing write-ahead fence. The lease still leaves `released` and shows no receipt while the provider reopen is in progress. So no admission, reuse or destructive cleanup can act in that window, and a crash still leaves a stop-only obligation. The original receipt is not deleted. It stays in the stop-only intent and is discarded only when the provider confirms the restart. This makes one simple rule: the suspended receipt exists only while no restart has been confirmed. I did not use "restore only when the provider confirms that the sandbox is not running". The Daytona plugin reports a sandbox in the `error` state as `active` in `inspectLease`, and its `stopLease` cannot stop that state. So with the current plugin, no provider confirmation exists for this exact failure, and the hold would stay. A separate change to the Daytona plugin's `stopLease` can recover an errored sandbox before it stops it. This PR does not depend on that change. If that change lands and its recovery succeeds, the existing stop attempt records a fresh receipt, and the restore path is not used. I also did not change continuation admission to accept a lease in `pending_cleanup`. Many gates treat pending cleanup as unfinished work (`hasPendingRunCleanup`, the remote stop check and others). A change there would be wider and more sensitive. The restored receipt is the provider's original receipt, bound to the same company, run, lease, provider, allocation and plugin. No process ID or run status takes its place. ## Verification All commands ran from `server/` in a clean worktree of `master` on macOS (Apple Silicon), with the embedded PostgreSQL test database. Vitest ran from the real path. - Baseline on `master` (`6452e964a36`), before any change: `npx vitest run src/services/workspace-recovery.test.ts` → exit 0, 331 passed. - The new tests fail on `master`. I reverted only the three source files and kept the new tests: - `npx vitest run src/services/workspace-recovery.test.ts -t "file-preserving restart fails|without repeating a restart|keeps the stop obligation once|interrupted restart that the provider"` → exit 1, 4 failed, 1 passed. - In the comment cases, no successor run starts (`expected [] to have a length of 1`). A temporary log (not committed) showed the lease in `pending_cleanup` in both cases. In the `unknown_failure` case, the comment was deferred with `remote_cleanup` / "Waiting for the previous environment to stop. Your message will start automatically." This is the same hold as in the incident. - The guard test "keeps the stop obligation once the provider has confirmed a restart" passes on `master`, as expected. It makes sure that the fix does not restore a receipt after a confirmed restart. - With this change, the same command → exit 0, 5 passed. - `npx vitest run src/services/workspace-recovery.test.ts` on the committed head → exit 0, 336 passed (331 existing and 5 new). - Neighboring suites (`task-retry`, `explicit-native-continuation`, `heartbeat/recovery`, `environment-runtime`, and the other server test files that import the changed modules: `live-recovery-work`, `retained-native-session-cleanup-work`, `execution-control-work`, `legacy-workspace-restore-recovery`, `deferred-execution-receipt-work`, `issue-agent-mutation-ownership-routes`, `native-workspace-export-retry`, `native-cancelled-workspace-preservation`, `native-workspace-warm-reuse`, `native-workspace-reconstruction`, `native-failed-workspace-recovery`, `heartbeat/workspace-preparation`) → exit 0, 16 files, 1456 passed. - `npx vitest run src/__tests__/chat-channels.integration.test.ts src/__tests__/slack-conversation-lifecycle.test.ts src/services/native-runtime/preserved-native-response-wait.integration.test.ts` (other importers of `task-retry`) → exit 0, 3 files, 2074 passed. - `pnpm --filter @paperclipai/server typecheck` → exit 0. - Not run: the full `pnpm -r typecheck`, `pnpm test:run` and `pnpm build` repo checks. CI runs them. No live Daytona probe ran. New tests: - `lets saved input continue after a file-preserving restart fails on a sandbox that cannot stop` (`disk_full`, `unknown_failure`): resume fails and stop fails. The lease keeps its original receipt and is `released` / `success`. Recovery is exhausted (after one attempt for a full disk, and after six attempts for an unknown failure). The saved comment starts one successor run with `fresh_isolated_workspace` readiness. The source files stay in the retained sandbox. - `continues a Retry in a fresh workspace without repeating a restart that cannot succeed`: after a full-disk exhaustion, Retry keeps the same generation and binds its intent. The restart is not called again, and the Retry starts one run in a fresh isolated workspace. - `keeps the stop obligation once the provider has confirmed a restart`: resume returns, a later check fails, and the stop fails. The lease stays `pending_cleanup` without a receipt, and the intent has no `priorStop`. - `restores the stop receipt after an interrupted restart that the provider never confirmed`: the lease starts in `pending_cleanup` with a suspended receipt, as after a process restart during the reopen. The stop fails, the receipt is restored, and the attempt continues to the normal reopen. Assumption and limits (runtime dependency rule): the provider calls are mocked, and the database is real. The tests do not exercise the real Daytona error text. The full-disk check matches the patterns that Paperclip already uses for sandbox disk-full errors. If the provider error text does not match, recovery uses the bounded retry path, which now also ends in a fresh-workspace continuation. The change adds one database write after each confirmed reopen and one after a failed reopen and stop. It adds no polling. ## Risks - A resume can fail after the provider started the sandbox, for example when the reply is lost. If the stop-only teardown then also fails, recovery restores a `stopped` receipt for a sandbox that can be running. The old run's processes do not survive the earlier stop, so the receipt is still correct as proof that the old execution ended. But the retained sandbox can run until the provider stops it or a later recovery or cleanup stops it. Before this change, the same case caused a permanent hold. - A full-disk restart failure now exhausts recovery at once. The task continues in a fresh workspace without the failed run's files. The files stay in the retained sandbox. An operator can still request workspace recovery explicitly after the disk is freed. - Retry no longer renews an exhausted generation that has a `use_alternative_strategy` failure and no fresh attempt. For destroyed or missing sources, the result is the same as before but without the extra attempt. - No schema or migration change. A suspended receipt is one extra field in the existing `sandboxStopAndRetain` metadata. Older servers ignore the field. The intent equality checks compare the stored value, so a mixed-version deploy keeps working. - #18820 also changes `server/src/services/workspace-recovery.test.ts`. The two PRs touch different tests. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (model ID `claude-opus-5-5`), running as a Claude Code agent with tool use (shell, file reads and edits, local test and typecheck runs) and extended reasoning. The runtime does not expose the context window size to the agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (affected suites; full repo runs are left to CI, as noted above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
895cee28b3 |
fix(daytona): recover an errored sandbox before stop-only cleanup (#19178)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Remote agent runs use sandbox provider plugins. Daytona is one of
them.
> - After a failed run, Paperclip stops the sandbox and keeps its files
("stop-only cleanup"). A task can continue only when every remote lease
of the failed run has a provider stop receipt.
> - Daytona rejects Stop for a sandbox in its `error` state. A sandbox
that filled its disk went to that state. The plugin called Stop on it,
so every cleanup sweep failed with the same generic error.
> - The lease stayed in pending cleanup without a stop receipt. Each
Retry or comment waited forever with reason `remote_cleanup`. A manual
Daytona Recover was the only fix.
> - This pull request makes the Daytona stop-only hook recover a
recoverable sandbox once, with a bound, and then stop it. When it cannot
stop the sandbox, it returns a typed failure. The server escalates a
terminal failure at once instead of retrying it like a transient error.
> - The benefit is that a full-disk sandbox no longer blocks its task,
and a sandbox that cannot recover is visible to operators on the first
attempt.
## Linked Issues or Issue Description
No public issue exists. Bug description:
**What happened?**
A task sandbox filled its 10 GB disk. Daytona then reported the sandbox
as `state: error`, `desiredState: started` and `recoverable: true`. Its
`errorReason` was a Docker `no space left on device` error. The Daytona
`stopLease` hook calls Stop on any sandbox that is not `stopped` or
`archived`. Daytona rejects Stop in the `error` state. The pending
cleanup sweep ran about every 30 minutes, and it failed on every pass.
The lease stayed `pending_cleanup` with `cleanup_status=failed` and no
stop receipt. Continuation admission
(`server/src/services/explicit-native-continuation.ts`) requires a stop
receipt on every remote lease of the failed run. So each Retry and
comment was deferred with "Waiting for the previous environment to stop.
Your message will start automatically." It never started.
A manual Daytona `POST /api/sandbox/{id}/recover` fixed it. The
sandboxes reached `started` in about one minute. The next sweep stopped
them, recorded the receipt, and the tasks continued.
**Expected behavior**
Stop-only cleanup recovers a recoverable errored sandbox and then stops
it. When the sandbox cannot be recovered, cleanup returns a clear, typed
failure that the server can tell apart from a transient error. Cleanup
never deletes the retained sandbox, and it never loops on Recover.
**Steps to reproduce**
1. Run a task in a Daytona sandbox until the sandbox fills its disk and
Daytona shows `state: error`, `recoverable: true`.
2. Let the run fail, so the server records a `sandboxStopAndRetain`
intent and calls `environmentStopLease`.
3. Before this change, Stop fails with `Sandbox failed to stop with
status: error` on every sweep, and the task cannot continue.
4. The new tests in `plugin.test.ts` reproduce this with a mock sandbox
whose Stop fails from `error`.
**Paperclip version or commit**
Observed on a hosted deployment. Reproduced in unit tests on `master` at
`6452e964a36`.
**Deployment mode**
Managed Daytona sandbox execution with plugin-backed stop-only cleanup.
Related: #18172 (Daytona exports under disk pressure) and #19050
(session reset on every Daytona restart path). This PR does not change
the resume path in `ensureSandboxStarted`, which already recovers a
recoverable sandbox.
## What Changed
- **Daytona plugin
(`packages/plugins/sandbox-providers/daytona/src/plugin.ts`)**: after
`stopLease` confirms the retention policy, a sandbox in `error` goes
through `recoverSandboxForStop`.
- Recoverable: the hook calls `sandbox.recover()` once. The SDK waits
for `started`. The plugin bounds the wait to 20 s, plus the existing
liveness margin for a hung connection. Then the hook stops the sandbox
normally and returns the usual `stopped` receipt.
- Not recoverable: the hook makes no Recover or Stop call. It throws
`PluginEnvironmentStopError("resource_unrecoverable")`.
- Recover fails: the hook reads the provider state once. A sandbox back
in `error` gives `recovery_failed`, or `resource_unrecoverable` if
Daytona now marks it so. A sandbox that is still starting, or a failed
read, gives `recovery_incomplete`.
- The recovered sandbox fails again before Stop finishes (for example,
the disk fills again): a fresh read that shows `error` gives
`recovery_failed`. The hook never calls Recover a second time in the
same call.
- The hook never deletes or archives the sandbox. The non-error stop
path is unchanged.
- **Plugin SDK (`packages/plugins/sdk/src/environment-stop-error.ts`,
new)**: `PluginEnvironmentStopError`, `environmentStopErrorData` and
`readEnvironmentStopFailure`. This follows the existing
`PluginEnvironmentRunnerIngressError` pattern. The envelope holds only
one closed reason. The reader validates the reason and derives
`terminal` from it. `resource_unrecoverable` and `recovery_failed` are
terminal. `recovery_incomplete` is not.
- **Worker RPC host (`packages/plugins/sdk/src/worker-rpc-host.ts`)**:
forwards that envelope as JSON-RPC error `data` for
`environmentStopLease` only.
- **Pending cleanup sweep
(`server/src/services/heartbeat/recovery.ts`)**: the catch reads the
validated envelope.
- On a terminal failure, it escalates on the first attempt. It writes
the one-time "needs operator attention" warning and uses the existing 30
minute retry interval. Before, this happened only at the fifth attempt.
- The failure log uses `errorKind: "provider_stop_terminal"` and
includes the closed `providerStopFailure` reason. The sweep still never
logs exception fields.
- The lease stays `pending_cleanup`. Automatic cleanup continues, and an
explicit Retry still bypasses the cooldown. Untyped and transient
failures keep their current behavior.
- **Docs**: the SDK README ("Stop failures"), the Daytona README,
`doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md` and
`doc/native-workspace-finalization-recovery.md` describe the new
behavior.
Shared contract note (module boundaries): the SDK envelope is the
smallest addition that lets the host tell a terminal provider stop
failure from a transient one. The worker host drops all other error data
across the RPC boundary, so the existing contracts cannot carry the
signal. The only consumer is the pending cleanup sweep. When a provider
does not throw the new error, or an older host ignores it, behavior is
unchanged. This PR does not touch
`server/src/services/workspace-recovery.ts` or
`sandbox-stop-and-retain.ts`.
## Verification
All commands ran in a clean worktree from `master`. Vitest ran from the
real path.
- Red on `master` (only the new tests added):
- Daytona, with `plugin.ts` from `master`: `npx vitest run --config
vitest.config.ts src/plugin.test.ts -t "provider error state"` exits 1.
All 7 new tests fail. The main case fails with `Sandbox failed to stop
with status: error`, which is the production error.
- SDK, with `worker-rpc-host.ts` from `master`: `npx vitest run
tests/worker-rpc-host.test.ts -t "typed stop failure"` exits 1 (the
typed case fails).
- Server, with `recovery.ts` from `master`: `npx vitest run
src/__tests__/heartbeat-pending-cleanup-sweep.test.ts -t "terminal
provider stop failure"` exits 1. All 3 cases fail (`errorKind:
"destroy_failed"`, no escalation).
- Green on this branch:
- `packages/plugins/sandbox-providers/daytona`: `npx vitest run --config
vitest.config.ts` exits 0 (534 passed, 26 skipped live tests). `pnpm
typecheck` exits 0. `pnpm build` exits 0.
- `packages/plugins/sdk`: `npx vitest run` exits 0 (222 passed). `pnpm
--filter @paperclipai/plugin-sdk typecheck` exits 0.
- `server`: 15 related suites exit 0 (1498 passed): pending cleanup
sweep, `heartbeat/recovery`, environment runtime, environment cleanup
work, native startup cleanup, native remote failure retention, observed
workspace preservation, retained native session cleanup, idle sleep
safety, explicit native continuation, task retry, workspace recovery,
native failed and cancelled workspace preservation, and Daytona close in
the plugin worker manager.
- `pnpm --filter @paperclipai/server typecheck` exits 0. `pnpm
check:module-boundaries` exits 0.
- New tests:
- Daytona: a recoverable errored sandbox is recovered once
(`recover(20)`) after the retention check and before Stop, then stopped.
An unrecoverable one returns `resource_unrecoverable` with no Recover,
Stop, start, delete or archive call. A failed recovery makes exactly one
Recover call for each outcome (`recovery_failed`,
`resource_unrecoverable`, `recovery_incomplete`). A Recover call that
never settles is cut off by the bound. A recovered sandbox that fails
again before Stop gives `recovery_failed` with no second Recover.
- SDK: the envelope holds only the reason. A forged or non-SDK error
gives no data. Unknown reasons, accessor fields and a sender-supplied
`terminal` flag are not trusted. The worker host forwards the typed
failure and drops a plain error's detail.
- Server: terminal reasons escalate on attempt 1 with the long retry and
one operator warning. `recovery_incomplete` keeps the short retry. The
lease stays `pending_cleanup`, and the provider message never reaches
the logs.
- Not run: the full `pnpm -r typecheck`, `pnpm test:run`, `pnpm build`
and `pnpm run typecheck:build-gaps` repo checks. CI runs them. No live
Daytona probe ran. The mocks model the Daytona SDK `recover()` contract:
it waits for `started` and rejects when the sandbox returns to `error`.
## Risks
- Time budget: the host gives `environmentStopLease` at most 60 s. A
recovered stop uses up to 20 s for Recover and up to 30 s for Stop, plus
short reads. A slower recovery returns `recovery_incomplete`. The SDK
call can continue at Daytona after that, so the next sweep finds the
sandbox started and stops it normally. If the host deadline fires while
the worker still stops the sandbox, the next sweep sees `stopped` and
records the receipt.
- Classification: after a failed Recover, the plugin cannot tell a
rejected request from a sandbox that started and failed again. Both give
`recovery_failed` (terminal) when Daytona still shows `error`. A hung
Recover call with a sandbox still in `error` is also terminal. Terminal
failures still retry every 30 minutes, and each retry makes at most one
Recover call. An explicit Retry also makes one attempt. There is no
tight loop. A sandbox that fails Recover every time gets about two
Recover calls an hour until an operator acts.
- Recover restarts the sandbox container before Stop. The plugin runs no
command in it, and the retention policy is confirmed before Recover.
- The continuation message for a terminally failed lease still says the
message "will start automatically". This PR does not change admission. A
follow-up can use the typed reason there.
- Compatibility: a host without this change ignores the new error data
and keeps the old retry behavior. A plugin without this change sends no
typed data.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, code execution for tests and typechecks).
The runtime does not expose the context window size to the agent.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (affected suites; full repo
runs are left to CI, as noted above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
|
||
|
DottaandClaude Opus 5.5
|
f1516e8009 |
fix(native): stop a session.open model rejection instead of a doomed bootstrap retry (#19190)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Runner runs start a provider session with the `session.open` command before any prompt > - A definite `session.open` rejection is treated as a pre-session failure and gets a bounded same-run bootstrap retry > - The runner also reports a closed classification. Two classifications say that the provider cannot run the selected model: `requested_model_unsupported` and `grok_model_mismatch` > - Paperclip ignored that classification. It scheduled a retry that must fail the same way. On a remote sandbox with a retained runner PID, the due retry entered restart recovery. Restart recovery has no checkpoint for a bootstrap retry, so it failed closed as `native_restart_recovery_blocked` / `ambiguous_remote_recovery_evidence` > - The task then stayed `blocked`. The recovery actions settled as no-replay holds, so no active action told anyone to change the model, and the parent task stalled too > - This pull request classifies those two definite rejections as the existing `native_provider_model_rejected` failure > - The benefit is a terminal failure on the first attempt, an accurate error, and an active board recovery action that names the next step: choose a supported model ## Linked Issues or Issue Description **What happened?** On a hosted staging instance, a Grok agent run failed at `session.open`. The runner log said `ACPX sidecar command session.open was rejected (retryable=false, classification=grok_model_mismatch)`. Paperclip recorded `recoveryMode: "bootstrap_retry"` and scheduled a same-run retry 30 seconds later. The run had a remote sandbox and a recorded sandbox PID, so the due retry went to restart recovery. Restart recovery found no checkpoint and no provider events and failed closed with `ambiguous_remote_recovery_evidence`. The run ended as `native_restart_recovery_blocked`. The task stayed `blocked` with no active recovery action for more than one hour. Its parent task stayed blocked on it. **What did you expect to happen?** The selected model cannot start, so a retry cannot succeed. The run should fail once with the model-rejection error. The task should show the existing "choose a model" action, owned by the board, until someone changes the model and retries. **Steps to reproduce** 1. Start a native run whose `session.open` command fails with `classification=grok_model_mismatch` (or `requested_model_unsupported`). 2. Read the `native_run_finalizations` row. Before this change it is `retryable_failure` with `recoveryMode: "bootstrap_retry"` and a `nextAttemptAt`. 3. On a remote sandbox run with a recorded PID, the due retry enters restart recovery and fails as `native_restart_recovery_blocked`. **Paperclip version or commit** `master` at `f5f5184f23`. Related: #18910 routes due remote retries with a retained PID to restart recovery. #18908 continues verified failed native preparation. Open PRs #19177, #19180, #19182 and #19184 change no-replay hold settlement and workspace hold release. They do not change this classification. This PR does not duplicate them. ## What Changed - `nativeSessionFailureSourceCode` (`server/src/services/native-runtime/native-session-executor.ts`) returns `native_provider_model_rejected` for a definite `session.open` failure or rejection whose runner classification is `requested_model_unsupported` or `grok_model_mismatch`. Other definite `session.open` rejections still return `provider_session_open_rejected` and keep the bounded startup retry. An indeterminate `session.open` outcome stays `native_session_interrupted`. - `native_provider_model_rejected` is already a permanent failure. The run stops on the first attempt with no `nextAttemptAt`, so it never enters the restart-recovery retry path. - When no provider session exists, the coordinator `nextAction` and the board recovery action `nextAction` say: choose a supported model for this agent, then retry the task. - The existing product paths for `native_provider_model_rejected` apply: the task presentation shows `choose_model`, and the recovery service treats it as `configuration_incomplete`. - `doc/execution-semantics.md` describes the rule under "Native provider model capacity". ## Verification Red before the fix (new test, unchanged production code): ```sh cd server && npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts # Tests 2 failed | 21 passed # failureCode: native_session_interrupted, phase: retryable_failure, recoveryMode: bootstrap_retry, # nextAction "Retry provider bootstrap on this same run ..." - the same record seen on staging ``` Green after the fix: ```sh cd server && npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts # Tests 23 passed cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts \ src/services/native-runtime/native-startup-retry.integration.test.ts src/services/task-interruption-presentation.test.ts # Test Files 3 passed, Tests 1186 passed pnpm --filter @paperclipai/server typecheck # exit 0 ``` The new integration cases use embedded PostgreSQL and the real executor failure path. For both classifications they check: - coordinator `terminal_failure`, `failureCode: native_provider_model_rejected`, `nextAttemptAt: null`, attempt 1 - run `failed` with `errorCode: native_provider_model_rejected` - task `blocked`, execution lock released - an `active` board recovery action with cause `native_provider_model_rejected`, return owner set to the agent, and the choose-model next action - `settleUnrecoverableExecutions` leaves the action `active`, so it does not become a silent no-replay hold - `claimNativeSessionResumptions` claims nothing, so there is no second executor The unit test also checks that `session_ensure_failed` keeps its bounded retry and that an indeterminate model rejection stays unclassified. A live Grok run with a mismatched model was not run. The failure record in the test matches the staging record field by field. ## Risks - Low risk. The change adds two closed runner classifications to an existing permanent failure code. Other `session.open` rejections keep their current retry. - A model mismatch that a retry could fix would now need an explicit retry. The runner already marks these rejections `retryable=false`. - This PR does not change restart recovery. A remote bootstrap retry with a retained sandbox PID for other `session.open` causes can still fail closed as `ambiguous_remote_recovery_evidence`. That needs a separate change with remote stop evidence. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code with tool use (shell, file edits, browser reads of the staging API). It did the investigation, code change, tests and PR text. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
46a50a3bdd |
fix(recovery): release a recovered workspace hold for the new assignee after reassignment (#19184)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a board user reassigns a task in the middle of a run, Paperclip
stops the old run and saves its sandbox files (copy-back).
> - If copy-back fails at the stop, Paperclip records a
file-preservation hold for the stopped run. The hold blocks all
execution on the task until the files are recovered.
> - The new assignee's wakes were already queued. The final dispatch
gate saw the hold and cancelled them.
> - Automatic recovery then saved the files. But the hold stayed
effective for the new assignee: explicit continuation accepts only the
stopped run's own agent, so no message, Retry or stranded-work sweep
could start the new assignee.
> - The task stayed `in_progress` with no run, no wake and no active
recovery.
> - This pull request releases a recovered file-preservation hold for a
different current owner.
> - The benefit is that a reassigned task continues with its new owner
after its files are safe, with no manual action.
## Linked Issues or Issue Description
**What happened?**
A board user reassigned a running native task from agent A to agent B
and posted a comment. A's run ended `cancelled` / `issue_reassigned`.
Copy-back was still in progress, so B's first run was cancelled
`workspace_busy`. Paperclip then recorded a resolved no-replay hold
(`native_continuation_requires_reconciliation`, fingerprint
`native-workspace:<A run>`). The final dispatch gate cancelled B's two
queued runs with `execution_reconciliation_required` ("Recovering saved
work…"). Automatic recovery then saved the files
(`workspaceRecovery.status = recovered`, `workspaceRepairRequired:
false`). The blocker still said "Saved work recovered. Send a message to
continue." with `canContinue: false`. The task stayed `in_progress` with
no run for hours. Two hosted reproductions had the same shape. A sibling
reassignment without a copy-back failure recovered normally.
**Expected behavior**
After the saved-file receipt verifies, the new assignee continues. The
hold must still block every executor while the files are not recovered.
The stopped run's own agent keeps its no-replay hold.
**Steps to reproduce**
1. Start a native run on a task with a retained sandbox.
2. Reassign the task to another agent while the run executes, so that
copy-back records a `workspaceRestoreRecovery` hold.
3. Let automatic workspace recovery finish.
4. Before this change, `getExecutionBlocker` still returns the hold for
the new assignee, and the stranded-work sweep queues nothing.
**Paperclip version or commit**
`b9fbbf8d30` (also seen on `818d2bb6`).
**Deployment mode**
Hosted, PostgreSQL. The test uses embedded PostgreSQL.
Related open pull requests (checked; none covers this path): #19177
(board reconciliation of sibling holds), #19169 (disposition repairs
behind a cancelled blocker), #19143 (park a workspace-busy wait), #19149
(interrupt stop identity).
## What Changed
- New `restoreHoldReleasedForCurrentOwnerPredicate`
(`server/src/services/execution-blocker.ts`) is a SQL predicate. It
matches a file-preservation hold (`workspaceRestoreRecovery`, not
`restore_unsafe_archive`) when all of these are true:
- the stopped run is terminal and finished;
- the task has an agent assignee, and that agent is not the stopped
run's agent;
- the stopped run's agent is not the current review participant;
- `completedWorkspaceRecoveryCondition` is true for the hold: completed
system recovery, `workspaceRecovery.status = recovered`, succeeded
preservation operations bound to this hold, and the `workspace_finalize`
receipt;
- every saved operation is a copyback into the task workspace
(`workspace_recovery`). A snapshot-only save keeps the hold, because the
new owner's ordinary turn does not attach snapshots.
- The predicate is wrapped in `coalesce(..., false)`. Its negation
therefore cannot hide an unrelated hold through NULL JSON reads.
- The proof is the durable completed-recovery receipt. It does not read
live workspace authority (epoch, writer, snapshot), so reuse of the
workspace by the new owner cannot re-arm the hold.
- `getExecutionBlocker` excludes matching holds from both the
file-preservation query and the general blocker query. There is no scan
limit and no extra query per hold.
- `admitExplicitNativeContinuation` excludes the same holds from its
source checks. A released hold therefore cannot reject a new owner's
message with "owner changed".
- `completedWorkspaceRecoveryCondition` also accepts a SQL expression
for the run id. Its behavior is unchanged.
- The hold row, its `automaticRecovery.replay = "blocked"` evidence and
its status are not changed. If the task returns to the stopped run's
agent, the hold applies again.
- `doc/execution-semantics.md` describes the rule next to the existing
"Reassignment does not discard the stopped run's files" sentence.
## Verification
New test in `server/src/services/workspace-recovery.test.ts`: `hands a
reassigned task to its new assignee once the previous owner's files are
recovered`. It uses the real retained-workspace recovery, embedded
PostgreSQL and the real stranded-work sweep. It builds the incident
shape: a cancelled `issue_reassigned` source run, a resolved no-replay
hold, a new assignee, and that assignee's gate-cancelled run.
- Before recovery, the blocker holds the new owner
(`workspaceRepairRequired: true`).
- After recovery, the blocker is `null`. The sweep creates exactly one
pending turn for the new owner. That turn is not a retry of the stopped
run. A second sweep adds no turn.
- The hold row keeps `status: resolved`, `outcome: blocked` and `replay:
blocked`.
- Live repair evidence on the stopped run is then changed, so
`hasVerifiedWorkspaceRepair` returns false (as after a remote reuse).
The blocker stays `null`.
- After a reassignment back to the stopped run's agent, the hold applies
again.
- An unrelated reconciliation hold for the same stopped run, without
file evidence, still blocks the new owner.
- New test `keeps a snapshot-only recovery held for a new assignee`: the
snapshot-only save does not release the hold. This test fails against
the first revision of this pull request.
Red before the fix (test kept, `execution-blocker.ts` restored from
`master`):
```
cd server && npx vitest run src/services/workspace-recovery.test.ts -t "hands a reassigned task"
AssertionError: expected { …(13) } to be null (blocker: "Saved work recovered. Send a message to continue.", canContinue: false)
```
With only the blocker assertion removed, the sweep also failed:
`expected [] to have a length of 1 but got +0`.
Green after the fix: 1 passed.
Related suites, all pass (991 tests): `workspace-recovery`,
`execution-blocker-read`, `legacy-explicit-continuation-recovery`,
`workspace-restore-reconciliation`, `native-failed-workspace-recovery`,
`workspace-restore-repair-route`, `execution-projection-recovery`,
`heartbeat-undispatched-execution-wait`,
`issue-update-comment-wakeup-routes`,
`heartbeat-observed-workspace-preservation`,
`native-workspace-reconstruction`, `explicit-native-continuation`.
Not run: a live run on a deployed instance. The hosted incident tasks
are not recovered by this pull request until it is deployed.
## Risks
- A new owner can start after the files are recovered, while the stopped
run's action outcomes stay unknown. This matches the existing rule for
holds without a workspace: Paperclip does not create those holds for a
reassigned task. The new owner gets a fresh turn, not a replay.
- The hold still blocks every executor until the exact receipt verifies,
so there is no second executor on the workspace during copy-back.
- Other execution holds (non-workspace holds recorded before a
reassignment) keep their current effect. This change does not alter
them.
- A reassignment after a snapshot-only save still needs a board action,
as before. That path is not changed here.
- The release is one correlated `exists` in the existing blocker
queries. Its subqueries use primary keys and the run and issue indexes.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, with tool use
and code execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandPaperclip
|
f7a4c5678f |
feat(runtime): pre-copy completed harness history before enrollment (#19137)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - An ephemeral web server needs provider history in shared object
storage.
> - Offline enrollment must preserve old backups before it can detach a
volume.
> - A large copied dataset still exceeded the maintenance deadline with
parallel transfers.
> - Most of its bytes belong to successful turns with released runtime
authority.
> - This pull request pre-copies that history while unrelated executions
continue.
> - The final offline migration still verifies every source and grants
enrollment authority.
## Linked Issues or Issue Description
Refs #19108, #19038, #18943.
**What happened?**
A copied instance has 1,953 local harness backups and 54.8 GB of
declared source bytes. A resumed parallel migration still reached its
25-minute deadline. It cannot safely enroll by moving all history during
maintenance.
**Expected behavior**
Copy eligible completed history while the server remains online. Defer
active, failed or changed sources. Keep all existing data and final
enrollment checks.
**Steps to reproduce**
1. Retain many local native harness backups on a volume-backed instance.
2. Attempt a full offline conversion within the maintenance deadline.
3. Observe that most time moves historical bytes rather than current
execution state.
## What Changed
- Add finite `native-harness-prestage.mjs` preparation from a normal
runtime build. Require explicit opt-in and shared object storage.
Process one source at a time with a 90-minute deadline.
- Use the recorded released runtime owner of each successful, finished
turn. Do not assert that all writers stopped.
- Recheck checkout, related queued/running work, successful result
ownership, session, Runner, exact lease stamp and unchanged backup at
database publication fences.
- Defer source changes, malformed source JSON and unsafe backup roots.
Log each such source and its reason. Keep database failures and
cancellation fatal. Keep completed objects valid for retry. Never return
an enrollment-ready receipt.
- Reuse the bounded migration source walker. Keep offline admission and
final verification unchanged.
- Keep private lifecycle state, task ownership and provider policies
unchanged. Document the operator path and its limits.
## Verification
- A zero-source scan now checks the existing database obligation query.
A known mismatch blocks with a source-missing receipt; empty
fresh/already-durable scans and nonempty deferred scans remain valid
advisory results. The prior command fails the mismatch regression. All
16 command and runtime-bundle tests pass for this follow-up.
- At head `7ba3dbc7aa`, 87 focused command, database-timeout, storage,
archive, publication and runtime-bundle tests passed with real
PostgreSQL and filesystem restoration.
- The current follow-up passes all 68 backup and archive tests. A source
renamed after preparation fails the new regression on the previous head,
then defers and permits the next backup with this fix. An archive output
ENOENT remains fatal. Existing database-error and cancellation tests
pass.
- New race tests fail against the previous storage owner: a changed
backup, queued turn or revoked lease otherwise publishes.
- Tests prove unrelated executions stay running, same-task work defers,
failed history stays deferred, retry verifies existing objects, and
private bytes restore after deleting the old home.
- Command review fixes require the configured existing vault key, retain
it for the process, set database lock/statement/idle bounds, close
clients on cancellation, and cap shutdown at ten seconds. Abort
immediately sets failure status and prints one blocked receipt. A real
child with real shortened timers reproduces the old missing-receipt exit
and verifies the fix. Tests also use a real PostgreSQL lock.
- Malformed manifest or runner JSON and symlink replacement fail the new
preparation and publication-fence regressions on the previous code. The
fixed command defers each damaged source and copies the next eligible
backup. Separate tests keep database SyntaxError and cancellation fatal.
Offline migration still rejects damaged sources.
- The bundle test runs the emitted preparation entry with plain Node
without starting the app.
- A prior copied-data experiment completed with 1,738 new publications,
nine verified current objects and 206 deferred sources. Later checks
found different database and filesystem capture times: 43 source runs
were absent from the older database snapshot, and one required
historical manifest was absent from the retained filesystem archive.
This experiment does not prove a consistent restore or enrollment
readiness.
- Full current-head CI at
|
||
|
DottaandPaperclip
|
dbbde744d2 |
fix(workspaces): preserve finalized recovery holds through offline import (#19180)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Offline workspace import must save files before local volume
removal.
> - A task recovery hold can remain after native file finalization
succeeds.
> - The old importer rejects that hold even when its file source is
exact and no controller owns the run.
> - Clearing the hold would discard the task's recovery obligation.
> - This pull request saves files and carries each exact recovery source
forward without resolving the hold.
> - Accepted-result finalizers keep their existing retry path.
## Linked Issues or Issue Description
Refs #19144, #18943, #19038. This PR builds on merged #19144.
**What happened?**
Offline import reports `workspace_recovery_unresolved` for terminal
native runs with finalized workspace references and active task recovery
holds. Some holds also name an exact historical file allocation.
**Expected behavior**
Save the workspace durably. Keep task warnings, recovery rows, results,
coordinator phases and task pointers unchanged. Retain exact future file
recovery and accepted-result finalization retries.
**Steps to reproduce**
1. Retain a local task workspace and an active recovery hold for a
finalized native run.
2. Stop all application writers through offline maintenance.
3. Attempt workspace import.
4. Observe that the old importer rejects the hold.
## What Changed
- Add an owner module that checks the exact terminal native run,
finalized reference, released ownership and task relationship.
- Require the normal finalizer's successful company/run/task receipt for
an accepted-result retry. Support historical receipts without a
workspace foreign key.
- Validate recorded file allocations with the existing recovery-source
owner.
- Journal and recheck hold evidence. Lock recovery, coordinator, run and
allocation rows before publication.
- Give finalized file sources a separate migration marker. Keep legacy
execution-replay suppression out of accepted-result retry paths.
- Use the owning parser for older finalized references that omit
optional fields.
- Redirect future file recovery to immutable snapshot preservation. Keep
provider-stop and source checks intact.
- Verify native copyback or snapshot receipts for each exact allocation.
Require proof for every source, including completed mixed repairs.
- Retain existing successful copyback receipts. Keep a historical
folder's obligation on its real source workspace.
- Retain the finalized migration marker through the execution blocker
projection. Discover finalized child-task sources from their exact
ancestor workspace reference.
- Document the receipt and lock contracts.
## Verification
- Fresh local run at head
|
||
|
Nicky Leach
|
8c062fa481 | fix(native): deliver plugin VM Runner usage to Core (#19162) | ||
|
Nicky Leach
|
53937be283 | fix(agents): select environment plugins only for agents on that environment (#19173) | ||
|
DottaandPaperclip
|
f5f5184f23 |
fix(workspaces): preserve retained branches and private commit sources (#19147)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Ephemeral web servers must preserve existing task files before
volume removal.
> - An old private checkout can be absent while its recorded branch
still exists.
> - A deleted branch can also retain its exact commit in the original
private object source.
> - The importer currently rejects these surviving sources.
> - This pull request saves the verified committed tree and records its
exact source proof.
> - It keeps task state, branch ownership and recovery obligations
unchanged.
## Linked Issues or Issue Description
Refs #19138, #19144, #18826.
**What happened?**
Existing Git workspace records can have missing checkout folders and
retained branch objects. The importer returns
`local_source_unsafe_or_missing`. A successful deletion receipt can also
have a valid private object source that the importer overlooks in favor
of a missing or changed project cache.
**Expected behavior**
Save the surviving committed tree from the exact recorded branch or
deletion receipt. Reject competing owners, live worktree registrations,
changed branches and invalid source identities. Do not use the current
default branch as a substitute.
**Steps to reproduce**
1. Retain an isolated workspace branch in its managed project cache.
2. Remove its clean checkout but keep the branch and original workspace
record.
3. Attempt workspace migration.
4. Alternatively, retain a deleted commit in its recorded private source
while its shared source is unavailable.
## What Changed
- Pin a retained branch in its verified company and project Git cache.
- Require the recorded private path, no competing workspace owner and no
surviving Git worktree registration or index authority.
- For a user-owned branch, require its exact successful unforced removal
receipt after archival. Preserve the branch ownership metadata.
- Prefer a verified recorded private object source for successful
deletion receipts when that source still exists.
- Recheck branch and receipt proof before publication. Preserve
compatibility with existing migration journals.
- Allow missing active records only through the existing offline writer
fence. Keep their status unchanged.
- Add byte-restoration and rejected-authority tests. Document the source
limits.
- Revalidate and honor interrupted journals' original Git source path.
Reject detached registrations and retained orphan index data. Reuse the
cleanup producer command formatter for paths containing spaces.
## Verification
- 172 focused migration, concurrency and isolated-source tests pass with
real PostgreSQL and Git repositories.
- The previous importer fails the retained-branch and offline-active
restoration cases. The previous source selection also fails the
private-cache and clean user-owned receipt cases.
- Tests remove the old home and restore the pinned commit and task file
bytes. New shared default-branch work is excluded.
- Tests reject competing owners, registered worktrees, missing branch
ownership, deleted branches, changed branch tips, forced removal and
revoked receipts.
- Read-only re-inventory of 79 actual failed source records makes 78
eligible with committed owner overlays. One older local test repository
remains blocked. This is source-overlay qualification, not full
released-image acceptance.
- A separate bounded archive replay completed six imports before its
25-minute deadline. That timing failure remains open; it does not
establish complete migration.
- Local server typecheck has no changed-file diagnostics. Stale linked
declarations prevent a clean local whole-server result. Full clean CI
remains required.
- Serving enrollment and empty-home recovery remain separate acceptance
gates.
- Latest rebased head
|
||
|
DottaandClaude Opus 5.5
|
512fbea0bb |
fix(native): deliver messages saved while a native run finalizes (#19183)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can reply on a task while the agent's run is still finishing > - A remote native run first commits its result and its status decision (for example `done`). Then its sandbox environment settles (harness backup, warm retention). > - A message that arrives after the final turn took its input is saved as a deferred receipt behind that run > - The saved-message recovery path holds the receipt while the source run still owns its environment lease or its workspace copy-back owner > - Nothing notified that path when the lease or the owner settled, so the messages waited for an unrelated company event > - This pull request notifies the existing deferred-receipt work when a run's environment lease settles or its workspace finalization owner is released > - The benefit is that messages sent while a run finalizes are delivered once, soon after the run settles, and a task closed in that window is reopened for them ## Linked Issues or Issue Description **What happened?** A user posted three plain comments on a task while the agent's remote native run was finalizing. The final turn had already called its finish tool. The comments were saved in one deferred receipt. The run then committed `done` and its status became `succeeded`. No follow-up run started. The task stayed `done` with three unanswered requests. About 20 minutes later, an unrelated event in the same company woke the deferred-receipt worker, and the messages were delivered. **Expected behavior** A message that arrives after the last turn took its input, but before the run releases, gets one follow-up turn as soon as the run settles. If finalization marked the task `done` in that window, the task is reopened for the message. Messages that the finished turn already consumed are not replayed. **Steps to reproduce** 1. Start a native run with a remote (sandbox) environment. 2. After the agent calls its finish tool, post a user comment on the task. 3. Let finalization commit `done` and project the run as `succeeded`. 4. Observe that the comment's deferred receipt stays `deferred_issue_execution` after the environment lease settles. **Root cause** A remote native run is finished by native finalization, which projects the terminal status. The in-process run completion (`releaseIssueExecutionAndPromote`) does not run for it. Its saved messages depend on the stranded-queue sweep in `resumeDeferredExecutionReceipts`, which calls the release with `resumeStrandedQueue`. That release correctly waits while `hasPendingRunCleanup` is true (an unsettled environment lease or a workspace finalization owner). Finalization notifies the deferred-receipt worker when it projects the run, but the cleanup is still pending at that time. The worker then drops the entry. Lease settlement and owner release did not notify the worker again. Refs #19122 (saves a redirect sent while a stopped run settles; this change reuses the same deferred-receipt delivery path for normally completing runs). ## What Changed - `environmentService.releaseLease` notifies deferred-execution work for the company when a run's lease leaves `active` or `pending_cleanup` (released, retained, or failed). - `environmentService.releaseLeasesForRun` notifies deferred-execution work in the same way. - `withNativeWorkspaceFinalizationOwnership` notifies deferred-execution work when it releases its owner. - Delivery uses the existing path: `resumeDeferredExecutionReceipts` → stranded-queue release → deferred wake promotion (with the existing reopen rule for user messages). No new delivery mechanism. - New regression in `deferred-execution-receipt-work.test.ts` with the real deferred-receipt worker, the real release adapter, and the real lease service. - `doc/execution-semantics.md`: one additive paragraph for messages sent while a run finalizes. ## Verification - Staging evidence (read-only): the task's saved queue held the three comments as one `deferred_issue_execution` receipt after the run succeeded. The run had no `run succeeded` lifecycle event (it was finished by native finalization, not run completion). The receipt was promoted about 20 minutes later with `reopenedFrom: done` by the same recovery path. - Red before the fix: the new test failed with `expected null to be truthy` (the receipt stayed deferred with no run after the lease settled). - Green after the fix: `cd server && npx vitest run src/__tests__/deferred-execution-receipt-work.test.ts -t "finishing native run"`. - Safety cases in the new test: - The receipt stays deferred while the environment lease is active (cleanup holds still apply). - After settlement, exactly one run carries both messages in order, and the done task is reopened once. - A second settlement signal does not deliver the messages again. - Related suites: `deferred-execution-receipt-work`, `environment-service`, `environment-cleanup-work`, `native-environment-settlement-work`, `idle-sleep-safety`. - Not done: a live staging reproduction at the fixed revision. Deployment is out of scope. ## Risks - The deferred-receipt worker reloads the company's deferred receipts after each run lease settlement. Each reload is bounded and runs only when deferred receipts exist. The notification is transactional, so a rolled-back release does not wake the worker. - Low risk: no schema change, no API change, and no change to admission or reopen rules. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, browser for read-only staging diagnostics). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
06826915d7 |
fix(dispatch): park a workspace-busy wait instead of recreating runs (#19143)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Child tasks can reuse their parent's execution workspace. Only one run at a time may write in a shared sandbox workspace. > - A run that finds the workspace held by another live run waits with a `workspace_busy` scheduled retry. > - That wait cancelled the run and created a new retry run every 60–120 seconds, until the holder ended. Each check made one cancelled run record. > - Three sibling tasks that waited for a parent made 15 cancelled runs in 13 minutes. They also got the workspace in random order, not in arrival order. > - This pull request keeps the waiting run as one parked retry while its holder is live, and wakes it when the holder ends. > - The benefit is a clean run history, a prompt handoff, and fair arrival-order access to a shared workspace. ## Linked Issues or Issue Description Refs #18691 (the workspace wait for a live holder). Refs #19075 (sends a stopped writer with a pending save into the same wait). Refs #12587 (an earlier proposal about fairness for this retry). **What happened?** A parent task delegated three subtasks that inherit its execution workspace. Each subtask run was created, then cancelled with `workspace_busy` ("Shared execution workspace is busy: run … is still running"), then created again about every 1.5–2 minutes. One subtask got 8 cancelled runs, one got 5, and one got 2. Every cancellation named a holder that was really running: first the parent run, then the sibling that got the workspace. So serialization was correct. The cost was the churn of run records, and the subtask that waited first ran last. **Expected behavior** A run that needs a busy shared workspace waits as one durable record. It continues when the holder finishes. Siblings that wait for one workspace get it one at a time, in arrival order. Two writers never share the workspace, and the user can still stop a waiting run. **Steps to reproduce** 1. Bind two or more issues to one shared execution workspace in a sandbox environment. 2. Keep a run live on one issue (the holder). 3. Wake the other issues. Before this change, each due check promotes the retry, cancels it with `workspace_busy`, and creates a new retry run. After this change, the retry stays parked until the holder ends. **Paperclip version or commit** Observed on staging at `53735db7`. The code is unchanged on `master`. ## What Changed - `finalizeWorkspaceBusyDeferral` records `contextSnapshot.workspaceBusyWait` = `{ key, holderRunId, waitingSince }`. The key is `execution:<id>` or `project:<id>`. `waitingSince` carries over when the run defers again for the same workspace. - New `server/src/modules/run-dispatch/adapters/workspace-busy-wait.ts`: - `parkWorkspaceBusyRetryInTx` runs in the scheduled-retry promotion transaction, after the existing gate allows the retry. If the recorded holder is still running and is not stale, or an earlier waiter for the same workspace is still scheduled, queued, or running, the retry stays `scheduled_retry`. Only its deadline moves. It does not create a run, an event, or a wake. The fallback deadline is 5 minutes, and never later than the holder's one-hour stale point. - `wakeWorkspaceBusyWaitersInTx` runs in the transaction that ends a running run (`writeRunStatus`). It makes due the retries that wait for that run, and the retries that wait for the same workspace as that run. It registers the scheduled-retry dispatch notification before the write. - `WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS` moves to the run-dispatch domain, so the deferral gate and the parked check use one value. `heartbeat/retries.ts` exports it again. - A retry without a recorded wait (for example, a retry from before this change) keeps the old behavior. - `doc/execution-semantics.md` describes the parked wait. ## Verification - Red before the fix, green after (`server/src/__tests__/heartbeat-workspace-busy.test.ts`, "reused shared execution workspace in a sandbox"): - "waits on a live holder in one parked run, then runs once when the holder ends": before, the retry was promoted while the holder was live. After, it stays parked through due checks at +1 s, +10 min, and +30 min. Ending the holder makes it due at once. It runs exactly once, and there is only one `workspace_busy` cancellation. - "hands the workspace to three waiting siblings in arrival order, one writer at a time": before, nothing woke the waiters when the holder ended. After, exactly one waiter is admitted per handoff, in arrival order. Each runs once, and there are 3 cancellations in total (one first deferral for each sibling). - "still lets the user stop a parked waiter": before, the waiter was promoted while the holder was live. After, Stop cancels the parked retry, and nothing brings it back. - Safety case (green before and after): "lets a stale holder reach the existing quiet-holder failure instead of parking forever". - `npx vitest run src/__tests__/heartbeat-workspace-busy.test.ts`: 35 passed. - `heartbeat-remote-workspace-busy`, `heartbeat-retry-scheduling`, `src/modules/run-dispatch`, `heartbeat/retries.test.ts`, `execution-projection.test.ts`: 266 of 267 passed. The one failure, "records pre-provider quota rejection…", is a timing flake on a loaded machine and passes when it runs alone. It does not use this path. - `tsc --noEmit` reports no errors in the changed files. - Staging evidence was read only. This change is not deployed. ## Risks - One extra indexed `SELECT … FOR UPDATE` on `scheduled_retry` rows when a running run ends. The update and the notification happen only when waiters exist. - If an earlier waiter stays queued for a long time (for example, its agent is paused), later waiters for the same workspace wait behind it. Each check is a deadline update, not a run. The fallback check runs every 5 minutes. - A remote writer that has ended but has not finished its save (#19075) is not parked, because its run is no longer `running`. Such a wait can still make a small number of deferrals until the save settles. - Low migration risk: no schema change. Old retries without the new context key keep the old behavior. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits, browser read of staging run records) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
c8013d06ad |
fix(workspaces): preserve task-only warnings during offline migration (#19144)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Ephemeral web servers must save existing workspaces before volume
removal.
> - Offline migration checks file ownership and recovery authority.
> - The same gate also rejects task assignment and configuration
warnings.
> - Those warnings do not require the old filesystem.
> - This pull request preserves known task-only warnings while it saves
files.
> - File recovery and execution obligations keep their existing
blockers.
## Linked Issues or Issue Description
Refs #18943, #19038, #19138.
**What happened?**
Offline workspace migration refuses tasks with assignment,
configuration, disposition or deliberate-wait warnings. It returns
`workspace_recovery_unresolved` even when the warning has no file or
execution recovery authority.
**Expected behavior**
Save local workspace bytes while retaining the task warning unchanged.
Continue to block unknown recovery kinds, file sources, export failures
and execution recovery work.
**Steps to reproduce**
1. Retain a local workspace on a task with an active assignment or
configuration warning.
2. Stop all app writers through the offline maintenance entry.
3. Attempt workspace migration.
## What Changed
- Admit four known task-only warning kinds during offline file
migration.
- Keep file recovery markers, system recovery, native failure markers,
export-failure causes and unknown kinds blocked unless their exact
legacy source is already preserved.
- Keep the existing live-migration recovery exclusion.
- Recheck warning evidence before publication. Do not resolve warnings
or change their owners.
- Add restoration, rejection and publication-race tests. Update the
migration plan.
## Verification
- 112 focused migration and concurrency tests pass with real PostgreSQL
and filesystem archives.
- The four task-only restoration cases fail against the previous
importer with `workspace_recovery_unresolved`.
- Tests delete the old home, restore its saved bytes and compare each
warning row unchanged.
- File and execution markers, unknown kinds and export failures remain
blocked.
- A task warning that gains file recovery during upload prevents
publication.
- Local server typecheck has no changed-file diagnostics. Stale linked
dependency declarations prevent a clean local whole-server result. Clean
full CI remains required before merge.
- The PR was rebased onto the merged Git preservation fix after
resolving a plan-document conflict. The rebased head passed full PR CI
(41943153422). Current head
|
||
|
DottaandPaperclip
|
7c66b6b1f6 |
test(onboarding): cover plan, hiring, and execution approval history (#19069)
## Thinking Path > - Paperclip manages agent work and human decisions. > - Onboarding can approve a plan, a hire, and an execution step in separate turns. > - Tests covered single approvals more closely than that full history. > - The repaired gate must retain valid plan approval through later decisions. > - Every consumer must recognize the saved confirmation kind and selected scope. > - This change tests that contract and registers a complete interview-to-execution journey. ## Linked Issues or Issue Description Refs #19043, #17914, #18283. The earlier tests admitted one accepted proposal at a time. Decomposition fixtures used standard confirmations. The direct-task E2E path forbids hiring. Those cases did not cover an accepted checkbox plan followed by hiring and a separate untargeted execution approval. ## What Changed - Add eight PostgreSQL contract cases for standard and checkbox plan approvals. Native tools create the cards. An authenticated human accepts them through HTTP. A later native run reads the approved revision and actual selected scope. - Exercise native task creation, generic REST creation, child REST creation, and accepted-plan decomposition. Check the assigned child and one creation audit event. Check durable replay on the three entry points that support it. A pending execution confirmation still prevents creation. - Register the explicit-only `onboarding-approval-history` suite for local native Codex and ACPX Claude. The browser chooses the interview, selects only drafting in the saved plan, approves one writer hire, and approves a separate Start drafting confirmation. The model must hire the writer and use that writer's own child execution to save the welcome document. - Grade each human decision, the unchanged plan revision and selection, approval timing, one assigned child and worker execution, the exact writer-authored saved revision, no pending requests, and persistence after reload. Calibrate the oracle against missing, forged, premature, duplicate, and changed outcomes. - Preserve synthetic option choices, document/run bindings, and saved revision authorship in the existing public first-task evidence projection. Keep private provider/account data excluded. Document the workflow, evidence, billing, and cleanup bounds. - Align the approval-history fixture with its completion oracle: the onboarding parent must be Done. Test both native profiles against graded completion evidence and reject a completed child whose parent remains In review. Count parent heartbeats separately from the writer: accept one to seven parent records while retaining the twelve-company-run cap and the independent three-approval/one-writer oracle. - Retain the existing 52-cell first-task suite. The new suite has one attempt, seven expected work turns, a twelve-run maximum, a fifteen-minute deadline, and the existing $5 company task budget. It is excluded from `--all`. - Update two stale cleanup assertions to include the existing thirty-second deletion timeout. ## Verification - Red control: temporarily restore the pre-fix admission module and run the new checkbox/native history case. It fails with the expected `First-task proposal requires human acceptance` error after plan, hiring, and execution acceptance. Restore the repaired module after the probe. - The final post-rebase history matrix and existing onboarding safety suite passed together: 74 tests, no skips. - Final focused Runner E2E support passed: 413 Vitest tests and all 370 Node support checks. This covers catalog discovery, journey controls, wrong-outcome calibration, public evidence, and existing first-task behavior. - Review controls: four authorship cases fail before the revision check. A delayed browser-response control fails before the approval-time fix. Both pass after repair. The final oracle requires the exact current document revision, writer ID, native child run ID, matching body, and readback after reload. - Credential-free discovery lists both new native cells. No provider call was made. `OPENAI_API_KEY` and `ANTHROPIC_API_KEY` are absent, so the real provider journey remains unqualified. - The earlier broad credential-free E2E support run reported 6,728 passing cases, three skipped cases, and nine failures in three unrelated files. All nine reproduce with the changed E2E source restored to the unchanged baseline: three expired overhead-fixture cases, one stale Pi profile-version assertion, and five workflow security assertions. This change does not alter those files, paid workflow policy, or provider pins. - Previous full workspace typecheck/build and [Linux CI](https://github.com/paperclipai/paperclip/actions/runs/41921754638) passed on `4f431f5aae99d072cc75027e90d7b847b4c342ca`, including the new eight-case PostgreSQL file. Ponytail subsequently found a fixture/oracle terminal-status mismatch. Both native-cell controls fail on that fixture (`done` observed versus `in_review` expected), while the parent-in-review negative control passes. A second comparison uses the actual `matchesRunCount` predicate: graded evidence with six parent runs and one writer run fails the old exact-seven-parent requirement. Both fixes pass these controls, all 413 focused support tests, all 370 Node checks, and the owning E2E project typecheck. [Fresh CI](https://github.com/paperclipai/paperclip/actions/runs/41936699686) passed on `91c980b110252da451b92eda597c168517de1276`: 48 successful jobs and one expected skip. Greptile reviewed this head at 5/5 with no unresolved findings. Ponytail's terminal-status finding is fixed and its thread is resolved; its current-head re-review remains queued. Live provider cells remain unqualified. ## Risks The production API and approval gate are unchanged. The new paid journey can expose model or product failures at an intermediate approval boundary. It retains those failures and uses the existing diagnostics and cleanup. Registration and deterministic tests do not establish a live provider pass. The unrelated baseline support failures remain explicit. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused tests; unrelated baseline failures are recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
5fdfd3f17a |
fix(recovery): bound repeated saved-record response bodies (#19168)
## Thinking Path > - Paperclip manages agent tasks and their saved execution history. > - A continuation carries completed API receipts to prevent duplicate work. > - Repeated saves can copy large record bodies into each new provider prompt. > - The resulting command can exceed the durable PRP frame limit before useful work starts. > - This change projects large successful saved-record responses as compact references. > - The original receipts and authenticated user direction remain available. ## Linked Issues or Issue Description **What happened?** A task with many saved document and comment responses can fail with `command_payload_too_large`. Same-run recovery can then show a runner identity failure that hides this first cause. Completed response bodies repeat content already stored in task records. **Expected behavior** Continue from saved records with completed-action evidence that fits the runner command. Keep user messages, answers and approvals intact. Keep the full original receipts. **Steps to reproduce** 1. Save 39 successful document, comment and task-write responses in a run's API receipt ledger. 2. Give each response a large record body. 3. Build and render a continuation, then queue it through the real durable PRP journal. 4. Before this fix, the regression throws `command_payload_too_large` with a 4.2 MB prompt. Related work: #16980 pages oversized assigned tool results. This change addresses earlier response bodies repeated in a later continuation. It does not change tool-result paging or protocol limits. ## What Changed - Project response data over 4 KiB for three known successful record-write operations. Keep record, revision and status metadata, HTTP outcome, receipt identity and a data digest. - Keep small, failed, unknown, mismatched and foreign-company results in their existing form. - Explain the reference format in shared prompt guidance and the execution contract. - Add database and real-journal regression coverage. Verify the original receipt ledger remains unchanged. The continuation service owns the projection. The shared adapter helper only explains the reference to native and legacy prompt consumers. There are no new dependencies, provider calls, timers or background scans. ## Verification - The unfixed database regression fails at the real PRP command limit with `command_payload_too_large`. - Continuation builder, native execution input and native continuation suites: 118 tests pass. - Shared prompt consumers: 142 tests pass. - Module boundaries and diff whitespace checks pass. - Greptile: 5/5 on `8109e7361c5b5e38f9aad35af9bf4ad7bdc5f338`, with no unresolved review threads. All CI checks pass. - Full workspace `pnpm -r typecheck` and `pnpm build`: passed. - Hosted verification: pending deployment. The source fix does not itself resume the affected hosted task. Local setup failures remain accounted for: the fresh checkout lacked an ignored root SDK link and embedded PostgreSQL library symlinks after an offline install that skipped lifecycle scripts. The maintained workspace-link helper, SDK build and platform hydration script repaired those prerequisites. A later test expectation assumed JSON insertion order across PostgreSQL JSONB; it now checks the digest against the unchanged stored receipt. These failures did not change runtime limits or recovery guards. ## Risks Large responses from these three successful operations now use explicit saved-record references in continuation context. Agents must read the record or source receipt when full content is needed. This is not a universal prompt-size limit: user history, unknown operations and other inputs retain their existing representation. No migration, permission, credential, provider-stop or replay policy changes. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), xhigh reasoning, through Codex in T3 Code. Used for investigation, implementation and local verification. No subagents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal task id - [x] I have run the targeted tests locally and they pass - [x] I have added tests and updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address review comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky Leach
|
737db73275 | fix(native): measure resumed run usage from the prior run's total (#19161) | ||
|
Devin FoleyandPaperclip
|
94fccc3180 |
fix(workspaces): preserve snapshots beside runtime sockets (#19159)
## Thinking Path > - A workspace can contain saved source files alongside transient runtime IPC endpoints. > - The portable archive supports files, directories and confined symlinks; treating a Unix socket as a fatal entry prevents every file beside it from being saved. > - Keep the existing archive owner and validation. Record socket paths as exclusions, leaving the source endpoint and adjacent work intact. ## Linked Issues or Issue Description **What happened?** Retained workspace recovery repeatedly fails with `remote_workspace_archive_unsupported_entry` when the source tree contains a Unix socket, preventing adjacent source files from being saved. A real socket reproduces the error in local capture and the serialized sandbox helper. **Expected behavior** Save regular workspace files, record runtime socket paths as exclusions, and leave the source endpoints and new file edits intact. **Steps to reproduce** 1. Create a temporary workspace with an uncommitted text file. 2. Open a Unix domain socket in that directory with a Node server. 3. Capture the workspace through the local archive function or generated sandbox helper. Before this fix both captures fail; afterward the saved file restores and the socket exclusion is recorded. **Paperclip version or commit** Reproduced on `6fa54d55779295d90c455389cebdd3da96607d20` before this change. The socket fixture establishes this failure class; it does not establish the entry type behind every historical unsupported-entry occurrence. ## What Changed - Exclude Unix domain socket endpoints from portable workspace captures and include their paths in the existing exclusion manifest. - Keep source sockets in place. Preserve normal files and the existing rejection of named pipes, device entries and unsafe links. - Exercise local and serialized capture, an unchanged source endpoint, restoration of adjacent uncommitted work, FIFO rejection, and a socket becoming a real file during capture. A changed source still fails safely and retains that new file. - Document the portable snapshot behavior. ## Verification - Red regression: both local and serialized captures failed with `remote_workspace_archive_unsupported_entry` before the change. - `pnpm exec vitest run server/src/services/remote-workspace-snapshot-archive.test.ts`: **43 passed** using real filesystem sockets, pipes, Git repositories and archives. - `pnpm -r typecheck` and `pnpm build`: passed. - Module boundaries and `git diff --check`: passed. Added source and PR text were scanned for credentials, private incident identifiers and personal data before push. - Snapshot storage and save-concurrency consumer suites pass **23 tests**. Their first local runs exposed missing workspace SDK build links and unhydrated macOS embedded-Postgres library symlinks in the new checkout. Repairing those prerequisites made the unchanged consumer suites pass; the earlier failures remain recorded and no assertion was weakened. - Greptile reviewed `e313565a5088abce448f11b29982462aa338a692` at **5/5** with no unresolved threads. All current-head CI checks are green and the branch is conflict-free. Hosted recovery needs a qualified deployment and original-task verification. ## Risks Socket endpoints represent runtime IPC and are not recreated by archive restore. Their paths are visible in the existing exclusion manifest. This adds no archive entry type, subprocess, recurring work, provider call or dependency. It does not alter source authorization, stop receipts, ordinary file validation, restore safety or recovery budgets. Unix socket/FIFO fixtures are skipped on Windows; existing portable archive coverage still runs there. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through Codex, xhigh reasoning. Used repository tools, code execution and real local filesystem/process fixtures. ## Checklist - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] Added focused behavioral and preservation regression coverage - [x] Updated the execution contract and documented limits - [x] Typecheck and build pass locally - [x] Scanned the diff and description before push - [x] Current-head CI is fully green - [x] Greptile is 5/5 with no unresolved comments Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
c2cc69d1e4 |
fix(recovery): resume and settle preserved native results (#19034)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution can save an accepted result before workspace export fails. > - Recover must preserve the files and settle that same result without another agent turn. > - The export failure can own the task's Blocked transition and its recovery notice. > - Recovery must prove preservation, stop, source identity and current authority before settlement. > - Recover can also transfer repair ownership to the system, which must keep explicit board export repair available. > - This change restores those paths while preserving later task state, live owners and unknown action outcomes. ## Linked Issues or Issue Description Related #17916 protects unfinished exports. Related #19004 preserves deliberate Retry authority. This change repairs accepted-result finalization and its recovery notice. **What happened?** A native run accepted its result, then exhausted workspace export. Normal Recover could save the files but leave the result and old recovery notice unsettled. A saved Retry could also use a different generation from preservation. Separately, Recover changed the action owner from board to system. Explicit export Retry then rejected that same source as stale, even when its accepted result and stopped allocation still matched. **Expected behavior** Preserve files, finish the original accepted result and close only its obsolete notice after exact proof. Keep explicit board export repair available after system adoption. Preserve current authority, Stop, later task changes, newer execution and every physical ownership fence. **Steps to reproduce** 1. Accept a native result and exhaust its workspace export. 2. Request ordinary Recover, optionally with an existing saved Retry. 3. Let file preservation complete. Inspect the task and source recovery notice. 4. To reproduce the explicit-repair defect, call export Retry after Recover adopts the action. Before this repair it rejects the valid system-owned source before any provider call. ## What Changed - Record the exact accepted result, contract, task status version and prior decision when export exhaustion creates Blocked. - Bind passive finalization to complete, source-bound preservation and the current chat/review authority. Recheck the proof under status commit locks. - Keep the saved Retry identity separate from its preservation generation. Retain coalescing and reject malformed or mismatched history. - Preserve an existing published GitHub assessment for the no-publication status-unproven wait. Keep all semantic, result, authority and publication checks. - Close an old export notice only after its own accepted result, export or preservation receipts, effects, projection and old Runner ownership settle. Keep newer work and files unchanged. - Permit explicit board export Retry after Recover transfers the exact source to the system. Require valid system and workspace recovery records for that same run, plus every existing result, stop, allocation, task and shared physical-lock check. Reject missing proof, another source and agent ownership before provider access. - Merge current master while retaining the stronger Stop, source and verified preservation checks. - Update the execution contract. No schema, endpoint, Runner protocol, model, trust or provider artifact change. ## Verification - Current head: `9be5a73489801095c9f39c9bfcb285a22b055912`. - The explicit-repair regression fails before the new guard change: ordinary Recover adopts the real database action, then export Retry rejects it as stale. It passes after the repair. - On the merged source, three owning PostgreSQL/file suites pass **220 tests**: `preserved-native-response-wait.integration.test.ts`, `native-workspace-export-retry.test.ts` and `idle-completed-history.test.ts`. The export suite includes four unrelated-owner/source rejection controls and both board-owned and system-adopted active-exporter controls. Results and run counts prove that repair creates no provider turn. - Server layered typecheck, module boundaries, `git diff --check`, and the added-line secret/private-incident scan pass. - Full workspace typecheck (`pnpm -r typecheck`) and build (`pnpm build`) pass. The complete local `pnpm test:run` is still running; its completed serialized chat partitions pass. Current-head CI is green (53 successful checks, 3 expected skips). Greptile reviewed `9be5a73489801095c9f39c9bfcb285a22b055912` at 2026-11-05 00:58 UTC with confidence **5/5**; all review threads are resolved. - Original qualification remains supporting evidence: the accepted-result fixture exercises HTTP Recover, real files, the finalizer, saved Retry coalescing, eight commit races, current principal/resource/head changes, Stop, later task changes, notice closure and repeated delivery. No accepted response, provider turn or publication is replayed. - Earlier failures remain recorded. Original red controls exposed wrong-action preservation, unrelated finalization barriers, generation mismatch and a lost GitHub assessment; the original repair added controls for them. The prior CI run [41908572248](https://github.com/paperclipai/paperclip/actions/runs/41908572248) failed an unchanged completed-history runtime fixture because its loopback service refused connections through the original ten-second readiness window. That owning suite passes unchanged in the 220-case run above. The earlier readiness failure's cause remains unproven; no wait or assertion was weakened. - Local and CI tests do not establish hosted task recovery. The serving revision and original-task outcomes require separate verification after rollout. ## Risks - Historical failures without the exact export-owned Blocked receipt keep their current task status. A saved result does not grant execution or publication authority. - Explicit export repair still resumes only the same retained allocation under the shared physical lock. It cannot take over an active exporter, create another agent turn, choose a replacement sandbox or discard saved files. - Retiring a source notice does not release physical resources or modify a newer task owner, later Done state or saved snapshot history. - The proof adds bounded reads only for concrete recovery obligations. It adds no recurring worker or polling timer. ## Model Used OpenAI GPT-6 through Codex for the original repair. OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), xhigh reasoning, for the merge, integration verification and system-adopted export repair. Both used repository tools, code execution and PostgreSQL tests. The runtime does not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and searched for duplicate or related work - [x] I have described the issue and linked related public PRs - [x] I have included no internal task identifiers or private incident links - [x] I have added behavioral and safety regression tests - [x] I have documented verification limits and risks - [x] I have updated the execution contract - [x] All Paperclip CI gates are green on the current head - [x] Greptile is 5/5 on the current head with no unresolved comments - [x] I will address review comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Devin Foley <devin@paperclip.ing> |
||
|
DottaandPaperclip
|
9e5039f6c8 |
fix(daytona): retain duplex close ownership after lease cleanup (#19150)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote agents use provider plugins to manage sandbox leases and duplex channels. > - The host verifies the exact route and session when a bound channel closes. > - Daytona lease teardown removed that binding before a later host close. > - The incomplete reply made the host retire the shared worker. A later command then failed during setup. > - This pull request retains bounded close receipts and joins the original cleanup. > - Later work can use the same worker while invalid or unconfirmed closes still fail closed. ## Linked Issues or Issue Description **What happened?** A normal lease teardown can close a duplex channel before the host closes its bound route. Daytona then replies with only the route ID. The host requires the original session ID too. It logs `duplex channel close not acknowledged; retiring worker`, stops the worker, and rejects a later `environmentExecute` call as stopped. **Expected behavior** A later close must identify the original route and session and join its existing cleanup. It must not revive the channel or restart the worker. Unknown ownership and an unconfirmed close must retain the existing failure behavior. **Steps to reproduce** 1. Acquire two leases through one Daytona provider worker. 2. Open a duplex channel on the first lease. 3. Release that lease, then close the bound channel through the host. 4. Execute a command on the second lease. **Paperclip version or commit** The real host/worker regression fails on `69dea878c0b920b335594debb0286443ba9f6c35`. This branch includes master `3a72efebe5b050e7594bb309e71a0d5f3072db00`. **Deployment mode** Managed Daytona sandbox execution in Cloud. The regression uses a local real host and SDK worker. Only the external Daytona SDK and PTY are simulated. Related work: Refs #19004, Refs #10429, Refs #18065, Refs #19050. These cover task Retry, managed worker startup, drain release, and stale provider sessions. ## What Changed - Retain at most 4096 retired route/session bindings with their original close promises. - Remove live channel authority before cleanup starts. Concurrent and later closes join the same promise and return the stored binding. - Reject reuse of a retained route ID. Keep unknown and evicted routes unable to provide a session acknowledgement. - Add a real child-process host/SDK-worker/Daytona regression with only the external Daytona SDK and PTY simulated. - Cover release, cancellation, stop, destroy, resume, a stalled close, concurrent closes, failed cleanup, forged IDs and bounded eviction. - Document close receipt scope and its separate provider-termination checks. ## Verification - RED before the repair: the actual host and SDK worker reproduced the retirement log, SIGKILL, stopped state, and later `environmentExecute` rejection. The direct provider cases also failed because the session ID was missing. - Combined-branch provider suite: 364 passed. - Combined-branch host manager, duplex, managed readiness and composed suites: 142 passed. The final test-only manifest-loading change passed all 6 composed cases again. - Full repository typecheck (`node scripts/typecheck-low-memory.mjs`), focused server types and the standalone Daytona build passed. The provider build uses pinned SDK `0.203.0` and TypeScript `5.9.3`. - The composed cases check the same worker/PID, one later command, preserved saved content, revoked channel writes/stops, and continued rejection of commands on closed leases. A stalled close still retires the worker and dispatches no later command. - Reproduce with `pnpm --dir server exec vitest run --config vitest.config.ts src/__tests__/plugin-worker-manager-daytona-close.test.ts` from a checkout with the SDK build dependencies prepared. Run the provider suite with its own Vitest config. - An initial local typecheck lacked the excluded providers' SDK links after an install with scripts disabled. The standard root postinstall linker restored those local links, and full typechecking then passed. The fixture loads its actual built manifest. Failed setup output is retained; no type configuration was relaxed. - The first two composed attempts stopped during fixture setup. They are not the behavioral RED above. Their failed outputs are retained. - The first broad local typecheck stopped at an IPC permission error during preflight. It passed after rerunning with allowed local IPC and the standard SDK links. - [Full repository CI](https://github.com/paperclipai/paperclip/actions/runs/41945022881) passed all 48 jobs on this head. Selection fell back to the complete task plan, so the run was unfiltered. The complete check inventory has 54 successes and two expected Storybook skips; the separate Snyk status passed. Fresh Greptile review on `daebbeb5005b9d9bd59d32ea30094279048b9213` is [5/5 with no actionable comments](https://github.com/paperclipai/paperclip/pull/19150#issuecomment-6646621914). Independent production and fixture review found no actionable findings. Hosted acceptance of this repair has not run. ## Risks - A receipt removed at the 4096-entry bound cannot confirm an old bound close. The existing host guard remains effective in that case. - Close acknowledgements identify channels. They do not certify remote process termination or clear unknown external work. Provider termination receipts and existing recovery gates remain in force. - Hosted evidence confirms the unacknowledged-close retirement class. The historical log does not distinguish a missing session ID from a timeout or another close error. The regression proves and repairs the missing-binding path; it also preserves timeout retirement. - The provider is outside the main pnpm workspace. A new Cloud image must compile its changed source. The unchanged plugin version alone does not prove activation. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, source editing and tool execution. The exact model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
11f2a5293d |
fix(native): name the proof that holds a recovered per-turn owner (#19061)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner per-turn sessions in a remote sandbox keep one conversation across task turns. > - A follow-up must prove that the previous Runner owner is finished before it starts a new Runner. #18846 added a retirement path for an owner that finished without a saved Runner checkpoint. > - On a test deployment, follow-ups still fail with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. The retirement path has about twenty separate fences, and each refusal gives the same error. > - The read-only run data cannot tell which fence held these runs. A code fix without that evidence would be a guess. > - This pull request names the first failed retirement proof in the error. Admission behavior does not change. > - The benefit is that the next failure on a deployment identifies the exact fence, so the follow-up fix can target it. ## Linked Issues or Issue Description Refs #18846 (retirement of recovered per-turn owners). Refs #18992 (settlement from live state). **What happened?** On a test deployment, 16 runs in one day failed before provider work with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. All were per-turn `acpx_runtime` runs in a Daytona sandbox, and each got a new sandbox. In the runs that I inspected, the previous run committed its result. In one of them, `harness_state.backup.persist` and `session.checkpoint.persist` ran for about 346 seconds and failed, so the Runner left no suspension receipt or failover backup. That is the case that #18846 addresses, but the follow-ups still failed. The run log does not show the retirement message, and the error is the same for each fence. Thus the cause cannot be found from the run data. **Expected behavior** When admission holds a follow-up, the error names the proof that failed. **Steps to reproduce** 1. Seed a per-turn sandbox owner that committed its result without a Runner suspension receipt. 2. Make one retirement proof fail. For example, leave the owner phase `observed`, or leave a `runner.suspend` command pending. 3. Admit a follow-up in a new sandbox. 4. Before this change, every case fails with the same error text. **Paperclip version or commit** Observed on a deployment before `e107a75755`. The same error text is on `master`. ## What Changed - `retireFinishedPerTurnRunnerdOwner` returns the refusal reason instead of `false`. `readSettledRunnerdProjection` returns the journal refusal reason instead of `null`. - When admission tries the retirement and it fails, the error is `prior_owner_terminal_state_indeterminate; retirement_refused=<reason>`. A warm session gets no suffix, because retirement does not apply to it. - Reasons use only fixed lists of known names: owner phase, recovery state, controller fence reason, unsettled command type and status, sandbox outcome, and quarantine copy result. Any other stored value appears as `other`. Reasons do not include payloads, file names, or paths. - The same proofs apply in the same order. Admission accepts and refuses the same states as before. - `doc/execution-semantics.md` describes the suffix. ## Verification - **Failure before the change:** I ran the updated `native-recovered-owner-followup.test.ts` against the unchanged executor. 18 tests failed. Example: expected `...indeterminate; retirement_refused=owner_observed`, received `...indeterminate`. - **After the change:** `cd server && pnpm exec vitest run src/services/native-runtime/native-recovered-owner-followup.test.ts`: 22 passed. Each held case now asserts its reason: `owner_retryable_failure`, `controller_remote_controller_lease_live`, `owner_recovery_resuming_session`, `journal_commands_unsettled(runner.suspend.pending)`, `journal_tool_input_unanswered`, `quarantine_journal_identity_mismatch`, `runner_lifecycle_not_suspended`, and `quarantine_runner_lifecycle_not_suspended`. - The success cases still retire the owner and admit the follow-up. - `pnpm exec vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-runner-restart-recovery.integration.test.ts src/services/native-runtime/remote-warm-session-recovery.test.ts`: 1227 passed, 20 skipped. - `pnpm exec vitest run src/services/native-runtime/native-controller-recovery.integration.test.ts src/__tests__/task-workspace-binding.test.ts`: passed. - `tsc --noEmit` reports no errors in the changed lines. This worktree did not build every workspace package, so the full-package typecheck shows unrelated missing-module errors. CI runs the full typecheck. - **Not verified live:** no deployment has run this change. ## Risks - Low risk. Admission logic is unchanged. Only the error text gets a suffix. - Consumers that match the error by prefix still match. `failed-run-retry.ts` uses `startsWith("runner_state_identity_mismatch: ")`. No code compares the full string outside tests. - This change does not fix the held runs. It finds the fence. A second change is needed after a deployment reports the reason. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, shell execution, and read-only browser access to a test deployment. The harness does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
5723fd82ca |
fix(native): release a fresh claim when task preparation fails (#19065)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run claims its finalization coordinator before task preparation. Then it starts the provider turn. > - Task preparation can fail after the claim and before any provider turn. > - In that case the executor kept its claim. Failure settlement read the claim as a live owner and stopped. Nothing failed the run. > - The 60-second execution-control watchdog then failed the run as `execution_finalization_deadline_exceeded`. The original error was lost. > - This pull request releases only that exact fresh, unstarted claim after renewal stops. > - The benefit is that ordinary settlement fails the run at once with its real error and keeps the workspace files. The watchdog stays a backstop. ## Linked Issues or Issue Description Refs #19028. That PR keeps the earlier failure code when the watchdog fails a run. In this case no earlier code is saved, so it cannot help. This PR removes the wait for the watchdog. I searched open PRs for duplicates. #18820, #19034 and #19038 touch near code, but none release a preparation claim. **What happened?** A native task with a prepared sandbox workspace failed during preparation, after the controller claim committed. The coordinator stayed `observed` and kept this controller's lease owner. The heartbeat failure path calls workspace preservation. Preservation refuses any lease owner without a matching restart claim, so it raised `NativeWorkspaceFinalizationBusyError`. The heartbeat then treated ownership as held and returned without a terminal write. About 60 seconds later the watchdog failed the run with a generic "verify the provider stopped" action. The real error (for example `native_harness_storage_owner_missing`) stayed only in the workspace operation record. **Expected behavior** When no provider turn started under a fresh claim, settle the failure like a failure before the claim: retain the source, fail the run with its own error, and do not wait for the watchdog. **Steps to reproduce** 1. Start a native run with a prepared remote workspace. 2. Make task preparation fail after `native.coordinator.claim` commits (for example, a preparation callback throws). 3. See that the run stays `running` until the control deadline fails it as `execution_finalization_deadline_exceeded`. ## What Changed - `native-execution-lease.ts`: add `releaseUnstartedNativeExecutionClaim`. It clears the lease owner and expiry only for the exact run, company, issue, owner and attempt, while the phase is `observed` and no result exists. - `native-session-executor.ts`: when task preparation fails after a fresh claim, mark that claim. The startup owner releases it after renewal stops. A restart-recovery claim, a startup-cleanup context, Stop, restart detach, or a different owner keeps the old behavior. - Tests: an executor regression for the fresh and restart cases, and a real-database test for release and workspace retention against the exact owner, a foreign owner, a different attempt and an accepted result. - `doc/execution-semantics.md`: state that the deadline is a backstop and describe the release. ## Verification - Red before the fix: `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "releases only a fresh claim"` fails with "releaseUnstartedNativeExecutionClaim ... Number of calls: 0". - Green after the fix, with the same command. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-failed-workspace-recovery.test.ts src/services/native-runtime/native-execution-lease.test.ts src/services/execution-control-reconciliation.test.ts` passes (embedded Postgres for the retention test). - `pnpm --filter @paperclipai/server typecheck` passes. - Not done: a live run against a real sandbox at this revision. The staging evidence is read-only. ## Risks - The release is fenced by the exact owner, attempt, `observed` phase and no result. Renewal stops before the release, so a renewal cannot see a missing owner and detach or cancel the run. - The fix covers failures inside task preparation only. Failures later in startup, or after a restart-recovery claim expires, still reach the watchdog. Those keep fail-closed behavior and need separate evidence. - If the release write fails, the run keeps the old watchdog path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
9cdbcbeddc |
fix(workspaces): restore archived commits from verified project objects (#19138)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An ephemeral web server must preserve historical task files outside its local disk. > - Archived isolated Git workspaces can have a successful branch-deletion receipt. > - Their private checkout can be gone while the project cache still owns the exact commit. > - The importer rejects that cache because it applies private-checkout validation. > - This pull request validates the managed object source for receipt-based restoration. > - The importer restores the recorded commit without using the current shared working tree. ## Linked Issues or Issue Description Refs #18826, #18943, #19038. **What happened?** An archived isolated workspace has a valid successful cleanup receipt and a retained commit. Offline migration rejects it with `archived_git_source_unavailable,local_source_unsafe_or_missing`. The private checkout was removed. The company and project cache still contains the exact commit. **Expected behavior** Restore the immutable commit from the verified project object cache. Keep strict validation for saved private edit recipes. Reject missing receipts, a foreign cache, a different origin and revoked cleanup authority. **Steps to reproduce** 1. Create an isolated Git workspace from a managed project cache. 2. Archive it and retain its successful branch-deletion receipt. 3. Remove its private checkout while the verified project cache retains that commit. 4. Attempt offline workspace migration. ## What Changed - Extract the existing managed Git object-source validator. - Use that validator only for exact successful deletion receipts that refer to a canonical company and project cache. - Keep private restore-point source validation unchanged. - Fetch immutable objects at the recorded commit into private scratch. Recheck receipt authority before publication. - Add real Git and database tests for restoration and rejected authority changes. Document the preservation path. ## Verification - 129 focused migration, concurrency and isolated-source tests pass across three files. - The new restoration case fails against the previous importer with the reported source-unavailable error. - Tests remove the old home and restore the retained commit. A newer shared working tree and unrelated files are excluded. - Tests reject a foreign company cache, wrong origin, missing receipt and receipt revocation during upload. - Local server typecheck has no diagnostics in changed files. Stale linked dependency declarations prevent a clean local whole-server result. Current-head full CI passes (54 passing checks, three skipped). Greptile is 5/5 with no unresolved review threads. - A replay of the 32 affected records is running in an isolated staging copy. Two committed owner overlays are under test. This is not released-image or serving-instance acceptance. ## Risks - A deletion receipt must identify an exact commit that still exists. Missing objects continue to block migration. - Shared working-tree contents must never substitute for archived private files. Only the recorded commit is fetched through this path. - The change does not recreate private edit authority, retire recovery records or alter serving configuration. ## Model Used OpenAI GPT-6.1 Codex, exact model ID `gpt-6.1-sol`. Used for reasoning, code editing, shell tools and test execution. The runtime does not expose its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7d46532c6d |
fix(runtime): bound parallel offline harness backup transfers (#19108)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - An ephemeral web server must retain provider history outside its
local disk.
> - Offline enrollment moves verified local harness backups to object
storage.
> - A cold rehearsal reached 699 current backup rows while processing
1,953 sources before its 25-minute deadline.
> - The serial transfer loop prevents this existing dataset from
enrolling.
> - This pull request overlaps at most four transfers within the
existing total scratch limit.
> - The benefit is faster preservation with the same ownership and
publication checks.
## Linked Issues or Issue Description
Refs #19038, #18943, #18932.
**What happened?**
Offline enrollment processed 1,953 local provider backups one at a time.
The normal Cloud image reached 699 current backup rows (including nine
pre-existing rows) before maintenance expired. The serving instance
remained unchanged during the isolated test.
**Expected behavior**
Enrollment must preserve every source within bounded time and disk use.
It must stop all in-flight transfers before it reports failure.
**Steps to reproduce**
1. Prepare a settled instance with multiple verified native harness
backups.
2. Run offline enrollment against shared object storage.
3. Observe that the serial owner waits for each archive, upload,
readback and publication before starting the next source.
## What Changed
- Process groups of up to four verified offline harness backups.
- Reserve space for the archive and encrypted copy before each group
starts. Keep the total scratch limit equal to one maximum-size serial
transfer.
- Derive a smaller archive limit from a closed manifest directory set.
Preserve unlisted files with an exclusive maximum-size reservation.
- Enforce the reservation while streaming. Keep the existing maximum for
ordinary runtime backups.
- Cancel and await peer transfers after failure. Keep completed
publications valid for retry.
- Preserve the original archive-stream failure when the tar writer also
fails after destruction.
- Document the migration bound and add integration regressions for
overlap, restoration, cancellation, retry, extra files and archive
limits.
## Verification
- Serial baseline: the new overlap regression fails with one active
transfer when four are required.
- Candidate: 59 targeted tests pass with real PostgreSQL, S3, filesystem
archives and restored provider bytes. The bundled-entry test is skipped
locally because this source worktree has no runtime bundle. The full
normal image rehearsal remains required.
- Follow-up: 48 archive/storage checks pass after correcting the
PAX-header allowance. The real encoder produces a 972,801,024-byte
full-entry projection, above the old 805,306,368-byte allowance and
below the new 1,024,001,024-byte bound. The existing global object limit
remains in force. The old formula fails the new regression. Exact-size
restoration and one-byte-over-limit rejection pass.
- The previous revision passed full CI after one setup-route timeout
retry. That failure remains recorded; the corrected current revision
|
||
|
|
fb96537e28 |
test(ui): verify disconnect recovery through browser outages (#18832)
## Thinking Path > - Paperclip lets people manage agent work through the board. > - Deploys can interrupt reads and writes while people use the board. > - Component tests cannot prove the browser keeps task content and saved drafts through a reload. > - The read and cache changes need a regression suite against the built app. > - This pull request drives those journeys through temporary API failures. > - CI discovers the new suite through the existing e2e shard plan. ## Linked Issues or Issue Description **What happened?** Outage recovery lacked one browser suite covering retained content, write recovery, cached reloads, access errors and sign-out clearing together. **Expected behavior** Keep loaded task content visible through a 20-second API outage. Show one connection banner and its recovery state. Save comments once and retain document edits. Restore approved reads after a reload. Keep real error responses readable and clear query records on sign-out. **Steps to reproduce** Open a task or settings. Return 503 with `tenant_app_unavailable` from browser API requests for 20 seconds. Type a comment, edit a document, or reload. Restore the API and compare the visible result with the saved server state. Repeat with real 403/404 statuses and sign-out. Stacked on Refs #18825, which depends on Refs #18822. Related durable-write foundation: Refs #17960. ## What Changed - Add `disconnect-resilience.spec.ts` with ten browser cases for comments, a queued comment across reload, document autosave, cached settings/sidebar/task reads, legacy Inbox reloads, 403 and 404 responses, sign-out clearing, restored-draft conflict detection, and first-socket outage control. - Assert one connection banner, readable recovery, no transient error boxes or raw codes, retained task text, one saved comment, and the saved document after reload. - Use the real local server and isolated embedded PostgreSQL. Intercept browser API traffic for outages. Control the logout/session response for the sign-out test. Create no agents or model runs. - Register realtime interception before navigation. Track both ends of current sockets, close them at outage start, and keep reconnects unavailable through the HTTP outage deadline. Preserve normal close forwarding and close tracked pairs during teardown. The regression captures the first browser socket without waiting for history or open, then checks that the outage closes it. - Prepare complete server-derived experimental settings and workspace preferences before the legacy Inbox navigation. Its route callbacks serve per-test snapshots across reloads, avoiding reads from disposed route.fetch responses. - Document the direct test command. The existing CI e2e shards already discover this file exactly once; no workflow or package script is added. ## Verification - Current head `228c1b68d30` is merge-ready: all 54 check runs completed (52 successful, one skipped, one neutral), the legacy Snyk status is successful, and both required gates pass. The single failed-jobs debug rerun in run `41752978283` passed on attempt 2, including Runner package job `126328280650` and `verify` job `126332119116`. Greptile Apex reviewed this exact head at 5/5 with zero unresolved threads; the branch has no merge conflicts. The three-PR stack is awaiting operator merge approval. - Original attempt 1 of current-head CI run `41752978283` had 46 successful jobs, one skipped and two failed: Runner package tests and the downstream `verify` gate. GitHub reports lost communication with the Runner host; the test step was cancelled, its log returns HTTP 404, and no Runner diagnostic artifact is available. The underlying test outcome and disconnection cause are unknown. One failed-jobs retry with debug logging was accepted after verifying the unchanged current head and passed on attempt 2. No source change was made; earlier passing CI and local tests do not certify this incomplete Runner result. Current-head CI now passes; the earlier disconnection cause remains unknown. - Preceding head `6531caccf62` reached merge-ready: all 54 check runs completed (52 successful, one skipped, one neutral), the legacy Snyk status was successful, and both required gates passed in run `41749512329`. Greptile reviewed that exact head at 5/5 with zero unresolved threads. Current head `228c1b68d30` carries later account-preference and bulk-selection fixes and now has its own passing current-head CI. - All 23 built-UI outage/write/loading cases pass in 4.5 minutes on exact current head `228c1b68d30`: ten outage, six durable-write and seven loading cases. The run used bundled Chromium, a fresh local-trusted server and isolated embedded PostgreSQL on macOS arm64/Node 25.6.1. Owned temporary server and database stopped. - Current integrated head `228c1b68d30` contains cache revision `fe4f682c879` and read head `09d1db29dea`. The exact application source passes all 1,198 focused tests across twenty-eight files, UI typecheck and both guards. Seventeen new cases cover restored-account settings, owned crown cleanup and pending bulk selections; sixteen fail the previous source and one preserves denial compatibility. All three browser specs pass standalone strict checking. All 23 current-head browser cases pass in 4.5 minutes. - Application revision `9c1a10f25d9` passed all 1,137 focused tests across twenty-one files, UI typecheck and both guards. Exact preceding application/spec head `4fec693ec94` passed all 23 built-UI outage/write/loading cases in 4.6 minutes and standalone strict checking of all three specs. It used bundled Chromium, fresh isolated PostgreSQL and local-trusted auth on macOS arm64/Node 25.6.1. Owned test processes stopped. - Preceding integrated head `870aa709657`, containing cache `cbb891dd37d`, passes all 23 built-UI outage/write/loading cases in 4.6 minutes: ten outage cases, six durable-write cases and seven loading cases. It used bundled Chromium, the built UI, a fresh local-trusted server and isolated embedded PostgreSQL on macOS arm64/Node 25.6.1. Owned test processes stopped. - The earlier head `763fe431b74` passes the affected legacy Inbox outage/reload case (34.3 seconds; 50.8 seconds including startup) and strict three-spec checking after the response-snapshot fix. The preceding head `973a12d77f4` passed all 23 built-UI outage/write/loading cases in 4.6 minutes: ten outage cases, six durable-write cases and seven loading cases. Its first-socket regression fails the previous helper because that socket stays open. The run used bundled Chromium, the built UI, fresh isolated PostgreSQL and local-trusted auth on macOS arm64/Node 25.6.1. That earlier isolated callback revision was verified with its affected case; the later application/spec head `4fec693ec94` also passes the full suite. Owned test servers and databases stopped. - The current base gives known skills/app errors priority over sibling loading. All 60 affected tests and both application UI typechecks passed. Its twelve new regressions failed the preceding source. All 47 recovery-worker tests and seven related Runner fixture variants also passed during the historical CI audit; those focused checks do not claim a full current server/Runner suite. - All 22 outage/write/loading cases passed in 4.7 minutes on preceding integrated head `b7deaaee5da`, including direct cache revision `97c0bc3966f`. The run used the built UI, bundled Chromium and a fresh local-trusted server with isolated embedded PostgreSQL on macOS arm64/Node 25.6.1. It covered all nine outage, six durable-write and seven loading cases. The owned temporary server and database stopped. - All 1,038 focused tests passed on that direct cache revision, with UI typecheck and guards. Six regressions failed the previous source: online retries could trigger a hidden-tab worker reload, and sign-out failed without crypto.randomUUID. Both modes and stamp states now pass. Standalone strict checking passes for all three browser specs. - The restored cached-A/draft-B/server-C test asserts zero replay PATCHes and retains the conflict. The queued-comment case holds fresh session verification after general API recovery and asserts zero POSTs until release. Denial, logout and cached-navigation cases require saved records. Sign-out seeds several company records to exercise native Storage key reordering. Healthy navigation waits for conversation-history readiness. - Earlier runs exposed real cached-header, storage cleanup and replay defects in the base. Those failures and corrected fixture assumptions remain in the discussion. A prior run passed 21/22 while real sockets remained available during simulated HTTP failures. A diagnostic trace established the contradiction. The first repair gated new sockets, but review identified the first pre-existing socket still escaping. This revision intercepts before navigation and closes existing pairs too. The new regression fails the preceding helper, and all 23 current cases pass without weaker assertions. Shared-memory exhaustion prevented two earlier diagnostic startups; only unused segments from completed owned test databases were removed. - All six separate cross-organization link/upload variants passed on an earlier integrated base. Standalone strict checking of that unrelated spec reports five existing errors in its shared fixture; this PR does not claim that check passes. - CI on older head `53d979e384c` failed the legacy Inbox case with `apiResponse.json: Response has been disposed` inside a route.fetch callback. Its other 25 cases passed and one skipped. The seven other browser shards succeeded; the aggregate only reports that child failure. The new snapshot callbacks remove those fetched-response lifetimes. Original CI evidence remains recorded in the discussion. - Greptile Apex reviewed exact current head `228c1b68d30` at 5/5 with no actionable findings and zero unresolved threads (summary updated 2026-11-04 at 01:26:14 UTC). The branch has no merge conflicts. Current-head CI now passes; the earlier disconnection cause remains unknown. Controlled local outages do not prove hosted authentication or a real Cloud restart/deploy. Commands run: ```sh pnpm exec playwright test --config tests/e2e/playwright.config.ts tests/e2e/disconnect-resilience.spec.ts --grep 'a legacy Inbox reload retains' --reporter=list pnpm exec playwright test --config tests/e2e/playwright.config.ts tests/e2e/disconnect-resilience.spec.ts tests/e2e/disconnect-write-recovery.spec.ts tests/e2e/task-loading-frame.spec.ts --reporter=list pnpm exec tsc --ignoreConfig --noEmit --strict --target ES2022 --module NodeNext --moduleResolution NodeNext --skipLibCheck tests/e2e/disconnect-resilience.spec.ts tests/e2e/disconnect-write-recovery.spec.ts tests/e2e/task-loading-frame.spec.ts ``` ## Risks The suite adds about three minutes to one e2e shard and depends on current accessible UI controls. It uses controlled outages and logout responses. A passing local run must not be treated as real deployment acceptance. There are no production behavior, schema, dependency, or API changes in this PR. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with xhigh reasoning effort, tool use, and code execution. The context window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for related PRs and linked them above - [x] I have described the issue following the bug report template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id - [x] All 23 built-UI cases pass on current head `228c1b68d30` - [x] All three browser specs pass standalone strict typechecking on the current head - [x] The current application source passes all 1,198 focused tests, UI typecheck and both guards - [x] I have added tests and updated relevant documentation - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Bender (Fable) <bender@agents.paperclip.ing> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
818d2bb661 |
fix(native): preserve closed post-result failure diagnostics (#19126)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native execution persists a result before it completes cleanup and workspace export. > - A later execution error enters the failure-write path. > - That write must reject an already selected result. The existing code reports lease loss for every rejected write. > - This can hide the category of the first error even when the same controller still owns an unexpired lease. > - This change adds closed diagnostic labels after exact ownership and accepted-result checks. It keeps the existing control error and recovery policy. ## Linked Issues or Issue Description **What happened?** A local real-PostgreSQL regression accepts a result through the production control plane, then injects a harmless filesystem failure. The production executor reports `native_session_lease_lost`. It loses the original failure category because the failure transition requires a null result ID. **Expected behavior** Keep the result guard and the existing lifecycle classification. Record a closed diagnostic category when the selected result is accepted and the controller still has exact, unexpired authority. Keep ordinary lease-loss fencing when that proof is absent or its diagnostic lock is busy. **Steps to reproduce** Run the new post-result cases in `server/src/services/heartbeat/runtime-execution.test.ts`. They use the real executor, control-plane result commit, heartbeat recovery path and PostgreSQL. The provider boundary is controlled. No live provider is used. **Paperclip version or commit** The regression fails on base `7d1e9b3477`. Related open PR #16917 changes legacy lease recovery. Open #19063 adds terminal finalizer reasons, and #15310 preserves database finalization errors. They do not change this executor diagnostic branch. This change only adds native diagnostic evidence and does not reclaim a lease. ## What Changed - Add a typed diagnostic with a closed stage, category and accepted-result flag. Do not retain the source message, path, arbitrary code or lease token. - After a rejected failure write, check the selected accepted result and the exact company, task, run, owner, attempt, generation, process identity and expiry. Use a coordinator-only `NOWAIT` lock. Fall back to the original control error if the diagnostic read fails. - Report the closed event after rollback. Keep `native_session_lease_lost` as the error message and leave every failure-write predicate unchanged. - Add real-PostgreSQL safety cases for stale and expired authority, owner replacement, Stop state, result selection, pending accounting, one provider invocation, closed labels and diagnostic sink failure. - Document the closed event, unchanged authority, and possible late diagnostic sink completion. - Add a competing-row-lock barrier. It verifies actual PostgreSQL lock refusal and control settlement before the competing lock is released. ## Verification - Permanent regression on unchanged base: RED because the closed diagnostic is absent. - NOWAIT mutation: RED when only NOWAIT is removed. PostgreSQL reports lock timeout instead of immediate lock refusal. The test uses barriers and the actual database error, with no elapsed-time assertion. - Final related suites: 56 tests passed across runtime execution and provider failure integration tests. - Full server source and test typecheck: passed. - Full `pnpm -r typecheck`: passed in an independent locked-dependency checkout. - Full `pnpm build`: passed. - Full `pnpm test:run`: running in the independent immutable checkout. All six chat shards passed; the remaining suites are still pending. - The first current-head CI attempt failed in two cases: workspace runtime initial fixture HTTP readiness and instructions-bundle error-message rendering. Both exact cases passed in isolation on unchanged PR head and base. The original causes remain unknown after bounded test-only observation. One controlled same-head rerun of each failed job passed. Current required CI and all other completed checks passed. Original failures remain recorded; no infrastructure or flaky-test claim is made. - Module boundary check and `git diff --check`: passed. - Exact-head Greptile: 5/5 with no actionable findings or open review threads. Automatic native review remains queued. That advisory check is not required by current master rules; no current-head native assessment is claimed. - Earlier fixture setup failures remain recorded. These included missing fixture fields and observing the Drizzle wrapper instead of its PostgreSQL cause. They are not the regression RED. An earlier broad typecheck was interrupted for the required NOWAIT revision; the final full typecheck passed. - No hosted export recovery or original hosted failure cause is claimed. ## Risks The extra read is diagnostic only. It uses NOWAIT and cannot wait for a competing coordinator lock. Its failure preserves the existing control classification. The event sink uses the existing 60-second execution-control bound. A slow sink can finish later and emit a late event. Result selection, Stop, accounting, retry eligibility and provider work do not depend on event success. No schema or lifecycle contract changes are required. ## Model Used OpenAI GPT-6.1, model ID `gpt-6.1-sol`, with reasoning, tool use and code execution. The host did not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
2ccc7d37ef |
feat(ui): retain approved reads across outage reloads (#18825)
## Thinking Path > - Paperclip lets people manage agent work through the board. > - The board already retains loaded reads during a temporary server failure. > - Reloading the page removes that in-memory data. > - A deploy can then leave a returning user without the shell or recent tasks. > - This pull request saves an explicit, limited set of reads in the browser. > - Cached identity supports display while fresh session checks protect writes. ## Linked Issues or Issue Description **What happened?** A reload during an API outage loses the board's loaded reads. An automatic worker update can also reload a hidden tab while a write is pending. **Expected behavior** Show the cached shell and recent tasks during a temporary outage. Limit saved content by user, company, age and build. Clear it on sign-out. Wait for a fresh session before sending account-bound writes. Defer automatic reloads while work is pending. **Steps to reproduce** Open a task and settings while online. Return 503 from the API and reload. Restore the API. Sign out and confirm that saved query records are gone. Discover a worker update while offline or during an autosave and confirm that it waits. Stacked on Refs #18822. Related durable-write foundation: Refs #17960. The outage browser suite follows in a separate test PR. ## What Changed - Persist an exact read allowlist through TanStack Query dehydration and hydration. Save shell preferences, company/sidebar data, agent/project lists, plugin UI metadata, the first inbox page, and 20 recent tasks. Task UUID and identifier reads share one recent-task slot. - Namespace records by user and company. Retain eligible reads across normal memory garbage collection, expire each read after 24 hours, and discard them after a build change. Unstamped builds disable hydration and persistence while retaining sign-out cleanup. Use the existing instance-wide plugin contribution query in the user's root record. - Exclude session credentials and bootstrap data, embedded plan documents, comments, preview work products, provider credentials and environment values, browser capabilities, pairing data, other queries, and mutations. Restore only display identity from session data. Preserve safe agent provider/model display fields and configuration object shapes. Apply the projection on save and restore. - Clear saved reads on sign-out, cross-tab account changes, and definitive session denial. Keep revocation working on plain HTTP without crypto.randomUUID. Use per-user revocation revisions to reject delayed writes and stale records. Preserve a newer account. Remove every UUID/identifier projection and nested inbox copy of a denied task. - Detach shared GETs when the account changes or signs out. Load complete, freshly authorized agents before opening bulk settings from a display-only roster. Retire a pending bulk-editor result or error when row, select-all or clear changes the selection, including a clear/reselect of the same agent. - Require a fresh server lookup before account-bound writes, pending task creation, or mounted autosaves resume. This includes automatic task read receipts, inbox archives, workspace preferences and primary-agent choices, with optimistic archive changes gated too. Cancel the old write scope when the verified user changes. Release completed archives through account-independent cleanup without restoring old cached rows; an old operation cannot clear a newer hide flag. Retry unconfirmed read receipts after a fresh session succeeds for the original account, and remember confirmed receipts across refreshes. Track pending reads per task so opening another task sends its receipt immediately; revisiting reuses an outstanding receipt and its original account scope. - Defer a hidden-tab worker reload until connectivity, mutations, durable sends, and autosaves are safe. Synchronously count each unresolved operation across online rate-limit retries. Release a cancelled target’s unused dirty token when an editor changes tasks; keep stored drafts, admitted saves and queued retries. Cover autosave debounce, newer edits, navigation and undo. ## Verification - Current head `fe4f682c879` is merge-ready: all 54 check runs completed (52 successful, one skipped, one neutral), the legacy Snyk status is successful, and both `ci / verify` and `ci / e2e` pass in run `41752927822`. Greptile Apex reviewed this exact head at 5/5 with zero unresolved threads; the branch has no merge conflicts. All three PRs now meet the merge-ready bar; the stack remains unmerged pending operator approval. - Greptile Apex reviewed exact current head `fe4f682c879` at 5/5 with no actionable findings and zero unresolved threads (summary updated 2026-11-04 at 01:24:40 UTC). Both settings-account and pending-selection findings are fixed, replied to and resolved. The branch has no merge conflicts. Current-head CI now passes in run `41752927822`. - Seventeen new cases cover settings restored from actual saved query records, matching/changed/signed-out identities, denied verification, late crown responses without overwriting newer reads, and bulk-selection changes during successful/failed reads. Sixteen fail the preceding source and one preserves denial compatibility. Both settings paths reuse the existing account-bound mutation guard; its optional explicit QueryClient supports the workspace hook’s existing detached-host mode. Primary-agent optimistic updates begin only after verification and cleanup checks ownership and the cache revision. - Earlier CI run `41749456480` on prior head `2cf7516f18c` has one real server test failure: the owned-runtime case in `idle-completed-history.test.ts` could not reach its HTTP child on port 34757 within 10 seconds (99 refused probes). Its other 1,829 cases pass and three skip; the only other unsuccessful job is the downstream `verify` gate. The exact selected case passes locally, and the entire 75-case file passes in 72.19 seconds on the exact `2cf7516f18c` source tree with isolated PostgreSQL and a real child process on macOS arm64/Node 25.6.1. The Linux CI startup cause remains unknown; these local results are not a fix or CI acceptance. One failed-jobs retry with debug logging was accepted on that unchanged head, then superseded by the later settings fixes. Current-head CI remains required; no source change was made for this failure. - Current cache head `fe4f682c879`, based on read head `09d1db29dea` and master `683fa4725e9`, passes all 1,198 focused tests across twenty-eight files, UI typecheck, token gates and the enforcing query-error guard. Dependent browser head `228c1b68d30` contains this exact application source and passes standalone strict checking of all three browser specs. Current-head CI now passes in run `41752927822`. - Three new navigation regressions fail the preceding application source and pass with the fix. They open three cached tasks before any receipt resolves, leave the page before success or failure, revisit an outstanding read without duplicating it, and recover an older failed receipt after a fresh session succeeds while a later task remains pending. - The fixture repair passes all seven deadline integration cases and the complete server typecheck, including the server test layer. The old assertion deterministically fails at 23:59:06.888 UTC; its one-second-old run also fails the first-second-of-day probe. The corrected fixture passes noon and both midnight probes. The permanent three-case coverage uses seeded 2099 clocks. - Nine new cases cover archive success/failure after sign-out and another account takes over, both mounted and unmounted; cleanup ownership when a newer archive uses the same task; session-check 429 exhaustion followed by matching, changed or signed-out identity; and a failed read receipt followed by a new session read. They verify no old-cache restoration, toast or navigation, retained newer hide flags, no failed-check loop and no duplicate confirmed receipt. Seven fail the previous implementation; two compatibility cases retain account-change suppression. Initial test error-constructor and query-notification timing assumptions were corrected before the passing checks. - The inherited skills/app error-priority fix passes all 60 affected tests and UI typecheck on both application branches. Its twelve new regression cases fail the preceding read implementation. The persisted-cache suite includes 48 cases. - Six new regressions fail against the preceding application source and pass with the fix. Actual worker-update and mutation-cache listeners cannot reload a hidden tab between online 429 retries; the original create key survives, and the update reloads once after a receipt. Real self-hosted and Cloud sign-out work without crypto.randomUUID, with stamped and unstamped builds. Saved reads are removed and account write scopes are invalidated. An initial test-fixture type annotation error was corrected before the passing typecheck. - The 48 cache tests cover actual five-minute memory GC, the 20-task bound across saved and live reads, original 24-hour expiry, denial before session verification, unstamped cleanup, cross-tab revocation and delayed flushes. Private GETs cannot cross account transitions. Bulk edits require complete current agent data. Undo releases the update blocker while preserving the Saved indicator. - All 23 built-UI outage/write/loading cases pass in 4.5 minutes on exact integrated head `228c1b68d30`, containing current cache revision `fe4f682c879`: ten outage, six durable-write and seven loading cases. The run uses bundled Chromium, the built UI, a fresh local-trusted server and isolated embedded PostgreSQL on macOS arm64/Node 25.6.1. All three specs pass standalone strict TypeScript checking. Owned temporary server and database stopped. - Earlier browser runs exposed and fixed cached-header, native Storage cleanup and replay/session defects. A prior run passed 21/22 because real WebSockets contradicted the simulated HTTP outage; the fixture now closes existing socket pairs and holds new sockets through the same outage deadline. An older CI run exposed a disposed response in the Inbox route callback; pre-navigation snapshots remove that lifetime, and the current full run includes that repair. Earlier failures and their diagnosis remain in the discussion. The current full run passes. Broader historical workspace failures remain documented in Refs #18822; this does not claim a clean full workspace suite. - Current-head CI passes on `fe4f682c879`. Controlled local outages do not complete hosted authentication or a real Cloud restart/deploy acceptance. Commands for the current cache change and inherited read fix (the current focused run also includes src/lib/inboxArchiveCache.test.ts and src/pages/Inbox.test.ts): ```sh pnpm --filter @paperclipai/ui exec vitest run src/pages/CompanySkills.test.tsx src/pages/apps/AppNotConnected.test.tsx pnpm --filter @paperclipai/ui exec vitest run src/hooks/useDurableSubmit.test.tsx src/hooks/useDurableAutosave.test.tsx src/hooks/useSignOut.test.tsx src/lib/persisted-query-cache.test.ts src/lib/service-worker-updates.test.ts src/lib/service-worker-cache-policy.test.ts src/lib/pending-send.test.ts src/lib/issue-execution-policy.test.ts src/components/InlineEditor.test.tsx src/components/MarkdownEditor.test.tsx src/components/NewIssueDialog.test.tsx pnpm --filter @paperclipai/ui exec vitest run src/components/agents/AgentBulkRoster.test.tsx src/components/agents/AgentBulkSettingsSheet.test.tsx src/hooks/useAccountBoundMutation.test.tsx src/pages/IssueDetail.test.tsx src/components/TaskChatThread.test.tsx src/components/TaskChatThread.task-creation-order.test.tsx src/api/client.test.ts src/api/agents.test.ts pnpm --filter @paperclipai/ui exec vitest run src/hooks/useCoordinatorWorkspaceEnabled.test.tsx src/context/PrimaryAgentProvider.test.tsx src/components/CoordinatorWorkspaceSetting.test.tsx src/components/primary-agent/PrimaryAgentPresentation.test.tsx src/components/MobileBottomNav.coordinator.test.tsx src/components/CoordinatorSidebar.test.tsx src/pages/Issues.coordinator-routing.test.tsx pnpm --filter @paperclipai/ui typecheck pnpm check:token-gates node scripts/check-query-error-rendering.mjs --enforce --summary ``` ## Risks This deliberately saves task content on the local browser for up to 24 hours. The allowlist excludes credentials and ephemeral capabilities. Local storage can be disabled or full; persistence then fails without blocking the app. Restored data can be stale and never grants server access. A user or access change clears cached account reads and invalidates pending writes. The browser outage suite and real hosted deploy acceptance remain release requirements. There are no migrations or new dependencies. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with xhigh reasoning effort, tool use, and code execution. The context window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug report template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id - [x] The focused local tests for this change pass - [x] I have added tests and updated relevant documentation - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Bender (Fable) <bender@agents.paperclip.ing> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
b7ef307a65 |
fix(workspaces): wait for an ended writer's save instead of failing setup (#19074)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent runs can use a remote task workspace in a sandbox. One writer run at a time holds that workspace. > - When a writer run ends, its workspace save and sandbox release finish a few minutes later. The writer hold stays until then. > - A new run that started in that interval failed with `setup_failed`: "The workspace still has an unfinished writer or save operation." > - A live writer already makes the new run wait (`workspace_busy`, added in #18691). An ended writer that is still settling did not. > - This pull request reports the ended writer as the holder too, so the new run waits instead of failing. > - The benefit is that work continues automatically after a short settle interval. Users do not see a failed run and a blocked task. ## Linked Issues or Issue Description Refs #18691 **What happened?** A task run failed during workspace setup with `setup_failed` and the error "The workspace still has an unfinished writer or save operation." The previous run on the same task had ended a few seconds before. Its sandbox lease was released 4 to 17 minutes later. A staging soak recorded 31 of these failures over 14 tasks in about 29 hours. **What did you expect to happen?** The new run waits for the workspace, as it does when the previous writer is still running. It continues when the save and release settle. **Steps to reproduce** 1. Bind a task to a remote workspace whose authority has `dirty: true` and a `writerRunId` for a run with status `failed` that ended now. 2. Wake the task's agent. 3. The run fails with `setup_failed`. No retry is scheduled. ## What Changed - `reserveRemoteWorkspace` now throws `RemoteWorkspaceWriterActiveError` for an ended writer whose save or sandbox release has not settled. Before, it threw a plain 409 conflict. The error keeps the same message, status and `remote_workspace_writer_pending` code. It uses the writer's `finishedAt` as its last activity and sets `ended: true`. - Workspace preparation already converts that error to a `workspace_busy` deferral with the one-hour holder cutoff. It now passes `holderEnded` to the deferral. - `WorkspaceBusyDeferral` says "ended, but its workspace save has not settled" for an ended holder, instead of "is still running". - A writer that is missing or has no task still gets the plain conflict, as before. - `doc/execution-semantics.md` describes the wait and its cutoff. ## Verification - Red before the fix: the new test "waits for a writer that just ended while its save or sandbox release settles" failed. The run had `status: failed`, `errorCode: setup_failed`, `executionRecovery.kind: bootstrap`. This is the same result as the staging failures. - Green after the fix: `cd server && npx vitest run src/__tests__/heartbeat-remote-workspace-busy.test.ts src/services/remote-workspace-reservation.test.ts` (16 tests pass.) - Safety cases in the new tests: - The waiting run does not change the workspace authority. The unsaved writer's files and hold stay unchanged. - A writer that ended more than one hour ago still causes the unfinished-writer setup failure, with no scheduled retry. The wait is bounded. - Existing cases still pass: live writer wait with Stop and reassignment, the concurrent reservation race (`remote_workspace_writer_pending`), and the unused-reservation takeover. - Also run: `remote-workspaces.integration.test.ts`, `heartbeat-workspace-busy.test.ts`, `remote-workspace-save-concurrency.test.ts`, `workspace-recovery.test.ts`. All 6 suites pass (442 tests). - Not done: a live run on staging at this revision. The fix is not deployed. ## Risks - A task whose ended writer never settles now makes up to about one hour of short `workspace_busy` waits (one scheduled retry every 1 to 2 minutes) before it fails. Before, it failed at once. The end state after the cutoff is the same. - Some of the staging failures have other causes that this change does not fix: - A previous run that stays `running` with no output for more than one hour still causes this failure, as intended. - A dirty workspace whose writer ended without a saved result (for example a workspace finalization retry that never runs) still blocks the task after the cutoff. These need separate fixes. - No schema, API or UI changes. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, browser reads of staging run records) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
aa8372fd5d |
fix(accounting): post late usage to its owning run (#18991)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner sessions keep one provider conversation across task turns, so Paperclip accounts usage per run. > - Codex can report more usage for a turn after its run's charge was posted. > - The Runner then marked the session `reconciliation_required`. If another run was attached, it also failed that run. Core refused the delivery with `native_usage_after_finalization_requires_reconciliation`. > - That usage has already happened. Blocking the session cannot undo it, and the debt waited for an operator while the run totals stayed wrong. > - Codex totals are cumulative, and the Runner already knows the owning run in most cases. So the correct amount can be posted automatically. > - This pull request reports the exact increment with its owner, and posts it to the owning run as a separate cost event. > - The benefit is that run accounting becomes correct without stopping work, and an old run's usage can no longer fail a newer run. ## Linked Issues or Issue Description No issue exists. I searched open and closed pull requests and issues for "late usage", `native_usage_after_finalization_requires_reconciliation` and `codex_usage_accounting_integrity`. I found no duplicate. **What happened?** Code review found this while diagnosing session recovery failures. When Codex reported usage above the known total for a turn that was not active, the Runner blocked the session. If that turn belonged to a previous run, the Runner also failed the attached run with `codex_usage_accounting_integrity`. Core kept the receipt as debt, and could correct it automatically only if the token counts were unchanged. **Expected behavior** Late usage does not stop any run or session. The charge goes to the run that used the tokens, and it is auditable. **Steps to reproduce** 1. Finish a Codex turn on a native Runner session and attach the next run. 2. Have Codex send `thread/tokenUsage/updated` with a higher total for the finished turn. 3. On the unfixed code, the attached run fails and later attaches are refused. `late_codex_usage_requires_durable_reconciliation_before_or_after_handoff` encoded this behavior. **Paperclip version or commit** `master` at `04a24f4e06`. **Deployment mode** Any deployment that uses a native Runner with Codex. ## What Changed - **Runner:** late usage no longer sets `reconciliation_required` or fails a run. For a turn that belongs to a previous run, the Runner emits `usage.reported` with the increment in `runDelta` and the owner in `reconciliation.priorOrigin`. `runDeltaAvailable` stays `false`, so an older Core does not bill it to the attached run. - **`CodexRunUsage::absorb_late_usage`:** treats only usage above the known thread total as late, and keeps the attached run's delta unchanged. A report at or below that total adds nothing, so the sum across runs always equals the thread total and a replay cannot charge twice. - **Delivery:** `captureLateUsageForOwner` stores the increment as a late receipt on the owning run. The owner must be a finished run in the same company and native session. If the Runner cannot name the owner, Core uses the session's last finished run and records the attribution as `inferred`. - **Local Runner:** late usage stays with the run that is active when it arrives, or the next run. Usage reported between two runs stays in the next run's delta, so the total stays exact. - **Accounting:** `reconcileLateRunUsageReceiptInTransaction` posts a token increase as a new cost event with the key `heartbeat:<run>:late:<receipt>`. It updates the run totals so the accounting integrity check still agrees, records `cost.late_usage_posted` in the activity log, and queues budget evaluation. - **No more refusal:** `persistUsageReceipt` no longer throws after it saves late evidence, so late usage cannot wedge a Runner outbox. - **Covered receipts:** an older same-source receipt at or below the accepted totals, with matching billing, is cleared because a newer cumulative receipt already covers it. Conflicting billing stays as debt. - **Older Runners:** a zero-filled invalidation stays on the delivering run. - **Unchanged:** a newer lower total, a changed model or billing type, and per-model receipts stay as visible accounting debt for an operator. - **Unchanged:** an unexpected provider message during the warm attachment drain still fails closed. A separate change addresses that gate. - **Docs:** added "Late provider usage" to `doc/execution-semantics.md`. ## Verification - `cd packages/paperclip-runner/runner && cargo test --locked -p paperclip-runner-core --test codex_provider`: 99 passed, 2 ignored. - `cargo test --locked -p paperclip-runner-core --lib codex_usage`: 7 passed. This includes new tests for the out-of-order and repeated report cases. - `cd server && pnpm exec vitest run src/services/run-cost-accounting.test.ts src/services/native-runtime/native-usage-delivery.integration.test.ts src/services/native-runtime/native-recovery-usage-delivery.test.ts src/__tests__/cost-data-quality.test.ts`: passed against embedded PostgreSQL before the last review fixes, and the accounting and delivery files were re-run after them. - `npx tsc --noEmit -p .` in `server` reports no errors. - New tests cover exact and inferred owners, repeated delivery, late usage that arrives before the owner's charge posts, another session's run, an older Runner's report with no increment, a same-run increase, and a lower total. They also run the accounting integrity check after posting. - **Review:** Greptile Apex scored 1/5, 4/5, 3/5, 2/5 and 4/5 while findings were fixed, then 5/5 at the current head. - **Local Runner:** driver tests cover usage reported between two runs, including across a restore. No end-to-end local run has exercised it. - **Not run locally:** the full test suite. CI provides that coverage. - **Not verified live:** a read-only check of a test deployment found no late receipts in its history, so this path has only run against fixtures. ## Risks - **Run totals can rise after a run finishes.** This is deliberate. The correction is a separate cost event and an activity log entry. - **Budgets are corrected after the fact.** A late charge can put an agent over budget after later work has started. Budget evaluation runs after each correction, but it cannot prevent that work. - **Local sessions accept imperfect attribution by design.** Their late usage can land on the active or next run, because local sessions have no durable path for routing it. This also applies after a Stop or a pause. The total stays exact. - **Out-of-order reports can leave tokens on the wrong run in the session.** If a prior turn's report arrives after the attached run has already counted past that total, those tokens stay on the attached run. Moving them would need a lower correction to the attached run. The total stays exact. - **Inferred attribution can name the wrong run in the session.** The usage still belongs to that session and the total stays correct. The attribution is recorded as `inferred`. - **A session with no finished run charges the delivering run,** as an inferred late charge after its own charge posts. - **Rollout order:** deploy Core before the Runner. An older Core would keep this event on the attached run as incomplete usage, and would still refuse a same-run late receipt. - **Migrations:** this change has no migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be reworked. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use and code execution. A human directed the design and approved the scope. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
2c39a2866b |
feat(plugins): let managed config grant worker variables to a plugin (#19049)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A managed instance gets its plugins from the control plane, and people do not install them > - Some of these plugins need deployment values, such as a service origin and a capability > - The plugin worker environment is scrubbed, and only a fixed list of first-party providers gets deployment variables > - Core also sends lifecycle events and project instance secrets only to plugins with a saved config row > - So a provisioned plugin cannot work until a person saves configuration for it > - This pull request lets the managed config grant named variables to a selected plugin > - The benefit is that a distribution plugin can run with no tenant configuration, and Core does not need to name it ## Linked Issues or Issue Description No issue exists. This describes the problem. I searched open pull requests for related work and found no duplicate. **What existing behavior does this improve?** How a plugin provisioned through `plugins.autoInstall` receives its deployment settings. **Current behavior** `buildPluginWorkerEnv` forwards variables only from a fixed list that names first-party packages. A distribution plugin cannot be on that list without Core naming it. A plugin with no `pluginConfig` row gets no resource lifecycle events, and project instance secrets refuse it. **Proposed behavior** The managed config can include `plugins.workerEnv`. It maps a catalog key to variable names. The worker of that plugin receives those variables from the server process environment. The managed config can also include `plugins.enabledCompanies`. It maps a catalog key to company IDs. In those companies, the plugin gets lifecycle events and can use project instance secrets with no config row. **Reason and benefit** The control plane stays the only place that decides which plugin gets which values. Tenants do not handle these values, and they are not in the database or in the managed config document. **Alternatives considered** - Create a config or enablement row for each company. The control plane would need a new tenant write API, and a row can stay after the document removes access or return after a database restore. The document is never saved, like the other managed settings. - Give a grant access to every company. This has no per-company opt-in. - Put the values in the managed config document. The document does not hold secrets, and this rule stays true because it holds only names. ## What Changed - `parseManagedConfigEnv` accepts an optional `plugins.workerEnv` object. Each key must be in `plugins.autoInstall`. Names must be uppercase variable names, with no duplicates. - Startup fails if a name looks like a host credential, such as a name with `KEY`, `TOKEN`, `SECRET`, `PASSWORD`, `DATABASE` or `JWT` as a word, or if it controls how the worker loads code, such as `NODE_OPTIONS`. - `parseManagedConfigEnv` accepts an optional `plugins.enabledCompanies` object. Each key must be in `plugins.autoInstall`. Values must be lowercase company UUIDs, with no duplicates. - `managedPluginWorkerEnvGrant` and `managedPluginEnabledCompanies` return values by plugin key. A managed instance only starts a selected plugin from the current image, so the saved package path is not part of this check. It can be old or null. - `buildPluginWorkerEnv` forwards granted variables. This does not depend on the `environment.drivers.register` capability. - Plugin lifecycle delivery includes a plugin with no config row only for the companies the document names. It still skips a company that disabled the plugin. The runtime event listener follows the same rule. - Project instance secrets accept a plugin with no config row only for a company the document names. - A variable grant alone gives no company access. - `doc/plugins/DISTRIBUTION-PLUGINS.md` describes both sections. ## Verification - `npx vitest run` in `server/` on `managed-config`, `managed-plugin-worker-env`, `plugin-lifecycle-delivery`, `project-instance-secrets`, `plugin-database` and `bundled-plugins`: 201 tests pass. - The new tests cover a digest-verified distribution plugin, a stale saved package path, a catalog key named after an inherited property, a company the document does not name, a company that disabled the plugin, a named company that does not exist, a grant with no enabled company, and host credential names. - `tsc --noEmit` in `server/` shows no errors in the changed files. My worktree does not have the built `@paperclipai/paperclip-runner` package, so unrelated files show errors locally. CI runs the full typecheck. - I did not run the full test suite locally. CI runs it. - I did not test this change on a running instance. ## Risks - **Older builds:** a build without this change refuses a document with `plugins.workerEnv` or `plugins.enabledCompanies`, because it fails closed on unknown keys. A control plane must send these sections only to a build that supports them. - **Trust:** the granted plugin gets the variable values in its process. The image owner and the control plane already trust this plugin. The name check stops a mistake with host credentials, but a name check cannot prove that a value is not sensitive. Core cannot use an allowlist, because it must not name the variables of a private plugin. - **Company scope:** the control plane sets which companies a plugin serves. A new company needs a new document, and the server reads the document when it starts. Self-hosted instances do not change. - **Rotation:** a new value reaches the worker only when the server restarts. Rotation without a restart is not part of this change. - **Background calls:** a grant does not add proactive company access. A plugin that makes background calls with no invocation still needs a saved config for that company. - This changes core plugin behavior, and I did not discuss it in `#dev` first. Maintainers can tell me if they want a discussion before review. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
c7390bef20 |
fix(server): stop one slow copy-back from starving workspace retries (#19111)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run can accept its result before the workspace copy-back finishes. A background worker retries a failed copy-back from the retained sandbox without another provider turn. > - Until that retry succeeds, the run stays `running` and keeps its task's execution lock. User answers and wakes for that task stay queued. > - The worker checked one entry at a time (`batchSize: 1`, no concurrent checks). A slow sandbox export held every other task's due retry for the whole export. > - Work without a saved retry time used a deadline of `0`. Copy-backs that a restart interrupted always sorted ahead of retries that were due hours earlier. > - On a busy server that restarts often, scheduled retries never got to the front of the queue. Their tasks stayed stuck. > - This pull request runs up to four checks at once, one per task, and orders pending work by when it became pending. > - The benefit is that one slow export cannot stall unrelated tasks, and an old retry runs before newer work. ## Linked Issues or Issue Description Refs #17734 (added this worker), #18421 (the same concurrency pattern for native reply recovery) and #19063 (open; records why finalization failed a run). This pull request does not change their behavior. **What happened?** On a hosted instance, sandbox exports were slow for a few hours. A harness-state save timed out after about 5.5 minutes in 27 runs, and worker copy-back attempts took 2 to 34 minutes each. Eleven native runs then waited in `retryable_failure` (or `workspace_finalizing` with an expired lease) for more than three hours. Each one had exactly one copy-back attempt and no retry. The worker's operation receipts show its attempts ran strictly one after another. After each restart, the worker first took the copy-backs that the restart interrupted. They have no `nextAttemptAt`, so their deadline was `0`. The server restarted every 30 to 40 minutes, so the overdue retries never ran. One stuck run held a task whose user had answered a question card. The answer's wake stayed queued for the same three hours. **Expected behavior** A slow copy-back for one task does not delay due copy-backs for other tasks. Due work runs oldest first. New work without a saved retry time does not go ahead of older due retries. **Steps to reproduce** 1. Create two native runs with accepted results on different tasks. Put both in `retryable_failure` with an overdue `nextAttemptAt`. The first is older. 2. Make the copy-back for the older run block. 3. Start the worker. On `master` the second task's copy-back never runs while the first one blocks. The new worker test runs these steps against PostgreSQL. **Paperclip version or commit** `master` at `69dea878c0b`. **Deployment mode** Any server with native runs that use remote sandbox workspaces. ## What Changed - `server/src/services/native-runtime/native-workspace-finalization-work.ts`: - Remove `batchSize: 1`. Add `concurrentChecks: { limit: 4 }` keyed by `issue:<company>:<issue>`. Runs that share a task keep their copy-backs serialized. Ownership is still checked per run by the existing advisory lock and owner receipt. - Work without a saved retry time uses the coordinator's `updatedAt` as its deadline, not `0`. The value is clamped to now, because the database clock can be slightly ahead of the process clock. Saved retry times and lease expiry behave as before. - `doc/native-workspace-finalization-recovery.md`: one paragraph on worker concurrency and ordering. ## Verification - Failing first: "retries another task while an older copy-back is still running". On `master` the second task's copy-back never ran while the older one blocked (the test timed out after 15 s). Now the second task finalizes while the first is still blocked. After release, the first one also finalizes. - Failing first: "orders work without a saved retry behind older overdue retries". On `master` the interrupted copy-back had deadline `0` and sorted first. Now it sorts after the older overdue retry, and its deadline is not in the future. - `cd server && npx vitest run src/__tests__/native-workspace-finalization-work.test.ts src/__tests__/scoped-deadline-work.test.ts src/__tests__/server-startup-feedback-export.test.ts`: 3 files, 159 passed. After rebasing onto `69dea878c0b`, the first two files: 38 passed. - `cd server && npx tsc --noEmit -p tsconfig.json`: pass. - Not verified: the worker on a live instance under slow exports. The diagnosis comes from that instance's operation receipts and run events. ## Risks - Up to four copy-backs can now run at once on one server. Each one uses its own dedicated database connection and sandbox transfer. Live runs already finalize in parallel, so this is within the existing load pattern. - The batch size is now the default (25). A pass ends when its slowest check ends, but the other lanes keep processing the rest of the batch. - `updatedAt` changes on any coordinator write. A pending entry with new writes moves later in the order, but it stays due. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs, read-only database and log queries). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
7d1e9b3477 |
fix(recovery): retain exact warm native export sources (#17916)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native task recovery must save unfinished workspace exports before continuation. > - A prepared export can still refer to a sandbox with a warm reuse policy. > - The strict recovery inspector rejects that policy and exhausts source verification. > - Ordinary reuse must not give the same unfinished allocation to another run. > - An existing user Retry can authorize bounded file maintenance after exact ownership checks. > - This change retains that exact source and preserves the original failure record. > - Recovery completes only when files, stop evidence and policy settlement agree. ## Linked Issues or Issue Description **What happened?** A failed native run has a prepared `keep_running` workspace export. Its exact sandbox lease still has `reuse_by_environment` policy. Native export cleanup can preserve that policy while it stops the sandbox. Generic workspace recovery then rejects the source with `lease_policy_not_retained`. A saved user Retry remains blocked before file export. **Expected behavior** Keep unfinished exports out of ordinary warm reuse. With current user authority and an unchanged saved Retry, retain the exact source for one bounded maintenance generation. Preserve files and return continuation to normal admission. Keep all ambiguous sources held. **Steps to reproduce** 1. Prepare a native workspace export with a warm sandbox and `keep_running` disposition. 2. Save a normal board Retry for the failed source run. 3. Persist its first exhausted source-verification failure before file export. 4. Run the normal maintenance worker. Before this change, source policy rejection leaves the saved files unrestored. Refs #17773 for source diagnostics. Related #17839 records a missing preservation obligation during blocked restart. This change handles an existing prepared warm export whose verification already failed. ## What Changed - Exclude unfinished physical allocations from reuse and obsolete-lease cleanup across every historical lease row. Reuse a clean candidate or acquire a fresh sandbox through normal selection. Preserve exact same-run reattachment, workspace-less execution and ordinary task admission checks. - Preserve the newer prepared remote-workspace writer proof for native and legacy runs. Share its binding check between single-source recovery and one batched allocation read. - Keep persisted recovery-action IDs when constructing continuation source checks. Synthetic sources cannot grant snapshot receipt authority. - Permit one audited maintenance adoption for the first exhausted source-verification attempt. Require an unadmitted canonical saved board Retry, current user authority, unchanged assignment and latest source run. - Keep all company, source, allocation, workspace writer, controller and physical finalization ownership checks. The public read-only inspector keeps its strict policy check. - Pin the exact source for file maintenance. Preserve the previous generation and reuse the existing bounded recovery worker. Create no provider turn or new user request. - Restore the warm policy in the same transaction as verified native copyback completion. A failed write or changed settlement keeps a recovery obligation. - When local merge metadata is missing, settle the exact verified snapshot while keeping its unfinished allocation retained. Deliver the saved Retry through normal fresh isolation and attach the verified files. Keep the accepted result as history. - Require native copyback proof before reopening an accepted result's original workspace finalizer. - Validate snapshot receipts through one shared boundary for warm, retained and ephemeral sources. Bind the receipt to the current generation's operation IDs, company, issue, run, recovery action, lease and provider allocation. - Retire fixture input before removing recovery proof, and use the same configured heartbeat for HTTP and worker dispatch. Keep the exact catalog refusal and no-provider-work assertions. - Document the recovery contract in technical documentation. Leave capability maps unchanged for this bug fix. ## Verification - Candidate-selection RED: all six clean-candidate and fresh-allocation controls fail on unchanged `f54e7baf538` because a pending export aborts the whole acquisition. GREEN: the focused suite passes **15/15 cases** after candidate filtering. Both provider paths preserve the protected lease and never resume or destroy its allocation. Obsolete fingerprints cannot send that allocation to cleanup. Historical-row fencing, modern native/legacy writers, exact same-run reuse, workspace-less execution and the single-read allocation-history check remain covered. - The existing reusable-runtime selection cases passed **40/40**. Driver/capability contract suites passed **5/5**. Reusable service controls passed **2/2**. After merging current master, the combined lease/provider suites pass 61 cases and the recovery integration selection passes 36 cases, including the upstream inherited-session and resolved-interruption assertions. This gives 97 passing targeted cases on the final source. A duplicate test-app declaration from the automatic merge was removed while retaining the upstream assertions. - The source-history repair has meaningful red/green evidence: three valid retired-handoff cases could not complete under the old blanket history-row rejection. Settlement now uses the same full source inspector as adoption. It accepts exact retired handoffs and still rejects a predecessor that becomes active after export. The 26 focused history, rollback, snapshot and ambiguity cases passed before this last merge. - Server typecheck passes on the final merged source. Module boundaries and `git diff --check` pass. Full local repository tests and a successful current-head full build are not claimed. Current-head CI is complete on `0ef974d76483e11fb0b0304bcb05bbd3b56e2ef9`: both required gates pass, with 54 successful checks, three expected skips, one neutral Paperclip review check and a successful security status. [Full CI evidence](https://github.com/paperclipai/paperclip/actions/runs/41904339370). Greptile reviewed that exact head at **5/5** with no actionable findings. Its check completed successfully. [Ponytail also reviewed the exact head at 5/5](https://github.com/paperclipai/paperclip/pull/17916#issuecomment-6640083408). No review threads remain unresolved, and the branch is mergeable. - Earlier local broad runs exposed a dropped recovery-action ID in the merged source projection and deferred test input being revisited after its records were removed. Both were repaired. Later local disk exhaustion caused manifest and saved-file reserve failures; generated caches were removed only from these checkouts. The affected saved-file control passed unchanged after cleanup. The original logs are retained. No disk reserve or timeout was lowered. - On the prior head, the first [full CI attempt](https://github.com/paperclipai/paperclip/actions/runs/41899038530) received simultaneous runner shutdowns. Four jobs were cancelled, and two late jobs have lost-runner annotations with unavailable logs. The Runner job had five earlier ACPX failures whose full assertion report was lost on interruption. Their cause remains unproven. Identical Runner source passed the [#19004 control](https://github.com/paperclipai/paperclip/actions/runs/41898305521/job/126814429725), including all seven enabled tests in that file. - That attempt's [browser shard](https://github.com/paperclipai/paperclip/actions/runs/41899038530/job/126817404595) returned to the exact saved task URL, but its conversation was still loading at the unchanged heading deadline. The [artifact](https://github.com/paperclipai/paperclip/actions/runs/41899038530/artifacts/13358840931) is retained. All five Cloud-auth cases passed locally on the unchanged head and in the [same-source CI control](https://github.com/paperclipai/paperclip/actions/runs/41898305521/job/126814429966). The delay remains unexplained. One rerun of failed/interrupted jobs was submitted after the workflow completed. This new source revision requires fresh CI. - The earlier [original-head workflow](https://github.com/paperclipai/paperclip/actions/runs/40552228175) passed all 48 jobs, and prior head `f54e7baf538` received Greptile 5/5. These results do not establish readiness for the new head. The earlier Superagent scan was neutral after a pinned-base checkout failure; its external analysis remains unverified. - These tests use real PostgreSQL and controlled provider boundaries. Hosted rollout and original-task acceptance remain unverified. ## Risks - Maintenance can resume and stop the exact sandbox for file export. It cannot admit a provider turn, choose another allocation or bypass normal continuation gates. - A saved Retry alone is insufficient. A changed source, newer run, active writer, competing lease, lost user authority or previous adoption keeps the hold. - Policy restoration can fail. Completion and restoration now commit together, so normal recovery remains responsible for a retry. - Snapshot preservation cannot repair missing local merge metadata. Its old allocation remains explicitly retained. A snapshot receipt grants no permission to reuse or destroy that allocation, or to replay the accepted result. - The allocation check uses one projected join per candidate over the existing provider lease index. It adds no polling timer or recurring scan. A fresh fallback can create another sandbox while the unfinished source remains retained; it grants no authority to execute a provider turn. - This change does not establish the cause of a prior provider failure or prove that any specific hosted sandbox still contains its original files. ## Model Used OpenAI Codex, GPT-6 family, for the original change. OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), xhigh reasoning, for the merge and integration repairs. Both used repository tools and PostgreSQL verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Devin Foley <devin@paperclip.ing> |
||
|
Devin Foley
|
69dea878c0 |
fix(sentry): prove a native capacity retry from its successor run (#19051)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server sends unexpected run failures to Sentry as run-failure events > - A native Codex run can fail because the model is at capacity. Paperclip then schedules a bounded automatic retry > - #18295 keeps that scheduled retry out of Sentry. It requires the committed finalization fields on the failed run row > - On staging, the failed run row can lose `finalizationReasonCode` after the decision is committed. The retry run exists, but the report still goes to Sentry > - This pull request uses the successor retry run as the proof, and stops reading the run's own finalization fields > - The benefit is that a capacity failure that has a scheduled retry does not make Sentry noise. Exhausted and held failures still report ## Linked Issues or Issue Description Refs #18295 **What happened?** A native Codex run failed with `native_provider_overloaded`. The status decision scheduled the capacity retry, and the retry run was created. The server still sent a run-failure event to Sentry. The server release contained #18295. The failed run row had `finalizationPhase: "committed"`, but it did not have `finalizationReasonCode`. The retry run was created about two seconds before the failed run's terminal write. **Expected behavior** When the committed capacity decision creates a retry run, the failure stays local. The retry owns recovery. A later failure of the retry reports its own outcome. **Steps to reproduce** 1. Make a native Codex run fail with a typed `codex`/`server_overloaded` diagnostic. 2. Let the status decision schedule the `native_provider_overloaded` retry. 3. Let a late terminal write replace the run's `resultJson` with a version that does not have `finalizationReasonCode`. 4. Report the run failure. Before this change, Sentry gets an event. **Additional context** Only the committed `schedule_retry` effect in `status-decision-committer.ts` creates a successor with `scheduledRetryReason = native_provider_overloaded` (through `materializeNativeProviderCapacityRetry`). Thus the successor row is a stronger proof than the run's own finalization fields. ## What Changed - `server/src/services/run-failure-report.ts`: `isScheduledNativeCapacityRetry` no longer requires `finalizationPhase === "committed"` or `finalizationReasonCode === "native_provider_overloaded"` on the failed run. It still requires a failed native run with `errorCode = native_provider_overloaded`, the typed `codex`/`server_overloaded` diagnostic, a native task, no signal, and a phase that is not setup. It still requires the successor run for the same task with `retryOfRunId = run.id` and `scheduledRetryReason = native_provider_overloaded`. The successor lookup and its index use do not change. - `server/src/services/__tests__/run-failure-report.test.ts`: two new cases. A scheduled retry stays local when the run row has no finalization fields, and when the row has an earlier finalization phase. The "finalization is not committed" must-report case changes to "finalization has no fields and no retry run exists". That case still reports. - `doc/observability.md`: describes the successor-run proof. ## Verification - `cd server && npx vitest run src/services/__tests__/run-failure-report.test.ts`: 284 tests pass. - Red check: with the `master` version of `run-failure-report.ts`, the two new cases fail. Each one captures one Sentry event. All other 282 tests pass. - `npx vitest run src/services/native-runtime/native-provider-failure.test.ts src/services/native-runtime/status-arbiter.test.ts`: 73 tests pass. - I could not load `native-provider-overload.integration.test.ts` in my local sandbox because of a missing environment setting. CI runs it. - Exhausted, held, other-lane, other-task, no-task, legacy, setup, signal and untyped-diagnostic failures still report. The existing must-report cases prove this. ## Risks Low risk. The change removes two field checks. The successor-run check stays. Only the committed capacity decision creates a successor with this retry reason, so a failure without a scheduled retry still reports. No migration. No change to the retry behavior. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Nicky LeachandClaude Opus 5.5
|
0173110ee3 |
fix(native): point failed run attachments at the docs (#19052)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a later run cannot attach to a native session, the run fails, and a later run may start a new provider thread. > - The first failure is easy to miss. In 5 of 11 cases on a test deployment, the run ended with a different error before it reported the attach failure. > - People and agents who diagnose a failed run look at its log and events, so that is where an explanation is useful. > - This pull request adds a pointer there at the first failure, with the real cause and a link to the docs. > - The benefit is that a failed attach explains itself, and readers can find the open follow-up in #19047. ## Linked Issues or Issue Description Refs #19047. I searched open and closed pull requests and issues for `prior_attach_failed` and related terms, and found no duplicate. ## What Changed - **Log:** `executeHeartbeatRuntime` writes one stderr line to the run log when a same-run retry follows a failed `run.attach`. - **Events:** `heartbeat.ts` adds a `lifecycle` event with the same message and the code `native_run_attach_failed`, using the stored redacted failure. - **Helper:** `describeFailedAttachment` extracts the real cause and returns nothing for other failures. - **Docs:** "Session settlement after a run" in `doc/execution-semantics.md` now explains this and links #19047. - **Link target:** the pointer links to the repo docs and not the issue. Agents read these logs, and anyone can comment on a public issue. - **Unchanged:** error codes and retry behavior. ## Verification - `cd server && pnpm exec vitest run src/services/heartbeat/runtime-execution.test.ts src/services/native-runtime/remote-warm-session-recovery.test.ts`: 57 passed. - `npx tsc --noEmit -p .` in `server` reports no errors. - New tests cover the log line at the throw site, and the helper's output for attach failures and other failures. - **Not covered:** the new event in `heartbeat.ts` has no test of its own. It uses the tested helper. - **Not verified:** I have not confirmed that agents read this log when they diagnose a failure. - **Not run locally:** the full test suite. CI provides that coverage. ## Risks - **Failure text in the log:** when the failure has no structured message, the log line includes the whole failure string. It passes through the run log's identity redaction. The event uses the stored failure that the executor already redacted. - **Link stability:** the link depends on the docs heading. Renaming that heading would break the anchor but not the page link. - **Migrations:** this change has no migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be reworked. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use and code execution. A human directed the design and approved the scope. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
31266d7159 |
fix(logs): drain existing admissions before finalization (#19055)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Run logs retain execution output for operators. > - Object storage uses bounded admission and asynchronous uploads. > - A diagnostic can wait for capacity when its native execution ends. > - Finalization closed the writer before that existing admission could finish. > - This pull request joins those admissions before it closes the log. > - The benefit is complete accepted output with the existing storage bounds. ## Linked Issues or Issue Description Refs #18944 and #18327. **What happened?** A runner close diagnostic began object-log admission before its recorder retired. The call waited at the 8 MiB pending-byte bound. Completion then closed the writer. The waiting call failed with `Object run log writer is closed`. A valid final log hash covered the stored prefix, but the diagnostic was absent. **Expected behavior** Finalization must retain calls that started before closure. It must reject new calls after closure. Cancellation and storage failures must still settle without a success claim. **Steps to reproduce** 1. Use the actual recorder and segmented object store with a controlled conditional upload. 2. Fill the pending-byte bound. Start native warm-session retirement while the diagnostic waits for capacity. 3. Complete the run through the real completion handler. Release the upload. 4. Check the diagnostic bytes, final hash, source coverage, and warning rows. ## What Changed - Track active admission promises in each object-log writer. - Reject new producers when finalization starts. Join existing bounded admissions, then flush their tail before manifest closure. - Add actual completion coverage and store controls for failure, Stop, global capacity, and independent writers. - Clarify the finalization contract in the storage documentation. Keep feature maps unchanged for this routine fix. ## Verification - Before the fix, the actual completion probe lost a pre-retirement close diagnostic and emitted `object_log_persistence_failed`. Useful canonical output, source coverage, and the final prefix hash remained valid. - After the fix, the exact probe retains the diagnostic and checks the stored bytes and full hash. Ordinary completion and stalled local append controls also pass. - Focused store controls pass, including shared 64 MiB capacity, new-producer rejection, storage failure, Stop, and independent writer progress. - Production and test source at `bc03f8d53a721f01f80a12105a69d8583865e651`: 3 actual completion controls and 53 store, recorder, journal, and diagnostic tests pass. Run with `node node_modules/vitest/vitest.mjs run` from `server/`; select the four safety files plus the completion file with `-t "retired recorder"` in a separate invocation. - Initial treatment repository typechecks and build pass. Current map and module checks pass. A broader four-suite run had 99 passing assertions but failed its existing 30-second completion teardown hook. A later isolated completion run passed all 61 cases at the same limits. The original teardown cause remains unresolved; no timeout was changed. - Repository typechecks and build pass at that source commit. Current head `6652adc7f3379ae250f7b726dad6c58c8482410f` removes only the incidental map paragraph; production and test source are identical. All current-head checks are terminal: 54 successful check runs, one successful legacy status, and three skipped checks. Required `ci / verify` and `ci / e2e` pass. Greptile reviewed exact current head at 5/5. Both review threads are resolved. GitHub reports the PR clean and mergeable. Full repository tests stopped in chat shard 2: a GitHub recovery fair-batch test expected no provider requests but observed `GET /app/hook/config`. Its exact unchanged case passes in isolation. The original cause remains unresolved; this full-suite run is red. First-head remote server lane 2 failed a native-reassessment backlog test: its quiet check observed two active work items instead of zero. The exact unchanged case passes locally. That original CI cause remains unresolved; no job retry or timeout change was made. Other first-head CI lanes passed. The first native review rated the earlier source head 4/5 for the same removed map note. That is not a current-head assessment. Current GitHub rules require zero approving reviews and only the two CI checks above; a fresh native advisory re-review is optional and remains unavailable while its earlier run has an unresolved workspace/accounting obligation. That obligation is tracked separately and is not claimed recovered. - Removed the incidental map paragraph identified by Greptile and the native review. The follow-up contains no runtime or test changes. - This proves the local production-path mechanism. It does not attribute any specific hosted warning to this cause. Hosted replay and served revision checks remain separate acceptance work. ## Risks Finalization now waits for existing bounded capacity admissions. Storage cancellation and the existing operation deadlines still apply. Tests check that failed uploads and Stop settle, and that a separate writer can make progress. No capacity limit, timeout, or execution authority changes. ## Model Used OpenAI `gpt-6.1-sol`, through Codex, with reasoning and tool execution. Context window size is not exposed by the host. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (56 focused checks; the full-suite red remains documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
86afc2048b |
fix(native): admit follow-ups after an owner finished in this process (#19071)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner per-turn sessions in a remote sandbox keep one conversation across task turns. > - Before a follow-up starts a new Runner, it must prove that the previous owner's controller stopped. #18846 added a retirement path for this proof. > - The native controller runs inside the webserver process. Thus the owner's recorded controller PID is the PID of the webserver. > - When the same server process serves the follow-up, that PID is alive and has the same start time. The proof reads this as "controller still alive" and refuses. The follow-up then fails with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. This continues until a deploy replaces the process. > - This pull request uses the controller's per-process boot identity to recognize that this process recorded the owner. The process then uses its own session registry to prove that the owner's controller stopped. > - The benefit is that follow-ups on a long-lived server no longer fail before provider work. All other ownership fences stay as they are. ## Linked Issues or Issue Description Refs #18846 (retirement of recovered per-turn owners). Refs #19061 (names the refused retirement proof). The two changes are compatible. Whichever merges second needs a small rebase in `retireFinishedPerTurnRunnerdOwner`. **What happened?** On a test deployment, 17 runs over 14 tasks failed before provider work with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. Each run had a per-turn lifecycle in a Daytona sandbox and got a new sandbox. In each inspected case, the previous run on the task succeeded and committed its result. Its run profile had no `nativeControllerRecovery` pointer. Without that pointer, `evaluateNativeControllerTakeover` is not lease-fenced, so it uses process liveness. The recorded `controllerPid` is `process.pid` of the webserver. When the follow-up is served in the same process, `process.kill(pid, 0)` succeeds and the start time matches. The result is `controller_still_alive`, and retirement refuses. **Expected behavior** A committed owner that this server process finished, released, and no longer runs is proven stopped. The follow-up then starts a new Runner, as #18846 intended. **Steps to reproduce** 1. Seed a per-turn sandbox owner that committed its result and released its lease. Record this process's controller identity (`currentNativeControllerIdentity()`) as the owner. 2. Keep the owner's file-backed journal in the local session root, and remove the database recovery pointer. 3. Admit a follow-up in a new sandbox. 4. Before this change, admission fails with `prior_owner_terminal_state_indeterminate`. **Paperclip version or commit** Observed on `e107a75755`. The same code is on `master`. ## What Changed - `native-restart-recovery.ts`: add `isCurrentNativeControllerProcess(owner)`. It is true only when the owner's `controllerBootId` is this process's per-start boot UUID and `controllerPid === process.pid`. A recycled PID or another container never matches. - `retireFinishedPerTurnRunnerdOwner`: when the owner was finished in this process and no `activeNativeSessions` or `nativeSessionStartups` entry exists for its run, treat its controller as stopped. Otherwise, use `evaluateNativeControllerTakeover` as before. The same in-process check runs again just before the archive. - All other retirement proofs are unchanged: committed result, released lease, no recovery state, settled journal, answered tool inputs, quarantine copies, and new or replacement sandbox. - `native-recovered-owner-followup.test.ts`: the seed helper accepts the owner's controller identity. New cases cover same-process admission (with and without a recovery pointer), an exited legacy controller, another live controller, and the same PID with a different boot identity. - `doc/execution-semantics.md`: describe the same-process stopped-controller proof. ## Verification - **Failure before the change:** with only the new tests applied, `cd server && npx vitest run src/services/native-runtime/native-recovered-owner-followup.test.ts` fails 1 case: "admits a follow-up in the process that finished the owner: local journal only". It expected admission and received `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`, the exact error from the deployment. The safety cases already passed on the old code. - **After the change:** the same file passes 27 of 27. - Adjacent suites: `native-recovered-owner-followup`, `remote-warm-session-recovery`, `native-runner-restart-recovery.integration`, `native-restart-recovery`, `native-session-resume`: 210 passed, 20 skipped. - `native-session-executor.test.ts`: 1128 passed. - `tsc --noEmit -p server/tsconfig.json`: no errors. - **Not verified live:** this change has not run on a deployment. Run data came from read-only API calls only. No recovery was done on the deployment. ## Risks - **Second executor.** The new proof applies only when this exact process recorded the owner, and this process has no session or startup for that run. The database proofs still require a committed result and a released lease. Another live controller, or the same PID with a different boot identity, still holds the owner. Tests cover both cases. - **Scope.** The change applies only to the per-turn retirement path. The warm remote recovery path (`inspectRemoteWarmSessionRecovery`) is not changed. Warm handoff in the same process uses the in-memory idle owner instead. - **Hypothesis limit.** The deployment API does not expose finalization rows. Thus I inferred that the same process served each follow-up from the evidence: no recovery pointer, a local journal present, and no other failing proof. The local reproduction gives the exact error. #19061 can confirm the refused fence on the deployment. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding in Claude Code with tool use (shell, browser for read-only staging API reads, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c19c4348c5 |
fix(server): record inspector provenance for dynamic import callback errors (#17844)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/49/changes/paperclip/49 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server sends errors to Sentry when an operator sets a backend DSN. An unhandled rejection is captured in `strict` mode, so the server process exits after the capture. > - A hosted server reported `TypeError [ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING]: A dynamic import callback was not specified.` two times, eight days apart, on two unrelated releases. Each event stopped the process. > - The stack has no first-party frame. An anonymous `setTimeout` callback at `<anonymous>:1:N` calls `import()`. The column changed between the two events, so the source text changed. > - In local reproduction, only two paths make that frame: `vm.compileFunction` without a filename, and an expression that a client evaluates through the Node inspector. No shipped server code or production dependency uses either path. The current events give no data about the source. > - This pull request adds bounded provenance to that one error: the inspector state when the event is reported (never its URL), allow-listed Node flag names, the process ids, the main-thread flag and the uptime. > - The benefit is that the next event gives evidence about the source of the anonymous code. A `closed` state rules out an expression from a still-open inspector session. An `open` state needs other evidence before a debugger is blamed. ## Linked Issues or Issue Description No public issue exists. This section follows the bug report template. **What happened?** A hosted server process stopped two times with an unhandled rejection: `TypeError [ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING]: A dynamic import callback was not specified.` The stack is `importModuleDynamicallyCallback (node:internal/modules/esm/utils:280)` ← `Timeout._onTimeout (<anonymous>:1:20)` (first event) or `(<anonymous>:1:41)` (second event) ← `listOnTimeout`. The events came 3 minutes and 17 minutes after process start, on two releases that changed only skill documents and UI files. The Sentry event has no data that shows where the anonymous code came from. **Expected behavior** The error report gives enough data to investigate the source of the anonymous code: whether an inspector session was open in the process, and which inspector-related Node flags were set. **Steps to reproduce** 1. `node -e "require('node:vm').compileFunction(\"setTimeout(() => import('node:fs'), 1)\", [])()"` gives the same error and the same `Timeout._onTimeout (<anonymous>:1:N)` frame. 2. An expression such as `setTimeout(() => import('node:fs'), 1)`, evaluated through the inspector (`Runtime.evaluate`), gives the same error and frame. 3. `vm.Script` (`evalmachine.<anonymous>`), `eval` or `new Function` in a `vm` context (`eval at …`), `new Function`, indirect `eval`, `node -e`, `node -p` and stdin give a different frame or import successfully. **Paperclip version or commit** `master` at `0cb3fc7db591ba23e21fbbc70042bc44ec7d6f20`. The hosted events came from builds `f96a1e64` and `3e7c3ee9`, on Node v24.22.0 and v24.22.1. **Relevant subsystem** Server Sentry error monitoring (`server/src/sentry.ts`). ## What Changed - New `server/src/sentry-dynamic-import-provenance.ts`: `addDynamicImportProvenance(event, hint)` acts only when `hint.originalException.code` is `ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING`. It adds: - the tag `dynamic_import_inspector` with the value `open`, `closed` or `unavailable` - the context `dynamic_import_provenance` with `inspector`, `nodeFlags`, `mainThread`, `pid`, `ppid` and `uptimeSeconds` - `nodeFlags` holds only the names of `--inspect`, `--inspect-brk`, `--inspect-wait`, `--inspect-port`, `--debug-port`, `--disable-sigusr1` and `--experimental-vm-modules` from `execArgv` and `NODE_OPTIONS`. Flag values and all other flags are dropped. - The hook never sends the inspector URL, because the URL contains a session secret. It loads `node:inspector` lazily, so a Node build without the inspector still starts and reports `unavailable`. The hook never throws. Every other event is unchanged. - The module comment and the doc say that the state is evidence at report time, not proof of origin. - `server/src/sentry.ts`: `beforeSend` is now `enrichServerEvent`. It runs the existing run-link enrichment, then the new hook. - `doc/observability.md`: the default capture set names the new tag and context, and says what is never sent. - `.github/workflows/sentry-contract.yml`: the Real Sentry SDK isolation job also runs the new test file. - Tests in `server/src/__tests__/sentry-dynamic-import-provenance.test.ts`: - the real Node error from `vm.compileFunction` - a real open inspector session: the URL and its secret are absent from the event. The test closes only the session it opened, and it skips when a developer already has an inspector open. - flag filtering with values and other flags present in `NODE_OPTIONS` - other codes and other hint shapes stay unchanged - a failed state read keeps the original event - the real `@sentry/node@10.71.0` SDK sends the original rejection to the server `beforeSend`, and enriches only the matching event. With `PAPERCLIP_REQUIRE_SENTRY_TEST_SDK=1`, a missing SDK fails the file. ## Verification - With the audited SDK on `NODE_PATH` and `PAPERCLIP_REQUIRE_SENTRY_TEST_SDK=1`, as in the contract job: `run-failure-sentry-real-sdk.test.ts` and the new test file pass (17 tests). - Without the SDK: the new test file, `sentry.test.ts` and `sentry-links.test.ts` pass (52 pass, 6 skipped). - With `PAPERCLIP_REQUIRE_SENTRY_TEST_SDK=1` and no SDK: the new test file fails at once, as intended. - Red: with `beforeSend` set back to `addSentryRunLinks`, the real-SDK test fails (`expected [] to have a length of 1`). Green with the change. - Hosted CI on this head: typecheck, build, all test partitions and Real Sentry SDK isolation pass. - Not run locally: the full Vitest suite and browser suites. A live hosted check is not possible before deployment. ## Risks - Low risk. The hook acts only on one error code. All other events keep their current content and grouping. - The process still exits after the capture, as before. This change adds diagnostics. It does not fix the source of the anonymous code. - `pid`, `ppid` and uptime describe the server process only. They hold no tenant, user or request data. - If a later Node version changes the error code, the hook stops acting and the events are the same as today. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through the Claude Agent SDK, with tool use (shell, file edit, code execution, GitHub and Sentry connectors). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
a3313f6944 |
fix(ui): complete read recovery and enforce query error rendering (#18822)
## Thinking Path > - Paperclip lets people manage agent work through the board. > - The board reads task, agent, settings, plugin, and run data from the server. > - A deploy can interrupt those reads after content is already visible. > - Local error branches can replace that content with an error or show raw codes. > - This pull request moves the remaining reads to the shared recovery behavior. > - The query-error guard prevents new local error patterns from returning. ## Linked Issues or Issue Description **What happened?** Loaded lists, settings, task panels, and manual reads could clear their content or show errors during a server restart. Some reads also disabled the shared retry policy. **Expected behavior** Keep loaded content during a temporary failure. Show the shared connection banner. Retry reads after recovery. Replace content after a real access denial or missing resource, with readable copy. **Steps to reproduce** Open a task, settings page, or run log. Return 503 with `tenant_app_unavailable` from the API. Restore the API and check that content refreshes. Repeat with 403 and 404 to check the failure state. Related merged work: Refs #15754, Refs #15772, Refs #17960. This change covers remaining reads and enforcement. Browser persistence and the outage browser suite follow separately. ## What Changed - Repair an existing PostgreSQL deadline fixture at UTC boundaries. A daily cap defers a heartbeat until both its original interval and the next UTC day allow it. Keep the counted run in its seeded day and cover noon, just before midnight and the first second after midnight with explicit clocks and real PostgreSQL. Production scheduling is unchanged. - Integrate the saved list, settings, shell, task, and manual-read recovery changes against current master. Preserve the current task, Inbox, and run-log structure. - Add `useResilientFetch` for component-local reads. Preserve loaded data, cancel stale requests, and retry through the shared connectivity policy. Keep browser and pairing capabilities in memory. - Retain run log chunks across stream changes and catch up after recovery without duplicate chunks or stalled requests. - Keep genuine authorization and missing-resource failures visible. Translate known single-word codes such as `forbidden`. Show known skills and disconnected-app failures before sibling loading states. Keep Retry bound to its failed request. - Make the query-error guard enforce by default and run it in the CI policy job. Document the narrow exceptions and the read recovery contract. - Invalidate pending OAuth discovery when the operator leaves setup or its context changes. Keep reserved-popup cleanup bound to the current attempt. ## Verification - Current read head `09d1db29dea` is merge-ready: all 59 check runs are completed (54 successful, three skipped, two neutral), the legacy Snyk status is successful, and both required `ci / verify` and `ci / e2e` gates pass. Greptile Apex reviewed this exact head at 5/5 with no unresolved threads; the branch has no merge conflicts. The current CI run is `41748339602`. - Current read head `09d1db29dea` includes master `683fa4725e9`. The latest change touches only the existing server deadline test fixture. Application source remains the previously verified `86376fa751e` revision: all 60 affected skills/disconnected-app tests and both application UI typechecks passed, along with token gates and the enforcing query-error guard. Those UI suites were not repeated for this test-only repair. - The fixture repair passes all seven deadline integration cases and the complete server typecheck, including the server test layer. The old assertion deterministically fails at 23:59:06.888 UTC; its one-second-old run also fails the first-second-of-day probe. The corrected fixture passes noon and both midnight probes. The permanent three-case coverage uses seeded 2099 clocks. - Twelve new regressions fail against the previous implementation. Both skills variants and disconnected-app pages show either request's 403 or 429 while the sibling pauses or remains fetching. Permission failures stay visible. Retry stays usable for its failed request and recovers independently. An initial test-fixture query-key typo was corrected before the passing run. The finding is replied to and resolved. Greptile reviewed prior application revision `86376fa751e` at 5/5 with no new actionable findings or unresolved threads. The current test-only head has now received its own Apex 5/5 review with no unresolved threads, and its CI is successful as recorded above. - The preceding OAuth fix passed all 237 setup tests on both application branches. Its ten regressions prove that Back, Cancel, unmount, or context changes retire initial discovery without creating a connection, caching old identities, navigating an abandoned popup, or closing a newer attempt. - Earlier master integration `e992f770339` passed all 309 tests across eight affected Slack, recovery, retry, submission and task files, plus UI typecheck, token gates, enforcing query-error scan and seven guard tests. The integration preserves Slack workspace reconnect and shared recovery, readable execution errors, source-inspection company scoping, and current submission behavior. Newly inherited retained-file recovery controls keep cached data through transient failures and hide it after definitive denial. - The preceding cache head `97c0bc3966f` passed 1,038 focused cache/write/editor/account/task/API tests. All 22 built-UI outage/write/loading cases passed in 4.7 minutes on exact browser head `b7deaaee5da`, with strict checking of all three specs. The latest skills/app rendering fix is covered by its affected component suites; those broader cases were not repeated for this separate read path. - Historical broader UI, CLI and build results and unresolved aggregate server/DB/adapter failures remain in the PR discussion. Interrupted CI runs were audited and recorded separately; they are not counted as passes. Controlled local outages do not prove hosted authentication or a real Cloud restart/deploy. Real hosted acceptance remains required after merge and deployment. Commands for the latest fix: ```sh pnpm --filter @paperclipai/ui exec vitest run src/pages/CompanySkills.test.tsx src/pages/apps/AppNotConnected.test.tsx pnpm --filter @paperclipai/ui typecheck pnpm check:token-gates node scripts/check-query-error-rendering.mjs --enforce --summary ``` ## Risks This migration touches many read surfaces. The main risks are initial loading, request cancellation, retained authorization state, and run-log paging. The tests cover those states. There are no server, schema, dependency, or API contract changes. The follow-up browser suite and hosted deploy acceptance are required before release acceptance. Recovered commits retain their original authors. If squash-merging, preserve `Co-Authored-By: Bender (Fable) <bender@agents.paperclip.ing>` and the Paperclip trailer. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with xhigh reasoning effort, tool use, and code execution. The context window size is not exposed. Saved source commits keep their original authorship. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug report template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id - [x] The affected local tests pass; broader limitations are recorded above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Bender (Fable) <bender@agents.paperclip.ing> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
61b2455319 |
fix(runtime): preserve failed sources and durable checkpoint authority (#19038)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A volume-free web server must preserve retained native session data. > - An old failed follow-up can keep its own snapshot while its controller file still belongs to the preceding successful turn. > - The Runner correctly refuses to import that file as the failed turn's execution authority. > - That refusal also blocks migration when no warm supervisor is present. > - This pull request preserves the file after it proves the older run's lineage, and uses existing typed database authority when present. > - Failed history and files survive, while normal execution authority stays unchanged. ## Linked Issues or Issue Description Refs #19024, which preserves the related older-supervisor case. The duplicate search found no PR for this older-controller case. **What happened?** Offline native-controller migration stops with `native_runner_checkpoint_binding_mismatch` for a failed retained follow-up. The local controller still contains the preceding successful run. A per-turn source can have no supervisor file, so the older-supervisor archive path does not apply. Read-only checks matched the database snapshots and controller-file hashes between the original source and the isolated rehearsal copy. **Expected behavior** Preserve the available file as evidence through the existing encrypted archive. Require a valid older controller, a completed successful prior run, and the same company, agent, task, native session, Runner and provider thread. Keep successful sources and active owners outside this archival fallback. When a valid typed database checkpoint already exists, derive only its missing recovery metadata and ignore obsolete local projections. **Steps to reproduce** Create a successful native turn and a later failed follow-up in the same session. Retain the successful controller file. Keep the failed snapshot and admitted input in the database. Leave the per-turn source without a supervisor. Run offline migration. The new positive regression fails with the binding mismatch before this change. ## What Changed - Share the scoped older-successful-run proof with the existing supervisor preservation path. - Validate the older controller through the Runner parser and artifact checks before archive publication. - Archive this failed legacy source without changing its snapshot, status, ownership, queued work or session pointer. - Preserve the normal refusal for foreign, changed, active, successful, invalid and already-durable cases. - Permit evidence archival for a stale observed coordinator only with exact host-owned stop receipts for every remote lease, no current task ownership, no result and no retry deadline. Recheck the proof before publication. - Use execution start for lifecycle ordering. A queued follow-up can predate its prior turn completion, and a remote stop can predate a later recovery timeout. - Resolve destruction receipts through the existing reusable-lease handoff verifier. Reject stopped tips, broken chains and competing tips. - Prefer a valid typed database checkpoint over a stale or absent local warm supervisor. Add only missing recovery metadata after current admitted-input and checkpoint binding checks. - Document the evidence and execution-authority boundary. ## Verification - Positive regression: red before the owner change and green afterward. - `pnpm exec vitest run server/src/services/native-runtime/native-controller-migration.integration.test.ts server/src/services/native-runtime/native-controller-recovery.integration.test.ts server/src/services/native-runtime/native-harness-backup-storage.integration.test.ts`: 81 checks passed across migration, recovery and durable storage. - The checks cover storage failure, repeated migration, exact file restoration after old-home deletion, unchanged history and ownership, and refusal to treat the archive as executable recovery authority. - A second regression reproduced the archive fence refusal for an observed failed source with a valid stopped-remote receipt. Negative controls cover missing, foreign, old and revoked receipts, live leases, pending results, retry deadlines and task ownership. Revoking the receipt during upload prevents publication. - Two additional regressions cover a follow-up queued before the prior turn finished and a remote stop before a later logical failure. Both fail with the earlier timestamp comparisons and pass with execution-start ordering. Genuine execution overlap and old receipts stay blocked. - Two historical source snapshots and controller-file hashes matched in a read-only live-versus-copy check. The first candidate reached the archive fence and exposed the observed-coordinator case. That failure is retained. A fresh candidate replay is required for the expanded proof. It does not qualify the full packaged migration or hosted Railway handoff. - Existing typed-authority checks cover stale and missing supervisors, repeat migration, byte-identical checkpoint rows, unchanged snapshots/owners/history, old-home deletion and ordinary queued-turn recovery. Foreign Runner, session and snapshot controls remain blocked. - Handoff checks cover an exact destroyed successor, a stopped successor, a broken proof and a competing tip. - Isolated replay of the two retained typed-checkpoint sources imported only derived recovery metadata. Checks confirmed unchanged checkpoint rows (12 and 9 parts), history, snapshots, finalization owners and task-session pointers. The replay used three source-module overlays with a blocked provider network; it does not qualify the released bundle. - Final-head CI attempt 1 passed typecheck, build and the migration suites but failed one unrelated budget-concurrency assertion in `decision-models.test.ts` (one provider call observed instead of two). The original log is retained. The isolated local comparison cannot classify the timing failure because this checkout has stale linked budget schema exports. The one retry of that failed CI job passed, and final-head CI attempt 2 is green. The original failure remains retained; its cause is not established. - Full local typecheck, repository tests and build are not claimed. The server typecheck reports pre-existing linked declaration mismatches outside the changed files; the full clean CI typecheck has passed. This checkout uses linked generated declarations; clean CI is required before merge. Prior head |
||
|
DottaandPaperclip
|
4740d251b7 |
fix(onboarding): retain plan approval across follow-up decisions (#19043)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Onboarding asks a person to approve a saved plan before execution tasks can start. > - A person can then approve a separate proposal for the first execution step. > - The creation gate checks only that latest card for the plan target. It loses the earlier approval of the unchanged plan. > - Decomposition and native context also omit checkbox plan confirmations. > - This pull request validates the latest execution decision and the latest plan decision together. > - The benefit is that approved work can start while human, revision, and proposal gates remain effective. ## Linked Issues or Issue Description Refs #17914, #18160, and #18283. The related-PR search found no duplicate repair. **What happened?** After a person accepted a checkbox plan card and a later execution confirmation, native task creation and REST child creation returned 403. Accepted-plan decomposition returned 422 because its lookup recognized only standard confirmations. Native task context returned no accepted plan for the checkbox decision. **Expected behavior** The latest execution proposal can use an independently approved, unchanged plan. A pending or rejected latest proposal must still stop execution. The current plan must retain valid human approval of its exact revision. **Steps to reproduce** 1. Save a plan on a server-created onboarding task. 2. Create a revision-bound checkbox confirmation and accept it as a human. 3. Create and accept a separate, untargeted confirmation for the first execution task. 4. Create the child through the native tool or REST API. 5. Separately, try accepted-plan decomposition or read native task context after step 2. **Paperclip version or commit** The workspace's registered hosted release is Core `09c05c55050fede448196637cbaa543a154da76e`. The three regressions also failed against local source `3cdb367369f29692b81cf19b514a2abdba90734f`. **Deployment mode** Hosted native Runner with ACPX Claude. Local regression proof uses actual PostgreSQL, authenticated HTTP routes, native tool authority, and synthetic task data on macOS arm64. No customer data or credentials are in the fixtures. ## What Changed - Validate the newest ordinary proposal and the newest plan proposal separately. Keep the existing human-card and conversational provenance checks for each. - Require the accepted plan target to match the current document and revision. A newer pending, rejected, invalid, or foreign plan decision cannot fall back to an older approval. - Share persisted confirmation parsing across admission, decomposition, and native context. Validate checkbox selections against the saved options and limits. - Return `acceptedPlan.approvedOptions` with the user's saved selected IDs and option text on later wakes, including revision-pinned context. Keep unchecked defaults out of scope. - Add behavioral and safety regressions. Document the approval rule. ## Verification - Three behavioral regressions failed before the repair: child creation returned 403, decomposition returned 422, and native context returned `acceptedPlan: null`. All three now pass. - Two scope-retention regressions failed before the review fix: native context omitted the selected option after a later approval, with and without a pinned plan revision. Both now pass. Unchecked defaults stay excluded. - Current admission, native-tool, and accepted-confirmation replay suites: 131 tests passed. Existing first-task completion safety checks: 17 passed, including the atomic decision recheck. - Safety cases cover missing and rejected plan approval, agent-only decisions, foreign source runs, changed revisions, invalid checkbox selections, newer pending or rejected proposals, and task-creation replay. - Full repository `pnpm -r typecheck` and `pnpm build` passed locally on final head `7c0e8ab784`. - [Full final-head CI](https://github.com/paperclipai/paperclip/actions/runs/41911580531) passed: all Vitest shards, Runner checks, eight browser E2E shards, build, release registry, and canary dry run. The required `verify` and `e2e` aggregates passed. Storybook checks were skipped for this backend change; the file-selection job was skipped because full CI ran. - Earlier local typecheck attempts exposed Copilot verification during build preparation, stale declarations across the rebase, and two test-fixture typing errors. The assets were verified, declarations rebuilt, and fixtures corrected before the complete final-head pass. - Full local Vitest attempts were interrupted for the rebase and review repair. The duplicate final local run was stopped after full Linux CI passed. No complete macOS Vitest-suite pass is claimed. - A fresh Claude-provider acceptance run remains unverified: the local Claude account is signed out and no Anthropic API credential is available. The actual native tool and HTTP paths are verified with real PostgreSQL persistence. This PR does not deploy or recover the reported production task. - Greptile: 5/5 on `7c0e8ab784`, with the review thread resolved. ## Risks A later acceptance must not revive rejected or changed scope. Admission validates both current decisions and retains source, company, human provenance, and document locks. The extra plan-proposal lookup is bounded and limited to accepted, untargeted first-task proposals. Ordinary delegation adds no new proof lookup. The native context adds an optional `approvedOptions` projection. Revision-pinned context now performs the existing scoped approval read to recover that selection. No database schema, request, provider, or background worker changes are included. ## Model Used OpenAI Codex, exact model `gpt-6.1-sol`, max reasoning effort, with repository, shell, browser, and GitHub tools. The current session metadata confirms the model and effort. The context window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
be20d4ad97 |
fix(projects): keep projects on the storage chosen at creation (#19036)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A project stores its work in Core/local storage or in a storage plugin > - Today, Core finds the installed default each time it reads a project > - So installing a default storage plugin moves existing projects, without a saved change > - Agents that cannot reach the new storage then lose access to new workspaces > - This pull request makes Core apply the default once, when it creates a project > - The benefit is that a project stays on its storage until a person moves it ## Linked Issues or Issue Description No issue exists. This describes the problem. I searched open pull requests for related work and found no duplicate. **What existing behavior does this improve?** How Core chooses the storage for a project when a plugin claims `defaultForProjects`. **Current behavior** Installing such a plugin changes the storage of every existing project with no explicit choice. Disabling the plugin changes it back. Nothing is saved, so the project has no fixed owner. Execution environments and storage are separate contracts, so an agent can lose access to new workspaces on a project that it used before. **Proposed behavior** Core saves the storage choice on the project when it creates the project. A project with no saved choice uses Core/local storage. A plugin-backed project stays attached to its plugin. A Core/local project can move to a plugin. **Reason and benefit** A project stays on its storage until a person moves it, so installing or disabling a plugin does not change where work goes. **Alternatives considered** - Keep the default at read time and add a process that saves a choice on existing projects when a plugin arrives. This adds a background step and a race with new projects. - Add a new database column. The existing policy field already holds the choice, so a migration is not necessary. ## What Changed - `resolveProjectWorkspaceProvider` reads only the saved choice. It no longer applies an installed default. - `withDefaultProjectStorage` applies the default when Core creates a project, including a personal private project. With more than one default, a project with no explicit choice is refused. This is the existing conflict rule, now applied at creation. - `projectService.update` refuses a change of storage on a plugin-backed project. An update that omits the field keeps the attachment. - Plugin lifecycle delivery sends project events to a storage provider only for projects attached to it, and for deleted projects. Core acknowledges the other events for that provider. Other plugins continue to receive all project events. - The project settings control has no "Inherit" option. It is locked on a plugin-backed project. - `ProjectWorkspaceProviderConfiguration.source` no longer has the `"default"` value. - A project with no repository or folder ignores its saved storage choice when a task runs. A storage plugin cannot manage this type of project, so its tasks use a plain task directory. Without this change, the second run of a task reset the agent session, and remote agents could not use an empty task directory. - The plugin SDK README and the plugin authoring guide describe the new rules. ## Verification - `npx vitest run server/src/__tests__/project-workspace-provider.integration.test.ts server/src/__tests__/plugin-lifecycle-delivery.test.ts server/src/__tests__/plugin-workspaces.integration.test.ts packages/plugins/sdk/tests/workspace-provider.test.ts`: 93 tests pass. - `npx vitest run src/components/ProjectProperties.concurrency.test.tsx` in `ui/`: 9 tests pass. - `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter @paperclipai/ui typecheck` pass. - `npx vitest run` on `heartbeat-session-preparation`, `workspace-preparation`, `remote-workspace-source`, `project-workspace-provider.integration`, and `plugin-workspaces.integration` in `server/`: 231 tests pass. - I did not run the full test suite locally. CI runs it. - I did not test this change on a running instance. ## Risks - **Behavior change at upgrade:** an existing project that uses a plugin only through the installed default, with no saved choice, moves to Core/local storage after upgrade. Its plugin files stay in place, but new work does not use them until an operator sets `storageProviderPluginId` on the project. This pull request has no automatic backfill. - **Missed lifecycle events:** a provider does not receive events for a Core/local project. If the project moves to that provider later, the update event starts its reconcile. - **One-way move:** a person cannot move a plugin-backed project back through the API or the UI. This is intended. If a plugin is disabled, the project fails closed, as it does today for an explicit choice. - This pull request does not stop a person from assigning an agent to a project with storage that the agent cannot reach. That is follow-up work. - This changes core plugin behavior, and I did not discuss it in `#dev` first. Maintainers can tell me if they want a discussion before review. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
8a0e7ab511 |
fix(runner): preserve ordinary append for unscoped governed terminals (#19044)
## Thinking Path > - Paperclip manages work done by AI agents. > - A native turn can park on a saved human question or approval. > - Its terminal event can return a committed receipt before observers finish. > - That option requires an explicit turn ID at the event port. > - The protocol also permits events without a turn ID. > - The runner requested the option for those events, which the port rejected. > - This change keeps those events on the ordinary append path. ## Linked Issues or Issue Description Refs: #19025. **What happened?** The native governed-wait observation can accept a schema-valid `item.completed` without a turn ID. If the later terminal also has no turn ID, the runner compares two absent IDs and requests `deferPostCommitObservers`. The event port rejects that option. The run then fails with `governed_wait_provider_terminal_unproven`. **Expected behavior** Only a terminal with an explicit turn ID can request the early receipt. An unscoped terminal must keep the ordinary append behavior and await its observers. The event port must retain its turn identity check. **Steps to reproduce** 1. Create a pending human-only interaction from the run. 2. Pass a schema-valid completed tool event without a turn ID through the real governed-wait observation. 3. Supply an interrupted terminal without a turn ID during the cancellation drain. 4. Run the actual native runtime with the PostgreSQL event port. The parent code rejects the deferral request and reports a failed terminal. **Paperclip version or commit** Reproduced on `4ea53f9eb5a24de10db3dfda18456132987c4af8`. The follow-up was identified by a review published after #19025 merged. No observed provider emission or original-task recovery is claimed. I checked current master and related PRs before this correction. **Deployment mode** Built from source. The regression uses real PostgreSQL and a deterministic provider boundary; it makes no model call. ## What Changed - Require a truthy terminal turn ID before the runner requests deferred observers. - Keep ordinary append behavior, the port identity check and all timeouts unchanged. - Exercise the production governed-wait observation with a persisted pending interaction and the real event port. - Retain exact-turn observer and checkpoint controls. Add a direct rejection control for unscoped deferral. - Document that unscoped terminals await their observers. ## Verification - First meaningful red: the actual observation, native runtime and PostgreSQL port fail on the unmodified parent runtime. The new case reports `NativeProviderTerminalFailure` in 143ms; this is not a setup or deadline failure. - Focused regression and authority controls: 12 passed, 69 filtered. Full runtime suite: 203 passed. Full PostgreSQL port and timing suites: 84 passed. - The tests leave the human-only interaction pending with continuation policy `none`, accept one result, and start only one turn. Existing scope, invalid evidence, checkpoint, abort, duplicate and restart controls remain. - Exact candidate: `8840ea2c9c308e5a60423e67fb3612d24029ba84`. Repository-wide `pnpm -r typecheck`, `pnpm build` and `pnpm check:module-boundaries` passed on this exact head. `pnpm test:run` is still in progress and is not reported as passing. Current-head CI passed: 55 success, 3 skipped and 1 neutral, including required `ci / verify` and `ci / e2e`. Greptile and Ponytail both reviewed this exact head at 5/5 with no blocking findings. The native review was source-only. No unresolved review threads remain. - No UI, database schema, agent permission or mapped user entry point changes. No deployment, live retry or provider replay was performed. ## Risks Unscoped terminals retain the existing observer wait and can still fail if it exceeds the existing drain deadline. They do not gain an early receipt. The change does not supply a missing turn ID or weaken source, company, run, session, cancellation or transaction checks. A PRP terminal is not physical-stop or accounting-completion proof. ## Model Used OpenAI `gpt-6.1-sol`, with code and shell tools. The host does not expose the reasoning mode or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
3cdb367369 |
fix(acpx): report how the sandbox agent process session ended (#16366)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Hosted `claude_local` runs use the ACPX engine. The agent process
runs in a sandbox, and a host-side process session bridge carries the
ACP stream.
> - Some hosted turns fail with only "The connection to Claude was
lost." (`acpx_turn_failed`, terminal session category `connection`). The
failure report shows one pending `execute` tool. It does not show why
the agent process went away.
> - The bridge already records how the session ended: the remote exit
code and signal, or a host proxy close. But it writes that evidence only
to the run log. The run failure report never receives it.
> - Without it, we cannot tell an OOM kill (`SIGKILL`) or a provider
process exit from a bridge or proxy close. Each cause needs a different
fix, so we must not guess.
> - This pull request keeps the first terminal evidence on the bridge
handle and copies it, as closed values, into the run failure report's
execution context.
> - The benefit is that the next recurrence names the cause, and the
correct fix can follow.
## Linked Issues or Issue Description
**What happened?**
Hosted `claude_local` runs on staging and production failed with "The
connection to Claude was lost." One run failed about 4 minutes in, 7–10
seconds after its last ACP event. Another failed after about 3.5 hours
with no ACP events for most of that time. Both had one pending `execute`
tool. The report had no information about how the sandbox agent process
ended.
**Expected behavior**
The run failure report says how the sandbox agent process session ended.
Examples: the remote process exited with a code, the remote process was
killed by a signal, or the host proxy closed.
**Steps to reproduce**
1. Start a hosted ACP `claude_local` run on a sandbox target.
2. Kill the agent process in the sandbox while a tool runs, for example
with `kill -9`.
3. See that the run log has an `ACP process session terminal:` line, but
the run failure report has no matching field.
**Paperclip version or commit**
`master` at
|
||
|
|
53735db7de |
fix(retry): resume verified preparation failures (#19004)
## Thinking Path > - Paperclip must preserve saved agent work while proving earlier execution stopped. > - A deliberate task Retry can observe a newer failure while an older preparation still holds execution. > - That older preparation can also have a finalized empty object log despite never reaching the provider. > - Binding only the latest run and requiring every log field to be absent keeps valid recovery deferred. > - This change follows the exact persisted predecessor chain and binds current Retry authority to eligible holds. > - It recognizes only an exact finalized empty-log receipt within the existing strict preparation proof. > - One fresh user turn can then start with saved work and the existing ownership, stop and accounting checks. ## Linked Issues or Issue Description Refs #18989, #18860 and #18997. The first waits for predecessor cleanup during automatic dispatch. The second selects the committed native session owner. The third retains admitted Retry authority when no source recovery action exists. None passes current board Retry intent to a held failed predecessor. A search of open repairs found no duplicate. ROADMAP.md was checked; this is a recovery bug fix. **What happened?** A failed native preparation retains a legacy run row and an inherited session. Its remote workspace is later saved and verified. An automatic successor also fails during preparation. Board Retry observes that newer run, but the effective hold belongs to its predecessor. The Retry does not bind the predecessor's recovery generation. The native historical-source guard then keeps the request deferred. **Expected behavior** Carry the deliberate Retry through its verified same-company, same-task and same-agent persisted predecessor chain. Start one fresh user turn after every source passes current execution, stop and saved-work checks. Keep incomplete or unrelated proof held. Preserve prior work and recorded action outcomes. **Steps to reproduce** 1. Fail native preparation with an inherited session and a remote workspace. 2. Finish its real workspace recovery and stop receipts. 3. Save a newer failed automatic successor with `retryOfRunId` bound to that source. 4. Submit ordinary task Retry for the newer run. 5. Before the change, no successor is admitted. The request remains deferred behind the older hold. **Paperclip version or commit** The regression fails on master `039e0dad2c51c80251e1302a19c5566e039c357a`, which includes #18989. **Deployment mode** Source checkout on macOS arm64 with real PostgreSQL, HTTP task Retry and the assembled heartbeat service. The remote filesystem boundary uses temporary files. Hosted rollout and an original-task Retry are not yet verified. This PR incorporates the reviewed empty-log follow-up #19021, preserving its commits and regression controls. Both parts are required for hosted preparation recovery, so the combined revision is qualified and deployed together. ## What Changed - Accept a finalized zero-byte `s3_segments` log only when its reference binds the exact company, agent and run, its digest matches empty content, compression is false, and no output progress or nonempty excerpt exists. Add a real-recorder HTTP regression and 14 negative controls. Unknown or nonempty logs stay held. - Read a complete persisted Retry chain in one recursive query, limited to 32 runs. Reject missing, cyclic, newer, unfinished, cross-company, cross-task and cross-agent sources. Transcript hints grant no authority. - Bind eligible historical recovery records to the current Retry intent. Keep complete or running recovery budgets. Leave successful ancestors and mismatched recovery-source records unchanged. - Recheck that same chain when selecting an ancestor's fresh-user strategy. - Admit only a held ancestor through the shared native preparation proof. Keep the closed preparation failure codes, no-provider state, profile, input, result, output and accounting checks. Treat inherited session input as prior context and accept the exact bounded host start marker. Verify current ownership, saved work and stop receipts before forcing a fresh session. - Add HTTP and PostgreSQL regressions and document the rule. - Give the HTTP Retry nudge and explicit test worker the same real heartbeat facade and provider boundary. Establish the revoked catalog receipt before either can dispatch. ## Verification - Combined integration head `91f021f233151192f69dd7a035d083abcc67c47c` requires fresh complete CI and review; qualification is in progress. - Repair validation: all **six** recovered preparation provider-boundary cases pass. The catalog cases still require the exact access refusal, zero provider inputs and zero provider turns. The success case still checks preserved file bytes and one native turn. - [Previous exact-head CI](https://github.com/paperclipai/paperclip/actions/runs/41883421120/job/126785508136) failed the catalog case with `remote_workspace_transport_unsupported`. The route created a default heartbeat that could race the injected worker and allocate outside the controlled transport. The repaired test shares its real heartbeat across both dispatchers and holds the agent start lock until the revoked catalog receipt is committed. It tracks and drains both Retry nudges during teardown. It changes no production access check or expected refusal. - The first broad local reproduction hit a full local disk. It passed 201 cases and failed 86, including explicit manifest free-space errors. Generated build caches were removed from these isolated checkouts. The owning run passed 670/671 cases. Its one failure was the saved-Files loader refusing insufficient temporary disk space. After cleanup, that unchanged test passes in isolation. The final shared-facade and start-lock version passes all six dispatch cases. Both legacy continuation suites pass all 29 cases. The final fixture teardown also retires deferred input before removing recovery proof, so global queue sweeps cannot revisit earlier cases. All seven dispatch and saved-Files controls pass together after that change. No disk reserve, assertion or timeout changed. The initial local typecheck also stopped on `ENOSPC` while materializing provider assets. Server typecheck now passes after cleanup. No test timeout or reserve was lowered. - Current-head CI is complete: 52 successful checks, three expected skips, and a successful security status. Both required gates (`ci / e2e` and `ci / verify`) passed on `0adc2e38103533dd45bcc56f957ac019642c57b1`. [Full CI evidence](https://github.com/paperclipai/paperclip/actions/runs/41898305521). Greptile scored final head `0adc2e38103533dd45bcc56f957ac019642c57b1` 5/5 with no blocking findings or unresolved threads. The earlier verification below describes the previous head. - RED on unchanged master `039e0dad2c`: the certified-preparation sibling passes, while the inherited-session case admits zero successors instead of one. A second isolated RED on latest base `8f06bd4e17`, including #18997, uses the exact final three-node test fixture. The HTTP route returns 202, then the test observes zero successors instead of one. All baseline production files remain unchanged. The initial dependency/setup failures are separate from these behavioral failures. - The positive cases cover the failed-failed-successful chain. They keep the completed result unchanged, verify the original saved file and validate fresh-session continuation authority. Repeated HTTP input and concurrent resume calls keep one successor. - PostgreSQL controls accept a complete 32-run chain. A 33-run chain and a newer-created predecessor return no usable ancestry and do not bind the older recovery record. - Safety controls cover company, task and agent scope, conflicting canonical bindings, missing and cyclic edges, transcript hints, live execution and controllers, unknown stop, failed saving, unsafe writer state, Stop, actor and generation changes, independent holds and mismatched recovery records. - Ancestor controls retain provider-event, native-input, session-checkpoint, controller-recovery, result, process, runtime, accounting and unknown-result guards. Host start and warning prefixes use exact origins, sequence and payloads. A fifth extra event is rejected. - The first complete owning run found 22 introduced admission failures. The initial historical-native allowance also applied to exact-source Retry. It admitted contradictory preparation evidence. The fix now requires a canonical ancestor and the shared strict startup proof. The original direct-source assertions remain unchanged. - The next focused run passed 82 cases, including both catalogue-warning controls. One new result fixture lacked the database's required owner tuple. Two existing controls reached their original 30-second limit. The fixture now uses the real native task and completion-contract owner tuple. A controlled nine-case run passes the result case, both unchanged timed-out controls, three failure-code controls and all three positives. The earlier timeout causes remain undetermined. All failed-run logs are retained. - Complete owning suites passed on previous head `137a8eb3febf41052e112fc3c4ffe1abb92bcab7`: all 675 tests across workspace recovery, task Retry, explicit continuation and legacy continuation. The command exits successfully with normal teardown. This includes every earlier introduced admission failure and the two unchanged timeout controls. Its base includes the merged #18997 authority repair and #18982 rolling-copy packaging repair. The rebase retained both test imports at their one adjacent-import conflict. - Node 24.19.0 and pnpm 9.15.4. Module boundaries and `git diff --check` pass. The initial PostgreSQL start lacked package library links after install with scripts disabled. The documented hydration restored the isolated dependency; that setup failure was not a behavioral red. - Previous-head [full typecheck](https://github.com/paperclipai/paperclip/actions/runs/41883421120/job/126785506215) and [full build](https://github.com/paperclipai/paperclip/actions/runs/41883421120/job/126785506094) passed. Earlier attempts received synchronized runner shutdown signals and a lost-runner annotation. The shutdown initiator remains unknown. A CI owner used the documented hosted-runner failover for attempt three and restored the original routing value after capture. That attempt exposed the catalog failure above. Source workflows and check requirements remain unchanged. The current head now has green CI, Greptile 5/5, no unresolved review threads, and no merge conflicts. Hosted rollout and the original task are not yet verified. - Combined recovery proof: 24 of 24 cases passed on this exact head in 43.13 seconds, with normal teardown and unchanged assertions and limits. They cover the real recorder, all 14 empty-output rejection controls, ancestor and direct Retry, saved messages, the configured native provider, and verified stop. The targeted TypeScript graph also passed; the preceding integration step passed 95 safety/provider/message cases and all 49 task Retry cases. - The incorporated #19021 was independently red/green with the real recorder: unchanged predecessor code returned HTTP 202 but admitted zero successors. Its 18 focused cases, all 690 owning recovery cases, full typecheck/build, 1,904 local chat cases, complete CI and Greptile 5/5 passed before integration. The unchanged CLI startup lane passed one justified retry; no hosted acceptance is inferred from those results. - Additional local qualification of the unchanged workspace fixture alongside the native bootstrap fixture passed 288 of 289 cases. One manual-admission HTTP case ended with `socket hang up` after 451 ms, without an assertion or HTTP status. That unchanged case passed once in isolation at the original 40-second limit, with normal teardown, and passed in exact `0adc` CI in 17.2 seconds. The earlier disconnect cause remains unknown; no assertion or timeout changed. - An earlier CI attempt failed the newly merged native ingress fixture because its shared Node command failed strict permission qualification. The current base includes #19020's owned executable fixture repair; no production permission check was relaxed. - Integration with current master had one documentation conflict. An automatic test merge also duplicated an app declaration; that syntax failure was retained and fixed by reusing the configured real HTTP app. Both new message-state assertions remain intact. Workflows and lockfile retain master's contents. ## Risks - A damaged or overlong chain remains held. The helper does not truncate it and grant partial authority. - The new query runs only for a deliberate Retry or its ancestor strategy check. It adds no background polling, table, credential or provider policy. - A fresh intent is not stop or file-recovery proof. The existing owners retain those checks. Prior effects remain fenced, and no old result is replayed. - Local tests do not prove hosted rollout. Live acceptance must use the served fixed revision and the task's original saved work. ## Model Used OpenAI GPT-6 through Codex for the original change. OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), xhigh reasoning, for the merge and CI repair. Both used repository tools and PostgreSQL verification. Context-window limits were not exposed by the harness. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Devin Foley <devin@paperclip.ing> |
||
|
DottaandPaperclip
|
22735cfcea |
fix(native): preserve governed terminal receipts during observer settlement (#19025)
## Thinking Path > - Paperclip manages work done by AI agents. > - A native turn can yield to a saved human question or approval. > - The provider must stop, and its terminal event must be saved. > - The old drain waits for both the event transaction and its later observers. > - A slow observer can make a saved terminal look unproved. > - This change separates the committed receipt from owned observer work. > - Checkpoint and result acceptance still wait for that work to settle. ## Linked Issues or Issue Description **What happened?** A native governed wait can fail with `governed_wait_provider_terminal_unproven` after its exact interrupted terminal is already in PostgreSQL. The event port waits for postcommit accounting, output and projection callbacks before it returns the receipt. Those callbacks can cross the provider drain deadline. **What did you expect to happen?** A committed terminal proves the event transaction. Its observers remain separate work. A checkpoint or result must not be accepted until those observers settle. A pending human-only question must keep its audience and continuation policy. **Steps to reproduce** Run the real PostgreSQL control-plane regression. Commit a same-turn interrupted terminal. Hold its postcommit observer beyond the existing provider drain. The parent code reports an unproved provider terminal. The fixed code retains the committed receipt and joins the observer before checkpoint or result acceptance. No overlapping repair was found in the current master or open-PR search. Related [#18977](https://github.com/paperclipai/paperclip/pull/18977) covers final conversation-response selection. It does not change this receipt boundary. The source reproduction is consistent with the observed failure. The exact hosted provider code is unavailable, so it does not prove that every observed failure used this branch. ## What Changed - Add an explicit terminal-only option to the internal event port. Ordinary appends keep their existing behavior. - Return that receipt only after the normal transaction and company, agent, run, session, source and revocation checks. - Retain the full observer promise. Join it before checkpoint and result writes. Keep its original rejection and observe it after an aborted join. - Reuse the existing provider drain and checkpoint deadlines. Preserve duplicate and restart replay without another model turn. - Document the receipt and duplicate-observer replay boundaries. Keep human-only and no-continuation cards unchanged. - Assert the actual Codex facade requests deferral only for its interrupted terminal. Isolate empty skill inventory in monitor-only process fixtures while retaining their real scheduler, lifecycle, selection snapshot and three-second deadline. ## Verification - Meaningful parent red: actual native runtime and real PostgreSQL event port. The terminal row commits, but a held observer causes `NativeProviderTerminalFailure` with `governed_wait_provider_terminal_unproven`. - Current candidate `0db24ae74ee42fdf92035d8407fc095e9f73cc6a`: full runtime suite, 203 passed. Real PostgreSQL control-plane, timing and monitor suites, 103 passed. Runner TypeScript and full layered server typechecks passed on the unchanged candidate source tree. The controls cover uncommitted and foreign evidence, explicit false and failed terminals, held and rejecting observers, abort and controller revocation, duplicate receipts, and new-port recovery. The real pending human-only card stays unanswered with continuation policy `none`. Recovery does not call `startTurn` again. - `pnpm -r typecheck`, the production server build (including the real Runner release binary), module boundaries and vendor dependency verification passed on `6d75`; the follow-up changes only tests and documentation. Diff checks pass on the current candidate. The prior `399ad3dc2b` repository-wide `test:run` was stopped as obsolete during the general-server phase after preserving its progress (exit 130, incomplete); its repository build was not reached. Its six completed chat shards passed 1,904 cases, but this is prior-head evidence and not a full-suite pass. Full current-head CI is required before merge. - CI attempt 1 on `6d75` failed one real three-second monitor-settlement assertion; the aggregate verify gate consequently failed. Other jobs passed. The actual CI merge tree was `3037bf6db162da3b0a9172046360401f8449ebc5` (`6d75` into `89703c3`). An unchanged full-file replay reproduced three timeouts. Original-deadline inspection found the legacy process still preparing; local phase timing measured cold skill inventory at 6.629 seconds. The original CI did not retain that phase detail, so its exact subcause remains unproved. The fixture correction supplies empty skill entries for these skill-free process tests and still invokes the selection-snapshot callback. It preserves all assertions and the three-second deadline, with no new drain wait. All 21 cases pass; deliberately restoring the old blocked-status rejection still fails the real blocked-check case with 409. No CI retry was issued. - The prior current-head native review was 4/5 with no production correctness finding. Its requested direct facade-options assertion and duplicate-observer wording correction are included. Current-head CI is green (55 success, 3 skipped), including required verify and E2E; current-head Greptile is 5/5 with no actionable findings. The old native advisory remains tied to `6d75`, and its requested findings are addressed. No new native assessment is claimed or required by protection. - The first draft broke three existing transaction-start timing controls. The no-debt path now keeps the original synchronous start, and all three pass. Two draft type errors were corrected. A further real-PG red found an extra default-append assertion after commit. Removing it restores the original committed receipt after a late abort; the same aborted signal still cannot write a checkpoint or result. Original setup and assertion failures remain in private verification receipts. - Normal merge of current master `3a2249dd5768313f04b2aa3860dfd363492eda16` has no conflicts. The affected production source blobs did not change in that baseline merge. - Local proof uses Node 24.22.2 on Darwin arm64, a real PostgreSQL instance and deterministic provider-boundary fixtures. It makes no model call. No deployed replay or original-task recovery is claimed. ## Risks Deferred observers must remain owned. A missed join could accept incomplete accounting or output. A rejected or stalled observer must prevent acceptance, and an abort must prevent late checkpoint/result writes. The tests exercise these boundaries. The port keeps the existing best-effort trace-publication handling. This change does not turn a PRP terminal into physical-stop or accounting-completion proof. ## Model Used OpenAI `gpt-6.1-sol`, with code and shell tools. The host does not expose the reasoning mode or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
e107a75755 |
fix(native): decide session settlement from live state (#18992)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner sessions keep one provider conversation across task turns, so a later run attaches to a session that an earlier run settled. > - Three gates decide if a session is settled: Core's journal check, the Runner's attach check, and the Runner's drain of provider messages after a turn. > - Each gate kept a list of harmless events and refused anything else. Every new harmless event became an outage. > - An example: Codex replays a settled turn's usage on `thread/resume`, so `run.attach` refused every session that restarted after its idle timeout. > - Events after a terminal describe the past. Refusing the session cannot undo them, and it hides them. > - This pull request asks a different question: is anything live or waiting right now? Past events are recorded and no longer block. > - The benefit is that harmless events stop breaking session continuity, with no event list for Core and the Runner to keep in step. ## Linked Issues or Issue Description No issue exists. I searched open and closed pull requests and issues for `run.attach requires a settled Codex provider session` and `prior_owner_terminal_state_indeterminate`. Related work: #18890 added one more accepted event to Core's list. This change replaces that list. #18991 handles late usage and is independent of this change. **What happened?** On a test deployment, 11 runs failed in 21 days when `run.attach` refused with `requires a settled Codex provider session with no pending events`. In all 11, the four pending events were two startup audit events, `session.resumed`, and a `codex_resume_usage_snapshot` diagnostic. The diagnostic is what the attach check refused. The retry then failed with `prior_owner_terminal_state_indeterminate` in 6 of them, which hid the first failure. **Expected behavior** A session with nothing live or waiting accepts the next run. If attachment fails, the run reports the real cause. **Steps to reproduce** 1. Finish a Codex turn on a native Runner session. 2. Wait for the idle timeout so that the provider process stops. 3. Start the next run. Codex replays the usage for the settled turn on `thread/resume`. 4. The attach fails. `durable_backend_attaches_after_a_settled_resume_usage_snapshot` reproduces this on the unfixed code. **Paperclip version or commit** Observed on `8fd917aedf`. The same check is on `master` before this change. **Deployment mode** Cloud deployment, with a native Runner in a plugin sandbox environment. ## What Changed - **Runner attach:** `attach_run` refuses only for live or waiting state: no thread, a closed session, a session that needs reconciliation, an active or ambiguous turn, an active goal, a pending tool call, or an unanswered provider request. The error now names the reason. - **Pending events:** they no longer block attachment. They stay in the retained queue, and the Runner commits them under the prior authority before it rotates. `session.resumed` is still discarded because it is superseded. - **Runner drain:** `drain_completed_turn_tail_for_warm_attachment` no longer needs a list of known messages. It still fails on a message that names another turn, on `turn/started`, on a provider request, and on a process exit. - **Core:** `isSettledRemoteWarmControlState` accepts any authenticated event after the terminal unless it starts work (`turn.started`, `turn.accepted`, `run.attached`, `tool.execution.started`, `semantic_tool.input`, `runtime_request.created`), or a goal snapshot or update shows an active goal or work in progress. All identity, source and protocol checks are unchanged. - **Visibility:** the next run's log names the event types that were recorded after the previous run finished. - **Real cause:** a same-run retry after a failed attachment now fails with `prior_attach_failed: <original failure>`, instead of `prior_owner_terminal_state_indeterminate`. - **Docs:** added "Session settlement after a run" to `doc/execution-semantics.md`. ## Verification - `cd packages/paperclip-runner/runner && cargo test --locked -p paperclip-runner-core --test codex_provider`: 102 passed, 2 ignored. `--lib`: 478 passed, 2 ignored. `--test durable_recovery`: 3 passed. - **Failure before the fix:** `durable_backend_attaches_after_a_settled_resume_usage_snapshot` failed on the unfixed code with the same error as production. - `cd server && pnpm exec vitest run src/services/native-runtime/remote-warm-session-recovery.test.ts src/services/native-runtime/native-session-executor.test.ts -t "settled remote warm control evidence|recovers a blocked|automatically recovers only a proven settled local session|retained|warm"`: 476 passed, 680 skipped by the filter. - `pnpm exec vitest run src/services/native-runtime/native-session-resume.test.ts`: 57 passed. - `npx tsc --noEmit -p .` in `server` reports no errors. - **Real data, read-only:** I decrypted the 11 failed runs' stored events on a test deployment and printed only event types, field names, and thread and turn IDs. All 11 show the same four events. - **Audit of live state:** I read the Runner's provider state and request tracking to build the attach check. It is a code audit. I have not proved that every kind of live work sets a flag. The Runner holds active turns, ambiguous starts, tool calls and provider requests in state today. - **Not covered:** the wiring that throws `prior_attach_failed` has unit coverage for its message only, not an end-to-end test. - **Review:** Greptile Apex scored 4/5, then 5/5 at the current head. - **Not run locally:** the full test suite. CI provides that coverage. - **Not verified live:** no live run has used this change. ## Risks - **Fail-open for unknown live work:** an event type that starts work, but is not in the short list and sets no state flag, would now pass. This is the main tradeoff. The previous list failed closed, but it caused real outages. - **More events reach the journal after a terminal.** The usage replay diagnostic is now delivered rather than refused. These events are informational, and they are named in the next run's log. - **A failed attachment is not retried on the same run.** Some refusals could clear on their own, such as a turn that finishes, and some attach errors are not about live work, such as a failed provider restore. The run now reports the cause, and it needs another run to continue. On a test deployment, the next successful run after each failure used a new provider thread and a new execution workspace. A later change could add same-run retry. - **Rollout order:** deploy Core before the Runner. An older Core would refuse the new events that a new Runner commits after a terminal. - **Migrations:** this change has no migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be reworked. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use and code execution. A human directed the design and approved the scope. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
25b5a072cc |
fix(recovery): let messages continue recovered preparation (#19026)
## Thinking Path > - Paperclip must preserve work while allowing people to continue failed tasks. > - A preparation failure can retain an ID from an earlier provider session. > - Workspace recovery and host settlement already prove that the failed attempt never dispatched a provider. > - The preparation classifier also required that inherited session ID to be absent. > - That extra condition kept new messages, saved messages, and Retry behind a resolved recovery hold. > - Classify execution from the existing execution evidence and admit one fresh turn through normal gates. ## Linked Issues or Issue Description Refs #18908, #19004, and #19021. The latter two address historical Retry ancestry and empty object logs. This repair addresses a current source with an inherited session and ordinary user messages; it needs neither ancestry nor an object log. ROADMAP.md and related open PRs were checked. **What happened?** Sending a message to a failed task saved it in the queue, but did not start another run. The failed native preparation had recovered files and durable host settlement, plus `sessionIdBefore` from prior work. The classifier treated that input as execution evidence and kept the recovery hold. **Expected behavior** Admit the new user turn once the existing saved-work and stop checks pass. Preserve previous failures and files. Show the current run instead of the old error, and consume messages when execution starts. **Steps to reproduce** Fail Runner preparation before runtime selection. Preserve its remote workspace. Retain an inherited `sessionIdBefore`. Send an ordinary message or click Retry. The three new HTTP regressions accept the request but admit zero successors before the repair. **Paperclip version or commit** Base `3bd52a9fa0`. **Deployment mode** Authenticated hosted Runner. Regression verification uses macOS arm64, Node 26.4.0, real PostgreSQL, HTTP routes, and real temporary workspace archives. The external sandbox transport and native provider boundary are controlled in the execution fixture. ## What Changed - Stop requiring the inherited session input to be absent from otherwise verified failed preparation. - Cover new messages, previously saved messages, Retry, request replay, one successor, recovered files, and current task presentation. - Exercise the existing preparation safety controls with inherited context, including a new output-session refusal. - Exercise native execution and finalization with the inherited session input. - Document the continuation rule. ## Verification - RED: all three new HTTP cases admitted zero successors before the production change. - All three final HTTP cases pass. The 41 preparation safety cases, including the new output-session refusal, pass with inherited context. - The native execution fixture passes with inherited context. It reads restored files, writes the continuation result, runs one provider turn, persists the accepted result, empties the queue, and resolves the notice. - Initial positive assertions expected an empty queue while the fixture deliberately held the agent's slot, then expected a null optional wait field. The final assertions use the actual queue contract and retain execution and presentation checks. - `pnpm build`, `node scripts/typecheck-low-memory.mjs`, module boundaries, and `git diff --check` pass locally. The final rebased head `56d2ef26` passes the four HTTP/native journey cases. All 634 tests in the four complete owning recovery, Retry and continuation suites pass. Current-head full CI passes: 53 successful checks and three expected skips, with no failed or pending checks. The full local command has the documented timeout below. - The full local command passed its first three chat shards, then stopped in shard four on a 15-second timeout in `answers each finding thread in its own run instead of one unattestable batch` (267 other shard cases passed). The exact unchanged test passes alone (about nine seconds of test work) and all three full CI chat groups pass on this head. It uses a custom adapter, so the changed native-preparation proof is inapplicable. The timeout cause remains unknown; the full local command is not claimed green and its later groups were not reached. - Initial CI attempt: all other checks passed, but server shard 2/12 failed on the unchanged `idle-completed-history.test.ts` owned HTTP runtime test. The ten-second readiness probe returned ECONNREFUSED. The source and runtime have no diff from the base and this test does not use failed-preparation admission. The exact local case passes; the underlying readiness cause remains unknown. Logs and annotations are retained at [the failed job](https://github.com/paperclipai/paperclip/actions/runs/41897539206/job/126811385017). The single failed-job retry passed on the same head, including the dependent verify aggregate. This proves recovery of that check; the original readiness cause remains unknown. - Greptile reviewed the exact head `56d2ef26` at 5/5 with no actionable findings and zero unresolved threads. Hosted recovery remains unverified until the repair reaches the serving revision. ## Risks The risk is mistaking earlier context for a stopped current execution. Current provider session, process, controller, native input/result/output, accounting, exact lease stop, workspace epoch/generation/writer, actor, assignment, pause and approval checks remain enforced. No migration, new polling, API, credentials or execution policy is added. ## Model Used OpenAI GPT-6 through Codex, using reasoning, code editing, shell tools, and browser diagnostics. The exact serving snapshot and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (634 owning-suite cases and final-head journey cases; full-local timeout is documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |