mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 13:24:55 +01:00
cb5d7e440cf7745618f8b08df98d33e2bfc6f2f2
339
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
e37ab23dbc |
fix: hold an agent's new work while its provider usage limit is active (#19246)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Routine-driven agents run on the native Runner through ACP harnesses such as Claude > - When the agent's provider account runs out of allowance, every new wake for that agent fails the same way > - The ACP harness reported the exhausted allowance as a generic "terminal limit failure" with an `unknown` category, so Paperclip could not tell it from a context or turn limit > - Paperclip recorded a reset time only for the failing run's own startup retry; other wakes of the same agent kept dispatching, each spent about two minutes on workspace setup, and then failed > - This pull request classifies the limit, keeps its reset time, and holds the agent's new work until that reset > - The benefit is that a limited agent stops burning runs, shows a clear "waiting for usage limit reset" state with a time, and resumes its queued work after the reset ## Linked Issues or Issue Description Refs #13148 Refs #13379 **What happened?** A routine-driven Claude agent on the native Runner failed 29 times in about 15 hours. 20 runs failed with `native_provider_terminal_failed` and "ACP agent reported a terminal limit failure." The recorded provider failure had category `unknown`. 6 runs failed with `provider_session_open_rejected`. Their `session.open` result is `command_execution_failed` with classification `provider_rate_limit`, and the provider failure is HTTP 429 `rate_limit` ("would exceed your account's rate limit"). The startup retry of one such run recorded a cooldown until one hour later. A new wake for a different task of the same agent dispatched 16 minutes later, spent two minutes on workspace setup, and failed with the same 429. Separately, 4 runs failed with "Project authentication conflicts with this agent's managed AI connection". That message did not say which file or setting conflicted. The same message was also shown when the check could not finish. **What did you expect?** A provider usage limit is classified distinctly. Paperclip records the reset time when the provider reports one. New work for that agent waits for the reset with a visible reason. Queued work resumes after the reset, and no doomed run starts. A configuration conflict is reported only when one is found, and the message names the file to fix. **Steps to reproduce** 1. Use a native Runner agent with a Claude ACP harness whose account is rate limited or out of usage. 2. Let a routine or assignment wake the agent on several tasks. 3. See each run fail with a generic terminal limit failure or a `session.open` rejection, and see new runs keep starting. ## What Changed - Runner: The ACPX turn now receives the typed AIR session failure (`onTerminalSessionFailure`) through the runtime host and the ACPX port. The sidecar adds the provider's title to the provider failure message. Only a rate-limit or quota title becomes `rate_limit` or `quota_exhausted`. Context, turn and configured budget limits stay `unknown`. The failure details stay private. - Server: A native `turn.failed` with a `rate_limit` or `quota_exhausted` provider failure is now `native_provider_usage_limit`, the same code as the Codex usage-limit terminal. - Server: `agentProviderLimitCooldown` (extracted from the startup retry) also reads a reset clock in the provider's wording, such as "resets 2am (UTC)". The clock parser moved from the recovery service to `provider-quota-reset.ts` without changes. - Server: New `heartbeat/provider-limit-wait.ts`. Before any workspace or provider work, a queued run for an agent with an active provider limit is cancelled as `provider_limit_wait`. The run records the reset time, whether the provider reported it, the category and the run that reported the limit. Paperclip schedules one retry of the same wake at the reset through the existing AI connection wait lane. That lane does not consume the failure retry budget. Manual runs, recovering runs, conversation turns and pool-routed connections are not held. - Server: The execution projection shows "Waiting for usage limit reset" with the reset time and the next step. - Server: Only a found project authentication conflict is a configuration failure. Its message names the project file and the settings to remove. A check that cannot finish (timeout or another exit code) is now an ordinary failure, not a configuration conflict. - Docs: `doc/execution-semantics.md` describes the agent-level limit wait. ## Verification Red before the fix, green after: - `server/src/__tests__/heartbeat-provider-limit-wait.test.ts`: An automation wake after a 429 `session.open` rejection dispatched the adapter before the fix. Now the run is cancelled as `provider_limit_wait` and a scheduled retry exists at the reset. After the reset passes, the retry is promoted and the work succeeds. A manual run, a later success, expired evidence and an unclassified terminal still dispatch. - `native-startup-retry.integration.test.ts`: "reads the reset clock from an ACP terminal limit failure's provider wording" (it fell back to one hour before). - `native-session-executor.test.ts`: An ACP `turn.failed` with a typed `quota_exhausted` or `rate_limit` failure is a usage-limit terminal; `unknown` is not. - `packages/paperclip-runner/src/drivers/provider-failure.test.ts`: A typed AIR limit title is kept and classified; context, turn and budget limits are not; details do not leak. - `execution-projection.test.ts`, `ai-connections.test.ts`: The wait label, and conflict versus unfinished check. - Safety cases: `runtime-host.test.ts` and `codex-runtime-adapter.test.ts` forward the callback. The existing startup retry, provider capacity, recovery classification, legacy reconciliation, retry scheduling and execution configuration suites pass. Commands: - `cd server && npx vitest run src/__tests__/heartbeat-provider-limit-wait.test.ts src/__tests__/ai-connections.test.ts src/services/native-runtime/ src/services/recovery/ src/services/execution-projection.test.ts src/services/legacy-execution-recovery.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/__tests__/heartbeat-execution-configuration.test.ts` - `cd packages/paperclip-runner && npx vitest run src/drivers/provider-failure.test.ts src/drivers/acpx/runtime-host.test.ts src/drivers/acpx/codex-runtime-adapter.test.ts src/cli/acpx-runtime-sidecar.test.ts` - `tsc --noEmit` for server and paperclip-runner; the ACPX sidecar and provider-failure contract checks pass. I did not run a live provider limit on staging. The live evidence is the staging run records for the affected agent, which I read only. ## Risks - The limit evidence is per agent, not per account. If the board changes the agent's AI connection, automatic wakes can still wait until the old reset (at most one hour when no reset was reported, and at most one day). A manual run is not held, and one successful run clears the evidence. - A new wake that arrives during a limit is now cancelled and retried at the reset, instead of failing. The task stays in its status, and the run shows the reset time. - The Runner change takes effect when the provider pack is rebuilt and deployed. Until then, the server still holds new work using the existing `rate_limit` evidence from `session.open` rejections. - A turn terminal with a typed rate limit or quota is now `native_provider_usage_limit` instead of `native_provider_terminal_failed`. Both are terminal and board-owned. The recovery text now asks the board to restore provider usage capacity. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser reads of staging run records) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
26391707fb |
fix(runner): never settle an OpenCode reply truncated at the output limit (#19242)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native OpenCode runs turn the provider's last assistant text into
the task's visible reply
> - A model can loop until it reaches the output limit. The provider
then reports `finish: "length"`
> - The driver used that cut-off text as the final answer, and the run
succeeded with a broken reply
> - This pull request treats a response that stopped at the output limit
as incomplete
> - The benefit is that users get the accepted completion summary or one
corrected answer, never a truncated wall of text
## Linked Issues or Issue Description
**What happened?**
In a QA staging run, an OpenCode agent on
`openrouter/deepseek/deepseek-v4.1-flash-1017` called `paperclip_finish`
correctly. Then it produced about 32,000 output tokens of repeated text
(`E2E_QUIP_QUIP_...`) until the output limit stopped it. The driver
selected this truncated text as the settled final answer. The run
succeeded, and the task reply was wrong.
**Expected behavior**
A response that the provider stopped at the output limit is not a
complete answer. If an accepted completion report exists, its summary is
the reply. If not, the driver asks for one concise final answer. If that
also stops at the limit, the turn fails.
**Steps to reproduce**
1. Run a native OpenCode turn where the model calls `paperclip_finish`.
2. Let the next assistant message run to the output limit (OpenCode
reports `finish: "length"`).
3. Before this change, that message text is emitted as the final
`agentMessage`.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
03abe1f1a1 |
fix(native): keep the transport cause on a closed notification stream failure (#19240)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs read provider events from a shared session notification stream (Codex, OpenCode and ACPX drivers use it) > - When that stream closes, the driver fails the session with `notification_transport_failed` > - The run error was only "Provider notification transport failed closed." The real cause was only in a diagnostic event > - In a staging soak, many runs on OpenRouter and Grok agents showed this error. Operators read it as a provider fault. The diagnostic events showed a host-side cause: a stopped sandbox plugin worker ended remote runner supervision > - This pull request keeps the redacted, bounded cause on the terminal failure message, and does not name the provider > - The benefit is that a failed run shows why it failed, so operators do not chase the wrong provider ## Linked Issues or Issue Description **What happened?** Native runs failed with `native_provider_terminal_failed` and the error "Provider notification transport failed closed." The run did not show a cause. All failures in one cluster happened inside one minute, on several agents and models (OpenRouter DeepSeek, Grok). Each run's `harness.diagnostic` event had the real cause, for example `runner_process_termination_unconfirmed: runnerd process supervision failed: Cannot call "environmentExecute" — worker for "<plugin>" is stopped`, or `native_adopted_runner_identity_unverifiable: runner liveness could not be revalidated`. The sandbox duplex transport reported a `provider_exit` loss about 30 seconds before (the read-only process probe budget). **Expected behavior** The terminal run error keeps the redacted cause. It does not blame the model provider when the cause is host-side runner supervision. **Steps to reproduce** 1. Open a native Codex-protocol session and start a turn. 2. Make the notification stream throw (for example, a supervision error). 3. Read the `session.failed` and `turn.failed` messages. Before this change they are the bare generic text. ## What Changed - `codex-session-notifications.ts`: the `notification_transport_failed` terminal message is now `Notification transport failed closed: <cause>`. The cause is redacted first, then whitespace is collapsed and it is bounded to 1,000 characters. The `Error: ` prefix is removed. The code and `recoverable: false` do not change. - New regression in `codex-protocol-integrity.test.ts`: one `session.failed`, the cause and code are present, the secret is redacted, the length is bounded, and `turn.failed` has the same message. - `doc/execution-semantics.md`: an additive note that this failure can come from host-side runner supervision, and that the message keeps the redacted cause. ## Verification - Red: before the fix, the new test failed: `expected 'Provider notification transport faile…' to contain 'runner_process_termination_unconfirmed'`. - Green: `cd packages/paperclip-runner && npx vitest run src/drivers/codex/ src/conformance/local-provider-fault-matrix.test.ts src/native-session-runtime.test.ts` (565 passed). - `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json`: no new errors. The existing `src/eval/workflow-harness.ts` errors come from an unbuilt workspace package. - Staging evidence was read with read-only GETs only (run, events). ## Risks - Low risk. Only the message text changes. The failure code, `recoverable: false`, and the safe-replacement rules do not change. The server matches on the code, not on the message. - This PR does not add a retry in the same turn. Runner termination is unconfirmed when supervision is lost. A retry or replacement could start a second executor. The remote process probe already retries read-only checks inside a 30-second budget. Recovery stays with the existing ownership and recovery checks. - The cause text can include a plugin id. It goes through the existing provider-diagnostic redaction. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code with tool use (shell, file edits, browser reads) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandClaude Opus 5.5
|
7f9ef38273 |
fix(runner): report the upstream cause of OpenCode provider failures (#19166)
OpenCode reports some OpenRouter failures as an UnknownError whose message is the provider's JSON error body. The provider failure now reads its HTTP status, error type and message from that body, so the run shows the upstream message instead of raw JSON. The OpenCode nativeTurnFailure diagnostic now carries the provider failure's category, retryability and HTTP status (closed values only; no provider text or code), and recognizes the driver's opencode_empty_response code. Run status, error codes and retries do not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
737db73275 | fix(native): measure resumed run usage from the prior run's total (#19161) | ||
|
Devin FoleyandClaude Opus 5.5
|
6452e964a3 |
fix(runner): ignore replayed history for restored terminal turns (#19035)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Runner executes native provider sessions (Codex app-server
and ACPX) and can resume a run from an exact persisted checkpoint on a
replacement sandbox.
> - When a turn completes without a semantic result, the server
schedules one same-run disposition recovery from that checkpoint.
> - On the replacement sandbox, the restored runner can deliver the
committed history of the checkpointed turn again as live notifications,
before the disposition turn starts.
> - Between turns there is no active turn. The identity classifier only
marks settled-turn notifications as `stale_turn` while a different turn
is active, so the replayed `turn/started` reaches the live-turn guard
and fails the session with `turn_binding_mismatch` ("Provider started an
unexpected turn."). Recovery then stops with
`native_event_replay_conflict`.
> - This pull request makes the driver drop replayed history for turns
that were already terminal in the restored checkpoint.
> - The benefit is that exact-checkpoint disposition recovery can
continue instead of failing closed on its own history.
## Linked Issues or Issue Description
**What happened?**
A hosted ACPX run (Claude model) completed its first turn without a
semantic result. The server persisted an exact-checkpoint disposition
recovery. The second attempt restored the harness state from the
verified failover backup on a new sandbox. About 3.7 seconds after the
runner transport was ready, and before any `turn.submitted` event, the
driver failed with `turn_binding_mismatch: Provider started an
unexpected turn.` The run then stopped with
`native_event_replay_conflict` and required workspace repair.
**Expected behavior**
Notifications that only replay a turn that the checkpoint already
recorded as terminal must not fail the resumed session. The disposition
turn must start normally.
**Steps to reproduce**
1. Open a session, start turn `turn-1`, and complete it without a
semantic result.
2. Take the snapshot and recover the session on a new transport.
3. Deliver `turn/started`, `item/*` and `turn/completed` notifications
for `turn-1` again, before or during the disposition `turn/start`.
4. Before this change, the session fails with `turn_binding_mismatch`.
If the replay arrives while `turn/start` is pending, the driver binds
`turn-1` as the new turn and fails with "Codex turn identity changed
during start".
**Paperclip version or commit**
Observed on a staging release built from `c2163b3ecf08`. Reproduced on
`master` at `89703c3`.
## What Changed
- `CodexSessionState` records the turn IDs that were terminal in the
restored checkpoint (`restoredTerminalTurns`). It also adds
`isRestoredTerminalReplay(turnId)`. This is true only when the turn was
restored as terminal, is not the active turn, and the session has not
failed.
- `codex-session-notifications.ts` drops `turn/started`, `item/started`,
`item/completed` and item or turn delta notifications for such turns. It
emits no turn or item events for them. It emits a bounded
`provider_notification_identity` diagnostic with `classification:
"restored_terminal_replay"`, and shares the existing limit of 32
diagnostics.
- A replayed `turn/completed` is unchanged. It still goes through
`mapTerminalTurn`, which compares the terminal fingerprint and fails on
a conflict.
- New regression file `codex-app-server-driver.terminal-replay.test.ts`.
It replays history before and during the disposition submission. It also
checks that a live session still fails closed on `turn/started` for a
turn that ended in that same session.
## Verification
- Red before the fix, on `master` with only the test file added:
- `npx vitest run
src/drivers/codex/codex-app-server-driver.terminal-replay.test.ts` gives
2 failed and 1 passed. The "before" case fails with
`NativeProviderTerminalFailure: Provider started an unexpected turn.`
(`providerCode: turn_binding_mismatch`). This is the production error.
The "during" case fails with `Codex turn identity changed during start`.
- Green after the fix: the same command gives 3 passed.
- Related suites: `npx vitest run src/drivers/codex
src/drivers/acpx/codex-acpx-driver.test.ts
src/conformance/harness-driver.test.ts
src/backends/harness-driver-backend.test.ts
src/live/live-session.test.ts` gives 567 passed and 1 failed. The
failure is `codex-boundaries.test.ts > defers provider-owned workspace
existence without weakening its assignment`. It also fails on unchanged
`master` in my sandbox, because the sandbox workspace path triggers the
sensitive-path check. It is not related to this change.
- `tsc -p tsconfig.json --noEmit` shows no errors in the changed files.
The only errors come from the unbuilt
`@paperclipai/paperclip-eval-kernel` package in
`src/eval/workflow-harness.ts`, because I did not run the eval build
prep step locally.
## Risks
- The change relaxes the guard only for turn IDs restored as terminal
from the checkpoint. Notifications for those IDs are now ignored and do
not fail the session. A provider cannot use a settled turn ID for new
work, so these notifications cannot be new provider activity.
- Live post-terminal notifications, pre-turn notifications, foreign turn
IDs, `paperclip/runResult`, and tool calls keep their current
fail-closed behavior. The new test covers the live post-terminal case.
- Limitation: this change makes the driver tolerant of the replay. It
does not change why the runnerd transport cursor replays the committed
window on exact-authority recovery. The transport-level reason was not
reproduced here. The new diagnostic makes the event visible in run logs,
so later evidence can confirm or reject that path.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as an agent with
tool use, shell and test execution, and repository search.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
3a72efebe5 |
fix(native): confirm no runner process when remote staging fails before launch (#19059)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native sandbox runs supervise a remote `paperclip-runnerd` process. When that process exits during a turn, the transport restarts it inside a reconnect grace window. > - The server launcher stages the runtime assets first. Then it sends one launch RPC that starts runnerd. > - When staging failed, the launcher reported the exit as unconfirmed. This happened although no launch RPC was sent and no process could exist. > - Reconnect recovery treats an unconfirmed exit as a possible second owner. So it stopped at once with `runner_process_termination_unconfirmed` and did not retry. > - This pull request confirms the exit for failures before the launch RPC. Failures after dispatch stay unconfirmed. > - The benefit is that a transient staging failure, such as a full sandbox disk, uses the normal bounded retry. A real failure ends with `transport_reconnect_grace_exceeded` and keeps the staging error as the detail. ## Linked Issues or Issue Description **What happened?** On a hosted staging deployment, 7 native ACPX runs failed in 24 hours with `Provider notification transport failed closed.` (`native_provider_terminal_failed`). In every run, the runner process exited during the turn. The replacement launch then failed while it uploaded the runtime context into the sandbox: `runner_process_termination_unconfirmed: replacement runner health could not be verified: ... .paperclip-upload-<uuid>.tar.gz: create: ... no space left on device`. The sandbox disk was full. No replacement process was started. But the launcher said that it could not confirm the exit. So recovery did not retry inside its grace window. **Expected behavior** A staging failure before the launch RPC proves that no replacement process exists. Recovery must retry within the configured reconnect grace. If the failure continues, the run must end with the reconnect-grace failure and the staging error. **Steps to reproduce** 1. Create a remote runner launcher whose `ensureArtifact` step throws, for example with an ENOSPC upload error. 2. Wait for the handle completion to reject. 3. Before this change, `isExitConfirmed()` returns `false`, and no launch RPC was sent. The transport then fails closed with `runner_process_termination_unconfirmed`. **Paperclip version or commit** Observed on staging revision `89703c3`. Reproduced on `master` at `be20d4ad97`. Related: #19035 and #19014 change other runner paths (replayed terminal turns and redacted external tool results). They do not change remote launcher exit evidence. #18999 adds a free-space reserve for host-volume seed copies. It does not change sandbox disk use or this classification. ## What Changed - `createRemoteRunnerProcessLauncher` records when it dispatches the launch RPC. If the completion fails before that point, the handle reports `isExitConfirmed() === true`. The original error is kept unchanged. - A failed or timed-out launch RPC still reports an unconfirmed exit, because the RPC may have started runnerd. - Added a launcher regression for a staging failure before dispatch. - Added safety cases for a failed and a timed-out launch RPC. - Added a transport regression with the real `paperclip-runnerd`. A replacement whose staging fails is retried, and the next launch restores the durable session without a notification failure. ## Verification - Red before the fix: `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "remote runner process supervision"` failed `confirms no replacement process exists when staging fails before the launch RPC` with `expected false to be true`. The two launch-RPC safety cases passed. - Green after the fix: `npx vitest run src/services/native-runtime/native-session-executor.test.ts` passes 1131 of 1131 tests. - `npx vitest run src/services/native-runtime/setup-readiness.test.ts` passes 98 of 98 tests. - `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport.test.ts` passes 246 of 246 tests with the real debug `paperclip-runnerd`. This includes the existing case where a replacement with a dispatched but unverifiable process still fails closed. - `npx tsc --noEmit -p server` and `pnpm --filter @paperclipai/paperclip-runner typecheck` pass. - Not run: a live hosted sandbox with a full disk. The staging data showed the exact error, but this change was not deployed there. ## Risks - Low risk. The change affects only failures that occur before the launch RPC. In that window, the launcher has not run any sandbox command that can start runnerd. - Retries now run for the reconnect grace window, which is bounded and uses the existing backoff. Each failed staging attempt cleans its own upload scratch. - This change does not free sandbox disk space. If the disk stays full, the run still fails, now with `transport_reconnect_grace_exceeded` and the ENOSPC detail. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use, shell execution, and code editing. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
aa8372fd5d |
fix(accounting): post late usage to its owning run (#18991)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner sessions keep one provider conversation across task turns, so Paperclip accounts usage per run. > - Codex can report more usage for a turn after its run's charge was posted. > - The Runner then marked the session `reconciliation_required`. If another run was attached, it also failed that run. Core refused the delivery with `native_usage_after_finalization_requires_reconciliation`. > - That usage has already happened. Blocking the session cannot undo it, and the debt waited for an operator while the run totals stayed wrong. > - Codex totals are cumulative, and the Runner already knows the owning run in most cases. So the correct amount can be posted automatically. > - This pull request reports the exact increment with its owner, and posts it to the owning run as a separate cost event. > - The benefit is that run accounting becomes correct without stopping work, and an old run's usage can no longer fail a newer run. ## Linked Issues or Issue Description No issue exists. I searched open and closed pull requests and issues for "late usage", `native_usage_after_finalization_requires_reconciliation` and `codex_usage_accounting_integrity`. I found no duplicate. **What happened?** Code review found this while diagnosing session recovery failures. When Codex reported usage above the known total for a turn that was not active, the Runner blocked the session. If that turn belonged to a previous run, the Runner also failed the attached run with `codex_usage_accounting_integrity`. Core kept the receipt as debt, and could correct it automatically only if the token counts were unchanged. **Expected behavior** Late usage does not stop any run or session. The charge goes to the run that used the tokens, and it is auditable. **Steps to reproduce** 1. Finish a Codex turn on a native Runner session and attach the next run. 2. Have Codex send `thread/tokenUsage/updated` with a higher total for the finished turn. 3. On the unfixed code, the attached run fails and later attaches are refused. `late_codex_usage_requires_durable_reconciliation_before_or_after_handoff` encoded this behavior. **Paperclip version or commit** `master` at `04a24f4e06`. **Deployment mode** Any deployment that uses a native Runner with Codex. ## What Changed - **Runner:** late usage no longer sets `reconciliation_required` or fails a run. For a turn that belongs to a previous run, the Runner emits `usage.reported` with the increment in `runDelta` and the owner in `reconciliation.priorOrigin`. `runDeltaAvailable` stays `false`, so an older Core does not bill it to the attached run. - **`CodexRunUsage::absorb_late_usage`:** treats only usage above the known thread total as late, and keeps the attached run's delta unchanged. A report at or below that total adds nothing, so the sum across runs always equals the thread total and a replay cannot charge twice. - **Delivery:** `captureLateUsageForOwner` stores the increment as a late receipt on the owning run. The owner must be a finished run in the same company and native session. If the Runner cannot name the owner, Core uses the session's last finished run and records the attribution as `inferred`. - **Local Runner:** late usage stays with the run that is active when it arrives, or the next run. Usage reported between two runs stays in the next run's delta, so the total stays exact. - **Accounting:** `reconcileLateRunUsageReceiptInTransaction` posts a token increase as a new cost event with the key `heartbeat:<run>:late:<receipt>`. It updates the run totals so the accounting integrity check still agrees, records `cost.late_usage_posted` in the activity log, and queues budget evaluation. - **No more refusal:** `persistUsageReceipt` no longer throws after it saves late evidence, so late usage cannot wedge a Runner outbox. - **Covered receipts:** an older same-source receipt at or below the accepted totals, with matching billing, is cleared because a newer cumulative receipt already covers it. Conflicting billing stays as debt. - **Older Runners:** a zero-filled invalidation stays on the delivering run. - **Unchanged:** a newer lower total, a changed model or billing type, and per-model receipts stay as visible accounting debt for an operator. - **Unchanged:** an unexpected provider message during the warm attachment drain still fails closed. A separate change addresses that gate. - **Docs:** added "Late provider usage" to `doc/execution-semantics.md`. ## Verification - `cd packages/paperclip-runner/runner && cargo test --locked -p paperclip-runner-core --test codex_provider`: 99 passed, 2 ignored. - `cargo test --locked -p paperclip-runner-core --lib codex_usage`: 7 passed. This includes new tests for the out-of-order and repeated report cases. - `cd server && pnpm exec vitest run src/services/run-cost-accounting.test.ts src/services/native-runtime/native-usage-delivery.integration.test.ts src/services/native-runtime/native-recovery-usage-delivery.test.ts src/__tests__/cost-data-quality.test.ts`: passed against embedded PostgreSQL before the last review fixes, and the accounting and delivery files were re-run after them. - `npx tsc --noEmit -p .` in `server` reports no errors. - New tests cover exact and inferred owners, repeated delivery, late usage that arrives before the owner's charge posts, another session's run, an older Runner's report with no increment, a same-run increase, and a lower total. They also run the accounting integrity check after posting. - **Review:** Greptile Apex scored 1/5, 4/5, 3/5, 2/5 and 4/5 while findings were fixed, then 5/5 at the current head. - **Local Runner:** driver tests cover usage reported between two runs, including across a restore. No end-to-end local run has exercised it. - **Not run locally:** the full test suite. CI provides that coverage. - **Not verified live:** a read-only check of a test deployment found no late receipts in its history, so this path has only run against fixtures. ## Risks - **Run totals can rise after a run finishes.** This is deliberate. The correction is a separate cost event and an activity log entry. - **Budgets are corrected after the fact.** A late charge can put an agent over budget after later work has started. Budget evaluation runs after each correction, but it cannot prevent that work. - **Local sessions accept imperfect attribution by design.** Their late usage can land on the active or next run, because local sessions have no durable path for routing it. This also applies after a Stop or a pause. The total stays exact. - **Out-of-order reports can leave tokens on the wrong run in the session.** If a prior turn's report arrives after the attached run has already counted past that total, those tokens stay on the attached run. Moving them would need a lower correction to the attached run. The total stays exact. - **Inferred attribution can name the wrong run in the session.** The usage still belongs to that session and the total stays correct. The attribution is recorded as `inferred`. - **A session with no finished run charges the delivering run,** as an inferred late charge after its own charge posts. - **Rollout order:** deploy Core before the Runner. An older Core would keep this event on the attached run as incomplete usage, and would still refuse a same-run late receipt. - **Migrations:** this change has no migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be reworked. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use and code execution. A human directed the design and approved the scope. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
8a0e7ab511 |
fix(runner): preserve ordinary append for unscoped governed terminals (#19044)
## Thinking Path > - Paperclip manages work done by AI agents. > - A native turn can park on a saved human question or approval. > - Its terminal event can return a committed receipt before observers finish. > - That option requires an explicit turn ID at the event port. > - The protocol also permits events without a turn ID. > - The runner requested the option for those events, which the port rejected. > - This change keeps those events on the ordinary append path. ## Linked Issues or Issue Description Refs: #19025. **What happened?** The native governed-wait observation can accept a schema-valid `item.completed` without a turn ID. If the later terminal also has no turn ID, the runner compares two absent IDs and requests `deferPostCommitObservers`. The event port rejects that option. The run then fails with `governed_wait_provider_terminal_unproven`. **Expected behavior** Only a terminal with an explicit turn ID can request the early receipt. An unscoped terminal must keep the ordinary append behavior and await its observers. The event port must retain its turn identity check. **Steps to reproduce** 1. Create a pending human-only interaction from the run. 2. Pass a schema-valid completed tool event without a turn ID through the real governed-wait observation. 3. Supply an interrupted terminal without a turn ID during the cancellation drain. 4. Run the actual native runtime with the PostgreSQL event port. The parent code rejects the deferral request and reports a failed terminal. **Paperclip version or commit** Reproduced on `4ea53f9eb5a24de10db3dfda18456132987c4af8`. The follow-up was identified by a review published after #19025 merged. No observed provider emission or original-task recovery is claimed. I checked current master and related PRs before this correction. **Deployment mode** Built from source. The regression uses real PostgreSQL and a deterministic provider boundary; it makes no model call. ## What Changed - Require a truthy terminal turn ID before the runner requests deferred observers. - Keep ordinary append behavior, the port identity check and all timeouts unchanged. - Exercise the production governed-wait observation with a persisted pending interaction and the real event port. - Retain exact-turn observer and checkpoint controls. Add a direct rejection control for unscoped deferral. - Document that unscoped terminals await their observers. ## Verification - First meaningful red: the actual observation, native runtime and PostgreSQL port fail on the unmodified parent runtime. The new case reports `NativeProviderTerminalFailure` in 143ms; this is not a setup or deadline failure. - Focused regression and authority controls: 12 passed, 69 filtered. Full runtime suite: 203 passed. Full PostgreSQL port and timing suites: 84 passed. - The tests leave the human-only interaction pending with continuation policy `none`, accept one result, and start only one turn. Existing scope, invalid evidence, checkpoint, abort, duplicate and restart controls remain. - Exact candidate: `8840ea2c9c308e5a60423e67fb3612d24029ba84`. Repository-wide `pnpm -r typecheck`, `pnpm build` and `pnpm check:module-boundaries` passed on this exact head. `pnpm test:run` is still in progress and is not reported as passing. Current-head CI passed: 55 success, 3 skipped and 1 neutral, including required `ci / verify` and `ci / e2e`. Greptile and Ponytail both reviewed this exact head at 5/5 with no blocking findings. The native review was source-only. No unresolved review threads remain. - No UI, database schema, agent permission or mapped user entry point changes. No deployment, live retry or provider replay was performed. ## Risks Unscoped terminals retain the existing observer wait and can still fail if it exceeds the existing drain deadline. They do not gain an early receipt. The change does not supply a missing turn ID or weaken source, company, run, session, cancellation or transaction checks. A PRP terminal is not physical-stop or accounting-completion proof. ## Model Used OpenAI `gpt-6.1-sol`, with code and shell tools. The host does not expose the reasoning mode or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
22735cfcea |
fix(native): preserve governed terminal receipts during observer settlement (#19025)
## Thinking Path > - Paperclip manages work done by AI agents. > - A native turn can yield to a saved human question or approval. > - The provider must stop, and its terminal event must be saved. > - The old drain waits for both the event transaction and its later observers. > - A slow observer can make a saved terminal look unproved. > - This change separates the committed receipt from owned observer work. > - Checkpoint and result acceptance still wait for that work to settle. ## Linked Issues or Issue Description **What happened?** A native governed wait can fail with `governed_wait_provider_terminal_unproven` after its exact interrupted terminal is already in PostgreSQL. The event port waits for postcommit accounting, output and projection callbacks before it returns the receipt. Those callbacks can cross the provider drain deadline. **What did you expect to happen?** A committed terminal proves the event transaction. Its observers remain separate work. A checkpoint or result must not be accepted until those observers settle. A pending human-only question must keep its audience and continuation policy. **Steps to reproduce** Run the real PostgreSQL control-plane regression. Commit a same-turn interrupted terminal. Hold its postcommit observer beyond the existing provider drain. The parent code reports an unproved provider terminal. The fixed code retains the committed receipt and joins the observer before checkpoint or result acceptance. No overlapping repair was found in the current master or open-PR search. Related [#18977](https://github.com/paperclipai/paperclip/pull/18977) covers final conversation-response selection. It does not change this receipt boundary. The source reproduction is consistent with the observed failure. The exact hosted provider code is unavailable, so it does not prove that every observed failure used this branch. ## What Changed - Add an explicit terminal-only option to the internal event port. Ordinary appends keep their existing behavior. - Return that receipt only after the normal transaction and company, agent, run, session, source and revocation checks. - Retain the full observer promise. Join it before checkpoint and result writes. Keep its original rejection and observe it after an aborted join. - Reuse the existing provider drain and checkpoint deadlines. Preserve duplicate and restart replay without another model turn. - Document the receipt and duplicate-observer replay boundaries. Keep human-only and no-continuation cards unchanged. - Assert the actual Codex facade requests deferral only for its interrupted terminal. Isolate empty skill inventory in monitor-only process fixtures while retaining their real scheduler, lifecycle, selection snapshot and three-second deadline. ## Verification - Meaningful parent red: actual native runtime and real PostgreSQL event port. The terminal row commits, but a held observer causes `NativeProviderTerminalFailure` with `governed_wait_provider_terminal_unproven`. - Current candidate `0db24ae74ee42fdf92035d8407fc095e9f73cc6a`: full runtime suite, 203 passed. Real PostgreSQL control-plane, timing and monitor suites, 103 passed. Runner TypeScript and full layered server typechecks passed on the unchanged candidate source tree. The controls cover uncommitted and foreign evidence, explicit false and failed terminals, held and rejecting observers, abort and controller revocation, duplicate receipts, and new-port recovery. The real pending human-only card stays unanswered with continuation policy `none`. Recovery does not call `startTurn` again. - `pnpm -r typecheck`, the production server build (including the real Runner release binary), module boundaries and vendor dependency verification passed on `6d75`; the follow-up changes only tests and documentation. Diff checks pass on the current candidate. The prior `399ad3dc2b` repository-wide `test:run` was stopped as obsolete during the general-server phase after preserving its progress (exit 130, incomplete); its repository build was not reached. Its six completed chat shards passed 1,904 cases, but this is prior-head evidence and not a full-suite pass. Full current-head CI is required before merge. - CI attempt 1 on `6d75` failed one real three-second monitor-settlement assertion; the aggregate verify gate consequently failed. Other jobs passed. The actual CI merge tree was `3037bf6db162da3b0a9172046360401f8449ebc5` (`6d75` into `89703c3`). An unchanged full-file replay reproduced three timeouts. Original-deadline inspection found the legacy process still preparing; local phase timing measured cold skill inventory at 6.629 seconds. The original CI did not retain that phase detail, so its exact subcause remains unproved. The fixture correction supplies empty skill entries for these skill-free process tests and still invokes the selection-snapshot callback. It preserves all assertions and the three-second deadline, with no new drain wait. All 21 cases pass; deliberately restoring the old blocked-status rejection still fails the real blocked-check case with 409. No CI retry was issued. - The prior current-head native review was 4/5 with no production correctness finding. Its requested direct facade-options assertion and duplicate-observer wording correction are included. Current-head CI is green (55 success, 3 skipped), including required verify and E2E; current-head Greptile is 5/5 with no actionable findings. The old native advisory remains tied to `6d75`, and its requested findings are addressed. No new native assessment is claimed or required by protection. - The first draft broke three existing transaction-start timing controls. The no-debt path now keeps the original synchronous start, and all three pass. Two draft type errors were corrected. A further real-PG red found an extra default-append assertion after commit. Removing it restores the original committed receipt after a late abort; the same aborted signal still cannot write a checkpoint or result. Original setup and assertion failures remain in private verification receipts. - Normal merge of current master `3a2249dd5768313f04b2aa3860dfd363492eda16` has no conflicts. The affected production source blobs did not change in that baseline merge. - Local proof uses Node 24.22.2 on Darwin arm64, a real PostgreSQL instance and deterministic provider-boundary fixtures. It makes no model call. No deployed replay or original-task recovery is claimed. ## Risks Deferred observers must remain owned. A missed join could accept incomplete accounting or output. A rejected or stalled observer must prevent acceptance, and an abort must prevent late checkpoint/result writes. The tests exercise these boundaries. The port keeps the existing best-effort trace-publication handling. This change does not turn a PRP terminal into physical-stop or accounting-completion proof. ## Model Used OpenAI `gpt-6.1-sol`, with code and shell tools. The host does not expose the reasoning mode or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3a2249dd57 |
fix(runner): retry proven ingress bootstrap failures (#18981)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution must retain its admitted model and execution authority during recovery. > - A sandbox provider can reject Runner endpoint acquisition before the Runner gets a credential. > - The old error path loses the structured HTTP status and leaves that startup failure ambiguous. > - A zero counter in an old checkpoint cannot prove that no credential was issued. > - This pull request adds a versioned producer proof and carries closed provider status through the existing RPC. > - A proved transient startup failure uses the existing retry budget. A terminal proved-unused run can settle its accounting. ## Linked Issues or Issue Description Refs #18018 and #18934. Those changes restore existing checkpoints and ready retained environments. This change covers endpoint acquisition before Runner dispatch. A search of related public PRs found no duplicate implementation. **What happened?** A provider HTTP 502 during Runner endpoint acquisition becomes an untyped worker error. Native recovery treats it as ambiguous and stops without retrying. The run's native usage receipt also stays pending after startup ends without model work. **Expected behavior** Retry a structured transient endpoint failure only when the complete current controller checkpoint proves that this exact run never issued a Runner credential. Preserve authentication, configuration, Stop, ownership and provider-progress gates. Settle zero usage only under the existing terminal accounting owner. **Steps to reproduce** 1. Use the controlled endpoint in the Daytona ingress integration test to return HTTP 502. 2. Register the actual native Runner driver and persist its bootstrap journal through the encrypted checkpoint writer. 3. Observe the old terminal ambiguous disposition instead of bounded startup recovery. **Paperclip version or commit** Reproduced on the pre-change master source. The original regression used the production native driver, a real HTTP fault, a real durable control plane and local Postgres. The new contract additionally tests the pinned Daytona SDK and real worker RPC. ## What Changed - The controller owns a v2 persisted state with retained origin and credential history. Current readers still accept v1. Older writers reject v2. Restore and checkpoint decoding do not add missing history. - The proof validates the complete bootstrap journal, all six identity fields and every delivery, credential and progress control. The server rechecks its fingerprint under the current controller claim before retry or accounting changes. - Daytona and the SDK carry a closed acquisition phase and HTTP status through the existing ingress RPC. Both public error wrappers omit response bodies and causes, including Pino cause serialization. - Only structured HTTP 429 or 5xx ingress failures can use the existing three-attempt startup budget. HTTP 401/403, configuration and unknown errors do not gain retry authority. - A terminal proved-unused run wakes existing accounting and records zero provider work. It does not create a semantic result, prove physical termination or release an unknown process/workspace hold. - The normal automatic second attempt rechecks its exact claim, frozen execution and matching complete local/encrypted journal before admitting absent remote Runner state. It keeps the original pending bootstrap command IDs. A proved unused Codex home can rematerialize sealed skills. Full raw local state is checked before streaming-event or policy projection. - Normal saved-state readers and the existing E2E journal evidence matcher accept the closed v1/v2 set. The separate legacy no-launch witness is unchanged. Technical docs describe the contracts. ## Verification Current head: `cd38f70409cdb8664e14bfbf4efa208f468a0419` (final delta adds only fixture artifact preparation). Production files are unchanged from the independently reviewed `93c8a33520` source. Full repository typecheck and build passed at `96417acb58`; the affected server typecheck also passes after the final two-line fixture delta. Focused protocol, raw-proof, real retry, SDK and RPC acceptance pass on this source. Earlier broad suite counts are listed with their limits below: - Durable Runner protocol: 187 tests. - Native executor, driver, startup/accounting and redaction: 1,211 tests, including real Postgres and encrypted checkpoint persistence. - Normal saved-state restore/projection compatibility: 21 focused cases. - SDK: 209 tests. - Standalone Daytona: 544 tests; 8 opt-in live tests skipped. - Explicit standalone `pnpm -C packages/plugins/sandbox-providers/daytona exec vitest run --config vitest.sdk-ingress.config.ts`: 6 cases with real pinned SDK 0.203.0 and worker RPC ingress acceptance. These include lookup 502/401/403/400, private-preview 502 after successful lookup/refresh, missing credentials and private-response exclusion. - E2E journal evidence: 4 cases across both supported formats, with wrong ownership, missing stimulus and unknown-schema controls. - The exact historical controller source rejects a new v2 checkpoint at decode and restore. A v1 control is accepted. This verifies the downgrade fence with the actual older parser. - Startup controls cover issued and pruned credentials, old/truncated journals, provider progress, exhausted retry accounting, and checkpoint/Stop/controller-claim races. Race cases retain the current execution and pending usage. - New exact admission controls: 19 cases through encrypted persistence and cold restore. They cover current/stale/expired claims, Stop, pause, model/trust/account changes, prior progress, issued/pruned credentials, adopted/old/truncated state, local/canonical fingerprint mismatch, hidden delivery, unsupported work and company scope. Two additional red/green cases prove that raw local stream progress and missing event policy are rejected before lossy checkpoint projection. No case mutates run, claim, result or accounting state. - Actual native executor, scheduler and Rust Runner retry: 2 cases. Codex uses its fake app server. OpenCode uses the real proxy and fake OpenCode HTTP server with the frozen DeepSeek model and allow permission configuration. Each path launches no Runner on attempt 1, starts one Runner and provider on attempt 2, reuses both original command IDs, commits one accepted result, validates the persisted local workspace, records nonzero synthetic native usage and settles accounting once. No zero-work accounting flag survives. The controlled SDK and protocol tests use local source, isolated PostgreSQL, a private HTTPS preview and fake provider servers on macOS arm64 with Node 24.22.2. They create no hosted sandbox and make no model call. The root workspace intentionally omits standalone provider dependencies; the explicit SDK target fails if its pinned dependency is absent and has no conditional skip. They do not qualify hosted rollout or repair an old unmarked checkpoint. The first full local typecheck found missing checkout SDK links from an install with scripts disabled; the repository link helper repaired that setup. Current-head Linux CI typecheck and build pass. Independent final review checked 12 source scopes and six raw-proof controls with no remaining finding. Fresh Greptile review is 5/5 with no unresolved thread. The full Runner CI suite found two older control-peer test doubles missing the new proof method. Both now return no proof, with all 27 permission, ordering and cancellation assertions retained and passing. The new Cargo-dependent native retry suite uses the existing Rust-cached CI lane. All 39 lane coverage controls pass and prove complete, duplicate-free suite coverage. The first full test run stopped at an admitted chat Retry polling failure. CI also found a stale retained-allocation fixture, now repaired on master by #18984, and a canceled runner job. A real SDK suite initially relied on a dependency absent from the root workspace; the explicit standalone target now owns that acceptance. The separate Product E2E support run had 12 failures in four untouched files. All 12 reproduce on the original clean base with independent dependencies, compiled base source and verified workspace resolution. These concern workflow expectations, cleanup timeout arguments, a profile snapshot and operating receipts. They are not passing acceptance. The later full local repository test command stopped at a 15-second timeout in the unchanged GitHub pre-allocation retry `controller` case. That exact case passed in isolation with unchanged assertions. The broad Runner run initially omitted `test:typescript:prep` and lacked debug fake-provider binaries. Required preparation then passed 370 + 44 package contract cases (11 skipped); the prepared full package run reported 4,932 passed, 9 failed and 13 skipped. All failed families then passed with unchanged assertions in isolation: 8 ACPX controller tests and 2 credential controls. The paired original-base ACPX control also passed. The full-run failures remain recorded; they are not passing full-suite evidence. The first Rust-cached native CI run found a missing generated OpenCode proxy in a fresh checkout. The fixture now runs the existing verified provider-entrypoint builder. Removing the generated proxy reproduces the exact ENOENT before the fix; both real retry cases pass after the fix from the same absent-artifact state. All current-head CI gates pass, including the aggregate verification check. Linux native CI passes all 10 suites: 177 tests pass and 14 opt-in cases skip. Both real provider retry cases pass. The full Runner package passes 297 suites and 4,950 tests, with 1 suite and 10 tests skipped. The CLI workspace groups pass 4,762 and 286 tests. The first CI attempt had Runner shutdowns in the Runner package, Canary and three general server jobs. One unchanged CLI shard also hit child-server startup and dense ZIP deadlines. Both CLI files passed unchanged in isolation: 18 tests. One bounded rerun of unsuccessful jobs and their dependents passed. The original failures remain recorded. ## Risks - Older controllers reject newly written v2 state on downgrade. Current controllers retain v1 restore support, but old or unmarked state cannot gain this new proof. - An older plugin that omits structured ingress evidence remains held. The server and provider must use the new producer contract for this recovery path. The Docker `cloud` target builds the SDK, server/Runner and standalone Daytona producer from one source tree. The default `production` target does not build the standalone producer. Hosted rollout must confirm the new image and active image-owned plugin worker; a registry version alone does not prove its source. An npm-installed producer needs an explicit matching artifact upgrade. - This prevents future proved startup failures. It does not release historical accounting based on an old checkpoint, change retry budgets, replace a model or weaken sandbox isolation. - Zero model usage is separate from process termination, saved files and resource release. Those owners keep their existing gates. ## Model Used OpenAI Codex with tool use and code execution. This runtime does not expose the exact backend model ID or context window to the agent. Read-only independent agents reviewed checkpoint compatibility and SDK/RPC error boundaries. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (affected suites; full local run limits are recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
039e0dad2c |
Connect TestOps protocol plans to verified GitHub attempts (#18985)
## Thinking Path > - Paperclip lets people coordinate agent work through a governed control plane. > - Runner protocol evals test real provider sessions against a seeded mock control plane. > - The saved TestOps protocol plan has no GitHub executor. Its dashboard shows historical results. > - A current one-case attempt also failed before any tool call or provider output. Its retained public-safe snapshot withheld the error cause. > - This pull request connects selected cases to the existing trusted workflow and retains closed operator failure facts. > - Each selected case receives a current result. Missing evidence remains broken. Existing assertions and machine grades remain unchanged. ## Linked Issues or Issue Description **Current behavior** The saved protocol plan cannot start a GitHub qualification attempt. Historical dashboard results can be mistaken for current execution. A failed provider turn can also lose its failure category when the live snapshot applies public evidence redaction. **Proposed behavior** The existing workflow accepts TestOps service inputs, validates exact case identities before provider work, runs the selected cases, and verifies their published statuses. Eval artifacts retain only validated failure phase, category, HTTP status, and retryability. **Reason and benefit** Operators can run the standing plan from TestOps and inspect current, source-bound results. A targeted pass cannot stand in for full-plan coverage. A failed startup can be diagnosed without publishing provider messages or request identifiers. ## What Changed - Add optional TestOps selection and protected reporting jobs to the existing Runner protocol workflow. - Keep the current actor, source, provider credential, and controlled accounting gates. TestOps retains the explicitly selected controlled or legacy mode and a separate 41-case Claude roster. Controlled mode preserves its existing reviewed policy, immutable bindings, tracing and zero-retry admission. Its paid workers explicitly use GitHub-hosted runners, matching the current manifest admission contract even when the legacy AWS fleet is enabled. - Bind the exact standing plan, workflow, job run, source revisions, selected cases, and saved statuses. Accept only verified workflow URL aliases. - Create a closed Allure projection inside the report job. Transfer only fixed case identities, check verdicts, numeric usage and statuses to the reporting environment. Controlled raw/chat/controller evidence remains private. - Publish one result for every selected case. Retain missing execution/report evidence as broken. Reject unsupported passes, model mismatch and source drift. Preserve the evidence schema literal, stop publication on cancellation, and retain closed API error codes/statuses. - Tag full selections separately from targeted attempts. Add bounded ingestion observation. Apply the exact TestOps filter before the unchanged controlled provider/cell-capacity check, so a one-cell registration fits a one-cell reviewed policy. - Retain up to eight eval-only closed provider failure records before board evidence redaction. Forward the original notifications unchanged. - Add selection, reporting, privacy, and admission regression controls. Document scope and remaining acceptance. ## Verification - Direct Node protocol publication, projection, metrics, and security tests: 56 passed. Focused controls exercise both mode admissions, missing controlled policy/retries/source rejection, closed result transfer, exact metadata, and missing-result handling. - `actionlint .github/workflows/runner-protocol-live-evals.yml` and `git diff --check` passed. - A proposed one-cell controlled policy passed the actual hosted validator with a synthetic prepared request and zero provider calls. The actual 41-case catalog filter selected case194/get-task-context and the real hosted admission accepted it under maxCells1. Negative controls rejected the unfiltered41, unsupported provider, source, selection, origin, expiry and lock drift. This does not verify provider auth, a runtime build or TestOps publication; final merged source/lock binding is still required. - The actual diagnostic TypeScript module passed positive, negative, secret-exclusion, and unchanged-notification controls through the existing tsx runtime. A local actual-CLI startup serializer probe passed with closed facts, secret exclusion, exit status and read-only fixture cleanup. This used an isolated resolver into the existing dependency cache; canonical exact-dependency CI remains required. - Prior-head CI passed typecheck, Runner static/Rust/native-server checks and 4946 Runner package tests. One new serializer fixture failed during Linux read-only directory cleanup; this head applies the existing neighboring cleanup pattern. Unrelated chat E2E and CLI import timeouts remain retained. Latest-head required gates are pending. - Original current-source one-case attempt: https://github.com/paperclipai/paperclip/actions/runs/41860234840 . It failed with no tool calls or output. That failure remains retained and is not a passing qualification claim. - Diagnostic candidate attempt: https://github.com/paperclipai/paperclip/actions/runs/41863410602 . It retained the new closed field but still failed before output with category unknown. This disproves the claim that typed retention alone identifies the cause. - The earlier passing run https://github.com/paperclipai/paperclip/actions/runs/40627274706 used controlled Braintrust routing. Both current failures used direct legacy credentials, a different auth path. Next acceptance is one explicitly controlled case with a reviewed current policy, followed by TestOps registration and a fresh full plan. None is claimed complete by this PR body. ## Risks The TestOps contract currently binds the existing 41 cases and one recorded Claude Sonnet 5 profile. Catalog changes fail closed until reviewed. This does not qualify other providers. Admission and execution qualify only the selected accounting mode; a controlled pass does not prove direct credential health. Model-request readiness retains pending operating reconciliation. TestOps ingestion can fail independently; persisted results are checked, and missing evidence is never accepted as a pass. The new failure field contains closed facts only; raw messages, identities, credentials, and tool payloads are excluded. ## Model Used OpenAI Codex, GPT-6. The exact hosted model ID and context window are not exposed in this session. Used reasoning, repository inspection, code editing, and tool execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the available capability details - [x] I have checked ROADMAP.md and confirmed this does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs - [x] I have described the issue with the enhancement template field labels - [x] I have not referenced internal or instance-local Paperclip issues - [x] My branch name describes the change and contains no internal ticket ID - [x] I have run the available targeted local checks and reported the remaining checks - [x] I have added relevant positive and negative regression controls - [x] I have updated the authoritative protocol documentation - [x] I have considered and documented the risks - [ ] All required Paperclip CI gates are green on the latest head - [ ] Greptile is 5/5 with no open actionable feedback on the latest head - [x] I have addressed all current review feedback --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
612d9931a6 |
Support per-turn Runner sessions on ephemeral web servers (#18932)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The web server controls native Runner sessions in remote sandboxes. > - An ephemeral web server must recover without its previous local filesystem. > - The existing admission rule accepts warm sessions. Managed Claude, Grok, and OpenCode sessions resolve to per-turn execution. > - Per-turn execution also needs durable provider-session bytes after the sandbox is retired. > - This pull request adds encrypted backup publication and removes the warm-only admission rule. > - Both lifecycle modes can use the existing native ownership and checkpoint contracts during a web deployment. ## Linked Issues or Issue Description Refs: #18518, #18529, #18826, #18835, #18841, #18864, #18887. **What existing behavior does this improve?** Native Runner execution on web servers with `PAPERCLIP_EPHEMERAL_SERVER=true`. **Current behavior** The admission rule requires a warm Runner lifecycle. Managed provider credentials can resolve an agent to per-turn execution, so enabling the ephemeral deployment profile blocks those turns. A database controller checkpoint does not preserve provider files after the old web home and per-turn sandbox are removed. **Proposed behavior** Admit native execution in either resolved lifecycle. Keep provider and credential choices unchanged. Publish encrypted provider state to shared storage before sandbox cleanup. Restore it from a fresh web home for the next turn. Use the existing controller lease and generation checks for active takeover. **Reason and benefit** Operators can remove the web server volume without excluding per-turn providers or forcing managed accounts into warm sessions. New native Daytona profiles automatically enter both rolling-deployment qualification lanes. **Breaking changes** The additive migration creates a backup publication ledger. Existing volume-backed local-disk installations retain their local backup path. S3 installations publish encrypted backups and require a stable secrets master key. Existing-volume enrollment must import surviving session data before detaching the volume. ## What Changed - Admit native Runner execution independently of its warm or per-turn lifecycle. Keep host-owned legacy execution excluded from ephemeral deployments. - Store private provider state as encrypted objects. Journal uploads before I/O, read them back, and fence publication against the current native owner. Keep the previous published backup when storage or ownership checks fail. - Restore exact provider bytes into a private cache. Require an exact durable backup before sandbox cleanup. Extend the existing bounded retention pass to retire stored objects without adding idle polling. - Import selected legacy controller and provider state in the existing offline maintenance process. Preserve queued work, task pointers, pending recovery, and original files. Prefer canonical database checkpoints. - Preserve already-failed legacy controllers with missing original artifact proof as encrypted archives. These archives do not become execution authority. Successful retained sessions still require proof. Report archival and executable imports separately. - Verify unmanaged Grok API keys through the existing xAI account validator. The live test exposed a readiness gap that kept remote Grok agents paused after platform checks. Preserve failed credentials and the separate runtime warning. - Ingest Grok whole-prompt usage at the owning ACP boundary. Bind the receipt to the exact outbound request/session; reject replay, conflicts, and incomplete authority. Separate turn cost from cumulative session cost in the sidecar schema and Rust normalization. Preserve managed billing identity and identify direct xAI API credentials. - Preserve published workspace directory names across cold allocations, including destroyed-allocation cleanup. Keep unpublished epoch/lease directories isolated and refuse to overwrite an occupied preparation. This fixes the cold-session identity regression exposed by #18922. - Increase bounded checkpoint-header inspection to 1 MiB. Retain the aggregate inspection limit. - Register all nine native Daytona profiles in per-turn and warm rolling-deployment suites. Delete the entire departing server home, exercise ordinary reads and writes, require active ownership transfer, then check a later turn on another fresh server. Permit only exact notification replay after reconnect. Ownership stays with the existing modules: Runner validates its protocol state; Core validates run scope and controller ownership; storage owns object I/O; the maintenance process owns offline conversion; the E2E wrapper owns process replacement and probes. No provider protocol is added to the shared admission rule. Grok receipt parsing stays in its provider module. The shared sidecar gains an explicit cost basis because its existing price field means session-cumulative spend, while Grok reports prompt totals. Existing consumers retain cumulative behavior when the new field is absent. ## Verification - Focused native admission, archive validation, publication fencing, cleanup, readiness, legacy import, and E2E observer tests pass. Real PostgreSQL cases verify that queued intent and historical recovery rows stay unchanged. - Full typecheck and build pass on the rebased branch. The review follow-up passes server typecheck, 29 focused PostgreSQL/archive/readiness tests, and module-boundary checks. Revision fb8d39a7a1 reached 55 successful CI checks and Greptile 5/5 with no unresolved threads. The Grok follow-up passes Runner/server typechecks, 238 ACP runtime/projection/usage tests, 16 sidecar-schema tests, the Rust cost regression, and Core billing regression. The |
||
|
DottaandPaperclip
|
a194166017 |
perf(ci): repair dependency caches and parallelize transport fixtures (#18883)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Required CI checks protect changes to its native Runner and control plane. > - Recent Runner jobs restored a Rust cache but still rebuilt dependencies for 74–92 seconds. > - New workspace install hooks disabled the materialized JavaScript dependency cache. > - A single transport file also kept 326 test cases on one worker. > - This pull request corrects the cache inputs, replays known install hooks, and separates independent transport test groups. > - The same checks can finish sooner without removing test cases or widening runner access. ## Linked Issues or Issue Description Refs #18794 and #18544. Related dependency-action version update: #17698; this change fixes the cache input contract used by the currently pinned action. **What happened?** Recent PR jobs such as #18783 and #18825 restored an exact Rust dependency cache from October 16. The virtual workspace's development profile changed on October 30. Rust Cache 2.9.2 hashes member manifests but omits that virtual manifest. Rebuilding cannot replace an immutable exact cache entry. The JavaScript cache also rejected the new server and Runner Copilot postinstall hooks. **Expected behavior** Changing a Cargo profile selects a new dependency cache. An exact JavaScript dependency restore still performs required workspace install hooks. Independent transport fixtures use the existing file workers while retaining all assertions and real deadlines. **Steps to reproduce** Inspect the Rust dependency cache hit and following `build:rust` in the recent Runner package jobs. Run `node scripts/ci-dependency-cache.mjs` on the previous revision to see `cacheable: false`. Run the original transport file to measure the serial test work. ## What Changed - Add the virtual workspace manifest and ancestor Cargo configuration to all shared Rust dependency keys. Keep reader/writer identities aligned and preserve master-only cache writes and CODEOWNERS-only AWS routing. - Recognize only the exact reviewed root, server, and Runner postinstall commands. Include the server/Runner installer scripts in the dependency key. Replay workspace postinstall hooks after a materialized restore. Unknown lifecycle hooks still disable reuse. - Move attachment recovery, warm attachment, live adoption, and suspension fixtures into separate files. Keep the large event-backlog cases in the original file to avoid starting another expensive fixture group at the same time. Share five stateless test helpers outside production source. - Extend the opt-in `profile-ci` benchmark with a PR-scoped Rust cache producer and a fresh-host consumer. Require exact dependency/workspace hits and reject any Cargo unit that is rebuilt. Retain the JSON build evidence. Existing setup qualification checks real JavaScript cache restores. - Schedule the ten measured files over 50 seconds first, in approximate duration order. Package CI uses up to four workers, bounded by available CPUs; local runs retain Vitest defaults. The AWS runner has four CPUs and 16 GiB RAM; the three-worker run averaged 44% CPU and peaked at 15% memory. Preserve Vitest selection and shard membership and default ordering for other files. Job count and AWS authorization are unchanged. - Scope the optional paired server benchmark to changes in its measured inputs. Cache-only changes qualify exact restores and run the shared package tests using restored dependencies. - Update the cache and test documentation and the existing native cache contract guard. ## Verification - All 586 `.github/scripts/tests/*.test.mjs` policy tests passed, including cache trust and runner routing guards. Actionlint passed for all three changed workflows. - All 169 original top-level transport test definitions remain byte-identical. The original file passed all 326 cases in 317.99 seconds on this macOS host. A clean preliminary split passed the exact same named cases in 183.01 seconds with the same four-worker cap. - Preserve the first failed experiment: its parallel run overlapped the policy suite and timed out in the 1,024-event settled backlog case at 60 seconds; 325 cases passed. A clean repeat passed, but that does not establish the earlier failure's sole cause. The final layout keeps the backlog cases in the original file. The final layout passed the same 326 cases in 186.35 seconds (41.4% faster than the 317.99-second baseline). The original test bodies and all timeouts remain unchanged. - Fresh-host Rust qualification passed: all 308 compilation units were fresh, and Cargo completed in 1.59 seconds on a new runner ([final-head run](https://github.com/paperclipai/paperclip/actions/runs/41745082837/job/126296908816)). JavaScript setup dropped from 46 seconds cold to 23 seconds with exact dependency/build restores; all 1,052 shared-package cases passed against the restored tree ([final-head job](https://github.com/paperclipai/paperclip/actions/runs/41745082837/job/126295937797)). The previous qualification also passed (43 to 19 seconds) and confirmed all three postinstall hooks replayed. These are qualification results, not an end-to-end PR timing claim. - Initial full AWS package run passed 4,907 cases with 10 existing skips. The original-file remainder fell to 118.7 seconds, but the 100-second deadline file started last, leaving the full package at 417.43 seconds. Starting only three long files first passed 4,911 cases (including four new sequencer guards) in 422.54 seconds, so reordering alone did not improve full-package time. Summed file execution was 1,202 seconds across three busy workers. The final four-worker configuration passed **4,912 cases, with the same 10 existing skips**, in **324.99 seconds**, a **23.1% reduction** from 422.54 seconds. The five added cases guard ordering, selection, sharding, and path handling. Peak memory was 17.95% of the 16 GiB runner; average CPU was 49.14% ([final-head package job](https://github.com/paperclipai/paperclip/actions/runs/41745082974/job/126296339480)). - Local workspace typecheck and build passed. The complete local `pnpm test:run` passed all six chat partitions but reported failures in one existing OAuth redirect case and one gateway authority case in the general-server group. Both focused repetitions passed (three OAuth cases and one gateway case), and a complete repeat of both files passed all 593 cases in 42.89 seconds; that does not explain the original failures. The broad local run was stopped after 61 minutes once full final-head CI had passed; its remaining general/workspace/serialized work was not completed locally, and it is not reported as a pass. Original failure logs were retained. Full final-head CI provides the complete suite validation. Sequencer selection/shard guards and the 11-case transport suspension smoke passed. The corrected release workflow guard passed all 71 tests. - The first CI run failed only the policy guard that still expected the old native cache key; downstream aggregate checks also failed. That guard now checks the new profile/config prefix and existing host/target/protocol suffix. A second policy run found that the new sequencer test imported Vitest in the dependency-free policy lane; it now runs in the Runner package lane, where all four cases pass. Greptile reviewed final head `8011f84189` at 5/5 with no actionable findings and confirmed this issue fixed. Final-head policy, typecheck, build, and review passed. At 23:52:46 UTC on November 3, several AWS jobs were terminated together: the Runner package process received SIGTERM and browser logs recorded a runner shutdown signal. The failed/cancelled jobs passed on the same-head retry, including required `ci / verify`; no assertion or timeout was changed in response. The final concurrency adjustment at `fe5a69bb0b` also received Greptile 5/5 with no actionable findings. Final-head required CI, including `ci / verify`, passed ([aggregate](https://github.com/paperclipai/paperclip/actions/runs/41745082974/job/126300554711)). Local timings exclude install, build, cache transfer, and CI queue time. ## Risks A new Rust key needs one successful canonical master build before ordinary PRs can hit it. The fresh-host qualification uses separate PR-scoped entries. Parallel test files can increase I/O contention; case execution inside each file, timeouts, and assertions are unchanged. Two extracted suites exceed 1,000 lines because they retain existing multi-stage recovery fixtures unchanged; one parameterized fixture alone is 1,059 lines. No production execution logic, lockfile, credentials, runner trust gates, or required check is removed. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository editing, command execution, and GitHub inspection. The host does not expose a more precise model snapshot or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — targeted checks passed; the broad-run failures and repetitions are documented above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
4e29f00139 |
fix(native): keep settled ACPX sessions across Runner upgrades and run scheduled resumes (#18879)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner keeps one ACPX provider conversation across the per-turn runs of a task. Each run cold-attaches the session that the previous run settled. > - The saved ACPX state binds the digest of the launch profile that wrote it. Every deploy ships a new provider pack, so that digest changes. > - After a deploy, the next run of such a task fails `run.attach` with `ACPX durable launch profile digest does not match runner startup`. Every same-run retry fails the same way. > - On one instance this happened seven times. Six tasks used their retry budget and ended blocked, or in progress with no runner. In the seventh, the scheduled same-run resume never ran at all: the task stayed `in_progress` with nothing running, and its monitor did not fire. > - This pull request re-binds a settled session to the upgraded launch profile, types the remaining mismatches so the server stops retrying them, and makes a scheduled remote resume survive the stale-lock sweep. > - The benefit is that a task continues its conversation after a deploy. When it cannot, it stops with a clear next action instead of stalling. ## Linked Issues or Issue Description Refs #18846, #18847, #18854 and #18860 (nearby native recovery changes). Refs #15795, an open change to the same stale-lock sweep function. It changes run teardown, not lock cleanup, so a second merge needs at most a small rebase. **What happened?** A per-turn native task used provider `acpx` with Claude, driver `acpx_runtime`, on a remote sandbox. 1. Run A succeeded and settled its provider session. The next run with the same build cold-attached it. 2. The instance deployed a new build. 3. A board approval woke run B in a fresh sandbox. Run B failed after about five minutes: `native_session_recovery_failed: Error: PRP command run.attach failed: {... "result":{"code":"command_execution_failed","message":"ACPX durable launch profile digest does not match runner startup"}}`. 4. The coordinator recorded `retryable_failure`, attempt 1, `native_session_interrupted`, recovery mode `exact_checkpoint_resume`, recovery state `resuming_session`, and a retry 30 seconds later. 5. Two seconds after the failure, the stale-lock sweep cleared the task's execution and checkout locks, because run B was `failed`. 6. Forty minutes later the attempt was still 1. The scheduled monitor had not fired. The task stayed `in_progress` with nothing running. **Expected behavior** A session that the previous run settled continues after a deploy. If it cannot continue, the run stops at once with a clear next action. A scheduled resume either runs or fails visibly. **Steps to reproduce** 1. Save a suspended ACPX session under one launch profile. 2. Change the sidecar bytes and the provider-pack authority at the same paths. 3. Cold-attach the next run in a new runner. Restore fails with the digest mismatch. 4. For the stall: leave a remote native run `failed` with a due `retryable_failure` coordinator, run the stale-lock sweep, then run restart recovery for that run. It returns `blocked: issue_execution_lock_changed` and writes nothing. The new Rust and PostgreSQL tests run these steps. **Paperclip version or commit** `master` at `b0c4e345267`. **Deployment mode** Native Runner, per-turn lifecycle, remote sandbox. The re-bind applies to every ACPX agent with an ordinary startup. ## What Changed ### Root causes - **Upgrade failure.** `AcpxLaunchProfile::canonical_digest` hashes the provider-pack authority, the sidecar command and arguments, each artifact path and SHA-256, and any candidate admission. For a remote sandbox the authority is the provider-pack manifest digest, which covers the Runner source revision and dist digest. So every deploy changes it. `AcpxDurableState` stores only the digest, and `validate()` rejects any difference. Only a Kimi candidate run binding had a carve-out. - **Silent stall.** The failure write keeps the task locks for `retryable_failure` and clears them only for `terminal_failure`. The stale-lock sweep treats any `failed` run as cleanable, so it cleared them. A remote retry goes through `claimNativeRestartRecoveries`. That claim requires `issues.execution_run_id` to name the run, so it returned `blocked: issue_execution_lock_changed` with no coordinator write and no deadline. The live recovery queue then dropped the entry. The local claim (`claimNativeSessionResumptions`) already reclaims a free lock, so only remote runs stalled. - **Monitor.** `noActiveNativeMonitorRun` holds a task's monitor while any native run for it has a coordinator that is not `committed` or `terminal_failure`. Run B's `retryable_failure` held it. That is by design; it ends when the resume runs. ### Runner (Rust) - `rebinds_settled_session` in `acpx_provider_backend.rs`. Restore re-binds a saved session to the current launch profile only when all of these are true: - The lifecycle is `suspended` and an identity exists. There is no active turn, pending runtime request, undelivered provider event, or unconfirmed provider exit. - The startup has no candidate admission. - The saved descriptor's sidecar command and arguments equal this startup's. - Any other digest mismatch keeps failing with the same message, plus `classification: "provider_state_incompatible"` in the durable command result. `DurableRunnerError` carries an optional closed classification for this. - A saved descriptor that names an agent release this build does not qualify (ACPX version, agent server package or version, or command digest) also gets the class. `validate_session()` and its message are unchanged; restore only adds the class. ### Runner (TypeScript) - `runnerRecoveryFailureClass` reads the classification from the authenticated `runnerCommandResult`. The Codex app-server driver (which also drives ACPX through runnerd) and `HarnessDriverBackend` pass it on as `failureClass`. - If replacement after a failed resume is not allowed, `executeNativeSession` throws `NativeProviderStateIncompatibleError` (message unchanged: `native_session_recovery_failed: ...`). If replacement is allowed, the existing path opens a fresh session in the same run, as before. ### Server - `nativeSessionFailureSourceCode` maps that error class to `native_provider_state_incompatible` with `instanceof`, not text. The disposition is permanent: `terminal_failure` at once, the task blocks, and the board owns the recovery action. Its next action says that a Retry continues in a fresh provider session with the same workspace. - `findNativeSessionResumeRun` also returns the barrier's `errorCode`. `rebindNativeSessionCheckpoint` does not resume a source with `native_provider_state_incompatible`. The existing fallback in `buildNativeExecutionWithCheckpoint` then starts a fresh normalized session with the full task bootstrap. - `sweepStaleIssueLocks` keeps both locks while a referenced run's coordinator owns a result-less `retryable_failure`, unless the task is done or cancelled. Its update also requires that no referenced run has a live status at the write, because a resume claim can return the run to running after the status read. - `claimNativeRestartRecoveries` reclaims a free execution lock for a `retryable_failure` resume, like the local claim does, when the checkout lock is free or held by this run and no other run for the task is queued, running or scheduled. It records `native.recovery.execution_reclaimed`. A lock held by another run still blocks. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` and `doc/execution-semantics.md`. ### Safety argument for the re-bind What the binding protects, and what this change keeps: - **Candidate admission** carries company, model, platform and expiry authority. A candidate startup never re-binds. The Kimi run-binding carve-out and all candidate rejections are unchanged. - **The agent that loads the conversation** stays exact. `validate()` runs on the saved descriptor before the digest check and requires this build's qualified agent profile: ACPX version, agent server package and version, and `commandDigest`. The saved identity must still match that profile and the provider mode. - **In-flight provider work** is never re-bound. A turn, runtime request, undelivered event or unconfirmed exit keeps the strict binding, so no old provider process or request continues under a new build. The next launch only loads a settled conversation. - **Launch shape.** A changed sidecar command path or arguments still rejects. Only the content at the same paths may change. - **Artifacts** are not trusted from the saved state. The current profile comes from the trusted controller at startup, and session start verifies every artifact's bytes and the restored provider identity, as before. What changes: a new provider-pack authority, or new sidecar or Node bytes, at the same paths no longer invalidates a settled conversation. The saved digest cannot show which part changed, so the rule does not depend on it. I did not add state fields to tell authority changes from artifact changes, because the rule would treat both the same. One existing assertion changes. `released_kimi_cold_restore_rejects_candidate_injection_and_changed_full_profile` expected a settled ordinary Kimi session to reject a changed authority, artifact digest, or saved digest. Those three cases moved to a new positive test. Command, arguments, every candidate grant field, a missing profile and run binding still reject. ### How a task in the reported state recovers after this deploys The task has old-digest saved state, a `retryable_failure` coordinator at attempt 1, and cleared locks. 1. Startup recovery, and the live recovery queue, run restart recovery for the due run. It reclaims the free execution lock and claims the run. The resume runs now. 2. Run B's failed `run.attach` is a durable outcome of B's exact run authority. If the resume replays that outcome instead of executing it again, it fails with the old, untyped text. The remaining attempts then fail the same way, and the run ends `terminal_failure` with `native_session_retry_exhausted`. The task blocks with a board recovery action. This takes about a minute. 3. If the task blocks, press **Retry** once. Retry selects run B as the source (it has process identity and controller evidence, so the #18860 skip does not apply) and rebinds B's checkpoint, which holds A's provider session. The new run cold-attaches under the new Runner, and restore re-binds the old digest because the session is suspended and quiescent. The conversation continues. Run B's failed attach never wrote the state file, so the state is still A's settled state. 4. Tasks that already exhausted their retries (blocked, or in progress after `native_session_retry_exhausted`) need the same single Retry. Their earlier attempts ran under the old Runner. Wait for the task to show blocked before pressing Retry. A Retry pressed while the resume is running joins that run. ## Verification - Rust, failing first: `settled_session_rebinds_to_the_upgraded_runner_launch_profile` failed on `master` with the production message. `unsettled_or_relocated_session_keeps_the_strict_launch_binding` failed on the missing classification. Both pass now. - `cargo test -p paperclip-runner-core --no-fail-fast`: all 41 test binaries pass. The library has 471 passed and 1 ignored. - `cargo fmt --check`, and `pnpm --filter @paperclipai/paperclip-runner typecheck` (TypeScript, generated contracts, `cargo check`): pass. - Server, failing first: `native-scheduled-resume-lock.test.ts` (PostgreSQL). On `master` the sweep cleared the locks and restart recovery returned `issue_execution_lock_changed`. Both negative controls (another run's lock, a queued run) already passed. All four tests pass now. - `pnpm --filter @paperclipai/server typecheck`: pass. - Runner vitest (`src/native-session-runtime.test.ts`, `src/backends`, `src/drivers/codex`, `src/drivers/provider-failure.test.ts`, `src/live/runnerd-codex-transport*.test.ts`, `src/control-plane/acpx-controller-recovery.test.ts`): 31 files, 1,011 tests pass. New: the typed error when same-session recovery is required, and the recovery class reader and pass-through. - Server vitest: the native executor, status corpus and provider-pack admission suites (1,268 tests) pass. New: the classifier makes the typed error permanent, and the same text stays retryable. Checkpoint selection returns `errorCode`, and an incompatible source starts a fresh session with the same workspace (unit and PostgreSQL). - Server vitest over the recovery area: stale-lock sweep, native resumption, process recovery, failed-workspace recovery, restart recovery (integration, diskless, preparation), controller recovery, startup retry, startup cleanup, recovered-owner follow-up, session resume, runtime selection and recovery deadlines. 926 passed. Four `heartbeat-process-recovery` tests failed in that loaded run. Three pass when I run them alone. `redacts opaque environment-bound credentials from Sentry diagnostics: result` fails the same way on unmodified `master` on this host (a local environment value is redacted). - Review round 1: a new sweep test returns the run to running before the clear. It cleared the locks without the write recheck and keeps them with it. The strict Rust restore test covers an older agent server release and an older ACPX release. The ACPX backend and durable runner unit tests, the lock, sweep, resume projection and controller recovery suites, `cargo fmt --check` and the server typecheck pass. In the second full cargo run, the two `stdio_mcp` relay tests timed out waiting 5 s for a child exit on this loaded host. They do not use the changed code and passed in the first full run. - After the rebase onto the current `master`: the new lock test, failed-workspace recovery, stale-lock sweep and session resume suites pass (189 tests). Server typecheck, `pnpm check:module-boundaries` and `git diff --check` pass. - CI on `929ef4508d2`: in the first run, `isolated-repository-rehome` hit its 15 s test timeout, `idle-completed-history` could not reach its fixture runtime service, and e2e shard 3 was cancelled by a runner shutdown. A rerun of the failed jobs passed. All 55 checks pass. - Not verified: a live remote sandbox deploy, and whether run B's same-run resume replays its recorded `run.attach` failure or executes it again. The Retry path is tested at source selection and checkpoint rebind (server) and at cold attach (Runner). It is not tested end to end through a sandbox. ## Risks - A settled session now continues under new sidecar bytes. If a sidecar change made an old ACPX record unreadable, session start fails as any failed resume does. It does not corrupt state, and the run follows its normal recovery policy. - The sweep now keeps locks for a failed native run with a scheduled resume. If that resume can never be claimed (for example the agent stays paused), the task keeps the lock until the coordinator settles or the task closes. Before, the sweep cleared it, and the resume still never ran. - If another run holds the task's lock, a scheduled remote resume still stays blocked and the coordinator stays `retryable_failure`. This change does not settle that case. It needs a separate decision about superseding the resume. - `native_provider_state_incompatible` is a new failure code. No UI or migration change is needed. Older servers never see it. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8771606fd8 |
test(e2e): cover browser task editing and reopening (#18782)
## Thinking Path > - Paperclip lets people manage tasks and agent work. > - Daily QA uses reusable Product E2E journeys against a named target. > - Existing task journeys cover creation and assignment. > - They lack saved title and label edits across completion and reopening. > - This pull request adds one browser-only lifecycle journey with independent saved-state checks. > - It also puts shared registry entries on separate lines so independent case PRs can use distinct insertion points. > - The registry formatting preserves every token, value, entry order and case body. ## Linked Issues or Issue Description Refs: #18655. **What existing behavior does this improve?** Product E2E coverage of ordinary task editing and completion. **Current behavior** The reusable catalog covers task creation and assignment. It lacks one journey that edits a task, reloads it, completes it and reopens it while retaining its saved context. Several shared registries put all entries on one line, so independent case additions conflict. **Proposed behavior** Create a task assigned to the signed-in person. Edit its title and create one label through the visible UI. Change its status to In Progress, reload it, mark it Done, reload it, reopen it to Todo and reload it again. Require the exact saved task and label throughout. Preserve its description, assignee, project and priority. Allow no agent run or model usage. Put the existing shared registry entries on separate lines without changing their contents or order. ## What Changed - Add `task-edit-lifecycle` and six required checks for browser creation, saved edits, reload, completion and reopening, preserved context and no agent work. - Add valid, wrong and missing evidence controls. Retain allowlisted observations and bounded GET failure receipts in private evidence. - Register the case, feature associations, catalog expectation, source-hash inputs and browser-only result validation. - Extend target ownership for run-owned labels and one exact predeclared title change. Require compatible scoped cleanup before selecting the case. - Open the real side panel or Properties control when needed. Exclude inert and aria-hidden Labels rows and require visible readback. Document inline title editing and the case prerequisites. - Keep priority editing disabled. Verify that the saved priority remains unchanged. - Keep Cursor test observations outside its existing Stop deadline, preserve the full rejection/Stop overlap matrix and add pre-release no-provider-call controls. Production behavior and deadline values stay unchanged. - Format five shared registry files with one entry per line: the JourneyId and AgentJourney unions, definition filenames, expected catalog IDs and agent run counts. No registry entry moves in this PR. ## Verification - Current head is `d1af3aae3712455c90db937fa7a5141555710506`. Its only change from integrated head d40f28 is the Cursor driver test-fixture ordering repair described below. The prior normal merge consumes actual master `d59791828482f213713ca55000c40e6c9c6c03e5`, including reviewed UI repair #18848 and the existing idle-history fixture repair from #18853. It has both real parents, no conflicts and the required trailer. All seven lifecycle case/oracle/catalog/documentation blobs and six selected registry constructs remain exact 10625b. The idle-history fixture is byte-identical to master. The PR diff has 13 paths, including the one Cursor test file, and no feature-map or production-driver changes. - 167 focused tests pass across catalog, target contracts, agent evidence, lifecycle evidence and observation reads. E2E TypeScript and credential-free discovery of21 Product E2E cases pass. The real merged idle-history and exposure-reservation suites pass all 109 tests in132 seconds, including the owned HTTP-process adoption assertion and explicit unknown-listener denial/drift controls. Canonical shared/SDK, database and Runner TypeScript dependencies were rebuilt for the changed master inputs. All4,220 tracked package/script/root inputs and 4,954 dependency outputs are recorded; the outputs remain unchanged after the tests and browser journey. The first database build stopped because child PATH selected a different global pnpm and its version switch could not fetch signed metadata. The preserved attempt made no package or source change. Using the already-installed verified pnpm9 on child PATH passed. No install, Rust or provider-build global setup ran. - One fresh canonical managed-local browser journey passes on exact d40f28: all six checks and authentication, setup, journey, collection and cleanup pass. Cleanup releases the disposable instance. The journey takes 12.059 seconds and has explicit zero-run readback at all five saved states. Its runtime commit isd40f28. Definition hash is `f262707de2e544cecdb9c5c4c5222d61f04a122575b438658dec069e1a93f9cb`. The prior10625b local6-check/5-phase pass remains separately recorded with its own source and definition hash. - A real closed-panel control previously reproduced the original label-interaction failure using only the presentation preference `paperclip:panel-visible=false`. The corrected journey passed all six unchanged checks through an ordinary Show properties click. A separate canonical743 local run passed all six checks and five phases with released cleanup. No domain outcome was supplied by that control. - A separate QA staging qualification passed all six checks and five phases on exact743, with closed cleanup and verified zero agent work. That original pass remains bound to743. It is not regraded or relabeled as a staging run of10625b. Daily admission remains a separate coordinator action. - Earlier material failures remain part of the verification record. The first managed-local setup failed before authentication with an embedded PostgreSQL exit code of null; its original cause remains unresolved. A later loader diagnostic found missing library aliases in this isolated checkout. Existing bundled libraries were hash-verified and the aliases were restored only there. Subsequent real runs exposed an overlong label name, a status picker name mismatch and the closed-panel issue. Each failed result and cleanup record is retained. The fixes preserve the full campaign name, validate the production48-character label limit before mutations, and use the actual visible controls. - Original full CI [41652807327](https://github.com/paperclipai/paperclip/actions/runs/41652807327) failed an unchanged Discord fixture assertion, attention-retention fixture limits and the CLI import/export startup limit. One bounded recheck recovered Discord but repeated the other failures. The reviewed merged retention-fixture repair was consumed normally through #18770. Full CI [41667811850](https://github.com/paperclipai/paperclip/actions/runs/41667811850) passed all E2E shards but exposed inbox/account-fixture failures. This branch consumed the actual reviewed repairs from #18798 and #18799. Original logs and failed assertions remain retained; no timeout or oracle was weakened. - Original743 CI [41682757346](https://github.com/paperclipai/paperclip/actions/runs/41682757346) failed the unchanged CLI suite's30-second child-server startup hook. A bounded local phase probe passed the same startup oracle but did not prove the original CI cause. One coordinator-approved official recheck followed. Timestamp comparison shows46 non-skipped jobs retained their original start and end times; only the failed CLI job and final aggregate have new start and end times. The skipped selection job remains unclassified. New GitHub job IDs alone did not prove new executions. The original startup failure remains preserved and its cause remains unproven. - Formatting-head CI [41726104217](https://github.com/paperclipai/paperclip/actions/runs/41726104217) failed the upstream idle-history process-adoption test and dependent verify job:46 jobs passed, one was skipped and two failed. Its actual synthetic checkout had parents masterf187 and10625b; the failed test is absent from 10625b itself. The assertion expected one adopted process and received zero. A pure production-function control proves a conditional reserved-port drift path, but the actual failed CI port and full reconciliation counters were not logged. The actual CI cause remains unproven. This branch consumes the already-reviewed bounded fixture-port repair in #18853 through its normal actual-master merge. The original failure is retained. No unchanged-code retry, timeout increase, skipped assertion or ownership-policy change was used. - Integrated-head CI [41730231620](https://github.com/paperclipai/paperclip/actions/runs/41730231620) passed the formerly failing server1/12 lane, Typecheck, Build and E2E. It failed the Cursor package fixture below and the dependent verify job. A separate server4/12 job lost runner communication; the exact runner instance and Spot-request readback confirm a capacity interruption. Its unavailable log is not treated as a test pass. The original failures remain retained. No old-head retry was made. - The Cursor fixture sets a 300 ms cancellation budget, holds a rejected-create checkpoint and originally performed directory/file observations while that clock ran. Controlled400ms observation latency on the original fixture reproduces the same fenced Stop error. The same diagnostic passes after moving file observations before the cancellation clock, or while the initial Stop-checkpoint gate still prevents it from starting. The driver,300ms configuration and timeout negatives are unchanged. Every original assertion AST remains, and three new assertions forbid provider observation/cancel calls before releasing the checkpoint. All81 driver tests,12 client tests and43 checkpoint-archive tests pass; Runner TypeScript build and E2E TypeScript pass. The actual CI inner error and observation duration were not logged, so its precise original cause remains unproven. - Final d1af3a full CI [41733938939](https://github.com/paperclipai/paperclip/actions/runs/41733938939) attempt3 is successful. Attempt1 lost four exact-bound Spot runners; attempt2 recovered the Runner and both server lanes but lost the replacement E2E runner to another confirmed Spot capacity interruption. Two separate coordinator-authorized official failed-job rechecks were recorded. Attempt3 passed the E2E path and final aggregate. Successful executions retain their original timestamps; new job IDs alone were not used as proof of re-execution. All original failures remain retained. - Fresh Greptile check126258142754 completed successfully on exact d1af3a with 5/5 and zero unresolved threads. Full recursive workspace typecheck, all workspace tests and the full build were not repeated locally; the corresponding hosted full CI gates pass. ## Risks The case requires target support for `label` ownership and scoped `taskEdit.revisedTitle` cleanup. Older target adapters must not select it. Cleanup must reconcile lost creation responses and exact predeclared title saves without touching other resources. Registry formatting changes the harness definition hash because that hash includes source bytes. Historical runs must keep their original source and hash. The formatting does not change runtime values, order, dispatch, case checks or product behavior. Later case PRs must verify their actual merge resolutions and final-head checks before admission. ## Model Used OpenAI GPT-6 through Codex with reasoning, tool use and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
11ac90e31f |
fix(build): typecheck server/src/__tests__ and fix every type error (#18739)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server has about 1,100 test and helper files in `server/src/__tests__` > - `server/tsconfig.json` excludes that directory, Vitest does not check types, and no script runs the tsconfig in that directory > - So a test can call a function with the wrong arguments and still pass, and no tool reports it > - This gap hid 2,168 real type errors, and some of them hid tests that did not test what they claim > - This pull request fixes all of these errors and adds the directory to the server layers from paperclipai/paperclip#18737 > - The benefit is that a type error in a server test now fails the typecheck, with no baseline and no backlog ## Linked Issues or Issue Description Refs paperclipai/paperclip#18737, which is merged and added the server layers. Refs paperclipai/paperclip#18738, which is merged and checks the test projects outside `pnpm -r`, so it is not a duplicate. Refs paperclipai/paperclip#18137, which records that this directory has no check. **What happened?** No tool checked the types of 1,074 of the 1,101 files in `server/src/__tests__`. I searched `package.json`, the workflows, and all tsconfig files. 27 files are in the program of another tsconfig, and these 27 have 0 errors. 11 of the 27 are in `helpers/maintenance/tsconfig.json`. No script ran that project when I measured this. paperclipai/paperclip#18738 now runs it. **Expected behavior** The server typecheck checks these files, and a type error fails the check. **Steps to reproduce** 1. On the base commit, run `tsc -p server/src/__tests__/tsconfig.json --noEmit`. It reports 6,536 errors and needs 12.3 GiB. 2. Add a type error to any file in `server/src/__tests__`. 3. Run `pnpm --filter @paperclipai/server typecheck`. It passes. **Paperclip version or commit** I measured the error breakdown below on commit `8dfd632452`, with TypeScript 7.0.2. The branch now includes master up to paperclipai/paperclip#18865. **Deployment mode** Not applicable. This issue applies to development and CI. No production source file in the server changes. ### Error breakdown Each row adds one change to the row above. | Step | Total errors | Change | Cause | | --- | --- | --- | --- | | Config as committed | 6,536 | | | | Add `src/types/*.d.ts` | 4,447 | -2,089 | Config. The Express request types were missing. | | Set `rootDir` to the repository root | 2,436 | -2,011 | Config. | | Set `allowImportingTsExtensions` | 2,181 | -255 | Config. Vitest accepts `.ts` imports. | | Add `ES2024.Promise` to `lib` | 2,138 | -43 | Config. Node 24 has `Promise.withResolvers`. | | Set `allowJs` for `.mjs` imports | 2,148 | +10 | Config. It removes 8 `TS7016` errors and shows new real errors. | | Set `declaration` to false | 2,142 | -6 | Config. Tests emit nothing. | | Count only files in `server/src` | 2,058 | -84 | Other packages: 53 Daytona, 29 Runner test, 2 UI. | So 4,478 of the 6,536 errors (69%) came from config or scope, and 2,058 are real type errors in 314 files. The 29 Runner test errors are also real, which gives 2,087 errors. Tests that master added after this measurement had 81 more errors, so this pull request fixes 2,168 errors. This table classifies the first 2,058 errors. | Kind of real error | Errors | | --- | --- | | Wrong argument or assignment type | 1,023 | | Stale or partial fixture objects | 525 | | Property that the type does not have | 317 | | Missing narrowing of null or unknown | 101 | | Implicit any | 60 | | Missing name or module | 16 | | Other | 16 | ### What the type errors hid Most errors were stale fixtures and loose mocks. These errors hid real test problems, and this pull request fixes them: - **A test that never ran.** `instrumentation.test.ts` required `sdk-trace-base`, which is not installed. So 3 span tests were skipped. They now run and pass. - **Assertions that could not fail.** - `issue-thread-interactions-service` used a company ID as an issue ID. - `status-card-evaluation-work` compared `[]` with `[]` in a rollback check. - `slack-conversation-lifecycle` read `item.issueId`, which does not exist. - `email-channels` used a permission key that does not exist. - `heartbeat-project-env` imported `LOW_TRUST_REVIEW_PRESET`, which is not exported. The value was undefined. - **Postgres processes that did not stop.** 11 test files called `stop` on the test database helper. That method does not exist, and the helper method is `cleanup`. - **Arguments that the service ignored.** - `methodKey` is now `connectionMethodKey` in the Daytona, Unikraft, 1Password, and Railway tests. - 108 calls passed `"owner-user"` to a parameter that expects an options object. - 9 chat wizard fixtures passed `managementContext` as `source`, so they ran `existing_app` and not `manifest`. - 2 tests passed a `{ userId }` actor that the function ignores. - Inserts into `environments` passed a `companyId` column that the table does not have. - **Problems in tests that master added during this work.** - `sandbox-creation-operations` read `database.connectionString` when an external test database was set. In that mode `database` is undefined, so the fixture would stop with a TypeError. It now uses the external URL. - `idle-completed-history` set a Dot operation to `dispatched`. That is a Muse status, and the Dot type does not have it. The fixture now uses `admitted`, which is a real in-flight Dot status. ### Possible product problems, not changed These may be real problems in production code or types. I did not change them, because this pull request does not change production behavior: - `decisions-service` `service.create` does not validate the lenient `cancel_issue_tree` input. - `createConnection` ignores `input.connectionPurpose`. - The `createConnection` route schema default stores `{}` for `transportConfig`. - `prepareManagedAiRuntime` accepted the invalid value `mode: "connection"`. - `createBinding` accepts invalid `projectionClass` strings outside class 3. - `ensureManagedProjectWorkspace` forwards an untyped `repoRef` by spread. - `routineVariableSchema.label` is optional, but `RoutineVariable.label` is required. - `ToolRunContext.projectId` may be too strict. - `withdrawInteraction` reads `issue.status`, which its type omits. - `issueService.list` and `reconcileNativeFinalizations` omit fields that they return at runtime. - `createGitHubExternalObjectProvider` handles `tokenProvider: null`, but its type forbids this value. ## What Changed - Fixed the 2,168 type errors. The diff changes 322 test and helper files. - Corrected `server/src/__tests__/tsconfig.json`. Editors and the typecheck now read the same options and report the same errors. - Added three `test-tree` layers to `server/scripts/typecheck-layers.mjs`. They read the server layers as declarations, and any error fails them. A hash of the path selects the shard, so a new file does not move other files. - Added `server/scripts/typecheck-stubs/daytona-sdk.d.ts`. Server tests reach Daytona plugin source, and the root install does not have `@daytonaio/sdk`. - Changed two imports in `ui/src/lib/issue-execution-policy.ts` to use `.js` extensions, because server tests import this file. - Made `declarationsConfig` and `--declarations-only` skip the `test-tree` layers, because these layers emit nothing. Other projects read only the source layers. Added a test for this. - Updated `doc/DEVELOPING.md`. ## Verification Run these commands: ```sh pnpm --filter @paperclipai/server typecheck pnpm exec vitest run server/scripts/typecheck-layers.test.mjs ``` Results on macOS (arm64), Node 24, 16 cores. Each check ran after the merge of paperclipai/paperclip#18738 unless the row says otherwise: | Check | Commit | Result | | --- | --- | --- | | Server typecheck, `PAPERCLIP_TYPECHECK_LOW_MEMORY=1`, all 7 layers | `6e45a08573` | Exit 0, 0 errors, 62 seconds, peak 3.52 GiB | | `node scripts/typecheck-test-projects.mjs --low-memory` | `a3306bd13e` | Exit 0, 0 errors, 42 seconds, peak 3.25 GiB | | The same script with `--reuse-declarations` | `1b878aabc0` | Exit 0, 0 errors | | `typecheck-layers.test.mjs` and `live-recovery-work.test.ts` | `01db5634e4` | 90 passed. These include real compiler runs for a pass, a type error, and an invalid tsconfig. | | `tsconfig.declarations.json` after a full server typecheck | `0fb1e4fb0a` | It references only `below`, `cycle`, and `above`. No generated config names a `tsconfig.layers.json` file. | | Owned runtime test in `idle-completed-history` | `e8f2f448ac` | Passed, with the fix that master later replaced | | `idle-incoming-wake`, `plugin-idle-manager`, and the changed `chat-channels` test | `6e45a08573` | 30 passed | | Runner `native-execution.test.ts` | `eab4dcc1f4` | 114 passed | | The 19 server test files that changed since `eab4dcc1f4`, and `typecheck-layers.test.mjs` | `259737b700` | 4,275 of 4,277 passed. The 2 failures are in `chat-channels` and are in the list of 7 below. | The peak is the largest single process, from `/usr/bin/time -l`. Peaks for the server typecheck were between 3.42 GiB and 3.56 GiB across 8 runs. Peaks for the test projects were 3.25 GiB and 3.37 GiB across 2 runs. I did not measure a cgroup. **Runtime check of the fixes.** Types do not prove that a test still works. So I ran all 309 changed server test files, 11,929 tests, before the rebase onto paperclipai/paperclip#18737: | Result | Tests | Detail | | --- | --- | --- | | Passed | 11,913 | Includes the 2 tests and the 54 `dot-runner` tests that passed in a later run, as described below. | | Failed | 7 | The same 7 tests also fail on the commit before the fixes. | | Skipped | 9 | These tests have their own skip conditions: `sentry` 5, and 1 each in `bundled-plugin-registration`, `customer-success-cloud`, `onepassword-connection`, and `workspace-runtime-https-live-exercise`. | The 7 tests that fail with and without this change, on my machine: - 3 tests in `chat-channels` and `email-channels` stop with `ENOENT` on a `SKILL.md` file. The path is in the parent checkout, above the git worktree that I used, so I expect that the worktree is the cause. I did not prove this. - `workspace-instance-cleanup` compares `/private/var/...` with `/var/...`. This is a macOS symlink difference. - 2 tests in `git-reproducible-managed-checkouts` get `unknown_base_git_files` and expect `unpushed_managed_checkout`. I did not find the cause. - 1 more test in `chat-channels`, `materializes direct external-chat finals once`, gets `undefined`. I did not find the cause. CI is the check for these 7 tests. `ci / verify` passed on `eab4dcc1f4`. Failures during this work: - My fix to `agent-runtime-connector-access` changed an invalid permission key in the setup, but not in the assertion. The runtime check found this. I corrected the assertion, and the file passes. - `heartbeat-plugin-environment` failed one wait in the full run. It passed three times alone, so I think that load caused the failure. It can fail again under load. - `dot-runner` skipped all 54 tests in the full run because `cargo` was not on my `PATH`. It then passed 54 of 54 with the toolchain on the `PATH`. - My first `lib` value removed the default DOM types, and the error count went up by 375. I added the DOM libraries. - My Daytona stub with bare `any` left 48 errors, because generic helpers then inferred `unknown`. I added shapes that return promises. - One fix to `agent-secrets-routes` used `class_3_static_lease`, and the test failed. That class allows only specific paths. I reverted to `unclassified`. - `ci / General tests (server (8/12))` failed on `259737b700` because of the flaky port in `idle-completed-history`. paperclipai/paperclip#18853 fixed this test on master, so this pull request keeps that fix. **Audit of the diff.** I counted these patterns in added and removed lines, to find fixes that only hide an error: | Pattern | Added | Removed | | --- | --- | --- | | `@ts-ignore` | 0 | 0 | | `@ts-expect-error` | 0 | 1 | | `.skip` | 0 | 0 | | `as any` | 9 | 9 | | `: any` | 11 | 2 | | `as never` | 8 | 2 | | `as unknown as` | 35 | 4 | | `Promise<unknown>` | 16 | 5 | | `expect(` | 128 | 113 | No hunk removes more `expect(` calls than it adds. Three removals of test content were intentional, and I checked each one: - `muse-control` removed `signalToken`. Line 1012 already asserts that it is absent. - `merged-pr-confirmation-sweep` removed legacy aliases. Other tests cover them. - `heartbeat-workspace-branch-containment` removed a `finalize` branch that no test row can reach. Some partial fixtures stay unchecked through `as unknown as` and `Promise<unknown>`, mostly in `issue-agent-mutation-ownership-routes` and `issue-comment-reopen-routes`. **Accuracy of the check.** One program over the tree and the three shards reported the same errors at the same positions. Memory, as the peak of the largest process on one thread, was 3.93 GiB for one program and 2.43 GiB for the largest shard. I measured these two values before the fixes. Not run or not proven: - I did not run the full `pnpm test:run`, or a root `pnpm build`. I expect CI to run the full suites. - I did not run the 4 GiB container check, or the full `scripts/typecheck-low-memory.mjs`. The peaks above come from macOS, so they are not a Linux cgroup measurement. - I did not run the UI typecheck after the merges. - `workspace-runtime-https-live-exercise` is skipped, so its `cleanup` fix did not run. ## Risks - **A large diff in test files.** 328 files change. A pull request that edits the same tests can get merge conflicts. New tests on master can also add type errors before this merges. Any such error fails this check and needs a fix here. - **A higher cost for changes to source types.** A change to a type can add errors in test files that the author did not touch. The author must then fix them. This is the purpose of the check, but it is new work. - **Changed test behavior.** Some tests now run paths that they did not run before, such as the 9 chat wizard fixtures and the 3 span tests. They pass here, but they can be more sensitive to timing or to the environment. - **The Daytona stub can go stale.** The stub is written by hand. If the plugin uses a new SDK member, the typecheck reports an error until someone updates the stub. - **The release path.** `release-verify.yml` and `scripts/release.sh` run `pnpm -r typecheck`, so they now run this check. - **The tsconfig change.** The test transform reads `server/src/__tests__/tsconfig.json`. The options that I changed are for type checks, and the runs above passed. - **A gap that remains.** Tsconfigs in `packages/paperclip-runner` and in most sandbox provider plugins exclude `*.test.ts`. This check does not read most of those test files, so they can still drift. That work is separate from this pull request. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with reasoning, tool use, code execution, and subagents. The runtime does not show the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass (7 tests fail locally, with and without this change. See Verification.) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
683fa4725e |
fix(evals): refresh native runtime and pin reviewed catalog repairs (#18780)
## Thinking Path > - Paperclip lets people inspect and manage their agents’ work. > - The timeline must report canonical ownership history, and OpenCode tool failures need bounded diagnostics. > - The measured candidate also had independently reproduced question-navigation and oracle defects. > - This change promotes the reviewed production fixes and pins one immutable, qualified source composition. > - Original queries, model, counts, clocks, privacy controls and qualification holds remain intact. ## Linked Issues or Issue Description Refs #18775, #18777, #18779, #18711, #18713, #18718, #18699, #18798, #18799. **What happened?** The measured candidate predates merged native-runtime fixes and contains fixture defects. The timeline omitted recorded reassignment/removal history. Finish-input errors lacked a closed correlation and shape witness. **Expected behavior:** project recorded ownership transitions without inventing facts, distinguish successful native execution from failed verification steps, and retain content-free finish diagnostics. **Steps to reproduce:** repeat an unchanged assignee while editing another field; then reassign or unassign it. Only recorded transitions should appear. Invalid OpenCode finish input should produce a correlated rejection witness without values or schema text. Related #18699 adds gated company OTLP diagnostics. This change uses the existing local callback; it adds no exporter or capture policy. ## What Changed - Promote canonical timeline creation, reassignment and removal projection from #18777. Initial child ownership requires a recorded fresh-creation receipt; delegation requires a readable recorded parent. Unknown legacy facts fail closed. User actors/owners require active company membership; agent actors/targets must exist in that company. Spans, events and edges share that admission filter. Remove the introduced all-history absence query and use guarded indexed UUID lookups. - Add the shared/OpenAPI `unassigned` event kind, preserving newer main contracts and issue-route/service code through reviewed semantic joins. Existing UI consumers remain generic; no new visual marker is claimed. - Add bounded OpenCode finish diagnostics: schema digest, hashed call identity, closed shape enums and validation category. A throwing sink does not change admission or cancellation. - Pin immutable candidate `fd5e3b1e5a379524c0d1651ed8273e75ffd86541`, composed from combined #18777 source `b734393eff0d7ef85b7ad8abd556d2a66ca3a4e6` plus the reviewed MCP diagnostic delta. It preserves actually merged #18775 `97c07d149b5670088faf15317dcf603c75e2af87` and #18779 `a352c057d475067b4fe67375f06785287dfbe0ce`, and merged runtime fixes #18711, #18713 and #18718. - Qualify narrative handling for adjacent repair/verification sentences, intervening native-failure wording, and negated repair objects. Questions, denials and hypotheticals retain their controls. Preserve Previous question beside the qualified Entity type import; separately restore the already-called `internalResultLink` goal helper binding. That import is a runtime fixture repair, not a type-only change. - Include the narrow older-baseline CI typecheck closure and causal heartbeat test observers in the candidate. All four default typecheck projects execute. Main already includes the protected upstream tooling and newer heartbeat fixture, so neither is overwritten with an older candidate file. - Preserve all 244 identities, original queries, model, native/human counts and clocks, including the explicit prior Studio accounting exception. The full catalog stays on its candidate branch; the 196/244 gate and historical SDK runtime holds remain. ## Verification The primary’s f478 retry repeated the real composer-hidden five-second assertion (1 failure,18 passes,1 skip). Investigation established a setup readiness gap: an outer route response can delay mounting the chat thread beyond the assertion. The successor adds the existing real-thread `aria-busy="false"` readiness barrier before the unchanged composer assertion. The original120-second scenario and digest, unread, source-link, history, and reply checks are unchanged. The full owning browser scenario passed naturally in10.2s and with its first real chat response held6000ms in12.0s. This synchronizes fixture setup; the underlying AWS late-mount latency remains unproved. **Current main gates passed:** exact head `823dab49fcd8470140c178771e5c0c6b78725e97` has [successful normal CI](https://github.com/paperclipai/paperclip/actions/runs/41724620555), including both required aggregates, fresh Greptile 5/5 and zero unresolved review threads. Paperclip Review is neutral with a completed substantive 5/5 and no actionable defect. The current optional Superagent Security Scan is neutral/inconclusive: Security Graph context retrieval returned HTTP503, so its assessment did not start. That result supplies no security PASS credit. The primary source PR retains its separate current-head gate before Root sequences normal protected merges. - Reviewed MCP controls: 130 on main and 129 on the older candidate. Meaningful parent comparisons failed before implementation/correlation correction. Controls cover actual offered schema identity, null/omitted admission, bounded rejection categories and throwing-sink cancellation. - Exact unchanged production postimages retain 63 Postgres/public-route controls, nine plan controls and the focused index-plan proof. Owner OpenAPI passes 32 controls; main’s newer contract passes 33. Earlier setup/sandbox failures remain recorded; unchanged database controls were not replayed. - Narrative owner controls pass 216, retaining all prior 207 controls. The independent peer passes 144 balanced cases; earlier 108 and 39-case matrices remain retained. Actual consumer parent failures reproduced the review findings before repair. - Main’s server NoEmit passed on its protected baseline; only two pin files changed afterward. The candidate’s prior full server and four-project checks passed. All server/package/script/root-lock inputs remain byte-identical; the changed Runner project’s guard passes on the current source. Compatibility driver controls pass 47 and six owning suites pass 82. The one explicitly reviewed Docker-only unit omission concerns an unimported image-build feature; all four intended project guards still execute. - Pin/adapter/contract controls pass 47. All 242 expanded declarations, the two legacy dispatch rows, and all 244 singleton private policies retain their authenticated identities. Descriptor, launcher and private-protocol bytes remain exact; declaration loading invoked zero fixture callbacks. - All 506 SDK input blobs match the meaningfully compiled `6abe5739` source vector. Prior real shared/SDK builds measured declaration/map changes and zero runtime-JavaScript changes; a compiled consumer accepts the new kind and rejects an unknown kind. The reviewed registry remains unchanged. Default source proof, 51 SDK controls, 20 custody controls and 16 UI consumer controls remain qualified; runtime/compiler/parser/Zod/canary pins and held runtime grades are unchanged. - The full Mac heartbeat run on prior observer source retained **58 passes / six failures in 802.78s**. Two nonassignee controls subsequently pass focused checks. Four late timeout causes remain unresolved locally; no flakiness/infrastructure cause or complete local pass is claimed. [Actual AWS CI](https://github.com/paperclipai/paperclip/actions/runs/41712023737/job/126182620050) ran this exact heartbeat file with **64 passes in 107.209s**. Its server/package inputs are unchanged in the current narrative-only successor. The source-identical f478 heartbeat file also passed all64 AWS tests in105.812s; source inputs remain unchanged. Current main ordinary CI passed; the primary source PR must independently close its latest-head ordinary CI gate. - Historical main [run 41713049555](https://github.com/paperclipai/paperclip/actions/runs/41713049555) recorded six synchronized runner-shutdown failures and cancellations; the shutdown cause is unknown. A separate chat passive-control assertion observed one `db.transaction` call where zero was expected. Finite local probes passed 2+2 but did not establish a cause or late stack. No external publication, cleanup cause, source fix or flakiness is inferred. The primary’s separate composer-visibility CI failure also remains an inconclusive historical result; no guessed fix is credited. No old-head retry was used. - Earlier UI/inbox fixture failures and the CLI startup timeout remain historical. UI/inbox repairs are now upstream in #18799/#18798 and main preserves those files verbatim. The CLI timeout cause remains unknown. Broader local repository/Rust qualification is not claimed; fresh hosted CI supplies broader coverage. - All four actual repair merges, an exact merged-tree/diagnostic source join, and fresh controller/source/private-policy/active-run guards precede the ten-case TestOps measurement. No paid or model eval was launched for this PR; the original three passes/seven failures remain unchanged. ## Risks The next ten-case model result is unproved. Earlier MCP timeout and continuation-shape causes remain unknown; diagnostics provide future evidence without relaxing validation or rewriting grades. Prior local/CI failures are retained, including the unresolved chat assertion and local heartbeat timeouts. Initial ownership requires canonical in-window audit facts. Missing legacy facts remain unknown. The additive `unassigned` contract is synchronized across shared/OpenAPI/server consumers, with no new UI marker. SDK runtime holds and the broader 196/244 qualification gate remain. Normal protected merges require fresh checks and resolved feedback; no CODEOWNER or branch-protection bypass is authorized here. ## Model Used OpenAI `gpt-6.1-sol` through Codex. Capabilities used: repository analysis, code changes, source validation, and local test execution. The context-window size is not exposed by this harness. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — current focused/type/pin controls pass; the prior full Mac heartbeat failures remain disclosed and receive no local PASS credit - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green — current main normal CI and required aggregates passed; optional neutral reviews and separate primary gate are disclosed above - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — exact current main head; full thread pagination verified - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
9fb06afbfa |
fix(runner): explain ACPX session-start failures and name typed causes (#18847)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner starts ACP providers (Claude, Codex, Pi and others) through the ACPX sidecar. `session.open` starts the provider session. > - When the provider session does not start, Runner reports `classification=session_ensure_failed`. If the provider failed with a plain error and not with an ACP `RequestError`, no provider cause crosses the sidecar. Then the run error, the run events and the run log do not say why the session did not start. > - A provider can also type the failure, for example with ACP's authentication-required code. Runner reported that failure with the same generic classification. The server could not tell a limit or a sign-in failure at session start from other startup failures. > - On one instance, a Claude agent hit its provider usage limit. The first run failed in the turn with a typed limit failure. Each later run for many hours failed at `session.open` with only `session_ensure_failed`. Operators could only guess the cause from the timing. > - This pull request keeps a redacted, bounded cause for every session ensure rejection. It names a typed provider category in the closed classification. > - The benefit: an operator can read why a session did not start, and a later recovery change can treat a typed startup failure by its category. ## Linked Issues or Issue Description Refs #18622, which added the `ProviderFailure` cause for startup and turn failures. This change closes the remaining startup gaps. Refs #17543, which is a related open change to the same classifier. It keeps the JSON-RPC code range as a closed code. The two changes edit the same lines, so the second one to merge needs a small rebase. **What happened?** A native ACPX run failed before its first prompt with `ACPX sidecar command session.open was rejected (retryable=false, classification=session_ensure_failed)`. When the ensure rejection was not an ACP `RequestError`, the run had no other cause data. When the provider typed the failure (for example ACP `-32000` "Authentication required"), the classification stayed the same generic value. **Expected behavior** Every session ensure rejection keeps a redacted provider cause that an operator can read on the run. A typed provider failure keeps its category in the cause and in the closed classification. Provider text never goes into the closed error text. **Steps to reproduce** 1. Make `runtime.ensureSession` reject with a plain `Error`, or with an ACP error that has code `-32000`. 2. Open a qualified ACPX runtime through the sidecar. 3. Before this change, the plain error has no `providerFailure`, and the `-32000` error has category `unknown` and `classification=session_ensure_failed`. **Paperclip version or commit** `master` at `e0276e0166c`. ## What Changed - `acpxProviderFailure` reads ACP's reserved authentication-required code (`-32000`) as category `authentication`. It also reads the `errorKind` that Claude ACP puts in ACP error data (`rate_limit`, `billing_error`, `authentication_failed`, `overloaded` and others). The category table is now a `Map`, so an `errorKind` such as `constructor` cannot select an inherited object key. - The session ensure path in `codex-runtime-adapter.ts` keeps a cause for every rejection. A rejection without an ACP error keeps its own message, redacted with the launch environment and the existing credential patterns, with category `unknown`. An abort reason is Runner cancellation, so it does not become a provider cause. A non-`Error` rejection now keeps its cause on the wrapper error. - `acpx_sidecar_transport.rs`: for `ACPX_SESSION_ENSURE_FAILED`, a schema-valid cause with a typed category gives `classification=provider_<category>` (for example `provider_quota_exhausted`, `provider_rate_limit`, `provider_authentication`). An untyped or invalid cause keeps `session_ensure_failed`. Only the validated enum value is used. The cause's text never goes into the error string. - `runnerd-codex-transport.ts`: when a command result has a provider cause, the reported error now keeps Runner's closed ACPX rejection text, so the classification stays in the run error. Before, it changed that text to "Provider request failed". Any other message (for example a Codex `RPC failed: ...` message) still gets the generic text. Provider text still travels only in `providerFailure`. - The credential-free Pi probe (`pi-closed-startup.test.mjs`, `scripts/pi-public-install-probe.mjs`) now expects `provider_authentication`. Pi reports its missing credential with ACP's authentication-required error. - Docs: `doc/run-log-events.md` and `doc/architecture/runner-pi-capabilities.md`. **What the Claude ACP agent returns at session start** (`@agentclientprotocol/claude-agent-acp` 0.73.0 with the repository patch, and the ACPX 0.13.1 patch): - `newSession` and `loadSession` call `createSession`. It throws `RequestError.authRequired` (`-32000`) only for a claude.ai subscription when gateway auth is hidden. It throws `invalidParams` for a bad `cwd` and `resourceNotFound` for a missing resumed session. - Any other error, for example a failed SDK `initializationResult()`, becomes a generic `-32603 Internal error`. The ACP SDK puts the raw message in `data.details`, or the parsed JSON body in `data` when the message is JSON. - The typed AIR `sessionFailure` (category `limit`, `access` and others) is sent only on prompt responses and on `session_info_update` notifications. `loadSession` history replay also sends one for each earlier usage-limit message, but that notification does not show whether the limit is still active. The `errorKind` data is attached only by the prompt loop. - Thus a usage limit at session start has no typed signal on the ACP error. The cause keeps the provider's message, and the existing `API error: NNN` status parse can type it. This change does not add more free-text matching. **Shape for a recovery consumer:** the server reads the cause with `providerFailureFromError(error)`, as it does now. A startup cause has `phase: "startup"`, a `category` from the `ProviderFailure` schema and its `retryability`. The run error keeps `classification=provider_<category>` for a typed ensure failure, or `session_ensure_failed` when the category is `unknown`. ## Verification - `npx vitest run src/drivers/acpx src/cli src/drivers/provider-failure.test.ts src/live/runnerd-codex-transport.test.ts src/live/runnerd-codex-transport-settlement.test.ts` in `packages/paperclip-runner`: 103 of 104 files passed, 2214 tests. The 2 failures are in `installation-integrity.test.ts`. They fail the same way on unmodified `origin/master` in this checkout (a temporary `server.js` cannot resolve a module). - New tests: a plain ensure error keeps its message with the launch-environment secret and an `sk-ant-` value redacted; a `-32000` ensure error gives category `authentication`; a non-`Error` rejection with `errorKind: "billing_error"` gives `quota_exhausted`; an aborted handshake gets no cause; typed and untyped ACP errors; a closed ACPX rejection keeps its classification and a changed one gets the generic text. - `npx vitest run src/control-plane/acpx-controller-recovery.test.ts`: 8 passed. - `cargo test -p paperclip-runner-core --no-fail-fast`: all passed. New unit test: typed categories give `provider_<category>`; `unknown`, no cause, a category outside the schema and an extra field each keep `session_ensure_failed`; other closed codes keep their own classification. New pipe test through the fake sidecar: the cause survives the rejection, the classification is typed or generic, and the cause text is not in the error string. In the first full run, two `stdio_mcp` relay tests timed out. They passed in the second full run. They do not use the changed code. - `node --test test/acpx-sidecar-contract.test.mjs test/protocol-contract.test.mjs test/pi-closed-startup.test.mjs`: 29 passed. The Pi probe is skipped without the Pi distribution. - `pnpm --filter @paperclipai/paperclip-runner typecheck` (TypeScript, `cargo fmt --check`, `cargo check`): passed. `pnpm --filter @paperclipai/paperclip-runner build`: passed. - Server, after the runner build: native provider failure, run completion, task interruption presentation, Antigravity probe and the related native session executor tests: 639 passed. - Not verified: a live Claude account at its usage limit. I could not run the credential-free Pi probe, because this checkout has no Pi distribution. Its new expectation follows from the Pi patch, which raises `RequestError.authRequired` for the missing credential. ## Risks - The run error for an ACPX startup rejection with a cause now shows Runner's closed rejection text, not "Provider request failed". This text has only closed codes and is the same text that the run error showed before #18622. Provider text stays in `providerFailure`. - A plain ensure error message now becomes a displayed provider cause. Redaction applies before the 4,000-character bound. A new credential format that the patterns do not know can still need a new pattern. The same risk applies to all provider causes. - The classification of a typed ensure failure changes from `session_ensure_failed` to `provider_<category>`. No server code matches `session_ensure_failed`. The Pi probe expectation is updated. - The category and retryability are evidence only. This change does not change retry policy. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
da77c1bcfa |
fix(runner): align ordinary document key contracts (#18824)
## Thinking Path
> - Paperclip helps people manage AI agents that do work.
> - Native agents use tools to read and write task documents.
> - A document key is a stable identifier. The service trims it and
converts it to lowercase before it checks the key.
> - The tool schemas accepted filenames and long keys that the service
rejects.
> - This pull request makes ordinary document tool schemas match the
existing service contract.
> - Agents can now see the key rules before they call the tools.
## Linked Issues or Issue Description
**What happened?**
The schemas for `read_document`, `write_document`, and
`list_document_revisions` accepted `reconciliation.md` and keys longer
than 64 characters after normalization. The task document service
rejects these values. The descriptions did not explain that the key is
an identifier rather than a filename.
**Expected behavior**
The model catalog and the live and scenario tool definitions must
describe the existing document key rules. Values accepted after trimming
and conversion to lowercase must remain compatible. Historical
case-document keys use a separate contract.
**Steps to reproduce**
Validate `{ "key": "reconciliation.md" }` against the previous
`read_document` schema. It passes. Call the document service with that
key. It rejects the key before it reads the database. The added
regression repeats this comparison for all three tools.
## What Changed
- Apply the same document-key pattern to `read_task_document` and
`write_task_document` at the API bridge. Keep their existing raw-input
limit of 100 characters; ordinary tools keep 120. Export the shared
schema through the existing Runner package and server vendor boundary.
Route permissions, normalization and document locks stay with their
current owners.
- Add one shared schema for ordinary task-document keys. Keep the
existing 120-character raw input cap and the 64-character normalized
identifier limit.
- Preserve uppercase input, ECMAScript trim whitespace, and the Kelvin
sign, which converts to ASCII `k`.
- Apply the schema to the model catalog and the three live and scenario
definitions. Regenerate the catalogs, seeded fixture identity, and
fixture manifest.
- Test schema parity against the real task document service. Keep
historical case keys and the backend validator unchanged.
## Verification
- Current-head required CI is green at
`e67d3f1f49393660f0c70b3d1d136e6f86af821d`: 56 successful checks, four
intentional skips, and two neutral checks. [CI attempt
2](https://github.com/paperclipai/paperclip/actions/runs/41715924252)
passed after one targeted rerun of five jobs interrupted by the same
runner shutdown; the original interruption is retained. Fresh Greptile
and Paperclip reviews are both 5/5 with no unresolved review threads.
Hosted recovery remains unverified until automatic deployment and
replay.
- Normal master refresh to `e67d3f1f49393660f0c70b3d1d136e6f86af821d`
includes the already merged external-object startup fixture correction.
The previous required canary failed downloading
`mdast-util-gfm-table@2.0.1` with npm E404; later registry metadata and
tarball HEAD both returned 200. A separate general shard failed the
ten-thousand-future-objects startup assertion. Both original failures
remain recorded; neither was bypassed or retried. At the refreshed
baseline, 122 document/bridge cases and the exact large-object startup
case pass locally. Six document cases skip by platform. Fresh required
CI and reviews must pass on this head.
- The current-head Paperclip review found the same mismatch in the
task-scoped bridge. Before the correction, the real advertised schemas
failed 22 parity cases while 87 controls passed. After the correction
and a normal master refresh, the document-key and bridge suites pass 122
cases, with six platform-specific skips. The tests compare advertised
JSON Schema, runtime Zod validation, and the actual normalized document
service. They retain the bridge raw 100/101 boundary and historical
case-key controls.
- Runner TypeScript build, all server typecheck layers, and the actual
production Runner and server build pass. The final server build stamp
binds commit `80c880709c5f5630799fb8a48d54ff11ebc16610`. Full
current-head CI and fresh review are required after the correction. The
earlier CI run was externally cancelled and rerun; its cancellation
remains recorded.
- The regression failed in 33 of 64 cases before the fix. The remaining
cases passed as controls.
- The fixed regression passes all 67 cases, including every ECMAScript
trim character.
- The four focused Runner contract suites pass: 116 tests. They cover
all three tool surfaces and existing case behavior.
- Generated catalogs and manifest checks pass. Module-boundary and diff
checks pass.
- All server production typecheck layers pass. The test layer passes
after correcting the new test’s Ajv import. The full production server
and Runner build passes at
|
||
|
Devin FoleyandPaperclip
|
f7fae6b738 |
fix(runner): preserve native Claude GitHub broker transport (#18850)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Shared agents use managed GitHub launchers to acquire credentials for the active run. > - Native Claude runs through the ACPX provider environment boundary. > - That boundary keeps the launcher paths but removes their broker transport settings. > - Shell commands then report `capability_missing`, even when connector tools work. > - This pull request retains the three transport settings for explicit Claude managed launchers. > - Shell commands can use the existing session broker and its current access checks. ## Linked Issues or Issue Description **What happened?** Native Claude through ACPX cannot authenticate shell `git` and `gh` commands through an eligible managed GitHub connection. The launcher reports `GitHub capability_missing`. Connector tools can still authenticate through their own path. **Expected behavior** Managed shell commands acquire credentials for the active run. Provider processes do not receive raw GitHub tokens or ambient host credentials. **Steps to reproduce** 1. Configure an eligible managed GitHub connection and a native Claude ACPX execution. 2. Start a task and run `gh api user` through its terminal. 3. Observe `capability_missing`, although the corresponding connector identity request succeeds. **Paperclip version or commit** Reproduced from source at `6ef37398c8d799c5142a3a6590ecc70ad352b663` and current master `0b87308d5de93b4b8f18eeea89afc0bb4d9480a7` with controlled local dependencies. The affected runtime is native Claude through ACPX in a managed sandbox. Related: #18212 preserves Claude launcher selectors. This change supplies the missing broker transport. #18081 and #17599 address a separate Legacy Codex shell-policy boundary. #15354 addresses Legacy ACPX session hashes. The duplicate search found no open PR for this native Claude boundary. ROADMAP.md already lists managed personal GitHub identities; this restores that existing behavior. ## What Changed - Retain the broker URL, session capability and bridge capability within the existing explicit Claude managed-launcher allowlist. - Continue to reject raw GitHub tokens, ambient broker settings, task-supplied overrides and transport for other ACPX providers. - Add real Bash and launcher subprocess controls with a loopback broker and fixture Git/gh executables. Cover isolated, prepared and preconfigured homes; changed operation identities; revocation; and recovery. - Confirm that broker capabilities stay out of persisted session environment data. Update the execution guide. The ACPX environment owner retains this behavior. Broker authorization and credential resolution do not change. No shared API or schema changes are needed. ## Verification - Passing: 84 focused tests across `environment.test.ts`, `github-launcher.test.ts`, `runtime-sandbox.test.ts` and `configured-environment.test.ts` in the Runner package. - Passing: `pnpm --filter @paperclipai/paperclip-runner typecheck:typescript` and repository-wide `pnpm -r typecheck`. - Passing: repository-wide `pnpm build`. - Passing: 60 existing adapter-utils launcher tests; one unrelated test is skipped. - Passing: `node scripts/check-maps.mjs`, `node scripts/check-module-boundaries.mjs`, and `git diff --check`. - The new regression failed before the fix: both shell commands ran anonymously and the unit check showed the three missing broker settings. After the fix, the local broker receives each operation, changes identity for later commands, and denies revoked access without reusing a prior token. - The prepared-home fixture first rejected a macOS temporary-directory alias. It now resolves that fixture directory to its canonical path. A launcher-control command also named a nonexistent package config; the corrected run uses the root Vitest config. - Both `pnpm test:run` attempts stopped in chat-suite setup when embedded Postgres initialization exited with no numeric status. No chat tests ran. An independent database-support probe between attempts succeeded. The startup cause is not established; the full local suite is not verified. No further identical retry was made. - [CI run 41713841404, attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41713841404) passed on `92fd931722486c111030a7efe19dfc73f530b4c8`: 48 successful jobs and one conditional skip. This includes browser shards, general and serialized tests, Runner static/Rust/package/native-server checks, build, typecheck and the canary dry run. No CI jobs remain pending or failed. - The first CI attempt was interrupted on multiple runners. Browser and server/chat jobs logged runner shutdown signals followed by cancellation. The Runner static check reported lost communication with its runner. No failing browser assertion was recorded before termination; the summary gates failed on interrupted jobs. The successful retry used the same source revision. It shows recovery but does not establish why the hosts stopped. - Greptile reviewed `92fd931722486c111030a7efe19dfc73f530b4c8` at 5/5 with no outstanding findings. All review threads are resolved, and GitHub reports the branch as mergeable with no conflicts. The source tests use fixture credentials and a loopback broker on macOS arm64 with Node 25.6.1. They do not qualify a live Claude process, deployed sandbox, Windows shell or GitHub write. No existing run or deployed configuration was changed. ## Risks - Claude shell commands can read the explicitly supplied broker capabilities. Managed launchers need those capabilities. The existing broker still enforces active session/run binding, responsible identity, trust and connection access. - The exception applies only when the controller supplies a Claude managed-launcher directory. Raw GitHub tokens remain excluded, and broker settings remain absent from persisted session environment data. - No migration is required. Existing provider processes need their normal retirement and restart to receive a new launch environment. Deployment and live sandbox verification remain separate steps. ## Model Used OpenAI `gpt-6.1-sol` through the Codex harness, with xhigh reasoning. Used repository inspection, code editing and shell execution. The context window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
1423d745e8 |
feat(server): expose owned sleep blocker diagnostics (#18632)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
3910352c92 |
feat(runner): add experimental Cursor Cloud provider (#18068)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner gives each provider durable execution, tool, and completion contracts. > - Cursor Cloud currently has a direct adapter outside that native contract. > - Cloud work needs exact ownership across HTTP, SSE, Stop, and recovery. > - This PR adds a qualified Cloud API driver and verified proxy to Runner. > - Product setup and hosted placement remain separate dependent changes. ## Linked Issues or Issue Description Refs #5664, #11803, #17299, and #17471. Product integration is #18069. Managed-only API placement is #18310. The related direct-adapter model/transcript work is not replaced. No duplicate native Cursor Cloud integration was found. ## What Changed - Add native `cursor_cloud` / `cursor_cloud_api` with explicit model/repository validation, bounded HTTP/SSE, durable per-turn checkpoints and a qualified Node proxy launched through the Rust facade. - Commit terminal custody before publication/release; coalesce consumers, preserve ambiguous create/Stop/recovery fences, reject late tools, and distinguish definitely-unsent oversized requests from ambiguous responses. - Support scoped cold attachment, current authority, reconnect and verified artifact staging. Keep unsupported warm/external and Windows paths closed. - Preserve early source cursors and verify immutable legacy prefixes before suffix replay, including supported compaction/masking. Optional proof publication failure falls back to complete cold verification. - Preserve canonical records at the durable native producer with JSON-safe privacy and the existing 8MiB bound; drain admitted successor flushes. Reconcile upstream semantic/pack/schema/lifecycle changes and retain S3 and legacy CLI checks. ## Verification Core head `37c442ec8dd4`: all 66 check contexts are terminal successful/neutral/skipped, including required `ci / verify` and `ci / e2e`; fresh exact-head Greptile is 5/5 with zero unresolved threads. Dependent Product `81e828d20a77` and Top `dd2959689627` include the later client handoff correction and are running their own fresh gates. Core: 47/47 Rust, 4/4 compiled-native retirement, 19/19 settlement and normal Runner build passed. Runtime Top `9af1a5de42a2`: build and recursive typecheck passed. Earlier source-scoped evidence includes 398 focused tests, five relay tests and token/type gates. The 251-assertion, 7/8-suite run failed physical PostgreSQL teardown; the unchanged stable WSP suite passed 128 through the documented wrapper. The downstream fixture corrections passed 7/7 browser and 121/121 startup tests without timeout/oracle relaxation; a broken-unsubscribe control fails. Hosted runtime `9af1a5de42a2` has a passing normal build and recursive typecheck. Latest dependent heads additionally contain a pure client fix for stale live/history overlap, with 48 focused tests, UI typecheck and token gates passing. Its new client is not deployed in the frozen hosted image; server/native/API/provider source is unchanged by these four downstream files. Fresh hosted Linux admission on `9af1a5de42a2` retired its owned runner/proxy/socket with zero prompts. The saved configuration survived reopening, and Test authenticated the bound key/catalog/selected model without a prompt. Artifact with `claude-sonnet-4-5` completed, downloaded the exact 27 bytes, matched 121 canonical identities against complete logs and independently reached FINISHED. The active Stop retry was clicked within its immutable boundary, produced one canonical interrupted turn and acknowledged run-only cancellation, retained a valid 35.076s quiet window (maximum gap 4.584s), and independently remained CANCELLED/artifact-absent beyond the original 180-second command deadline. All provider-observer credential callbacks, DB clients and loaders closed; accounting is settled and explicitly unpriced. The first fresh Stop retains its failed 10.373s sampling gap and every sample. On the upgraded9af server, the historical F226 artifact was reopened and downloaded with its exact original hash after its provider agent had been deleted/GET404 earlier today. Historical F226 full Artifact/Stop cleanup, Question, corrected Plan and reconnect retain their original identities. Deleting the three newly created disposable provider agents and the new-case post-delete grade remain blocked on explicit user approval after automatic review rejected expanding the two original approved IDs. No new-case cleanup or complete fresh matrix pass is claimed; this is separate from the directly observed task/Stop behavior and retained exact cleanup proof. Retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook failures; Mac 17 SQL failures/skips; earlier startup-title, title-save, custody/Stop and deletion failures with unknown causes and bounded recoveries. `dea023` Artifact passed 17 checks but Stop failed its false warning and 50 duplicate log identities. Pre-repair ACK failure and original Core CI failure remain recorded; retry alone establishes no cause or full local-suite pass. Provider dollar cost stays UNKNOWN. ## Risks Experimental: Cursor independently acknowledges neither the effective model nor instruction digest before the first prompt. The caller’s explicit values are submitted and behavior checked live. Unpriced provider dollar cost stays UNKNOWN. Only qualified per-turn controllers are admitted. Uncertain work and incomplete cleanup stay fenced. Existing damaged/duplicate archives remain failed evidence. Prefix proof adds bounded object-store I/O; the 8MiB producer bound does not guarantee future providers’ output fits. ## Model Used OpenAI Codex, GPT-6 family, assisted with reasoning, repository tools, code execution, browser verification, and delegated code review. The serving model’s exact API ID and context window are not exposed. No value is inferred from the model catalog. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the limits of available model metadata - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch describes the change and contains no internal ticket ID - [x] I have run the scoped local tests reported above; pending gates are stated separately - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All required CI gates pass on the current head - [x] Fresh Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
07721e33f7 |
fix(runner): distinguish new skill authoring from source imports (#18717)
## Thinking Path > - Paperclip lets people manage agents and shared company skills. > - Agents can author new skills or add existing skills from approved sources. > - Source import and catalog install retain the source used by update, audit and reset flows. > - The create_skill tool described saving complete Markdown, without explaining this choice. > - An agent can therefore read an existing source and create an unrelated managed copy. > - This PR explains the distinction in the existing tool contract and tests its actual model-facing projections. ## Linked Issues or Issue Description **What existing behavior does this improve?** The model-facing description of create_skill and the choice between authoring and source import. **Current behavior** The tool says it saves a complete single-file skill. It does not distinguish new authored content from adding an existing skill. The Company Skills reference already directs external sources to import and catalog skills to install. **Proposed behavior** The tool explains that it saves newly authored content. It directs existing-source work to the existing import or install APIs through search_api and call_api. It retains the existing assignment and retry guidance. **Reason and benefit** A managed copy does not retain the original source locator. Clear guidance helps an agent choose the source-aware operation without adding a tool or changing permissions. This is a guidance repair. It does not claim that the import backend failed. ## What Changed - Clarify the canonical documentation, live descriptor and scenario descriptor for create_skill. - Test the actual discovered contract and provider projection, including API claim and company-boundary negatives. - Refresh generated descriptions and the seeded catalog digest and manifest entry. ## Verification - A focused regression fails on the parent because the actual discovered contract lacks the authoring versus import distinction. - Three focused pure suites pass: 48 tests. They include existing skill creation, retry, schema and authorization controls. - The source-owned serializers and protocol fixture validator regenerated the inventories without an app or native build. - Both generated inventories change only create_skill description. Arguments, required claims, modes, idempotency and all other tools remain byte-equivalent. - Current head `bca441726a6cba80e6e7da0321db59e27183c5b8` passes required `ci / verify` and `ci / e2e`; the full main CI run has 49 successful jobs. Windows, Docker, security and review checks also pass or are intentionally skipped. Greptile and the final Paperclip review are 5/5, with zero unresolved threads. This PR remains deliberately unmerged for the separately reviewed evaluation candidate. - Original head CI preserved: one unrelated UI clipboard-confirmation assertion failed after the clipboard call passed; 47 other executions succeeded and the final verify check failed as its dependent. The two UI owners are unchanged from the base. Exact React scheduler/timer timing is unknown. No unchanged-head retry was used before the necessary master conflict sync. - The conflict sync preserves incoming task-assignment guidance and regenerates metadata. It is not a fix for the UI test failure. The synchronized current head passes CI and review. No same-head recovery was consumed. ## Risks The description changes model context and the catalog digest. It may affect tool choice, but actual same-request and same-model acceptance is still pending. The change does not alter authorization, API behavior or grading. Local checks did not run the app, database, native runner, full graph typecheck or build. Intermediate test-fixture claim and placement mistakes were corrected; the final focused run passes. Do not treat these pure controls as a feature pass. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through Codex, with reasoning, source inspection and tool execution. The host does not report a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs; narrow create_skill/import and skill-import-guidance searches found no duplicate. - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b22ef22d56 |
fix(runner): show provider failure details on tasks and runs (#18622)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner starts providers and records their run outcomes. > - Provider failures can contain useful details below a generic protocol error. > - Those details were lost at the sidecar, command receipt, and run projection boundaries. > - A blocked task therefore showed a generic failure even when the provider gave a clear reason. > - This pull request carries a bounded, sanitized provider failure through those boundaries. > - Operators can read the actual cause on the task and run pages. Existing recovery policy still controls retries. ## Linked Issues or Issue Description Refs #17543 for the related closed JSON-RPC classification work. Refs #17272 for provider recovery in the legacy OpenCode adapter. This change adds display evidence for native Runner providers and keeps recovery policy separate. **What happened?** A native Claude session returned an HTTP 429 message below an ACP `Internal error`. The task and run showed only a generic startup failure. Operators had to inspect the provider response to learn that the account was rate limited. **Expected behavior** The task and run show the sanitized provider message, provider name, startup or turn phase, and available HTTP status. Unknown categories keep the message. Display evidence does not grant a retry or replace a recovery decision. **Steps to reproduce** 1. Start a native Claude run whose account is rate limited during model selection, or return a structured provider failure from a Codex or OpenCode turn. 2. Let the run fail and open its task page. 3. Before this change, the page shows a generic failure. With this change, the page shows the provider explanation. The run page shows the same explanation after reload. **Paperclip version or commit** Source checkout based on `f8a888a072`. ## What Changed - Add one JSON Schema for `ProviderFailure` and generate matching Runner and shared TypeScript readers. Keep the Runner package independent of the app workspace at runtime. - Normalize known ACPX, Codex, and OpenCode error fields. Preserve unknown messages. Redact known credential values and credential patterns before applying the 4,000-character limit. Do not copy response bodies, headers, stacks, or prompts. - Carry startup failures through the sidecar, Rust Runner, durable command receipts, and private transport error details. Keep provider text out of generic transport error strings and Sentry failure messages. - Keep turn failures in authenticated run events. Recover the same evidence from persisted events during server finalization. Store the cause in the company-scoped run result. - Show one shared notice on failed tasks and run details. Keep existing retry permissions, scheduled retry state, and resolved-task behavior. - Reapply registered run-secret and current-user redaction to task display data. Batch secret lookups for only the visible provider causes. - Document the contract and update the capability maps. Add regressions for nested Claude 429 errors, OpenAI quota and rate limits, OpenCode errors, redaction, durable receipt replay, restart finalization, and UI rendering. ## Verification - `pnpm -r typecheck` passed. `pnpm build` passed. All GitHub Actions build, test, package, and validation jobs passed on `d49c79ab4a`. The security checks passed. Greptile gave this head 5/5 with no open review threads. Paperclip Review remains pending. Its review worker exhausted six workspace recovery attempts with export_interrupted. A fresh review request was accepted and is waiting for that saved workspace. The ordinary Retry action rejected reuse of the stopped request. This is an external review-worker blocker; no check was overridden. The duplicate local `pnpm test:run` was stopped after the CI test shards passed. - Focused server settlement, provider evidence, and restart tests: 93 passed. Runner admission and normalization tests: 103 passed. Focused Runner wire and terminal tests: 29 passed. The real Rust/sidecar command receipt replay test passed. - The new projection test failed before implementation because the task had no provider cause. The startup producer regression also failed before extraction moved ahead of the generic error-code replacement. Both pass with this change. - Live check: a rebuilt local sidecar received an actual Claude HTTP 429 during startup. The new cause kept the full provider message. A synthetic task in an isolated local app then reached the same provider error through the native Runner path. The dedicated run result, task API, task banner, and run detail all showed the same message after reload. The later reporting fix keeps run.error generic; a regression proves the display cause still survives restart and stays out of Sentry capture. - The first isolated app attempt failed earlier in `admission_credential`. An ordinary manual Retry reached the provider and produced the 429 above. This PR does not claim to fix that separate admission failure. An initial parallel local database test setup also failed; the serial restart suite passed. - Security review found that copying display text into run.error could forward it to Sentry. A regression failed with that behavior. The fix keeps the cause in its display field. All 38 settlement/restart tests and the server typecheck pass. - All 29 protocol contract tests pass, including bounds and extra-field rejection. An earlier full Rust core run passed. The later full run hit one 5-second receipt-limit timing assertion; its isolated rerun passed. The full Linux and Windows Rust CI jobs passed on this head. - Review regressions pass for explicit Codex driver identity, native Rust startup and resume receipt propagation, oversized OpenCode/ACPX errors, task read-time registered-secret redaction, and the standalone source closure. The focused Runner run passed 103 tests; projection/restart tests passed 16. - Read-time redaction edge cases passed: a missing decryption key omits only the unsafe optional explanation; expanded redacted messages remain visible within the Unicode character limit. All 14 task projection tests and server typecheck pass. - The package Build check found an old Pi assertion against Error.message. The probe now checks the exact closed classification in the authenticated receipt and requires the new provider cause. Both polling transport cases pass. A local credential-free Pi probe stopped at admission_installation because this checkout lacks the Pi distribution; the installed Linux package probe passed in CI on this head. - UI coverage includes startup and turn text, unknown categories, literal rendering of provider text, read-only retry behavior, and resolved-task suppression. Token gates and map checks passed. - The live check covers Claude startup on macOS. Codex, OpenCode, terminal failures, and restart paths have deterministic regression coverage. The change is not deployed. ## Risks - Provider messages are untrusted text. The boundary selects specific fields, redacts credential patterns and known environment secret values, bounds the message, and renders plain text. New provider-specific secret formats may need added redaction patterns. - Optional fields preserve compatibility with older producers and stored runs. Old runs without the field keep their current fallback text. No database migration is required. - The cause stays in company-scoped run data. This change does not send provider text to first-party Telemetry or add it to OpenTelemetry span attributes. - Normalized categories and retryability are display evidence. Existing recovery classifiers, retry limits, ownership checks, and approval gates keep their authority. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and browser verification. The host does not expose the exact serving variant or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
1bc3ed6dd5 |
fix(opencode): keep native tool calls pending until Runner settlement (#18711)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native OpenCode sends semantic tool calls through the durable Runner protocol. > - The Runner retains a call until its result is known. > - OpenCode and its HTTP bridge each stopped waiting after 30 seconds. > - A slow response could leave a call pending after the provider had moved on. > - A later run then failed the pending-call guard during attachment. > - This pull request keeps native calls under their Runner owner until settlement or cancellation. > - The benefit is one delivered result and a safe next run. ## Linked Issues or Issue Description Related: #18343 applies the durable wait policy to native ACPX calls. This change applies that policy to native OpenCode. **What happened?** An assigned tool call took more than 30 seconds. OpenCode reported an MCP timeout while the Runner still retained the request. The source turn could finish, but suspension failed. A later `run.attach` failed because provider tool calls were still pending. **Expected behavior** The native proxy must wait for the authoritative result under the Runner owner. Stop, bridge close, and lease expiry must still end the wait. Unknown effects must still block attachment. **Steps to reproduce** 1. Start a native OpenCode run with an assigned semantic tool. 2. Hold the controller's result for 31 seconds. 3. Observe the provider's MCP result and attach a later run to the session. **Paperclip version or commit** Reproduced against `6d72243d8af0362d73fbcb5331fd42a384800d4e`. The branch is based on `36df8933f0`. ## What Changed - The native OpenCode proxy enables a driver capability for durable tool requests. - That capability removes the bridge's separate request timer and sets OpenCode's fixed MCP client ceiling to one hour, consistent with the native ACPX sidecar. - Ordinary driver calls keep their 30-second deadline. Assigned external MCP connections keep their existing budget. - Cancellation, result replay, tool authorization, and the pending-call attachment guard keep their existing contracts. - The native regression holds a result for 31 seconds, then completes a second run with the same Runner process and no pending provider calls. - The deterministic provider fixture honors the emitted MCP budget and retains the initial completion contract across ordinary follow-ups. - Execution semantics and the session verification maps describe the wait and its safety checks. ## Verification - Red before production edits: the bridge aborted a durable request at the ordinary 30-second boundary. The driver emitted `30000` instead of the expected `3600000` client budget. - The final combined bridge, driver, proxy, and native protocol run passed: 95 tests in five files (71.17 seconds). - The native delayed-result probe passed with the current Rust Runner and current bundled proxy. It completed the later run with the same Runner PID and an empty pending-call ledger. - Actual provider check passed on macOS arm64 with `opencode-ai@1.18.34`, the changed source, and a local deterministic OpenAI-compatible model. One assigned write waited 31 seconds and saved one file. Both turns completed. The second turn retained the saved tool receipt. The write ran once across five model requests. - Bridge safety cases cover result replay without another action, ordinary timeout behavior, MCP cancellation, and teardown of an unanswered durable request. - Both runner TypeScript projects passed typecheck. The first attempt lacked the eval-kernel build; it passed after building that declared dependency. - Map alignment, module boundaries, and `git diff --check` passed. - The first native follow-up probe failed because its synthetic provider forgot the initial completion-contract revision. This caused a repair turn and another discovery call. The fixture now retains the contract, and the same probe passed. The failure is retained here. - Runtime acceptance used the implementation at `583d316ad4`. The later clarification commit changes comments, documentation, and a test name only. It does not change runtime behavior. - Full current-head CI passes, including typecheck, build, general and serialized tests, Rust/Runner checks, all eight E2E shards, and the `ci / e2e` and `ci / verify` aggregate gates. Greptile is 5/5 on this commit, with no unresolved review threads. No deployment or retry of the affected historical task was performed. The actual provider check used a local model; it does not qualify a deployed Linux sandbox or a live model service. ## Risks - A native call can wait longer while its Runner owner remains valid. Stop, teardown, and lease expiry still end the wait. The MCP client has a separate fixed one-hour ceiling. Runner ownership can renew beyond that ceiling. If the client deadline expires first, the operation outcome remains unknown and the existing hold must still block replay and run attachment. - This prevents new premature timeouts. It does not classify or replay an old request whose effect remains unknown. - No schema, migration, credential, or permission change is required. ## Model Used - OpenAI Codex, GPT-6 family. The harness does not expose the exact model ID or context window. Capabilities used: code generation, tool-assisted diagnosis, local execution, and test analysis. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
09162fe893 |
refactor(runner): clarify native task assignment guidance (#18704)
## Thinking Path > - Paperclip lets people assign work to AI agents. > - Native task creation can start the assigned agent as soon as the task is ready. > - The default assignee is the agent that creates the task. > - An agent can omit the requested teammate, create work for itself, and reassign it too late. > - The server already supports setting the requested agent and project during creation. > - This pull request makes those fields and the early-start risk clear in both tool catalogs and the delegated-task example. > - The benefit is clearer guidance for the existing delegation contract. ## Linked Issues or Issue Description Related: #13281. **What happened?** An observed delegation attempt called `create_task` with only a title and retry key. The task used the documented current-agent default and started work before a later reassignment. The server honored its contract. The original failed QA result remains failed. **Expected behavior** When a user names an existing teammate and project, the agent should put their exact IDs in `assigneeActorId` and `projectId` during task creation. The requested worker can then start with the correct task ownership and project. **Steps to reproduce** 1. Ask an agent to create work for a named existing teammate in a named project. 2. Create a ready task with only its title and retry key. 3. Observe that the creating agent is assigned and can start before a later reassignment. This is a guidance mitigation for agent tool misuse. It does not change the default or enforce model compliance. A fresh deployed qualification is still required. ## What Changed - Explain initial teammate and project selection in the canonical `create_task` description and its two field descriptions. - Add a representative delegated-task call with an explicit agent, project, and saved plan. - Synchronize the same three descriptions in the sibling semantic-action catalog. Keep its schema constraints intact. - Keep the global execution prompt byte-for-byte equal to the base. Revision `paperclip-execution.v8` already directs the agent to create delegated work with the teammate ID. - Refresh the generated catalogs, seeded catalog identity, and protocol manifest. - Make the collaboration contract test use the declared scenario example, as the general contract test already does. Retain separate live-example validation and strict scenario input checks. ## Verification - All 79 existing native controls passed across `chat-project-tools`, `paperclip-runner-tool-authority`, `task-creation-backlog`, and `tool-receipt-recovery`. These cover alternate-agent readback, current-agent defaults, backlog holds, company boundaries, and dispatch recovery after commit. - All 108 protocol-action, semantic-catalog, catalog-doc, runtime-context, and collaboration checks passed. All 11 contract fingerprint checks passed. - The original CI collaboration-test failure reproduced locally before correction: the test sent live-only fields to the older scenario schema. After selecting the declared scenario example, all 13 collaboration cases passed. Malformed titles, undeclared project fields, and arbitrary unknown fields still reject. No schema or live example was changed for this correction. - Generator checks passed for semantic contracts, the semantic-action catalog, operation groups, and protocol coverage. - `pnpm -r typecheck`, `pnpm build`, `node scripts/check-maps.mjs`, and `git diff --check` passed. - A structural comparison confirms that types, required fields, lengths, modes, permissions, outputs, and defaults are unchanged on both catalog surfaces. - No new text-matching tests were added. These local checks passed again after the global prompt was restored. Fresh exact-head Greptile and Paperclip reviews are both 5/5 with no actionable findings. [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/41565349812) completed successfully on `2e5f93e122191e17b316a7780a74ff404fc468f3` (attempt 2), with all 49 workflow jobs successful. - Current-head CI first failed only the unchanged native-process retention case at its original five-second limit. The shard passed 2,130 other tests and skipped eight. One unchanged local Node 24 control passed. The original test and retention dependency blobs match the base. The exact timeout cause remains unknown. No source, assertion, or time limit changed. One authorized recheck of only that runner shard and its normal final gate passed. The recheck passed 2,131 tests and skipped eight. The other 47 successful job results were carried forward with their original start and finish times. - The prior head `42207172dd4bc376dab880dbccb2c698acccf6c4` passed all 49 CI jobs and fresh 5/5 reviews. Its first CI attempt lost one server runner to a reported shutdown. One authorized retry of only that lost shard and its normal gate passed with unchanged assertions and time limits. The original failure and recovery remain recorded. - A normal merge of that prior head was rejected by the base policy because the global prompt required a code-owner review. This revision drops that optional global change. No policy override or bypass was used. - The new catalogs have semantic catalog digest `sha256:5af904b7fc088509b98dde41611c042ff3c22c881e252190817978de0d1b733a`. The authorized `create_task` operation digest is `sha256:f5dad10c7969680d91514603966705ac09ceaa3f2cff21fd6c199e2de0d7a63d`. - A new run can report admitted prompt revision v8 and its digest. The ordinary run-profile tool fingerprint does not hash these descriptions. Source/Core and matching provider-pack identity prove deployed code, not model delivery or compliance. A fresh campaign with new agents is still required. - No model run or deployed acceptance was performed for this source change. The original failed qualification remains failed. This tool-guidance change is a mitigation hypothesis. ## Risks The longer tool descriptions change the catalog identity. A model can still omit a requested field. The global prompt remains v8. Existing assignment, authorization, wake, replay, and session behavior stays intact. A new campaign is required because a stored provider session can retain old tool declarations. A native-process retention test has a recurring five-second timeout on unchanged code. One unchanged local Node 24 control passed. This does not prove CI reliability or explain the timeout. No timeout or assertion was changed. ## Model Used OpenAI GPT-6 through Codex with reasoning and code execution. The runtime does not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
16643a6e03 |
fix(runner): recover pending Stop cleanup after controller restart (#18682)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server owns execution, Stop delivery, and usage settlement. > - A Stop can remain pending when the controller restarts. > - Recovery must reach the existing session without starting more paid work. > - Startup and budget fences currently block that cleanup, and interrupted runs leave the restart selection. > - This pull request gives exact audited Stop cleanup a shared authority check and records its real acknowledgement. > - The benefit is safe cleanup while usage debt remains honest. ## Linked Issues or Issue Description **What happened?** A pending native run-only Stop can retain an unfinished execution owner after controller loss. Its startup cancellation fence prevents recovery admission. If teardown marks the run interrupted and clears its task lock, restart selection also omits it. A recovered session can receive cancellation without the durable acknowledgement that the normal Stop request would save. **Expected behavior** Recover only the existing remote session for the audited pending Stop. Keep fresh provider work blocked, including during budget pause. Keep incomplete usage pending until a real complete receipt arrives. **Steps to reproduce** 1. Start a native run with a held reservation and an authenticated remote checkpoint. 2. Persist a run-only Stop and its startup and execution cancellation fences before acknowledgement. 3. Lose the controller. Leave the coordinator unfinished, mark the run interrupted, and clear its task lock. 4. Attempt restart recovery and observe the missing claim or cancellation admission rejection. Related work: #14733 binds caller Stop requests. #18658 handles plugin startup cleanup obligations. #18035 proposes broader accounting changes. This change repairs existing-session Stop recovery and preserves current accounting guards. ## What Changed - Add one shared check for the exact run, company, issue, pending Stop, intent audit, live recovery lease, generation, and remote reattachment disposition. - Include eligible interrupted Stop owners in restart selection. Keep generic interruptions and another task execution owner fenced. - Reuse only the original held reservation for cleanup. Keep budget pause effective for fresh, bootstrap, dead-runner, and unproven recovery work. - Apply the same cleanup authority at heartbeat runtime selection and native entry. Skip attachment staging and warm-owner retirement only for that exact cleanup. Revalidate at session publication and fail closed if the proof changed. - Send the existing Stop through the audited cancellation producer before turn submission. Close the coordinator only after acknowledgement, including system-owned run-only Stops. - Give verified cleanup a transport purpose that retains pending semantic inputs without replay, dispatch, or a semantic reply. Add an explicit Core retain-only intake mode; preserve authenticated identity, schema, digest, sequence and replay checks. Ordinary execution still requires its dispatcher. - Add PostgreSQL, native protocol, and startup-fence regressions. Update the execution contract and recovery maps. ## Verification - The combined Stop regression fails on unchanged production at the cancellation reservation guard. Interrupted restart selection also fails before the repair while its running sibling passes. - An assembled `heartbeatService.recoverNativeRunsAfterRestart` regression exposes the earlier runtime-selection fence with the rest of the repair present. The fixed path reaches selector/claim, runtime selection, original held-budget reuse during pause, dispatch, real native protocol, audited cancellation, and heartbeat settlement. It starts no provider turn, acquires no new sandbox, and does not reclaim the task lock. - Fail-if-called attachment staging and turn boundaries remain untouched. A retained warm-owner regression prevents retirement. Invalidation at session publication fails closed. Actual backend/driver constructor tests verify cleanup-purpose forwarding, absent default options, and denial under foreign ownership. - Missing usage retains accounting debt and the hold. A later complete receipt records one 25-cent charge and settles that same hold. Concurrent duplicate cleanup is denied. Both user-owned and system-owned Stops pass. - Real transport tests: 9 pass on Node 24.22.1. They preserve ordinary saved-tool and completion recovery, prove the authority-denial race before cancellation, and qualify cleanup with pending ordinary/completion inputs and either handler ordering. - Real authenticated Core intake: 11 pass. Callback-present and callback-absent retain-only cases persist and acknowledge unanswered input without dispatch, recovery or result. Default missing-dispatcher rejection and digest-integrity/suspension guards remain effective. Omitting the callback alone first failed by disconnecting; the explicit retain-only mode fixes that defect. - Repository `pnpm -r typecheck`, the server/runner-vendor build, map checks, and module boundary checks passed locally before integrating the latest retained-startup change. That integration passes 11 native startup/cleanup compatibility cases and 3 assembled recovery/accounting cases. The final broad affected-suite attempt was stopped without a verdict; it is incomplete. Full repository tests and build are delegated to current-head GitHub CI. - Review exposed an additional real-process close failure with unanswered input. The corrected path requires a fresh exact provider-stop receipt confirming exit, followed by durable event drain and exact runner suspension. All 11 focused transport cases pass, including missing/false exit-proof negatives and ordinary recovery siblings. Runner TypeScript and map validation pass after this correction. - The assembled test controls provider, workspace-transfer and GitHub bootstrap I/O. It does not qualify hosted credentials, physical provider termination, workspace preservation, or historical receipt recovery. Local socket-permission setup failures are separate from semantic regression results. No hosted replay has run on this repair. - CI exposed an ordinary factory-call compatibility regression and an obsolete authority-read test fixture. The repair preserves single-argument ordinary calls and updates the race fixture without changing its Stop assertions. The complete native executor suite now passes all 1,049 tests; the ordinary factory boundary and all five Stop-accounting cases pass. A separate local environment-default case twice exceeded its five-second wait while other work ran; its prior-head CI case passed. The same unchanged case passed an isolated diagnostic without changing its assertion or timeout; the precise contention source remains unproven. - The final fixture edit initially used the wrong overloaded promise type. CI rejected it; the annotation-only correction passes the exact `pnpm run typecheck:build-gaps` entry point locally, including server service tests and remaining workspaces. No runtime assertion changed. - Final exact head `007fd4d04a2d6338c6dc073e97ab470a277fc3d0`: required `ci / verify` and `ci / e2e`, build, typecheck and test groups pass. Greptile is 5/5 with both findings resolved. Three expected checks are skipped; the optional native check is neutral, while its final current-head comment gives 5/5 from source inspection without a checkout or test execution. The first server-shard attempt lost its self-hosted runner connection; one scoped rerun passed. Its original annotation and unavailable-log evidence are retained. ## Risks - This changes execution authority. Cleanup requires an audited, exactly bound pending Stop, an unfinished coordinator, a live remote reattachment claim, and the original held reservation. It never authorizes a replacement provider turn. - Missing checkpoint, original held reservation, termination, or usage evidence can keep cleanup or accounting pending. The repair does not manufacture acknowledgement, replies, receipts, or zero cost. - Retain-only intake is an internal trusted constructor option. It changes no agent tool permission. Ordinary callback requirements and semantic dispatch remain unchanged. - No schema, public API, provider profile, or deployment setting changes are required. ## Model Used OpenAI Codex, `gpt-6.1-sol`, Extra High reasoning, with source inspection, code execution, and regression testing. The context-window capacity is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug report template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket ID - [x] I have run affected tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address all reviewer comments before merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
f8a888a072 |
feat(runner): define bounded startup lifetime contracts (#18490)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native sessions and plugin workers have separate startup lifetimes. > - A caller timeout does not prove that a worker or an accepted host write has stopped. > - Shared startup code needs an exact deadline and an actual settlement contract. > - This PR adds those contracts and tests them before any production caller uses them. > - The dependent integration in #18405 will use the contracts with verified cleanup. > - This split lets reviewers assess the lifetime rules separately from runtime activation. ## Linked Issues or Issue Description Refs #18405. This PR extracts the foundation from that change. The integration will depend on this PR. **Problem or motivation** A bounded startup must distinguish the caller's wait from actual worker completion. A late worker reply or an accepted host request can outlive the original RPC promise. A shorter timeout must not discard that responsibility or authorize a replacement run. **Proposed solution** Add one Core execution context, optional Runner startup deadlines, and worker settlement and retirement APIs. Test the contracts with real child-process IPC. Keep production context creation, transport projection, capability negotiation, durable cleanup, and Stop/recovery integration in #18405. **Alternatives considered** One combined PR mixed the lifetime contract with its activation and cleanup policy. Activating a shorter deadline before the complete transport and cleanup path is unsafe. Splitting only the SDK declarations would not provide a useful behavioral review boundary. **Roadmap alignment** This supports the existing plugin, sandbox execution, and bounded recovery capabilities. It adds no new product workflow, endpoint, migration, provider or runtime dependency. ## What Changed - Add `NativeExecutionContext`. It retains the selected input, binds one transport target, rejects work after cancellation or expiry, and tracks actual worker settlement. Expiry alone does not complete pending work. - Add worker startup groups. A caller timeout retains the group's responsibility. Retirement revokes obsolete invocation scopes, waits for accepted host handlers, and retires only the verified group's bookkeeping. Each cleanup wait is bounded; an unfinished handler remains pending for a later retry. - Add an optional Runner startup signal and absolute deadline for session open and retained attachment. Clean up abort listeners after synchronous failures. Existing callers keep their default timeout path. - Expose the remaining confirmed controller-lease budget without a new query or renewal timer. - Add optional SDK context and preparation-retirement fields. No production bridge sends or advertises them in this slice. - Keep the required lease mock update with the new `remainingMs()` contract so this foundation typechecks independently. Production activation files, including the native executor and environment task bridge, match the current master base. No production caller constructs `NativeExecutionContext` or passes the new Runner startup option. This PR adds no durable cleanup producer and claims **no standalone TTFT improvement**. The dependent integration must enable the context, transport and verified cleanup together. ## Verification Initial split verification on `8ebc8a2f9c6c02e90f920356138b3742a4067b28`, based on `9a3f395737e7a92faf99971459979061d423531a`: - From `server`: `../node_modules/.bin/vitest run src/services/native-runtime/native-execution-context.test.ts src/services/native-runtime/native-execution-context-worker.test.ts src/services/native-runtime/native-execution-lease.test.ts` — **17 passed**. Includes real child IPC, caller timeout with late reply, worker death, accepted host-handler draining, obsolete cleanup invocation rejection, unrelated work preservation, context scope, expiry, and confirmed lease budget. - From `packages/paperclip-runner`: `../../node_modules/.bin/vitest run src/native-session-runtime.test.ts` — **188 passed**. Includes existing callers without the new option and new synchronous-failure/expired-deadline cases for fresh and retained sessions. - Direct TypeScript checks for `packages/plugins/sdk/tsconfig.json` and `packages/paperclip-runner/tsconfig.json` — passed. Built only their shared/eval TypeScript prerequisites. - From `server`: `../node_modules/.bin/tsc --noEmit` — passed. - Checked the partition against the reviewed combined candidate. The twelve selected files match exactly. The thirteenth file carries only the required test-mock update. The production executor, bridge and target wiring remain equal to the base. - `git diff --check`, added-line secret/private-data scan, and added-code-comment scan — passed. No generated files or run artifacts are committed. - The first old-base split passed its focused tests but failed server typechecking because the unchanged executor test mock lacked `remainingMs()`. The original failure is retained. Moving that one-line mock update into this foundation resolved the split dependency; the final current-master checks above pass. - Full local CI suites, Rust builds, deployments and provider calls were not run, as requested. GitHub CI will provide full regression coverage. Current-head GitHub CI, standard review and Apex are complete; exact acceptance is recorded below. Review follow-up `d7635f4d280dab757a3666703737bcda955c9361` fixes three reproduced lifetime edge cases. A retiring startup group rejects new calls and shares one retirement promise; failed bounded retirement remains retryable. Transport identity uses a fixed field tuple rather than object-property order. Finishing a context rejects pending lookup waits without aborting an already-started backend or pretending the worker completed. All three new regressions failed before the fix; the updated foundation selection passes **20 tests**, including actual IPC and failed-retirement retry. Server typechecking and added-line checks pass. These replace the earlier 17-case selection, rather than adding 20 unique cases to it. The original failures and passing reruns are retained. Apex follow-up `cb8860bab5ab9d6357a352e11f276be0d0cdb26d` keeps removed worker handles reachable while they own startup calls, accepted host handlers or retirement work. Drain/retirement includes active and retained handles; completion callbacks remove settled handles. Stop and replacement remove only the exact handle they observed. This adds no database/network operation, polling loop or generic shutdown wait. The SDK, Runner, context and transport bridge are unchanged. Seven additional IPC regressions failed on the prior manager and pass after the fix, covering removal/replacement, Stop-all, host-handler settlement, bounded retirement failure/retry, pruning, and replacement ownership. The updated foundation selection passes **27 tests**, with server typechecking passing; these supersede the earlier 20-case selection. A separate real-PostgreSQL/child-IPC characterization uses the integration's actual SDK host, storage registry and run-control path with the exact corrected manager bytes. Before the fix, cleanup released run ownership while an accepted host write was blocked, then the late write restored obsolete metadata. After the fix, ownership and cleanup remain pending until the write completes; cleanup then removes the stale metadata before releasing ownership. The original failure, corrected result and source snapshots are preserved. Final head `cb8860bab5ab9d6357a352e11f276be0d0cdb26d` passes GitHub CI (53 successful checks, 2 expected skips, Snyk success), standard Greptile **5/5** and fresh Apex **5/5**. All review threads are resolved and the PR is conflict-free. ## Risks The shared Runner helper now observes synchronous operation failures through its asynchronous cleanup path. The affected runtime file's 188 tests pass, but full regression coverage still depends on CI. The new lifetime APIs require honest settlement signals. A future caller must not treat a deadline, worker crash or Stop intent as verified retirement. Production activation remains dependent on the complete integration and its cleanup tests in #18405. This foundation introduces no shorter production startup budget or new cleanup policy by itself. The context's transport projection is process-local. It must not be persisted or used as a general authorization cache. No production caller supplies it in this PR. There is no database migration, new dependency, or deployment requirement for merging this foundation. ## Model Used OpenAI Codex, based on GPT-6, assisted with repository inspection, the split, tests and command execution. The exact serving model ID and context-window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal issue identifier - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes — this description documents the inactive contracts; operational activation documentation remains with #18405 - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green on `cb8860bab5` - [x] Current-head standard Greptile and Apex are 5/5 with no unresolved findings - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
baa4a2e64e |
fix(runner): restore Codex auth before source recovery (#18612)
Restore disposable Codex authentication and safe provider configuration before cold source recovery. Preserve source receipts and conversation state until reconciliation, and rebuild assigned tools from the admitted launch binding. Validate remote launch destinations, clear stale authentication when no selected credential exists, and atomically publish SSH file transfers without following retained links. Red/green recovery regressions, a real two-turn Codex/MCP probe, and all current PR checks pass. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
42c64d4ec4 |
fix(runner): preserve deliberate OpenCode approval waits (#18610)
## Thinking Path > - Paperclip coordinates agent work and human approval. > - A saved Plan must wait for the user before work continues. > - The native runner interrupts the provider after it saves that wait. > - OpenCode can report the interruption with an idle event and no final answer. > - Empty-answer recovery treated that intentional pause as an incomplete answer. > - This change stops answer recovery for an interrupted turn. > - The runner still requires a real provider terminal before it settles the wait. ## Linked Issues or Issue Description **What happened?** OpenCode could send a corrective model request after Paperclip interrupted a turn for a saved approval request. The native runner then failed with `governed_wait_provider_terminal_unproven` because it did not receive the required terminal event within the existing drain window. **Expected behavior** An interrupted turn must stop requesting answers. A provider idle or cancelled event must remain available to the native wait settlement. An HTTP abort acknowledgement alone must not count as a terminal. **Steps to reproduce** 1. Start an OpenCode turn that has no final answer. 2. Interrupt that turn after saving a durable approval request. 3. Have the provider return `session.idle` or an idle `session.status`. 4. Observe that the old driver sends a second prompt instead of emitting the terminal event. The controlled regression fails before the fix for both idle event forms: it observes two prompts instead of one. The affected driver blob is identical in the deployed source and the current base. Related: #18364 added the empty-answer recovery path. ## What Changed - Record interruption intent for the active OpenCode turn before asynchronous settlement. - Cancel its answer-recovery timer and prevent corrective requests for that turn. - Preserve provider terminal checks and avoid a delayed abort of a later turn. - Cover both idle event forms, an existing correction, durable question handoff, in-flight tools, missing terminal evidence, and a later authorized turn. - Document the interruption contract. ## Verification - The new regression failed before the fix for the expected extra prompt. A second red test confirmed the sibling durable-question handoff also sent that unwanted prompt. - Interrupted idle events now produce explicit cancellation terminals. Tests reject a failed terminal and require normal completion on the later authorized turn. - All 305 targeted tests passed across the OpenCode driver, normal empty-answer recovery, interruption, backend publication, and native session runtime suites. - Runner TypeScript check passed: `tsc -p packages/paperclip-runner/tsconfig.json --noEmit`. - `git diff --check` passed. - Greptile reviewed the current commit at 5/5 with no findings. Both earlier Paperclip review findings are fixed and their threads are resolved. - The additional Paperclip advisory reviewer could not assess the final commit because its earlier workspace save did not finish. Its check is neutral; this is not a passing assessment. No reviewer configuration or workspace safety gate was changed. - Repository CI passed on run 41470761437, attempt 2. Attempt 1 retained a 15-second timeout in an unchanged voice-report test; only the failed jobs were retried once, on the same commit. The identical test also passed in an isolated baseline check against disposable PostgreSQL with its original timeout. - Live QA acceptance is pending. No eval prompt, model, allowance, timeout, or assertion changed. ## Risks The change affects OpenCode interruption and answer recovery. A silent provider must remain unproven; the negative control checks that case. Ordinary answer recovery remains enabled on authorized turns. No schema, permission, or provider configuration changes are required. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code execution, and browser tools. The runtime does not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7751fc3107 |
fix(server): complete volume-free controller admission and recovery (#18543)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its volume-free web server must admit remote warm execution without
depending on a local disk.
> - Fresh run rows retain the initial legacy label during workspace
preparation.
> - The native runtime selector commits the execution mode later.
> - The earlier guard rejected valid remote runs before that selection.
> - This PR checks storage during preparation and the resolved lifecycle
during runtime selection.
> - Agent-file recovery also has to reconstruct disposable cache paths
from the retained remote directory.
> - The deployment test removes the complete predecessor home, including
immutable caches, to enforce that contract.
## Linked Issues or Issue Description
Refs #18518, #18529, #18533.
**What happened?**
A real Codex/Daytona deployment test failed during setup because the
volume-free workspace guard checked the initial run label. After fixing
admission, controller retirement first failed on an immutable skill
cache. Removing that cache exposed a production recovery failure: B
rejected the saved agent-directory path from A, even though the remote
execution identity was unchanged. The next live attempt recovered that
directory and transferred the same runner, but adoption then tried to
recreate provider launch assets from the deleted controller home.
**Expected behavior**
Allow remote workspace preparation before native selection. Reject
actual legacy and per-turn execution at the runtime boundary. Delete the
entire stopped predecessor home, including immutable caches, without
following links to shared dependencies. B must reconstruct its local
agent-file projection from the exact registered remote lease while
preserving the save baseline and unsaved remote edits. Reattaching an
already-running provider must not prepare a new launch or require the
old launch files. The full backend must recover the admitted
instructions from the matching durable run checkpoint.
**Steps to reproduce**
Run the explicit
`rolling-web-deployment.runner-codex.daytona.recover-controller` Product
E2E cell with a complete server/UI build, disposable PostgreSQL/S3, and
an owned Daytona fixture. The new boundary regressions also reproduce
the premature guard and missing runtime rejection without a model call.
**Paperclip version or commit**
The failures were observed on
`247552fe0719dcc92f7b42c50b658be6f624986d`. This branch is based on
`e122478368`.
## What Changed
- Validate Runner and remote workspace authority before runtime
selection without trusting the initial run label.
- Reject resolved legacy execution and local-source fallback, preserving
the native warm-policy check before admission.
- Rebind remote agent-file caches on ephemeral controllers and retrieve
their validated contents without uploading over the retained execution.
Preserve pending saves when the remote collector is unavailable.
- Restore over-quota remote directories without applying save quotas to
run admission; ordinary saves still enforce quotas.
- Skip provider launch asset materialization when adopting an existing
Runner. A real-process regression deletes the old skill/instruction
sources before reattachment.
- Recover admitted system instructions through Runner-owned checkpoint
validation for the same run, session and context digest. Fresh launches
still compose current instructions.
- Make the attention-retention test advance past its database-assigned
deadline, fixing a clean-master clock mismatch.
- Serialize remote collection and cleanup across controllers with an
operation-scoped Postgres advisory lock; fence stale cache-path updates.
- Keep synthetic deployment probe tasks in a separate disposable company
so they cannot block the business completion oracle.
- Share the launcher's directory cleanup with rolling retirement. Reject
a symlink root and skip linked descendants.
- Test admission with real PostgreSQL transactions and cleanup with real
read-only directories and external symlinks.
- Update developer documentation and the canonical qualification record.
Broader deployment acceptance remains in progress.
## Verification
- `pnpm -r typecheck` and `pnpm build` passed.
- All 156 workspace preparation, runtime selection and ephemeral
contract tests passed. Two new regressions failed before the source fix.
- Seven focused Product E2E cleanup/replacement tests and `pnpm
test:e2e:runner:typecheck` passed.
- `pnpm test:run` stopped at an unrelated 15-second chat integration
timeout after 338 passes. That exact test passed in isolation without
increasing its timeout. The later full-suite shards did not run.
- `pnpm test:e2e:runner:unit` reported 5,898 passes and 13 failures.
Clean master reproduced 11 failures in unchanged onboarding
expectations, operating receipts, Pi catalog assertions, workflow
bootstrap counts and embedded PostgreSQL startup. The other two workflow
assertions were fixed by the intervening master commit. These failures
are retained, not counted as passes.
- All 122 workspace-preparation tests pass after the reviewed
host-fallback fix. The first broader attempt exposed a test environment
leak; restoring the ephemeral flag between cases resolved it.
- Agent-directory and instruction regressions passed 179 tests. Five
focused cache-loss cases pass: different home, same path on an empty
filesystem, foreign lease, concurrent controller, and missing collector.
- The live fixture at `a18f6b85a3` gracefully stopped A and deleted its
entire home, then failed the 420-second remote reattachment deadline on
the saved local agent-directory path. That failure is retained; exact
owned sandbox cleanup was independently confirmed. The next live attempt
at `e14ca5f3e9` restored the cache and transferred the same runner, but
failed while rematerializing launch assets from A. Both failures are
retained and their owned sandboxes confirmed absent. No active-overlap,
ten-deploy, or production-canary acceptance is claimed.
- All 71 related Runner adoption/recovery cases pass, including the new
removed-context regression (failed before the source fix).
- The revised full agent-directory suite passed 110 cases and failed
three cleanup timing cases. All 20 cleanup/cache-loss cases then passed
in isolation with unchanged timeouts, including over-quota recovery.
- Server and Runner TypeScript checks pass after the latest corrections;
the live retry remains pending.
- The live test at `67714af9ff` adopted the same remote execution after
A was deleted, then failed while the full backend read the retired
instruction bundle. All owned sandboxes were independently confirmed
absent. Its regression now passes with actual backend recovery and no
new thread or turn. All 59 backend factory tests and focused checkpoint
identity tests pass. Runner and server TypeScript checks pass; the exact
committed server build passes. A fresh paid-model retry followed.
- The attention-retention CI timeout reproduced on clean master. The
deadline correction passes all 53 tests in that file with unchanged
timeouts.
- The real `cd690de4d3` run transferred the same runner/session after
deleting A, completed, and completed the next revision. During handoff,
372 reads and 38 saves had zero failures; all 38 saves were verified.
The full attempt was interrupted because the probe's open task
contaminated the business company oracle. This remains incomplete
evidence. The scope regression fails before the fixture fix, then
passes; all nine deployment support tests and E2E typecheck pass. All
three owned sandboxes were confirmed absent and temporary
credentials/storage were cleaned up. A clean corrected attempt is
running.
## Risks
- The runtime guard must still reject legacy and per-turn execution
before dispatch. Tests cover these refusals and a valid resolved warm
policy.
- Directory permission changes apply only during cleanup of stopped,
test-owned homes. Traversal does not follow symlinks. This changes no
production cache permissions.
- Cache reconstruction must not change remote execution authority or
rebase the original save baseline. Tests verify foreign-lease rejection,
preserved receipts, and absence of local-to-remote upload.
- Postgres locking is scoped to active remote file work and adds no idle
polling; local/durable-server locking remains unchanged.
- No schema or public API changes. External conversation exemptions
remain unchanged.
## Model Used
OpenAI GPT-6 Codex, with reasoning, repository editing, shell execution
and live integration-test tools. The exact runtime model variant and
context-window size are not exposed in this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Latest real qualification at
|
||
|
DottaandPaperclip
|
66e8fab944 |
fix: complete native chat responses and improve API discovery (#18364)
## Thinking Path > - Paperclip must perform work and report its outcome accurately. > - A chat response and its delegated or operator-owned work are separate stages. > - Native chat contracts instead required literal completion of the user's whole request. > - Agents kept searching or waiting even when a truthful handoff should finish the response. > - OpenCode also has separate acknowledgement backlog and empty-response failure paths. > - This PR aligns chat completion with its current stage and makes OpenCode completion truthful. ## Linked Issues or Issue Description Refs #18098 for retained OpenCode evidence and #18377 for separate eval setup, oracle and reporting repairs. **What happened?** Real pinned OpenCode 1.18.34 reproduced an MCP timeout behind 800 text fragments. Pilot [41222196525](https://github.com/paperclipai/paperclip/actions/runs/41222196525) separately exposed a fabricated successful reply after a reasoning-only response. Later restart attempts timed out without an answer. Their native criterion literally required restarting the instance, although the chat instructions and authority required an operator handoff. General authority guidance alone did not resolve it. **Expected behavior:** A standing chat completes its current authorized response. It performs and verifies permitted actions, verifies delegated handoffs, or explains the specific unavailable action and operator path. The worker's task remains separate. Ordinary execution tasks must still complete their requested work. Empty provider responses must not become invented success. **Version and reproduction:** Actual pinned OpenCode and native Rust Runner, with retained full-stack AWS attempts. Product baseline master `51ea624344`. This PR adds no historical catalog to master. ## What Changed - Build standing-chat completion contracts from the server-owned conversation identity. Scope each current request to the response stage; preserve request text, criterion IDs, order, human-response scope and source revisions. Require verified authorized actions and truthful handoffs. Ordinary task contracts stay strict. Existing admitted inputs retain their saved contract during recovery; new contract content uses the existing revision/hash mechanism. - Guide prospective defaults to their owning resource and preserve existing work unless backfill is requested. Guide unavailable board/operator operations to the supported handoff. These are general chat instructions with no eval identities or expected answers. - Batch adjacent assistant text for at most 50 ms and 4,000 UTF-8 bytes. Preserve text, metadata and order; flush before non-text frames and shutdown. - Give an empty OpenCode response one correction within its existing turn, model, reasoning, context and deadline. Accept actual completed answer parts with incomplete headers. Preserve actual active tool work, bound idle evidence grace and fail truthfully if correction stays empty. - Reject empty OpenCode terminals in Rust without actual prose or an accepted semantic result. Never invent successful replies or evidence. Refresh generated project-discovery contracts. - Preserve accurate public API-discovery descriptions for existing project defaults and managed instance restart operations, including their field scope and authority limits. ## Verification **Qualified: [pilot 41256592937](https://github.com/paperclipai/paperclip/actions/runs/41256592937) passes 3/3, with all 35 independent checks passing.** | Case | Checks | Full case duration | Evidence and cleanup | | --- | --- | --- | --- | | Instance restart | 9/9 | 140.1 s | complete | | Project defaults | 11/11 | 204.2 s | complete | | Project work | 15/15 | 204.0 s | complete | All three are original-attempt results from source `c51e079f7f2364305b131ffd549e263ff78c5335`, containing the exact product files from `13c2b61f31d873a475ea0c3c026b19926ff42301`. Actual saved conversation contracts were inspected, as were ordinary native worker contracts where present. Each protected archive includes the actual API state and final PNG; cleanup completes. Original queries, `openrouter/deepseek/deepseek-v4.1-flash-1017`, finite case clocks, concurrency one and zero eval retries are preserved. Earlier passes do not transfer into this qualification. Both current heads pass all checks: product **62** (56 success, one neutral, five skipped), catalog **56** (54 success, one neutral, one skipped). Both have exact-head Greptile **5/5**, zero unresolved review threads and clean mergeability. The three trusted private-evidence protocol files remain byte-identical to the dispatch workflow. Raw evidence stays protected. - **2,044 API discovery tests pass.** Red controls showed that “restart instance” selected unrelated drain/settings operations and “save project defaults” selected repository replacement. Accurate public summaries now rank the existing operations correctly; route registration preserves descriptions. No endpoint, permission or field behavior changes. - **112 real-database, authenticated HTTP, native Runner and OpenAPI consumer tests pass locally**, using disposable task-owned Docker PostgreSQL. It was removed afterward; shared IPC was untouched. These cover persisted contract selection, admitted native input and the model envelope together. The isolated CI suites also pass. - **77 focused contract/input/context tests pass.** Red controls catch the wrong conversation predicate; canonical identity requires both agent and user IDs, including the actual heartbeat projection. Ordinary tasks, agent-only issues and planner tasks remain strict. Coalesced requests preserve criterion identities, source hashes and human-response scope. - **144 OpenCode driver tests pass**; native core **441 pass, one ignored**; packaging **56 pass, six skipped**. Server typecheck and build pass. - Real pinned OpenCode/native Runner: a delayed 25-second acknowledgement completes within the original 30-second MCP deadline. The text-fragment control changes 817 notifications / 54.1 seconds / timeout to 45 / 19.0 seconds / accepted finish. Separate fast, six-minute and real authenticated API probes complete 32 tool calls plus finish. These do not reproduce every remaining historical MCP timeout, and no common cause is claimed for all such failures. Retained failures remain attributed to their original source. Earlier runs exposed fabricated empty-response success, prospective-default backfill, a literal whole-task chat criterion, fixture readiness and oracle/reporting defects. The immediately preceding [41250502608](https://github.com/paperclipai/paperclip/actions/runs/41250502608), product `c0bf249807` / source `7b1de24ce1`, is **1/3**: work 15/15; restart 4/9 with the repaired contract actually admitted but no answer; defaults timed out before settlement, then exceeded the unchanged 256 MiB original-evidence allowance. Defaults domain behavior on that attempt remains inconclusive. #18377 retains the full attempt ledger, including missing archives and pre-paid cancellations. The fresh 3/3 establishes these selected outcomes; it does not establish full-catalog reliability. ## Risks The shared contract change applies to standing native chats across providers. It changes model instructions and completion scope, while permission, approval and review gates remain enforced by their existing owners. Incorrect model claims remain possible; the product evals independently require actual mutations, native work and delivery. The correction adds at most one provider request within the existing deadline; text batching can delay adjacent text by 50 ms. Three qualified attempts do not establish full-catalog reliability. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, tool use, code execution and live eval tooling. Exact hosted model revision and context-window size are not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge - [x] Final same-source three-cell pilot passes 3/3 with required evidence and cleanup --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
75fa37f30f |
fix(server): expire native backups from retention deadlines (#18380)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
baf651c8d0 |
perf(heartbeat): drive retry promotion from durable deadlines (#18387)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
8bcc5a01ed |
fix(runner): preserve OpenCode written-answer defaults (#18456)
## Thinking Path > - Paperclip lets people answer questions while native agent work is active. > - Runner drivers translate provider question forms and answers into the shared contract. > - Pinned OpenCode 1.18.34 enables written answers when `custom` is omitted. > - The driver required explicit `true`, so it removed that input from ordinary native questions. > - The same branch could override a native opt-out or add a separate custom field to a text-only question. > - This pull request preserves the native default, explicit opt-out, original choices, and exact answer delivery. ## Linked Issues or Issue Description **What happened?** A native OpenCode question with choices and no `custom` field loses its written-answer control in Paperclip. The pinned provider enables that control by default. A legacy alias can also override native `custom:false`. A question with no choices and `custom:true` is rejected because the driver adds a separate custom field to text mode. **Expected behavior** Follow the [pinned OpenCode question contract](https://github.com/anomalyco/opencode/blob/v1.18.34/packages/schema/src/v1/question.ts). Preserve the written-answer default for choices. Honor native `custom:false`. Keep optionless questions text-only. Preserve every supplied choice and return the exact written answer to the same native request. **Steps to reproduce** Use the normalizer regression or the local OpenCode driver fixture. Emit a native question with choices and omit `custom`. The unfixed driver omits `customAnswer`; its existing written-answer roundtrip assertion fails. The separate negative controls reproduce native opt-out precedence and text-only normalization errors. **Paperclip version or commit** Reproduced on main `f2db909393de124caf6a5380a52d4e0e1d621661`, with pinned `opencode-ai@1.18.34`. ## What Changed - Apply the native `custom` flag first, then the legacy alias, then the pinned default of `true`. - Add a separate custom field only when choices exist. Keep optionless questions text-only. - Add default, opt-out, alias, and exact response validation regressions. Make the driver fixture omit the optional native flag and retain its exact custom-answer roundtrip assertion. - Document the pinned native behavior. ## Verification - Before the fix, the real normalizer fails three targeted assertions: omitted custom input, native opt-out precedence, and optionless text with `custom:true`. - The local driver subprocess regression also fails before the fix. Its native request omits `custom`, and the driver drops the expected written-answer control. - After the fix, all **79 tests in three files pass**: the new nine-case normalizer suite, the full 65-case OpenCode driver suite, and the five question-set response tests. The driver sends the unchanged written answer in the provider's ordered answer array and records one terminal request resolution. - Runner TypeScript check passes. A first attempt lacked the isolated eval-kernel declarations; building that local dependency resolved the setup gap. Locked dependencies were installed from the existing cache with scripts disabled. - Diff checks pass. Exact-head [canonical CI run 41203866644](https://github.com/paperclipai/paperclip/actions/runs/41203866644) passes build, typecheck, release registry, Runner checks, server tests, and all eight E2E shards at `72b21207e8590b78e24fe2cef4e3f6c374be53a4`. All current PR checks are terminal with no failures. Storybook is skipped for this scope and Paperclip Review is neutral. Full repository checks were not repeated locally for this narrow provider translation change. - [Fresh Greptile review](https://github.com/paperclipai/paperclip/pull/18456#issuecomment-6543742512) gives 5/5 on that exact head with no actionable findings. All review threads are addressed. GitHub reports `MERGEABLE` and `CLEAN`. - Local tests use a controlled provider fixture. They do not establish a paid model pass or attribute a deployment incident without its original native arguments. No QA run, runtime deployment, protected pin, journey oracle, prompt, budget, run bound, or model setting changed. ## Risks Native OpenCode choices now expose the written-answer control that the provider already enables by default. Explicit native opt-out wins over the legacy alias, and disabled custom responses remain rejected. Existing choice IDs and labels remain unchanged. Deployed users need the repaired Runner artifact through the normal delivery path. Local driver proof does not replace deployed journey qualification. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, source inspection, and code execution. The host does not expose the exact underlying model version or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with available version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs; none repairs this native default - [x] I have described the issue in-PR following the bug template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
492d537340 |
fix(native): recover controllers across webserver replacement (#18343)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native execution runs in a remote Runner while the webserver owns its controller. > - Encrypted PRP checkpoints and session snapshots already survive a webserver restart in Postgres. > - Ordinary warm and per-turn admission still needs local markers and supervisor metadata to find that state. > - Replacing the webserver with an empty native controller directory can therefore lose a usable retained session. > - This change adds bounded recovery metadata to the existing snapshot transaction and reconstructs local projections from the selected durable source. > - The benefit is native execution continuity across webserver replacement, with the existing ownership and agent lifecycle policies. ## Linked Issues or Issue Description **What existing behavior does this improve?** Native remote controller recovery and retained-session continuation during a webserver upgrade. **Subsystem affected** server/ — native runtime, checkpoint persistence, recovery scheduling and internal upgrade reporting. Product E2E qualification also changes. **Current behavior** Without both the local PRP marker and supervisor file, normal continuation can return before discovering the saved Postgres checkpoint. The snapshot already exists in Postgres, but its local reuse compatibility envelope cannot be reconstructed. **Proposed behavior** Select the existing durable source, validate its identity and ownership, restore its PRP state and supervisor projection, then use normal authenticated admission. Missing or inconsistent evidence holds recovery. It does not authorize fresh provider work. **Reason and benefit** A replacement webserver must recover active execution and later continue its retained session without the old server's native files. This is one component of deployment continuity. HTTP traffic switching, workspace storage migration and Railway rollout remain separate. **Breaking changes** No new public endpoint, schema migration or database pool. The private run-profile envelope is additive. Disk-independent rollback also requires a reader for the staged controller format and compact checkpoint part; the first production release containing these readers is the floor. Older file-backed sessions need an owner checkpoint or the bounded internal importer. **Additional context** Related foundations: #17401, #17916, #17254, #14621, #14622. The implementation contract and open acceptance gates are in [the dated plan](doc/plans/2026-10-30-native-controller-checkpoints.md). ## What Changed - Persist a versioned recovery envelope, at most 4 KiB, beside the redacted session snapshot in its existing transaction. Keep it out of public run projections. - Compare frozen admission separately from prepared attachment instructions during staged recovery, preserving exact template/identity/authority checks. - Bound the existing lazy checkpoint pool at three connections after measured two-slot burst backlog; keep request/renewal capacities and SQL operation counts unchanged. The tuning is a separate commit. - Discover and restore the exact selected retained source without local native markers. Preserve the 32-candidate window, progress barriers and current authority checks. - Fence snapshot and result writes. Acquire coordinator, issue and run locks in order and check expiry after lock waits. Combine snapshot ownership/read work into one statement. - Reuse existing recovery deadlines and drain/resume scheduling. Retained-owner waits preserve the original unstarted run and actual lease deadline. Stop, ownership and provider-progress fences precede deferral. Add bounded native evidence reporting, retained-source import and retention protection. - Admit a settled warm source with an authenticated idle goal-read suffix while retaining active-work and mixed-authority holds. - Permit lifecycle `turn.stop` checkpoints after accepted results under existing ownership fences. Claim the existing successor coordinator before retiring the old warm session, and stage shared attachments only after verified retirement. Preserve omitted isolated-home defaults and existing fresh skill-selection order during recovery without hiding real configuration changes. - Carry the exact durable six-field PRP binding during active recovery after a healthy warm handoff. Keep the Runner mismatch guard and preserve opaque attachment IDs. - Preserve an already verified active or idle warm process during remote preparation; revalidate its proof/liveness and retain later authenticated PRP attachment. Ordinary resumed leases retain suspension checks. - Drain retained source events under their original authority, reconcile immutable source accounting and block unknown source tool dispatch before target work. - Persist compact staged handoff intent through authenticated adoption and recover direct/listener command-loss windows without another Runner or duplicate attachment. - Rebuild suspended lifecycle receipts for resumed prepared tasks, and retry missing lifecycle reads after partial PRP materialization without overwriting contradictory evidence. - Add normal warm/per-turn diskless continuation coverage, real Postgres races and SQL-cost checks. Add a reproducible paired load workload. - Give only the explicit outage fixture a 15-minute provider idle window, with a red/green catalog regression preserving all production timing and lifecycle policies. - Publish the goal-less task-session pointer in the existing post-acceptance terminal checkpoint transaction, with current ownership, accepted-result, reset and newer-progress fences. Keep a private bound receipt so late completion cannot duplicate or resurrect publication. - Use a retained warm sandbox in the dedicated continuity fixture, and accept an exact output token followed by an explanation. Keep the strict identity oracle. - Extend the explicit Daytona cell to a five-minute active outage and another replacement after completion. Require retained provider identity, a new tool receipt and a useful follow-up result. Add an owned loopback Postgres E2E fixture for hosts that cannot start embedded Postgres. ## Verification **Merge recommendation — `fc68584ce6231a02e47ecd743f55b036ccea034f`: ready for the qualified native-controller component.** Full CI, fresh review, local application paths, controlled performance qualification and the strict live Linux/Daytona case pass. Independent resource cleanup is complete. Broader rollout, browser reconnection and the unexecuted transport/platform combinations remain integration gates. Integrated 484 passed full local recursive typecheck/build and ordinary active→completion→cache-loss→C component qualification. Fresh review then found a valid attached-wake failure: preparation appends attachment instructions before the verifier compares execution to the frozen admission. 9f4 fixes that by separately checking the pre-preparation admission and exact prepared execution. Both hashes are captured privately per admission; the full template, identity, configuration and authority checks remain. Its regression fails before the fix; 36 focused cases, 42 staged source/target cases and server typecheck pass afterward. The review thread is resolved. Exact-head full CI passes ([run 41168228822](https://github.com/paperclipai/paperclip/actions/runs/41168228822): 48 successful jobs, one intended skip); the complete `pnpm test:run` partitions are forced, with the five native suites in the Runner lane. This is not a claim that an unsharded local command passed. The fresh exact-head Greptile check succeeds with 5/5 and all six threads resolved. The original two-connection database comparison closes **steady PASS and burst FAIL**. Every relative CPU/WAL/request-p99 gate passes and all operations/ownership remain correct, but candidate arm 3 drains in 1,015.99 ms and baseline arm 4 in 1,059.67 ms against the unchanged 1,000 ms absolute gate. Those results remain failed. A separate 2→3→4→2 diagnostic reproduces drain times 1,051.57 / 956.79 / 882.02 / 1,065.72 ms: three is the smallest tested passing cap, and the repeated two-cap failure survives run-order/cache effects. Separately reviewable `fc68584ce6` changes the existing lazy checkpoint pool default to three connections. Request capacity stays ten and liveness two; idle release, shutdown fencing, transaction counts and pool count are unchanged. Thirteen DB lifecycle/cap tests and DB typecheck pass. Full recursive typecheck/build pass at exact |
||
|
DottaandPaperclip
|
59da15066a |
feat(runner): deliver native Kimi Code (#18391)
Deliver native Kimi Code setup, exact models, scoped credentials, verified packages and runtime packs. Preserve current upstream release attestations and restore verified same-run remote retry dispatch without weakening process ownership. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b866fa4bdc |
feat(runner): add pinned Kimi Code native runtime (#18028)
Add the pinned Kimi Code runtime, native instruction and model admission, scoped credentials, and pending-candidate recovery contracts. Retain current master release attestations and provider-pack compatibility. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8ff812cdde |
feat(chat): deliver proactive task messages to agent chats (#18374)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agent Chat gives each person a persistent conversation with each
agent.
> - Tasks and routines produce results outside that conversation.
> - Those runs need a safe way to bring a requested result back to the
person.
> - The server must select the agent's own chat and check the source
task's access rules.
> - This pull request adds proactive chat messages and unread
indicators.
> - People can see the result, open its source task, and reply in the
same chat.
## Linked Issues or Issue Description
**Subsystem affected**
Server, native runner tools, shared contracts, database, and UI.
**Problem or motivation**
An agent can report progress on a task or comment on an authorized task.
It cannot send a routine result to its own standing chat with the
responsible person. A morning review routine therefore cannot leave a
visible digest in that person's chat.
**Proposed solution**
Add `send_chat_message({body, idempotencyKey})` and `POST
/api/agents/me/chat/messages`. The server selects the company, agent,
and responsible person from the active run. It saves an attributed
comment and audit record in one transaction. The chat shows a
source-task link and per-person unread state.
**Alternatives considered**
A destination parameter on a generic comment tool would expose
unnecessary routing authority. A new delivery worker would add another
durable delivery path. This change uses a dedicated tool and the
existing comment and live-event paths.
**Roadmap alignment**
This extends the experimental Agent Chat capability in `ROADMAP.md`. The
existing `enableAgentChat` setting controls it. Related public work:
Refs #13282, Refs #13283, and Refs #14408. Searches for proactive chat
and unread chat found no duplicate PR.
## What Changed
- Add a focused messaging service shared by the native tool and HTTP
route. It checks current run ownership, execution mode, recipient
membership, task access, and task privacy.
- Reuse the canonical company/agent/person conversation. Save the
comment and audit record together. Keep task completion and chat
execution separate from delivery.
- Add source-task provenance and source-scoped retry keys. Identical
retries return the saved receipt. Changed content returns a conflict.
- Enforce the conversation's fixed agent at the shared comment-write
boundary. Keep normal reply cancellation and session checks.
- Add chat unread state, accessible agent dots, aggregate Chat
indicators, and visible-message read cutoffs. Refresh through existing
live events.
- Add the request-key constraint, native tool contracts, documentation,
capability maps, stories, and regression coverage.
## Verification
- Local delivery/security proof: all 38 proactive-message cases pass.
Eight credential-scope regressions and four quota regressions fail
against the prior source. The repaired delivery and canonical
cross-issue cap suites pass all 49 cases; all 31 OpenAPI tests, server
typecheck, and capability-map checks pass. The shared per-run cap covers
native and HTTP sends, mixed ordinary mutations, concurrent last-slot
contention, and retry/recovery without double charging.
- Credential tests preserve the authenticated person and authorizer;
reject bridge/test credentials and persisted skill-test runs after
task-mode changes; and recheck API-key revocation, scope, and
responsible person after destination locking. The earlier
delivery/native-authority suites passed all 74 cases before the four
quota additions.
- Four PostgreSQL contention cases cover native and HTTP sends plus
retries during a native chat mutation. All four deadlock against the
earlier source and pass with destination-before-source/run locking.
Recovery and human-reply delivery enqueue only the human comment; a
proactive digest never wakes the chat agent.
- Passed 108 companion conversation, native-authority, and OpenAPI
tests; 112 runner catalog/workflow-report tests; 47 completion-delivery
tests; and 25 project-handoff tests. Focused runner tests cover native
tool availability without generic API tools.
- All 327 affected UI tests, UI typecheck, token gates, and catalog
checks pass. The read hook retries at most three times, respects server
retry delay, coalesces only seen cutoffs, recovers on reconnect, and
cancels on scope changes.
- Real browser journeys with a deterministic process agent verify
routine-to-chat delivery, first-chat creation, unread state across
reload, the source link, secondary chat navigation, and a follow-up
reply referring to the digest. This does not qualify live GitHub
retrieval or model output quality.
- The Storybook showcase uses production components with deterministic
network fixtures. Rendered interactions pass for the landing, ordinary
read success, transient read recovery, and persistent failure after
bounded retries. Both Chat navigation variants update from two unread
chats to one while only Kodi's dot clears. Mobile and light variants
were reviewed. The existing Coordinator Recents label contrast finding
is unchanged; no Linux visual baseline archive exists.
- Current master's independent sandbox migration retains 0418.
Regenerated the feature migration as 0419 from the combined schema and
retained exactly the newest five snapshots; numbering/safety, replay,
and schema-drift checks pass.
- Workspace typecheck and build passed before PR preparation; canonical
server/UI typechecks passed again on the combined tree. Migration
`0419_slippery_impossible_man.sql` passes safety and schema drift checks
on disposable PostgreSQL, including interrupted index repair, concurrent
migrators, one-time owner read-state backfill, preserved receipts, and
two SQL replays. A default five-second local snapshot comparison timed
out; a diagnostic 30-second run passed without drift.
- Earlier failures are retained in the record: full local `pnpm
test:run` and broad shards encountered embedded-PostgreSQL startup,
timing, and provider-recovery failures. Thirteen selected
external-channel cases pass in isolation on the original and changed
sources; an unordered comment-row assertion now explicitly orders both
reads. The earlier route registry, native tool count, runner catalog,
and accessible Chat badge-name failures were fixed and passed full
remote CI on `920b3bb7db`.
- Repeated managed-skill server8 failures came from CI's newer synthetic
merge: its Cloud guard correctly rejected the old local-disk fixture.
Master independently repaired the fixture in #18385; this branch keeps
that repair unchanged. All 57 managed-skill and Cloud prerequisite cases
pass on the combined tree, and that fixture has no remaining PR diff.
- Current reviewed head `dc16f314739a6d977bed21cb1446d0e80a541578`:
Greptile 5/5 with no findings, Superagent security assessment passed,
advisory Paperclip Review 5/5 with no findings, and zero open review
threads. All 55 current-head check runs pass, with five expected skips
and the advisory review neutral at 5/5; Snyk also passes. No checks are
pending or failing. Protected squash auto-merge is enabled and is
waiting for required code-owner approval. A normal exact-head merge was
refused by the base-branch policy; no admin bypass was used. The
previous head's Superagent quota finding is fixed and covered above.
## Risks
- Chat remains company-visible. Delivery rejects a source task with
private-task, project, or ancestor restrictions.
- Read state uses a comment timestamp cutoff. The server retains
database precision and never moves a read receipt backward.
- The new unique constraint changes comment query planning. An existing
regression now uses an explicit row order. The migration builds the
unique index concurrently, then takes a brief table lock to attach it.
It uses the existing top-level migration and interrupted-index recovery
path.
- The upgrade backfills missing chat-owner read-state rows at the latest
existing agent comment. Existing read receipts remain intact. New chats
can still be unread before the human participates.
- Quota observation commits atomically with delivery. A rejected send
rolls back its attempted denial audit row and any first-chat creation;
the canonical warning remains in server logs, and previously spent quota
remains enforced. Identical/conflicting retries and receipt recovery
spend no extra slot.
- This version delivers in-app only. It adds no external-channel, push,
or email delivery.
- Local full-suite acceptance remains limited by the failures recorded
above. The feature stays behind `enableAgentChat`.
## Model Used
OpenAI Codex. PR preparation and CI work used `gpt-6.1-sol` with tool
use and code execution. The earlier implementation also used Codex, but
its exact model ID was not recorded in the visible session. The
context-window size was not provided.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run focused tests locally and they pass; the broad local
failures are recorded above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
|
||
|
Nicky LeachandPaperclip
|
493e229d93 |
perf(runner): cache verified native executable digests (#18390)
## Thinking Path > - Paperclip manages AI agents and their task execution. > - Runner checks qualified native programs before it starts a provider. > - Installation checks and command opening can hash the same program again. > - A new sidecar process repeats this work on a later warm task. > - This pull request stores the computed digest in a private filesystem cache. > - Every cache hit still checks the independently selected trusted digest and current file metadata. > - An unchanged program can start without another full native file read. ## Linked Issues or Issue Description **What existing behavior does this improve?** Repeated **qualified ACPX** native executable admission in Runner. Claude, ACPX-Codex, and Grok use this verifier in both Core-owned and plugin-backed launches. The ordinary native Codex app-server launcher is a different path and is not changed. A retained provider that opens no new native command has no saved hash work here. The measured Claude task path does not require a comment backlog. **Current behavior** Installation verification and final command opening each read and hash the native executable. The qualified Linux x64 Claude file is 241,667,256 bytes. A new verification and command lease reads 483,334,512 native bytes. Another Node process repeats both reads. **Proposed behavior** Store the actual computed SHA-256 digest by device, inode, size, nanosecond modification time, and nanosecond change time. Reuse it across Runner and sidecar processes. Compare each hit with the current build-owned digest. Hash the file on a miss or changed identity. This replaces the earlier Claude-only deferral design in this same PR. **Reason and benefit** The first verification and command lease reads the native file once. Later processes read none of its native bytes while its identity stays the same. Related: #16372 established qualified executable admission. This change is independent of other preparation PRs. It does not change provider pins or profile versions. ## What Changed - Add a shared filesystem digest cache under the absolute OS temporary root at `paperclip-runner-<uid>/executable-digests-v1`. - Use private directories and files, versioned hash-only filenames, bounded JSON entries, and atomic temporary-file replacement. Cache files contain a version and the computed digest. They contain no credentials or source paths. - Fall back to full hashing when the cache is missing, unreadable, malformed, or unusable. Keep current expected-digest selection outside the cache. - Keep cache data outside task homes. Neither `HOME` nor task cwd selects the path. The OS temporary-root selection honors absolute platform `TMPDIR`, `TMP`, or `TEMP` overrides. A relative root disables caching. Hosts must keep that root outside task workspaces. Temp cleanup causes a later rehash. - Keep the no-follow executable open, regular-file and size checks, execute permission, descriptor ownership, before/after identity checks, and preparation-to-launch identity comparison. Cache publication occurs before the final identity checks. - Use this path for standalone installation checks and final command opening. Remove the earlier pending-digest type and Claude-only launch helper. Runtime host and profile selection keep their original structure. - Test changed content, all fingerprint fields, symlinks, mode changes, inode replacement, mutation during lookup/publication, concurrent cache writers, wrong trusted pins, and cache failures. - Include a deliberate trust-boundary test: a process running as the same OS user can plant a matching cached digest for changed bytes. It opens a test descriptor and executes nothing. In plain language: Runner remembers the hash for the same file. It still checks which file it opened and which hash the build requires. The cache trusts the OS user who can write it. ## Verification From `packages/paperclip-runner`: - `../../node_modules/.bin/vitest run src/drivers/acpx/executable-digest-cache.test.ts src/drivers/acpx/installation-integrity.test.ts src/drivers/acpx/runtime-host.test.ts` — 177 passed, 6 platform-specific skips. - `../../node_modules/.bin/tsc -p tsconfig.json --noEmit` — passed. - `git diff --check` and the added-line secret, PII, private-address, and code-comment scan — passed. The offline comparison uses baseline `45a3a92e6dc52d20923b9aa34a22fee2ec5443f9` and treatment `c4356df12032448de90516ae329d8f939af10984`. Each sample starts a different Node process. Each normal sample runs the actual qualified installation verifier and opens a command lease. Seven samples per case follow one warmup. The cache is removed before each cold case. The next process reuses its entry. Baseline order alternates around the treatment pair. Command digest and package paths match in every pair. | Verifier plus command opening | Baseline | Cold cache | Later process hit | | --- | ---: | ---: | ---: | | Full native reads | 2 | 1 | 0 | | Native bytes read | 483,334,512 | 241,667,256 | 0 | | Local median, ms | 1041.26 | 602.70 | 130.17 | | Local range, ms | 994.20–1133.11 | 580.27–999.83 | 121.96–179.45 | | Median whole child process, ms | 1379.96 | 988.75 | 523.01 | Raw verifier-plus-open samples in ms: - Baseline: 1133.11, 1074.40, 1061.99, 1041.26, 994.20, 1036.12, 1037.70. - Cold: 647.13, 646.33, 999.83, 584.83, 580.27, 583.37, 602.70. - Later process hit: 179.45, 138.37, 134.39, 130.17, 121.96, 122.33, 126.68. Separate direct-opener fault probes use a disposable copy of that same native file: | Change | Native bytes read | Result | Local open, ms | | --- | ---: | --- | ---: | | First copied file | 241,667,256 | Accepted | 526.14 | | Next process, different HOME and task cwd | 0 | Accepted | 24.91 | | Changed timestamps | 241,667,256 | Accepted after hashing | 531.40 | | New inode, identical bytes | 241,667,256 | Accepted after hashing | 536.70 | | Corrupt byte, same size and exactly restored mtime | 241,667,256 | Rejected | Whole process: 841.36 | | Different independent trusted pin | 0 | Rejected | Whole process: 374.76 | The corruption probe keeps `mtimeNs` exactly equal before and after, at `1000000000000000000`. Its changed `ctimeNs` invalidates the entry. Cache expiry is not required. The changed-pin probe rejects the old actual digest without a native read. The cached image is `sha256:99ce80690a0c13328f41fd84156cc73e1ac562eaed3120fd41d04d7acdfb325d`. The Node binary is `/opt/paperclip/runner/libexec/node`, version `v24.20.0`. The artifact is Claude SDK `0.3.286`, with expected digest `sha256:fe503f65c6289d59c23e5b21ae44f03583f997dd33a2cbfc75ab4f96fb8fc73f`. The container used a read-only root, a disposable writable tmpfs, and disabled network. No provider process, model request, credential, or deployment was used. **These elapsed values are diagnostics from Linux x64 CPU emulation on an ARM64 Docker host. They are not production latency or TTFT estimates.** The exact byte counts, cross-process reuse, output equality, and invalidation results are the portable evidence. The saved phase 0 exports have no directly comparable native executable hashing span. Their broad `runner.transport.attach` spans include other work. A production comparison within 20% is therefore unavailable for this seam. The earlier local 919 ms control was also an emulation diagnostic; it is not a production baseline. The initial Docker attempt was denied access to the local socket by the sandbox. The same offline command then ran with local socket permission. An initial fault probe restored mtime through a `Date`, which did not prove exact nanosecond equality. Its raw report is retained as historical; the final probe sets and verifies an exact mtime before and after corruption. The prior Claude-only implementation and its checks and measurements remain historical evidence. They do not qualify this replacement. The superseded home-directory cache head had one browser recovery test timeout in its GitHub workflow. It expected a saved title after session recovery. The failure is retained. No browser code was changed; the current head reruns CI after moving the cache outside task homes. No full local CI suite was run, as requested. GitHub CI supplies full regression coverage. Current head `c4356df12032448de90516ae329d8f939af10984` passed the complete [GitHub workflow](https://github.com/paperclipai/paperclip/actions/runs/41145223331), including all browser shards. Its latest checks have 56 successes and 2 intended skips, including both security scans started when the PR became ready. Standard Greptile reviewed this exact head at 5/5 with no open findings. Apex then reviewed the same head at 5/5 under the documented OS-user trust policy. It reported no new findings. There are no unresolved review threads. The PR is ready for review and remains unmerged. ## Risks - This deliberately weakens protection against the same OS user. That user can write the cache and can plant a chosen digest for a file fingerprint. A cache hit trusts that record; metadata identity is not a fresh content hash. The explicit fault test documents this accepted policy. - A read-only administrator-owned Runner can use this user-writable cache. The cache has no signature, secret, or daemon. It does not require a Launcher or ROM change. - Other users cannot use normal cache permissions to write these entries. Cache errors fall back to hashing. Removing the cache directory restores full hashing. Old fingerprints leave small unused entries until the cache is removed or OS temp cleanup evicts it. Temp eviction costs one later full hash. - A changed file normally rehashes. A digest that differs from the independent current pin still fails admission. Existing command, package, credential, process-lifetime, and final dispatch checks remain. - No schema, public protocol, package dependency, profile version, or provider pin changes. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code editing, test execution, and tool use. This session does not expose a more precise serving model ID or context-window capacity. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with the details available to this session - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the enhancement template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket ID - [x] The targeted local tests pass; the full local suite was not run as requested - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green for the replacement head - [x] Standard Greptile is 5/5 with no open findings for the replacement head - [x] Apex is 5/5 on the current head with no unresolved findings - [x] I will address review findings before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
18b1637b63 |
fix(runner): accept late Antigravity login URL during drain (#18254)
## Thinking Path > - Paperclip manages work performed by AI agents. > - The native Runner owns Antigravity subscription sign-in. > - Sign-in receives completion on stdout and the browser URL on stderr. > - The two pipes can deliver the completion result before the valid URL. > - The owner rejected that URL after it entered the completed phase. > - This pull request accepts one validated URL while the same child drains. > - A real child regression now proves the order and the failure controls. ## Linked Issues or Issue Description Refs #18188, #18189, and #18190. Related qualification work: #18345 and #18356. The duplicate search found this existing PR, so this update preserves its history and adds the real child proof here. **What happened?** A valid sign-in can fail when the authenticate response arrives before the authorization URL on the other pipe. The completed phase then rejects the valid URL. The same existing success case failed in [current follow-up CI](https://github.com/paperclipai/paperclip/actions/runs/41138618869/job/124347859445). That log has no stream-order trace, so it does not prove its inner failure branch. **Expected behavior** Both valid delivery orders must succeed. Invalid or duplicate URLs, unexpected protocol requests, failed retirement, and failed credential verification must still fail. **Steps to reproduce** 1. Run a synthetic native child that completes authenticate on stdout. 2. Wait for stdin EOF before it emits the authorization URL on stderr. EOF proves that the parent accepted the result and began retirement. 3. The valid case fails on frozen master `45a3a92e6dc52d20923b9aa34a22fee2ec5443f9`. The invalid and duplicate controls pass there. 4. Run the same regression with this PR's existing production fix. All three cases pass. **Paperclip version or commit** Existing reviewed PR parent: `646f5cdf37898ad54e1fa1cdfe39f59844da3e7c`. Frozen actual master for the real child RED: `45a3a92e6dc52d20923b9aa34a22fee2ec5443f9`. ## What Changed - Accept one fully validated authorization URL in the authentication or completed-drain phase. Keep duplicate rejection and all existing URL checks. - Retain the ten existing in-memory controls for both valid orders and eight failure paths. - Add three real child cases for valid, invalid, and duplicate late URLs. Use stdin EOF to order the streams without sleeps or a mocked retirement handler. - Check exact protocol requests, child exit, close-once behavior, Google-call counts, and persisted refresh rotation. This update changes only the existing test file; it adds no production delta to the reviewed parent. ## Verification - Frozen actual master: the new valid real child case fails, while both negative controls pass. The original eight-control parent calibration also remains recorded in this PR's history. - Existing fixed source plus the new tests: 24 passed. This includes 11 existing authentication tests, ten stream controls, and three real child cases. - Late invalid and duplicate URLs fail with zero Google calls. The valid case makes two mocked identity calls and saves the rotated synthetic refresh token. All three retire the real child and close the command lease once. - Scoped owning TypeScript `noEmit` passes for the auth module and both test files. The temporary configuration is not committed. Diff and credential-pattern checks pass. - The tests use real child pipes and private synthetic files. Google responses and the command installation are mocked. No provider account, network request, model run, app, or QA deployment was used. - Full monorepo typecheck, tests, and build were not run locally. Fresh [normal CI](https://github.com/paperclipai/paperclip/actions/runs/41140757050) passes on `1744c9f3374f78b10c2f15c42f3cf5d3af8c3119`: 56 successful checks, two skipped, and one neutral. Typecheck, build, Rust, static, Runner, server and browser lanes pass. Greptile gives that exact head a fresh 5/5 with zero unresolved threads. The PR is mergeable and clean. The earlier parent's passing CI remains historical evidence only. ## Risks The change accepts a URL while the same authenticated child drains. It still validates the exact OAuth URL and rejects a second URL. Success still requires the authenticate result, verified credentials, and proven process retirement. No provider pins, contracts, timeout bounds, retries, or public APIs change. Existing documentation already specifies these sign-in semantics; no documentation change is needed. ## Model Used OpenAI Codex, based on GPT-6, with tool use and code execution. The exact runtime model ID and context window were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0e15c92073 |
feat(skills): make managed content independent of web-server disk (#18334)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents use installed skills and exact version pins during execution. > - Managed skills still depended on mutable directories on the web server. > - A replacement server could lose acknowledged edits or remove an unassigned skill whose directory was absent. > - This pull request makes immutable versions the authority for managed content and installed repository reads. > - The server can rebuild delivery files from durable content after replacement. > - This establishes skills readiness for ephemeral web servers. General server draining and workspace recovery remain separate work. ## Linked Issues or Issue Description **Subsystem affected** Skills services, shared contracts, database receipts, Skill Studio, and runtime delivery. **Problem or motivation** A fresh web server must discover, read, edit, and deliver protected installed skills using the same database, object storage, secrets, and supported application artifacts. It must not need its predecessor's files or process memory. **Proposed solution** Use `managed` ownership with a null source locator. Read installed content by immutable version. Prepare packages before a short publication transaction. Persist execution selections before materialization. Recover exact pinned runtime assets and reject deletion while active or recovering pins exist. **Alternatives considered** Reconstructing a mutable installed directory preserves the old source-of-truth ambiguity. Permanent dual writes create two competing authorities. This change uses one managed model after a bounded legacy conversion. **Roadmap alignment** This improves the existing Skills Manager, Studio, version history, and Cloud execution capabilities. The maintainer supplied the implementation contract. This does not add a separate skills product or deploy fleet changes. Related work: #18082 supplies immutable package storage. #17273 addresses edits outside the API for linked local skills; this PR keeps that source category separate. #2648 is a closed earlier ephemeral-filesystem proposal. ## What Changed - Add managed ownership and focused content publication and execution materialization modules. Keep the archive format and package storage tables. - Publish managed create, save, delete, fork, rename, restore, and catalog content with version checks. Add durable create receipts. Preserve file-save receipt responses. - Return file version identities. Keep Studio drafts after a conflict and provide an explicit reload action. A delayed reload cannot overwrite newer typing or navigation. - Persist exact runtime selections, including empty selections. Rebuild pinned assets and verify their admitted digests on a replacement server. - Preserve source access on forks and disconnected copies. Keep the existing workspace policy category for managed skills. - Add bounded conversion, eligibility, and independent verification phases. Load persisted Cloud configuration in the standalone CLI without seeding or changing it. Reject partial current versions against installed inventories; recover only complete saved content or the exact source. Document storage migration blockers and the compatible rollback floor. - Add separate-process PostgreSQL/S3 qualification and update domain, retention, policy, and UI regressions. - Pass the plugin worker manager into Studio execution, admit restricted `skill_test` mode through native Runner contracts, and preserve admitted scope across native and REST tools. Native tests cannot create ordinary tasks, browse company data, or use external tools. - Select the exact durable Studio revision even when unassigned or Beta skills is off. Persist Studio output and audit atomically with native task completion, preserving the result on replay. ## Verification - Final revision `13e8af480b88ba6fea5737bac4bd81104d604792` passed full CI: all 49 workflow jobs completed successfully ([run 41127396098](https://github.com/paperclipai/paperclip/actions/runs/41127396098)). This includes typecheck, test suites, Runner checks, browser checks and build. Greptile review 15 is 5/5 on that same revision, with zero unresolved threads. The PR is ready for review. - Local workspace typecheck and build passed after integrating master. Focused checks passed: 33 execution/completion cases, 52 Runner HTTP cases, 16 managed-content cases, 17 Studio cases, the new real-process CLI regression, map checks and UI token gates. The CLI regression proves inspection does not seed missing Cloud configuration. Server typecheck passed after that fix. - The independent-process PostgreSQL/S3 matrix passed through normal HTTP skill routes. It covers A → B → C replacement, lost commit responses, upload crashes, concurrent saves, binary contents and executable delivery. Metadata listing and warm selected delivery retain zero object downloads. Local timing measurements are recorded; they do not establish hosted latency. - Hosted A → B → C passed on the owned disposable test tenant with external Postgres, S3 and a real remote Paperclip Computer. Predecessor temporary homes were absent. C read and executed B's acknowledged managed save. New runs used repository v1 while v2 was installed. Hashes matched an independently checked exact upstream commit. - Compatible image rollback to `6518b323` passed. Studio retained the unassigned fork's C save. A fresh remote run executed the newer managed content and historical repository pin. The actual shell event confirmed expected hashes, stdout, exit 0 and mode 0555. - Runtime candidate `9200e3fcb4` deployed successfully ([Cloud run 41124147876](https://github.com/paperclipai/paperclip-cloud/actions/runs/41124147876)). The final Studio test delivered only the unassigned fork's exact v3, executed its script and persisted a successful output snapshot. Independent command-event inspection confirmed all three hashes, stdout, exit 0 and mode 0555. All 14 installations were version-backed. Independent verification checked all 25 versions: 23 remote objects, two bundled database snapshots, zero blockers. - The deployed image is `9200e3fcb4`. Later master integration, the client-only delayed-reload draft guard and the CLI bootstrap have separate automated proof. The CLI bootstrap was also exercised against the tenant's stored configuration and storage through the same read-only services. This PR does not claim that the deployed image is the final PR revision. - Live Studio checks exposed worker wiring, admission, access scope, exact selection and completion-publication gaps. They are fixed. Failed attempts remain in the evidence. The full local suite exceeded 80 minutes under host contention and was stopped. Later combined/isolated local attempts also timed out; they are not counted as passes. Full CI covers the integrated suites. Earlier CI found an obsolete restart-scope fixture, which was corrected and passed all 52 HTTP cases. - Reproduction and migration: [`doc/skill-ephemeral-web-servers.md`](doc/skill-ephemeral-web-servers.md). Source revisions, deployments, versions, run IDs, hashes and failure history: [`doc/plans/2026-10-30-ephemeral-web-skills.md`](doc/plans/2026-10-30-ephemeral-web-skills.md). ## Risks - Legacy conversion requires stopped admission and retired filesystem-authoritative writers. Pre-conversion binaries are not supported rollback targets afterward. - Cloud readiness requires remote object storage. Local provider receipts and provider mismatches remain blockers until objects are copied and verified. - Linked local sources retain filesystem semantics and are outside the cloud guarantee. - Cold reads still fetch the existing archive format. Warm runtime preparation retains the verified-cache path. - Bundled reconciliation preserves an installed head. Existing release snapshots remain selectable. An older server must not replace a newer head with its shipped files. - No production volumes, fleet defaults, or general deployment coordination change in this PR. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution, and browser use. The exact backend model ID and context-window size are not exposed in this session. No subagents were used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
6174b4d7c8 |
fix(runner): allow session reset after confirmed task retirement (#18373)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native warm sessions retain a Runner owner between turns. > - Closing that owner must stop execution and preserve valid recovery state. > - A failed suspension can leave a rejected close promise after the task has stopped. > - A session reset then retries that same promise and blocks every fresh session. > - This PR separates task retirement proof from checkpoint reuse for a new session. > - A reset can proceed after exact stop proof, completed handoff, and file collection. ## Linked Issues or Issue Description Refs #18011. That change preserves checkpoint authority during warm preparation. This change handles a stopped owner whose checkpoint cannot be reused. Related: #14695 covers managed files retained with warm processes. A search found no duplicate stopped-owner reset fix. **What happened?** A warm Runner can stop and complete its task handoff while close reports `runner did not durably suspend before checkpoint`. The transport caches that rejected close promise. Core retains the warm owner. Each session reset retries the same promise before it can launch a provider. A controller restart clears the in-memory owner and permits recovery. **Expected behavior** A reset to a new normalized session should release the old owner when its exact task has proved that execution stopped and its handoff completed. Core must collect stopped instruction files before it admits the new session. An unconfirmed owner must continue to block replacement. **Steps to reproduce** 1. Complete a warm native run through a plugin task provider. 2. Change configuration so Core retires the warm owner. 3. Let the task stop and complete, but fail durable suspension proof. Close keeps its rejected promise. 4. Reset the session and start another run on the same task environment. 5. Observe the old close error before any new provider work starts. ## What Changed - Keep a lease-bound retirement receipt in the plugin task owner. It requires `executionStopped: true` and successful task completion after credential handoff. - Accept an unrecoverable suspension error only during retirement for a different normalized session, with that exact owner's receipt and settled Core tool results. - Collect stopped instruction files before releasing the warm owner. Preserve old checkpoint evidence. - Keep same-session, active-owner, incomplete-handoff, unrelated-owner, unresolved Core tool results, and other close-error fences. - Report Core tool settlement on the close error after ingress retirement and drain. Default missing proof to false. Expose the error through the server's Runner shim. Add regression tests and document the recovery boundary. ## Verification - The reset regression fails with the new handling disabled and passes with it enabled. - Both affected suites passed: 1,032 tests. The final 15 focused owner/reset cases pass after adding the tool-settlement guard. - The assembled regression uses the real task-owner and backend binding. Its controlled task provider proves stop and completion. Close returns the same rejected promise on every call. - Negative cases cover incomplete handoff, other close errors, same-session continuation, provider change with the same session ID, unrelated receipts, unresolved Core tool results, and failed file collection. Collection can later succeed without replaying stop or reading the old checkpoint. - Five real-process transport cases pass. They cover an insufficient suspension deadline, the cached rejected promise, late tool completion within and beyond the deadline, interruption, and result-persistence failure. Unsettled callbacks produce `semanticToolsSettled: false`. - `pnpm -r typecheck` passed with the repository's pinned Rust toolchain on PATH. After the final contract change, Runner TypeScript build and server `tsc --noEmit` passed. - The local full-suite run was stopped at the requester's direction. Full regression and build coverage are delegated to GitHub CI. - Full GitHub CI passed on `ef0040d80af679ce6d4a130468e15f75678c1f4c`. Standard Greptile reviewed the same commit: 5/5, zero comments. Apex then reviewed that same commit: 5/5, zero comments. Final checks confirm no unresolved review threads and no merge conflicts. - Initial full CI on `00c002fd5a` failed one scheduled-retry route test at 18/20 cancellations. Controlled cancellation latency reproduced the short poll-budget failure. Before the next cycle, master advanced with #18322, which coordinates that test at the first completed batch. The separate local timeout commit was dropped during rebase. The upstream test passes locally. The rebased run passed 46 targeted owner/reset and scheduled-retry route cases. This PR has no scheduled-retry source or test diff. - No deployed provider run or tenant rollout is claimed by these source tests. ## Risks The exception applies only to a new normalized session and an exact completed plugin task. The old checkpoint remains evidence and cannot authorize reuse through this path. The original suspension failure can remain visible on its run. This fix prevents repeated startup failures after a reset; it does not hide unconfirmed cleanup or make every suspension succeed. Missing task receipts and failed file collection continue to block replacement. No schema migration, plugin protocol change, Rust Runner binary change, or new polling is required. Tests use a controlled task boundary; they do not certify every external provider implementation. ## Model Used OpenAI Codex, GPT-6 family, with tool use and local code execution. The exact runtime model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal issue or instance details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] Greptile Apex is 5/5 with no unresolved findings - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
9f524ec1a7 |
fix: persist native Runner usage before transport acknowledgement (#18354)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent execution must preserve usage so Paperclip can price work and enforce budgets. > - Remote Runner execution can outlive its web receiver. > - The receiver previously acknowledged Runner events before its later usage recorder saved the accounting receipt. > - This pull request saves required native usage receipts before that transport acknowledgement. > - A replacement receiver can accept the same retained report and create one ledger charge. ## Linked Issues or Issue Description **What happened?** Native remote Runner usage could leave its durable outbox before the web receiver saved its accounting receipt. The later recorder required a local receipt directory. A replacement web receiver could lose the only remaining report when the database write failed. **Expected behavior** The Runner retains unaccepted structured usage. Postgres owns accepted receipts. Repeated delivery preserves one receipt for each source report and one charge for the run. Missing usage keeps accounting pending. **Steps to reproduce** 1. Start a supported built-in native remote Runner. 2. Make the receipt database insert fail after the provider reports usage. 3. Replace the web receiver and its local directory. 4. Reconnect the same Runner and inspect the saved receipt and ledger. **Paperclip version or commit** The initial investigation used `ebd05e9f4c88c66af7473db86bbaacfd1544aa0a`. **Deployment mode** A web server with a remote native Runner and shared Postgres. Related work: #18327 owns output logs and provider traces. This change has no dependency on its S3 export path. #18031 covers interrupted Codex receipt recovery. #18005 covers legacy spool reads. #17516 covers Codex stop-boundary retention. ## What Changed - Add native producer receipt delivery through the existing Runner callback before transport acknowledgement. - Reuse `run_usage_receipts`, stable source sequences, and deterministic report identities. Reject conflicting replay and retry a lost commit response with the same receipt. - Restore per-turn usage from structured receipt facts. Preserve provider turns that share one PRP envelope turn. - Bind retained warm transport callbacks to the current run authority epoch, allowing old attachment control acknowledgements while preserving accounting ownership. - Persist Codex session baselines and cumulative totals in the existing provider checkpoint. Mark qualified whole-run deltas, preserve them across resume/restart/attachment, and leave old unmarked reports incomplete and unpriced. - Normalize the bounded final usage tail before the terminal event. Retain later increases for explicit reconciliation, block further attachment, and fail an affected successor without changing its baseline. - Complete runs from the accepted immutable receipt and its exact saved rate-card amount; preserve existing live-budget enforcement. - Skip the local receipt recorder for supported built-in native remote Runner execution. Keep local, legacy, and external Dot/Muse behavior unchanged. - Keep pricing, reservations, ledger settlement, source fencing, and budget holds with their existing owners. - Put the actual Runner integration test in the existing Rust-cached native test lane. Build before opening Postgres and give cold compilation its own bounded budget. ## Verification - Fresh Greptile review is 5/5 on `e7376a50c8162770da1d581ecf25c8cffac64520`, with no unresolved findings. Parent and independent peer review approved the final changes. - Real PostgreSQL 18.6 and a real Rust Runner exercise failed receipt insertion, receiver-directory loss, replay from the same producer, and exactly one 25-cent ledger charge. The actual accounting spool path stays absent. Optional logs/traces are disabled. The provider is controlled; no paid request is made and the fixture never pre-saves the expected receipt. - Producer checks passed 32 usage/provider tests,37 terminal controls, corrected completion, all 3 previously failing exit cases, and both full transport goal-continuation cases. Rejected active-goal completion retains its 120-token tail before failure. Baseline 100→successor150→33 duplicate old snapshots→old120 retains explicit incomplete reconciliation evidence and positive budget exposure. - Heartbeat executor checks passed 982/982. PostgreSQL tests preserve exact 1.002 USD pricing, the 100.2-cent ledger amount, budget holds, source takeover, partial/final receipts, cost-only corrections, equal rate-card replay, and stale-completion interleaving. Newer incomplete usage remains pending rather than settling an older snapshot. - Runtime replay and strict usage-schema checks passed 2/2; generated contracts and Rust formatting pass. Test-partition contracts passed 32/32. - [Final combined-head evidence](https://github.com/paperclipai/paperclip/pull/18354#issuecomment-6531526158) records full workspace typecheck/build and 41/41 accounting tests against all five cloud PR heads, including this final accounting commit. It includes commands and evidence limits. - Exact-head CI is green: 53successful checks, 6intentional skips, and the legacy Snyk status passed. Typecheck, build, Runner, server, browser, and aggregate gates pass. Runner Vitest1/2 initially failed an unchanged mocked Antigravity login fixture; its focused case passed locally, then one unchanged job retry passed 131 files/1995 tests, including all 11 Antigravity tests. The initial failed attempt remains recorded in CI. Initial local attempts exposed the host SysV semaphore limit and corrected harness path, IPC readiness, and queue-counter defects. A temporary test loader selected an existing POSIX-semaphore PostgreSQL 18.6 build without changing repository dependencies. An optional expanded debug provider run was interrupted at its unrelated 8192-turn rollover stress case; no full-file local pass is claimed. Required failure cases were run directly, and the full Runner CI remains required. ## Risks - The guarantee covers built-in native remote Runner execution. Local native execution, legacy adapters, external Dot/Muse execution, and server-originated model calls retain their existing contracts. - This does not provide traffic switching or general web-server failover. The existing controller identity, checkpoint, and generation gates still apply. - The receipt JSON gains optional structured producer facts. No schema migration or accounting S3 spool is added. Text output and raw trace payloads are excluded. - Codex accounting authority requires a Runner artifact built from this provider baseline implementation and qualified by digest. Update receivers and strict usage-schema readers before releasing that Runner; PRP does not negotiate this payload field. Existing unmarked Codex reports remain pending/unpriced. No deployed Runner artifact is claimed fixed. - Usage arriving beyond the bounded terminal tail may require manual reconciliation or a fresh session. Its evidence remains in the existing producer checkpoint/outbox; no adjustment ledger or recurring scan is added. ## Model Used OpenAI `gpt-6.1-sol` assisted with the implementation, tests, and documentation through code execution and repository tools. The context window size is unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
8362dbff4d |
feat(runner): make every ACPX provider declare its release attestation (#18342)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native ACPX runs admit a provider only when its saved profile version and `commandDigest` match the current profile exactly. > - #18325 made every provider digest attest its upstream release, and bound the shared ACPX layer and Runner source through the pack `contractDigest`. > - The model is documented, but it lived in prose. A new provider, such as one in an open PR, could still choose another digest model without any check or guidance. > - This pull request makes the model the default path: each provider declares its attestation in one registry, and the generator, a test and a scoped rule explain the pattern where contributors meet it. > - The benefit is that new providers and upstream bumps follow one model, without a growing list of blocked legacy patterns. ## Linked Issues or Issue Description Refs #18325, #18237. **What happened?** After #18325, only Copilot, Grok and Pi went through `acpx-release-attestations.mjs`. Cursor and Antigravity had separate inline checks in the generator, and Claude and Codex had none. A new provider could add a profile with any digest. Nothing told the author how to compute it, and nothing pinned the profile versions that now change only for deliberate resets. **Expected behavior** Every ACPX provider declares how its `commandDigest` attests the upstream release. Tools and review guidance describe the pattern when a contributor adds a provider, bumps an upstream release or changes a version. **Steps to reproduce** 1. Add a new provider profile to `packages/paperclip-runner/acpx-profiles.json` with an arbitrary `commandDigest`. 2. Run `node packages/paperclip-runner/scripts/generate-acpx-profiles.mjs --check`. 3. Before this change, the check passes. ## What Changed - Add `ACPX_RELEASE_ATTESTATIONS` to `scripts/acpx-release-attestations.mjs`. Every profile declares one kind: - `installed-command` (Claude, Codex): installation hashes the upstream launch command. - `launcher` (Grok): the SHA-256 of the launcher shipped verbatim. - `closure` (Copilot, Pi): upstream versions and complete platform closure pins. - `declaration` (Cursor, Antigravity): a checked-in release declaration, as canonical JSON. - `generate-acpx-profiles.mjs` checks every profile through the registry. The separate Cursor and Antigravity digest checks are now part of the same path. No digest changes. - A profile without a declared attestation fails generation. The message says to declare one and points to the rule. - A stale digest message now says that a digest changes only with its upstream release, and that an upstream change keeps the profile version. - Add a test that pins the profile versions. It explains that a bump resets every saved session for that provider, and that a new provider starts at 1. - Add `rules/acpx-profile-identity.mdc`. It describes the identity model, how to add a provider, when to bump a version, and review criteria. `AGENTS.md` and `.greptile/files.json` route the profile, attestation, contract-registry, ACPX driver and provider paths to it. - Update the profile identity section in `packages/paperclip-runner/docs/durable-recovery.md` with the attestation kinds and the version policy. ## Verification - In `packages/paperclip-runner`: - `node scripts/generate-acpx-profiles.mjs --check` passes. Running it without `--check` changes no digest. - `node --test scripts/acpx-release-attestations.test.mjs scripts/contracts/contract-fingerprints.test.mjs`: 22 tests pass. New cases: every profile declares an attestation, a new provider without one fails with guidance, and Cursor and Antigravity declaration changes affect only their own digests. - `vitest run src/drivers/acpx/profile-compatibility.test.ts`: 15 tests pass, including the version table. - `tsc -p tsconfig.json --noEmit` passes. ## Risks - Low risk. No digest, version or runtime admission behavior changes. - A new provider PR must add one registry entry. The generator error states what to add. - The version table needs an edit for each deliberate reset. Upstream bumps do not touch it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
abc270ee4c |
fix(runner): attest upstream releases and bind ACPX uniformly for every harness (#18325)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native ACPX runs admit a provider only when its saved profile and `commandDigest` match the current profile exactly. > - For Copilot, Grok and Pi, `commandDigest` hashed dozens of Paperclip source files. Claude, Codex, Cursor and Antigravity hash only their upstream release. > - Any change to a bound source therefore advanced these profiles. #18212 alone moved Copilot from v88 to v91 and Grok from v15 to v19. Each bump reset in-flight sessions and added immutable identity fixtures (about 160 on master). > - #18237 added the provider pack `contractDigest`, which now guards Runner source compatibility at the pack boundary. > - This pull request changes the Copilot, Grok and Pi digests to upstream release attestations, as the other providers already use. > - It also binds the shared ACPX layer the same way for every ACPX harness: the profile's `acpxVersion` and the pack `contractDigest`, which now covers the Runner's ACPX patch. > - The benefit is that Runner source changes no longer reset these sessions or need new profile versions and fixtures. ## Linked Issues or Issue Description Refs #18237, #18212, #17972. **What happened?** A Runner change that touched a bound source file changed the Copilot, Grok or Pi `commandDigest`. The author had to advance the profile version, regenerate an identity fixture and keep every old fixture. On deploy, saved sessions for that provider failed the exact admission check and started fresh. **Expected behavior** A provider's `commandDigest` changes only when its upstream release changes: an upstream version, a platform closure, the ACPX patch or the Grok launcher. Runner source compatibility is the job of the pack `contractDigest`. **Steps to reproduce** 1. On master, change a comment in `src/drivers/acpx/runtime-host.ts`. 2. Run `node packages/paperclip-runner/scripts/grok-profile-identity.mjs --check`. 3. The check fails. The Grok profile must advance, and the next deploy resets Grok sessions. ## What Changed - Add `packages/paperclip-runner/scripts/acpx-release-attestations.mjs`. It computes the release attestation for each provider: - Copilot and Pi: a canonical-JSON hash of the profile version, upstream package versions and the platform closure pins (Pi also binds its Node version). - Grok: the SHA-256 of `launcher.cjs`. Admission already hashes these bytes. - The attestation binds no Paperclip source, shared ACPX layer, absolute path, pack manifest digest or install layout. - Bind the shared ACPX layer once for every ACPX harness (Claude, Codex, Copilot, Cursor, Grok, Pi, Antigravity): - Add the Runner's ACPX patch (`patches/acpx@<runner acpx version>.patch`) to the pack `contractDigest`. A patch change now retires older image packs for every harness. Other packages' ACPX patches are not read. A build context without the Runner's patch fails. - Remove the ACPX patch hash from Cursor's release declaration (`cursor-contract.json`) and advance Cursor once to profile 17. Copilot, Pi and Cursor no longer bind ACPX in their own digests, like Claude, Codex, Grok and Antigravity. - Saved inputs for every harness already must match the current `acpxVersion`. - Claude and Codex digests do not change. - `generate-acpx-profiles.mjs` writes these digests to `acpx-profiles.json`. `check:acpx-profiles` refuses a stale digest. - Move the Copilot closure pins to `copilot-closure-pins.ts`, so that the generator can import them. `copilot-installation.ts` re-exports them. - Advance each profile once and then freeze it: Copilot 92, Grok 20, Pi 37. A version change at the switch keeps the rollout on the same path as every earlier profile bump. - Replace the historical version lists in `profile-compatibility.ts` with a range. Every revision from 1 through the current one decodes. Claude and Codex still decode revisions 1 through 5. Launch and recovery admission still require the exact current profile and digest. - Delete: - `scripts/copilot-profile-identity.mjs`, `scripts/grok-profile-identity.mjs` and its test - the Grok launcher source stamp - 159 Copilot, Grok and Pi identity fixtures, including three unreferenced `copilot-mainline-v*` fixtures. The Cursor fixtures stay. - Rewrite the affected tests as behaviour checks: - Each provider admits only the exact current profile. Every earlier version decodes and is refused at admission. - The last source-bound digests, and a changed digest at the current version, are refused by recovery and by the backend factory. - The Rust Pi descriptor test uses the generated profile instead of fixtures. - New `acpx-release-attestations.test.mjs`: a relocated copy of the inputs gives the same digests, each upstream input changes only its own providers, and the generator refreshes and checks the manifest. - Remove the fixture exceptions from `.dockerignore`. Add the attestation script and the generator to the Daytona image content inputs. - Update the Pi install probe and the Rust integration constant for the new Pi digest. - Update the docs to describe the current attestation model: `doc/grok-native-runner.md`, `doc/RUNNER-INTEGRATION-PLAYBOOK.md`, `doc/observability.md`, `packages/paperclip-runner/docs/durable-recovery.md`, and the Copilot, Pi and rich ACP architecture reports. Each report now opens with the current profile identity; earlier source-bound profiles remain as dated chronology. No doc references a deleted file. ## Verification - In `packages/paperclip-runner`: - `node --test scripts/acpx-release-attestations.test.mjs scripts/contracts/contract-fingerprints.test.mjs`: 19 tests pass, including a Runner ACPX patch edit that changes the contract digest, other patch edits (including `acpx@0.12.0` for `adapter-utils`) that do not, and a build context without the patch that fails. - `node scripts/generate-acpx-profiles.mjs --check` passes. - `vitest run src/drivers/acpx src/backends src/contracts`: 1647 tests pass, 9 skipped. - `tsc -p tsconfig.json --noEmit` passes. - From the repository root, `vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/daytona-image.test.ts`: 11 tests pass. - In `server`, the 8 test files that use ACPX profiles, pack digests or `contractDigest`: 1617 tests pass. - Not run locally: the Rust tests and `cargo fmt`, because this machine has no Rust toolchain. CI runs them. - Before this change, a synthetic upgrade check built two Runner revisions that differ only in shared source. A Claude checkpoint saved under the first revision rebinds under the second, and a changed Claude `commandDigest` starts fresh. This confirms that the provider JSON has no other build-dependent field. ## Risks - **One reset at rollout.** The Copilot, Grok, Pi and Cursor digests and versions change once. Saved sessions for these providers start fresh on their next run, as after any earlier profile bump. After this change, Runner source changes no longer cause this reset. - **Lost "source changed, so requalify" signal.** The old digest forced a new profile version when bound source changed. Nothing enforced requalification: Copilot was last qualified at v47. Pack-boundary contract changes are now caught by the pack `contractDigest`. - **Image packs are retired once for every provider.** Adding the ACPX patch changes the pack `contractDigest`, so the controller uploads its bundled pack (about 16 s per remote start) until images are refreshed. Claude and Codex sessions are not reset: their digests and versions do not change. - **Grok launcher and closure pins.** The Grok digest covers only the launcher bytes. Grok's runtime version is still checked separately at admission, as before. - **Grok digest equals early Grok profiles.** The launcher bytes have not changed since profile 1, so the new Grok digest equals the published profile 1 digest. The profile version still separates them: launch, recovery and the backend factory require profile 20. An image pack from that period has no `contractDigest`, so the pack gate refuses it. `remote-provider-pack-verification.test.ts` now covers this case. - **Docker context checks** run `generate-acpx-profiles.mjs --check`, which now imports `.ts` pin files through Node type stripping. `materialize-pi-distribution.mjs` already relies on this on Node 24. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
fba0ab8162 | fix(runner): preserve GitHub helpers and Codex probe policy (#18212) | ||
|
|
e345b9217d |
fix(runner): paginate native task searches without hidden truncation (#18282)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agents search tasks to find work that already exists. > - The search schema permits 200 results, but the server returns at most 100. > - The response does not tell the agent that more results exist. > - This pull request adds pages and reports the page size and continuation cursor. > - Agents can read the full accessible inventory before they decide that a task does not exist. ## Linked Issues or Issue Description **What happened?** A native `search_tasks` request with `limit: 200` returns only 100 tasks. Historical scans recovered 167 and 173 tasks with status partitions. This proves missing inventory and extra work. It does not prove duplicate task creation. **Expected behavior** Honor the declared limit. Report more results and give the caller a continuation cursor. **Steps to reproduce** Create 205 matching tasks in a test company. Call `search_tasks` with `limit: 200`. The baseline returns 100. The fixed path returns 200, then five when the caller uses `nextCursor`. **Paperclip version or commit** Baseline: `81ebe479e8db8a3d59c86cdfb894bd917734492f`. Related changes: #16979 changes UI status filters before pagination. #13699 adds public task URLs. Neither repairs this native limit. An exact `search_tasks` plus pagination PR search returned no result. Pattern: [Task searches omit cases beyond the first hundred](https://www.braintrust.dev/app/Paperclip/p/Paperclip%20Staging/patterns/c20462d6-05fd-4862-bfbd-7bec1f2bc159). ### Hypothesis and thesis for a fix The native authority uses `boundedLimit` with its default maximum of 100. The tool schema permits 200. The response contains only `tasks`. The raw request and result counts agree with this cause. Fix the native authority and its canonical tool contract. Keep the existing `tasks` field. Add `limit`, `returnedCount`, `hasMore` and `nextCursor`. Use ascending task IDs so task activity does not move results between pages. Apply current company, privacy and Muse read checks before each page. A cursor grants no access. Callers must keep the same filters and read every page. Concurrent task changes require a fresh scan. ## What Changed - Honor the native search limit of 1 to 200, with a default of 50. - Add ID cursor pagination and page metadata. - Describe complete inventory reads in the tool contract and access map. - Refresh the generated semantic contracts and seeded catalog identity. - Add regression coverage for complete inventories and access-filtered pages. ## Verification - Red: the database integration test requested 200 of 205 persisted tasks. Baseline returned 100. - Green: the same regression returns pages of 200 and five without duplicate IDs. - The native authority and Muse suites cover company exclusion, status filters, empty pages, invalid cursors and readable-only metadata. - Command: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/paperclip-runner-tool-authority.test.ts src/services/native-runtime/muse-native-visibility.test.ts`. - The test uses the real run-bound native authority and disposable PostgreSQL. It does not use an external model or change a deployed scanner. - Both native authority and Muse suites pass: 71 tests. Server direct `tsc --noEmit`, Runner `typecheck:typescript`, `node scripts/check-maps.mjs` and generated semantic contract checks pass. - Full typecheck and build cannot finish here because `cargo` is absent. The full local test command was stopped after about five minutes when the complete CI matrix was running. It has no complete verdict. Current-head CI passed, including typecheck, build, all general/serialized/Runner test suites and E2E. Greptile is 5/5 with no actionable findings at `392cb7424a1fa0ec73401d5e8a10a22b504fde32`. Optional Storybook and manual Docker workflows were skipped and are not executed coverage. The success-example correction also passed all 82 contract tests locally. ## Risks Task order changes to ascending ID. Pagination is not a snapshot. New or changed tasks can require a fresh scan. Each page retains the existing full company-list read and Muse authorization cost. This change adds no database query loop or new background work. Existing callers can still read `tasks`, but must follow `nextCursor` for a complete inventory. No scanner implementation was supplied. No migration or deployment is included. ## Model Used OpenAI GPT-6 through Codex, with tool use and code execution. The exact provider deployment ID and context limit are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge The task requires the supplied branch to stay unchanged. All applicable CI and review checks pass at the current head. Unmerged and not deployed. --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
fa760dbce4 |
fix(runner): reuse image packs only with matching contract digest (#18237)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote native runs use a provider pack. The controller reuses a protected pack from the sandbox image when the pack is compatible. > - #17749 gates this reuse on a hand-maintained interface revision in `provider-pack-contract.json`. > - Later changes altered the runner/pack boundary (#17922, #18188), but nobody bumped the revision. A stale Claude or Codex image pack can still be reused across a sidecar contract change. This is the failure class of #17972. > - A manual revision has a "forgot to bump" failure mode. Source-byte hashes change on every refactor, so they are too brittle. > - This pull request declares the sidecar `session.open` contract in its JSON schema and computes a contract digest from the JSON contracts and Rust shapes. > - Image-pack reuse now requires an equal digest. > - The benefit is that an incidental change keeps image packs reusable, and a real contract change retires them automatically. ## Linked Issues or Issue Description Refs #17749, #17972, #17922, #18188. **What happened?** The controller reused an image provider pack when `compatibility.acpx` and the selected agent's profile digest matched. For Claude and Codex, that profile digest binds only the upstream executable. The ACPX interface revision stayed at 1 after later sidecar contract changes. A pack built before those changes passed the reuse check. **Expected behavior** The controller reuses an image pack only when the pack uses the same runner/pack contracts as the controller's bundled pack. Comments, formatting and refactors do not change that result. **Steps to reproduce** 1. Build an image pack from a commit before a sidecar input change, such as a new `session.open` field. 2. Run a controller from a later commit that sends the new field, with the same interface revision and Claude version. 3. Before this fix, `verifyRemoteProviderPack` accepts the old image pack. ## What Changed - Declare the sidecar `session.open` parameter set as a closed `sessionOpenParams` definition in `acpx-sidecar.schema.json`. - `generate-acpx-sidecar-contract.mjs` now generates `GENERATED_ACPX_SIDECAR_OPEN_PARAMS` from it. - `parseOpenParams` uses the generated list instead of an inline `Set`. The accepted set is unchanged. - This is the surface of #17972. The schema is now its source of truth. - Declare the nested `session.open` contracts that the sidecar parses strictly: - `protocol/provider-schemas/runtime-context.schema.json` closes every runtime context object. `parseNativeRuntimeContext` uses the field lists that `scripts/generate-runtime-context-contract.mjs` generates from it. - The sidecar schema declares the closed `providerPolicy` object, and `parseProviderPolicy` uses its generated field list. - The accepted fields do not change. - Add `packages/paperclip-runner/scripts/contracts/contract-fingerprints.mjs`. It fingerprints these contracts: - the PRP and sidecar JSON schemas, with annotation text removed - the generated catalogs in `generated/` - the Rust serde shapes, `json!` payload keys, match literals and env keys in `runner-core/src`, without test-only modules - the reserved env key lists in `configured_environment.rs` - Read only the trees that every pack build context contains. Fail when a registered source is missing, so two build contexts cannot silently produce different digests. - Use only Node built-ins. The registry adds no dependency. - Write `contractDigest` as a top-level payload field in `build-provider-pack.mjs`. The manifest digest algorithm and the existing field layout do not change. - Require an equal `contractDigest` for image-pack reuse in `remote-provider-pack-verification.ts`. A pack without a digest is not reused. The controller uploads its bundled pack, as it does today for any incompatible pack. - Re-attest the source-hashed profiles, because this PR changes their bound sidecar sources: - Grok becomes profile 15 and now also binds the generated allowlist, so a Grok field change in the schema still invalidates the Grok identity. - Copilot becomes profile 88, and also binds the generated runtime context fields. - Earlier profiles stay decodable. - Add the registry script and `generated/` to the Daytona image content inputs. - Document the digest and the schema-first rule in `docker/daytona-runner/README.md` and `doc/DEVELOPING.md`. ## Verification - `node --test scripts/contracts/contract-fingerprints.test.mjs` (in `packages/paperclip-runner`): 4 tests pass. - These changes keep the digest: - schema description text - schema reformatting - renaming `parseOpenParams` - Rust comments, `#[serde(skip)]` fields and `#[cfg(test)]` modules - edits to the `#[path]` test module `tests/provider_api_validation.rs` - These changes alter the digest: - a new `session.open` schema field - a changed constraint on the `remainingWork[].description` property - a new reserved env key - a new sidecar env key - a new checkpoint field - A build context without a registered source fails. - `node --test test/acpx-sidecar-contract.test.mjs`: 13 tests pass. A new case rejects an unknown `session.open` parameter and accepts each declared parameter. - `node --test scripts/grok-profile-identity.test.mjs`: 4 tests pass. A change to the Grok field in the generated allowlist invalidates the Grok identity. - `generate-acpx-profiles.mjs --check`, `generate-acpx-sidecar-contract.mjs --check` and `generate-protocol-manifest.mjs --check` pass. - Runner vitest suites pass: 470 passed, 1 skipped. They cover the sidecar, sidecar input and protocol, Copilot and Pi installation, profile compatibility, native execution, recovery identity, the backend factory and the environment. - Server tests: pack verification, executor and artifact preparation pass (1,004 tests). - `tsc --noEmit` passes for `packages/paperclip-runner` and `server`. - `tests/runner-e2e/daytona-image.test.ts`: 10 of 11 tests pass. The failing test also fails on `master`. It expects a literal `profile-v19-identity.json` line in `.dockerignore`, which uses a `profile-v*` glob. - Not run locally: - A complete `build-provider-pack.mjs` run. The builder rejects the local Homebrew Node because it is not portable after relocation. - Rust tests and `acpx-controller-recovery.test.ts`, because `cargo` is not installed. - The full `pnpm -r typecheck`, `pnpm test:run` and `pnpm build`. ## Risks - Only JSON contracts and Rust shapes are part of the digest. The remaining TypeScript-only definitions are the `paperclip_finish` and `paperclip_block` tool schemas in `completion-result.ts` and the TypeScript env key lists. They are not covered until they move into a schema. The nested `expectedIdentity` and `tools` values ignore unknown fields, so a change there cannot cause a startup rejection. - A change there can let a stale image pack be reused. The sidecar then rejects the input with its strict parser error. - Today, Claude and Codex image packs have no contract check at all. - The digest covers all registered contracts for the whole pack. A contract change for one provider kind also retires image packs for the other. Each replacement costs about 16 s per remote start until the image is refreshed. - Existing image packs have no digest. After deploy, the controller uploads its bundled pack until the image is rebuilt. - The Grok and Copilot re-attestations start fresh sessions for those agents once after deploy, as each profile bump does today. - The Daytona image content ID formula changes, so the next image build does not reuse a cached image. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code in the desktop app, with repository tools, shell execution and research subagents. The session does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |